Clean linked-worktree full index stalls silently for workspace-scale projects with persisted stored_hashes >= 267k

[jaamores]

Issue Publication

schema_version: 1

Tooling owner

project:     DeusData/codebase-memory-mcp
host:        https://github.com/DeusData/codebase-memory-mcp
tracker:     https://github.com/DeusData/codebase-memory-mcp/issues
version:     0.9.0
binary:      /home/ubuntu/.local/bin/codebase-memory-mcp
update URL:  https://github.com/DeusData/codebase-memory-mcp/releases/latest/download

Local authentication

$ gh auth status
github.com
  ✓ Logged in to github.com account jaamores
  Token scopes: 'admin:public_key', 'gist', 'read:org', 'repo'

jaamores does NOT appear in the DeusData organization. The token
scopes allow repo-creation and issue-creation on personal repositories
but the tooling is owned by a different organization. Public issue
creation on a public repository is allowed for any authenticated user
without write access to the repo, so this is not a blocker.

Local evidence surveyed

gh issue list --repo DeusData/codebase-memory-mcp --state open --limit 200
  (30 most recent issues inspected)

related open defects that may overlap with this incident:
  #1273 Mode upgrade silently keeps weaker graph: incremental routing
       ignores stored index capability level
  #1287 index_repository reports parse_partial_count: 0 after incremental/
       artifact reload despite persisted coverage gaps
  #1296 MCP session serves a stale graph indefinitely after the
       background watcher re-indexes

the exact failure described in this incident (workspace-scale clean
full index stalls silently after incremental.edge_snapshot with no
response file) is not among the open issues surveyed.

Issue title

Clean linked-worktree full index fails silently for workspace-scale
projects with persisted stored_hashes >= 267k, blocking external
workspace index authority

Issue body (sanitized, ready for gh issue create)

### Version

codebase-memory-mcp 0.9.0 on Linux x86_64, statically linked binary at
/home/ubuntu/.local/bin/codebase-memory-mcp.

### Install channel

GitHub release archive, installed via `codebase-memory-mcp install`.

### Binary variant

standard.

### What happened, and what did you expect?

`index_repository --mode full --persistence true --repo-path <path>`
exits 0 but never writes the persisted artifact and never writes the
worker response file. The same failure reproduces on four legitimate
invocations: three against a recursively-scanned root containing nested
repositories and one against a clean linked worktree (no nested
repositories, no p2a-workspace, no `.codebase-memory` inputs) where
the only source files are the workspace repository contents.

Expected: a clean exit either publishes the artifact and writes the
response file, or fails non-zero with a clear error message in the
worker log.

### Reproduction (small, deterministic)

1. Create a small synthetic repository:

       mkdir /tmp/cbm-repro && cd /tmp/cbm-repro
       git init -q --initial-branch=main
       git config user.email "test@example.com"
       git config user.name "Test"
       echo 'def hello(): return "hi"' > src/hello.py
       echo 'name: repro' > config.yaml
       git add src/hello.py config.yaml
       git commit -q -m "init"

2. Index the repository root with `mode=full`:

       codebase-memory-mcp cli index_repository \
         --repo-path /tmp/cbm-repro \
         --mode full \
         --persistence true

3. Create a clean linked worktree outside the repository root:

       git worktree add --detach /tmp/cbm-repro-authority HEAD

4. Index the clean linked worktree with `mode=full`:

       setsid nohup codebase-memory-mcp cli index_repository \
         --repo-path /tmp/cbm-repro-authority \
         --mode full \
         --persistence true \
         </dev/null >/tmp/cbm-repro-logs/cli.stdout \
                  2>/tmp/cbm-repro-logs/cli.stderr &
       disown

### Actual behavior

Worker log stops after `incremental.edge_snapshot` (or after a long
`gbuf.delete_by_file` per-file reconciliation when the persisted
graph has 267,909+ stored hashes). No further log lines. No response
file. Supervisor and worker exit cleanly with `exit_code=0`. No
`/tmp/cbm-repro-authority/.codebase-memory/` directory is created.

### Expected behavior

Either:
- The engine reaches artifact publication and writes the response file, OR
- The engine returns a non-zero exit and writes a clear error to the
  worker log explaining the stall.

### Last successful phase

`incremental.edge_snapshot` (or `gbuf.delete_by_file` for previously-
indexed files).

### First missing phase

artifact publication / response publication. No log line is emitted in
this phase.

### Environment

- Binary: codebase-memory-mcp 0.9.0
- OS: Linux x86_64, kernel 6.8.0-111-generic
- Memory: 64 GB host; VmRSS pinned at ~6.8 GB single-thread
- Workload: 576 to 564 source files; previously-stored graph with
  267,909 stored hashes and 3,086,265 nodes / 16,538,130 edges

### Related open issues that may overlap

- #1273 "Mode upgrade silently keeps weaker graph: incremental routing
  ignores stored index capability level" (capability mismatch + inverse
  case from this defect)
- #1287 "index_repository reports parse_partial_count: 0 after
  incremental/artifact reload despite persisted coverage gaps"
  (response-shape mismatch while persisted state is inconsistent)
- #1296 "MCP session serves a stale graph indefinitely after the
  background watcher re-indexes" (stale persisted state observation)

### Security and privacy redactions

- No private repository contents are quoted.
- No secret values, credentials, or tokens are quoted.
- Only public CLI surface, install output, and version metadata are cited.
- No source code excerpts are quoted.

### Requested maintainer guidance

1. Is the silent stall after `incremental.edge_snapshot` a known bug
   for projects with persisted stored_hashes >= 267k?
2. Should the engine emit at least one log line per minute in the
   stalled phase so operators can diagnose the failure?
3. Should the engine write a non-zero exit code when no artifact is
   published but the worker exits cleanly?
4. Is there a supported non-destructive migration path from a path-
   derived workspace project to a stable logical identity?
5. Can the artifact output be directed to an external stable authority
   root outside the repository being indexed?

We do not request a workaround that requires manual artifact editing,
manual artifact copying, hook bypass, recursive workspace indexing, or
deleting valid existing projects.

Issue creation status

attempted:   NOT YET
reason:      the user explicitly requested a clean local incident package
             BEFORE creating the upstream issue, so the package can be
             referenced in the issue body if the user later authorizes
             publication.
next step:   once the user authorizes, run:
               gh issue create --repo DeusData/codebase-memory-mcp \
                 --title "Clean linked-worktree full index fails silently
                          for workspace-scale projects with persisted
                          stored_hashes >= 267k, blocking external
                          workspace index authority" \
                 --body-file docs/tooling-incidents/codebase-memory-clean-worktree-project-identity/issue-publication.md \
                 --label "bug,stability/performance,parsing/quality"
             then record the returned issue URL/number in this file.

Blocked-publication classification (current)

CODEBASE-MEMORY TOOLING INCIDENT:
LOCALLY_RECORDED (not yet OPEN on the upstream tracker)

INCIDENT PUBLICATION:
BLOCKED_PENDING_USER_AUTHORIZATION (the body is sanitized and ready)

The incident is fully captured locally with the exact failure evidence,
the exact reproduction specification, and the requested maintainer
guidance. Once the user authorizes the upstream publication, the
sanitized body above can be submitted verbatim with gh issue create.

[DeusData]

Thanks for preserving the operational evidence and separating the last completed phase from the first missing one.

Current main confirms both observability gaps. After incremental.edge_snapshot, the per-file purge loop has no progress event until incremental.purge completes (src/pipeline/pipeline_incremental.c:836-849). The supervisor reads the response only after a nominally successful worker exit, but a missing response is not classified like an oversized/rejected response (src/mcp/index_supervisor.c:731-742). That matches the clean-exit/no-response failure shape you observed.

I have routed this as a priority/high 0.9.1 stability bug. The synthetic steps do not recreate the 267k-hash state by themselves, so the large persisted-state measurements remain the binding evidence. Thank you for including them.