True incremental indexing: clean-rebuild parity, 3× edit-indexing speedup, and measured search-quality gains - #1245
Open
ahundt wants to merge 756 commits into
Open
True incremental indexing: clean-rebuild parity, 3× edit-indexing speedup, and measured search-quality gains#1245ahundt wants to merge 756 commits into
ahundt wants to merge 756 commits into
Conversation
Extend active overlay Cypher relationship reads across additional MATCH and OPTIONAL MATCH patterns when every relationship pattern is fixed one-hop. Reuse the qn-keyed active edge-node store helper for terminal-bound expansion and keep OPTIONAL MATCH no-edge rows on the shared preservation path. Variable-length relationship traversal and id() projections remain canonical-only with caller-visible warnings. Validation: /private/tmp/cbm-build-active-multipattern-relationship-20260704T-current-3.log; /private/tmp/cbm-test-mcp-active-multipattern-relationship-focused-20260704T-current-2.log; /private/tmp/cbm-test-mcp-active-multipattern-relationship-suite-20260704T-current-2.log; /private/tmp/cbm-test-cypher-active-multipattern-relationship-20260704T-current-2.log; /private/tmp/cbm-test-store-nodes-active-multipattern-relationship-20260704T-current-2.log; /private/tmp/cbm-source-safety-active-multipattern-relationship-20260704T-current-2.log; /private/tmp/cbm-prod-build-active-multipattern-relationship-20260704T-current-2.log.
Use the existing qn-keyed overlay BFS helper for directed variable-length Cypher relationship patterns in active query_graph reads. Keep undirected variable-length traversal and id() canonical-only with explicit warnings until those semantics have a separate tested contract. TDD: /private/tmp/cbm-test-mcp-active-variable-relationship-before-20260704T-current-2.log failed before the production wiring because FreshVarSource was absent. Validation: /private/tmp/cbm-build-active-variable-relationship-20260704T-current-2.log; /private/tmp/cbm-test-mcp-active-variable-relationship-focused-20260704T-current-2.log; /private/tmp/cbm-test-mcp-active-variable-relationship-suite-20260704T-current-2.log; /private/tmp/cbm-test-cypher-active-variable-relationship-20260704T-current.log; /private/tmp/cbm-test-store-nodes-active-variable-relationship-20260704T-current.log; /private/tmp/cbm-source-safety-active-variable-relationship-20260704T-current.log; /private/tmp/cbm-prod-build-active-variable-relationship-20260704T-current.log.
Fix canonical and active-overlay variable-length Cypher traversal for undirected relationship patterns. The store BFS string API now maps historical directions onto the existing CBM_STORE_EDGE_DIR constants, so canonical and overlay BFS share the same inbound/outbound/any decision instead of duplicating string checks. query_graph can now answer MATCH (a)-[:TYPE*min..max]-(b) from ready overlay edges, while id() Cypher queries remain explicitly canonical-only until overlay id semantics or compaction exists. Validation: red canonical TDD /private/tmp/cbm-test-cypher-any-variable-relationship-before-20260704T-current.log; red MCP TDD /private/tmp/cbm-test-mcp-any-variable-relationship-before-20260704T-current.log; green consolidated suites /private/tmp/cbm-test-cypher-any-variable-relationship-consolidated-suite-20260704T-current.log, /private/tmp/cbm-test-mcp-any-variable-relationship-consolidated-suite-20260704T-current.log, /private/tmp/cbm-test-store-nodes-any-variable-relationship-consolidated-20260704T-current.log; source-safety /private/tmp/cbm-source-safety-any-variable-relationship-consolidated-20260704T-current.log; production build /private/tmp/cbm-prod-build-any-variable-relationship-consolidated-20260704T-current.log.
Overlay rows intentionally do not expose stable canonical node or edge ids until compaction. Rename the active-read guard around canonical identity, document the id() boundary, and add a Cypher regression proving active overlay rows are used for supported queries but disabled for id() projections. Validation: CBM_ONLY_SUITE=cypher ./build/c/test-runner; CBM_ONLY_SUITE=mcp ./build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check; make -j8 -f Makefile.cbm cbm.
Allow long-lived MCP servers to start a later overlay compaction pass after a prior background worker has finished but has not been explicitly joined yet. The start path now reaps one finished worker through the existing join/reset logic, preserves false while a worker is active, and keeps failed reaps on the existing log path only. Add a regression that uses the existing two-overlay fixture to prove a finished unjoined worker does not block a second compaction start. Validation: build/c/test-runner rebuild /private/tmp/cbm-build-overlay-reap-20260704T1818.log; CBM_ONLY_SUITE=mcp /private/tmp/cbm-test-mcp-overlay-reap-20260704T1825.log (171 passed); source safety /private/tmp/cbm-source-safety-overlay-reap-20260704T1826.log; diff check /private/tmp/cbm-diff-check-overlay-reap-20260704T1826.log; production build/sign /private/tmp/cbm-prod-build-overlay-reap-20260704T1828.log.
Expose background overlay compaction lifecycle through index_status so long-lived MCP clients can distinguish idle, running, and finished workers without relying on internal join helpers. Add focused MCP regression coverage for the finished-worker state after background compaction completes, and tighten local project-name snapshots to use checked snprintf handling.
Expose when exact incremental frontier telemetry is capped so index_repository callers can distinguish a known affected-path count from a frontier that hit incremental_exact_max_affected_paths. Preserve existing publish routing and response fields while adding affected_paths_limit and affected_paths_truncated only when available. Also centralize planner fallback reporting and cover oversized inbound-frontier fallback metadata in the pipeline suite.
Prevent unsafe scoped overlay/containment publication for cross-LSP frontier-too-large edits until scoped parity is proven. FastAPI/Python showed active-overlay and containment mismatches for routing.py; this routes that class through the existing full rebuild path, preserving correctness while leaving performance work open. Also suppress weak suffix fallback for Python super().__init__ when LSP resolved an external base, and recompute scoped complexity derived fields for changed in-scope nodes instead of preserving stale recursive/transitive loop-depth values. Validation: git diff --check; source-safety; ASan/UBSan test-runner build; CBM_ONLY_SUITE=pipeline 333 passed; production build/sign; isolated FastAPI oracle passed with full fallback and canonical equality.
Add an internal frontier no-op mask for exact file-delta batches so graph-equal frontier files can prove coverage without recursively expanding or being rewritten. Reuse the existing preserved-inbound-edge helper for scoped changed files and keep the old batch planner/apply APIs as wrappers for non-scoped paths. Focused validation: ASan/UBSan test-runner build passed, pipeline suite passed with 335 tests, source-safety passed, and product build passed. Synthetic python_reexport exact matrix passed. Real FastAPI routing.py still falls back with inbound_edges_require_full, so default-readiness and the broad FastAPI speed blocker remain open.
Route scoped-LSP exact scratch publishes to the existing full-reindex fallback when the scoped planner cannot prove active/full graph equivalence. FastAPI probes showed the no-importer shortcut was faster but produced divergent IMPORTS/CALLS/THROWS edges, so this commit preserves correctness and records the remaining performance blocker. Also build scratch package maps for overlay/exact delta routes to match full-pipeline resolver context, add opt-in inbound-edge debug logging through cbm_log_debug, and update Python route-decorator tests to expect the guarded full route while retaining fresh-rebuild equality checks. Validation: focused pipeline suite 335 passed (/private/tmp/cbm-test-pipeline-scoped-gap-kind-fix-20260705T.log); source-safety passed; git diff --check passed; sandbox full suite reached 6391 passed with 13 known HTTP listen failures plus one stale assertion fixed here; escalated HTTP suite 28 passed (/private/tmp/cbm-test-httpd-scoped-gap-kind-fix-20260705T.log). No tag created.
Mirror the production exact-route path-alias and package-map setup in the test-only scratch builder so lower-level exact-delta tests exercise the same resolver inputs.\n\nAdd a Python package exact-delta regression that applies the lower-level scratch delta and compares the result with a fresh FAST rebuild. This proves the small synthetic package is safe while the larger FastAPI scoped-LSP mismatch remains covered by external benchmark artifacts and the production full-reindex guard.\n\nValidation:\n- CBM_ONLY_SUITE=pipeline CBM_ONLY_TEST=incremental_exact_scratch_python_package_matches_fresh_rebuild build/c/test-runner\n- CBM_ONLY_SUITE=pipeline build/c/test-runner\n- bash scripts/check-source-safety.sh\n- git diff --check
Add a real FastAPI matrix scenario that copies git-tracked HEAD files from an existing checkout into an isolated case repo, applies a routing.py probe insertion, and records source git metadata. Cloning is explicit opt-in via --clone-missing-real-repos; normal runs reuse --fastapi-repo, CBM_FASTAPI_REPO, or common local paths. Current evidence: /private/tmp/cbm-fastapi-matrix-current-20260705T-postpatch.json passed with canonical_equal=true, publish_kind=full, exact_reason=frontier_too_large, and speedup_full_rebuild_over_incremental=0.43653250773993807. This proves the safe fallback path and preserves the FastAPI scoped-LSP performance blocker as a rerunnable benchmark target. Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; uv run python scripts/benchmark-incremental-speed.py --binary build/c/codebase-memory-mcp --matrix --matrix-scenarios fastapi_insert_probe --fastapi-repo /private/tmp/cbm-fastapi-cli-inbound-debug-20260705T-repo --work-root /private/tmp/cbm-fastapi-matrix-current-20260705T-postpatch --out /private/tmp/cbm-fastapi-matrix-current-20260705T-postpatch.json --timeout 240 --include-logs; bash scripts/check-source-safety.sh; git diff --check.
Share the route path/handler scanner between sequential and parallel call passes, add FastAPI/Starlette websocket route-registration suffixes, and let sequential unresolved route-registration suffix calls emit Route registration facts when the existing route-literal guard accepts the path. Add a sequential/parallel parity canary for FastAPI websocket decorators. Validation: make -f Makefile.cbm -j16 build/c/test-runner; CBM_ONLY_SUITE=parallel build/c/test-runner; CBM_ONLY_SUITE=pipeline CBM_ONLY_TEST=fast_route build/c/test-runner; make -f Makefile.cbm -j16 cbm; bash scripts/check-source-safety.sh; git diff --check. FastAPI matrix evidence: guarded production remains safe by full fallback in /private/tmp/cbm-fastapi-matrix-websocket-parity-current-20260704T2215.json. Forced exact cap64 now preserves the previously missing tests/test_ws_router websocket Route nodes and 9 route-registration edges, but still fails canonical edge parity; keep scoped-LSP full fallback until the remaining edge-attribution mismatch is fixed.
Treat exact-delta inbound_edges_require_full as a hard fallback instead of allowing the incremental containment path to publish a graph after exact upsert has already reported that unsupported inbound edges require a full rebuild. Strengthen the focused incremental regression to assert full fallback behavior and compare the result against a fresh FAST rebuild. Validation: make -f Makefile.cbm -j16 build/c/test-runner; CBM_ONLY_SUITE=pipeline CBM_ONLY_TEST=incremental_fast_mixed_unowned_edge_frontier_falls_back_to_full_rebuild build/c/test-runner; CBM_ONLY_SUITE=pipeline CBM_ONLY_TEST=inbound build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check.
Insert the FastAPI benchmark probe as an APIRouter method instead of a top-level function before an indented method declaration. The old mutation produced invalid Python and contaminated scoped-LSP overlay diagnostics with parser fallout. Compile the mutated source before writing it so future FastAPI probe changes fail fast with a precise error instead of producing misleading benchmark data. Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; uv run python -m py_compile /private/tmp/cbm-fastapi-valid-probe-current-20260704T2250/fastapi_insert_probe/repo/fastapi/routing.py; bash scripts/check-source-safety.sh; git diff --check; FastAPI matrix /private/tmp/cbm-fastapi-valid-probe-current-20260704T2250.json passed with canonical_equal=true and publish_kind=full.
Match sequential CALLS property capacity to the parallel pass for edge JSON emitted through calls_emit_edge so changed-file overlay/exact paths preserve later call arguments. Require THROWS/RAISES source nodes to be callable scopes through a shared pipeline helper, preventing sequential file/module fallback edges from diverging from parallel full-index behavior. Add regression coverage for the eighth CALLS arg and top-level raise sequential/parallel parity. Validated with focused canaries, full pipeline and parallel suites, source-safety, product build, guarded FastAPI matrix, and an isolated guard-removal experiment that still failed active equality.
Add a focused regression for Python scoped overlay/exact behavior when receiver-method calls need broader resolver context. The test keeps overlay_publish=small_deltas on, edits one Python file, and asserts the production scoped_lsp_gap path falls back to full reindex, avoids replacement overlay publication, preserves the expected CALLS target, and matches a fresh FAST rebuild. Also correct stale cross-LSP comments so the code describes the current split between full FAST fused cross-LSP, sequential changed-file cross-LSP, and parallel scoped no-op cross-LSP.
Remove stale language-specific wording from the cross-LSP pass header comment. Java and Kotlin are now wired through fallback cross-LSP paths, so the previous example was misleading for scoped resolver diagnostics.
Add an internal helper that rebuilds CBMLSPDef rows from persisted graph nodes by parsing existing node properties and routing through the existing cross-LSP conversion path. Cover the Python receiver-method case with a focused pipeline test that combines changed-file defs with persisted provider defs and resolves the call through the existing Python LSP cross resolver. This is a diagnostic foundation only; it does not relax scoped-LSP fallback guards or claim FastAPI/default incremental readiness.
Add a bounded property-bearing batch lookup for candidate qualified names using the existing VALUES CTE pattern from node-id batch lookup. Update the scoped cross-LSP persisted-def diagnostic to use the batch node loader, and cover ordering, duplicates, missing inputs, project scope, and properties preservation. This is a resolver-context primitive only; it does not relax scoped-LSP fallback guards or claim default incremental readiness.
Add a bounded store helper that expands imported symbol QNs to exact and dotted-member QNs in project scope, preserving input order and reporting truncation. Use it in the scoped cross-LSP diagnostic so persisted Python defs are discovered from the imported Logger scope instead of a hand-built member list, while proving OtherLogger remains excluded. Validation: make -f Makefile.cbm -j16 build/c/test-runner; CBM_ONLY_SUITE=store_nodes; CBM_ONLY_SUITE=pipeline; focused store/pipeline tests; scripts/check-source-safety.sh; git diff --check.
Add an opt-in store-backed scoped cross-LSP path that builds bounded persisted defs from import QNs, loads property-bearing nodes through the store batch lookup, and refines import values only when the store proves module.local_name is an unchanged supported symbol. Keep the path default-off behind explicit ctx metadata and scope cap, preserve the existing scoped-LSP fallback guards, and add a pass-level Python canary for resolving Service.run to provider.Logger.log without selecting OtherLogger.log. Validation: build/c/test-runner rebuilt; focused scoped-LSP canaries passed; CBM_ONLY_SUITE=pipeline passed 340 tests; CBM_ONLY_SUITE=store_nodes passed 111 tests; source-safety, diff-check, and product build passed.
Extend the test-only exact scratch helper so a caller can opt into all-file metadata and the store-backed scoped-LSP cap while preserving the existing wrapper for default-off callers. Add a Python receiver guardrail that proves the store-backed exact scratch can publish the desired provider.Logger.log edge, while recording the current strict oracle gap between lsp_method metadata and the fresh full suffix_match metadata. This keeps scoped-LSP guard relaxation blocked until the oracle path is improved or resolver metadata equivalence is explicitly designed. Validation: rebuilt build/c/test-runner; focused guardrail passed; CBM_ONLY_SUITE=pipeline passed 341 tests; source-safety and diff-check passed.
Add a shared cross-LSP import-value refinement helper that upgrades module import QNs to exact imported symbol QNs only when the active LSP def set proves module.local_name exists. Use the helper in both sequential and parallel fused cross-LSP paths. The parallel path keeps ordinary registry import caches unchanged and bounds refinement through the existing module-def filtered subset when available. Tighten the store-backed scoped exact oracle back to strict fresh-rebuild equality, closing the prior lsp_method versus suffix_match metadata mismatch for the Python receiver fixture. Validation: rebuilt build/c/test-runner; focused strict oracle passed; CBM_ONLY_SUITE=pipeline passed 341 tests; CBM_ONLY_SUITE=parallel passed 29 tests; source-safety, diff-check, and product build passed.
Reject the unsafe FastAPI scoped-LSP exact publish path while preserving reusable resolver and frontier improvements. Move the field-type hint resolver into shared pipeline internals so sequential exact and parallel full paths use the same bounded refinement. Avoid importer-frontier expansion for unchanged exports, keeping changed-export expansion covered by tests. FastAPI matrix evidence showed forced scoped exact could be faster but failed canonical graph equality, including routing.py call/complexity drift. Keep the production scoped_lsp_gap full-reindex guard and do not expose the ineffective scoped-symbol cap as user config. Validation: pipeline suite 341 passed; parallel suite 29 passed; store_nodes suite 111 passed; source-safety passed; git diff --check passed; product build passed; FastAPI safe-guard matrix passed with canonical_graph.equal=true and publish_kind=full.
When LSP resolves a call to an external or unindexed target, keep strong registry evidence such as same-module/import-map matches but drop weak non-import-reachable short-name fallbacks. This prevents false CALLS edges like scope.get -> local *.get while preserving Python re-export/import behavior such as fastapi.Header. Add a focused pipeline canary for external LSP fallback suppression and keep the existing Python re-export and super().__init__ canaries green. FastAPI matrix remains correctness-safe through the scoped_lsp_gap full-reindex guard, so this is not a default-performance closure.
Exact scratch can now resolve field-type-hint CALLS targets through the existing store-backed node lookup instead of requiring every unchanged target to be preloaded into the scratch graph. Parallel and full gbuf paths keep the gbuf-only lookup behavior. Add a FastAPI-like exact-scratch regression test for request.body -> GzipRequest.body and keep the scoped_lsp_gap production guard in place. The guarded FastAPI matrix remains canonical-equal but full-reindex, so this is a correctness enabler rather than default-performance closure.
Suppress Python file-node self.* CALLS edges when the registry result is only a weak short-name match. This targets exact-scratch cases where a missing enclosing function lets an instance-style call fall back to a file-level suffix edge that fresh full indexing does not emit. Add a focused pipeline canary for self.add_api_route file-node fallback. The normal FastAPI matrix remains guarded to full reindex, so this is a correctness slice toward scoped exact parity, not a default-performance claim.
Suppress Python file-node dotted call edges when the registry result is only a weak short-name guess and the target is not import-reachable. Keep route-literal handling, strong registry strategies, field-hint materialization, and import-reachable suffix fallbacks intact. Add canaries for the FastAPI-style request.headers.get false edge and the import-reachable preservation case.
When a dotted or namespace-qualified callee has import evidence but no candidate is import-reachable, stop before weak same-simple-name fallback. This prevents external calls such as email.message.Message or receiver calls such as response.get from resolving to unrelated project symbols while preserving import_map, import_map_suffix, qualified_suffix, and bare-name fallback behavior.
Add scripts/benchmark_fact_comparisons.py and docs/schema/benchmark-comparisons-v1.schema.json to group repetitions by effective implementation, capability, scope, cache, host, harness, terminology, and correctness contracts. Emit elapsed ratios only for parity_manifest_and_contract_v1 pairs; emit capability differences without ratios for capability_delta_manifest_v1 pairs; preserve source run and occurrence IDs and outer lifecycle wall-time rows. Extend fact_result_rows() with graph, freshness, and retrieval-quality outcomes and append generated comparison artifacts from generate_report(). Retained reports without fact bundles remain readable and report comparison status as unavailable. Define the versioned join and formula IDs in benchmark terminology 1.1.0. Verified 198 passed, 1 skipped, and 34 subtests; Draft 2020-12 schema validation, Ruff, source-safety, and git diff checks pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Move incremental_speed.py, run_experiments.py, summarize_results.py, fact_comparisons.py, autotune.py, active v2 schemas, terminology.json, configuration spelling data, and shell harnesses into benchmarks/. Keep scripts/_benchmark_compat.py plus historical Python and shell filenames as thin compatibility frontends. Preserve docs/schema/benchmark-facts-v1.schema.json because retained v1 bundles embed that exact URI; accept the former v2 URI only while loading retained results. Regenerate docs/BENCHMARK_TERMINOLOGY.md and src/foundation/profile_terms_generated.h from benchmarks/terminology.json. Update CONTRIBUTING.md, MAINTAINERS.md, docs/BENCHMARK_EXPERIMENTS.md, docs/EVALUATION_PLAN.md, and tests to use canonical paths. Verification: 205 passed, 1 skipped, 34 subtests passed; Ruff E4/E7/E9/F passed; comparisons-v1 schema validation passed for 42 retained bundles and 91 comparisons; generator --check, bash -n, git diff --check, and scripts/check-source-safety.sh passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore scripts/benchmark-index.sh, scripts/benchmark-search-graph.sh, and scripts/clone-bench-repos.sh to their upstream/main paths and exact blob hashes 756bda0, cc94147, and 883e078. Rename the branch-created single-run harness to benchmarks/run_benchmark.py. Remove historical Python frontends and scripts/_benchmark_compat.py so canonical benchmark execution has no shim or sys.path mutation. Add resolve_benchmark_script_path() and validate_benchmark_script_digest() in benchmarks/run_experiments.py. Retained scripts/benchmark-incremental-speed.py plan entries resolve at execution time without rewriting archived plans; missing cells fail closed when the resolved harness SHA-256 differs from the recorded benchmark_script_sha256. Ignore benchmark-results/ and *.facts/ while retaining the existing .worktrees/ ignore. Document run_benchmark.py versus run_experiments.py and preserve retained campaign flag and directory spellings. Verification: 207 passed, 1 skipped, 34 subtests passed; retained 42-cell audit reported 42 complete, 0 missing, 0 corrupt, 0 unplanned; Ruff E4/E7/E9/F, terminology generator --check, comparisons-v1 schema validation, git diff --check, and scripts/check-source-safety.sh passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/depindex/depindex.c mapped auto_dep_limit=0 to INT_MAX, then multiplied max_results by 5 for the manifest query and allocated INT_MAX entries for vendored dependencies. Saturate the query limit, size manifest results from returned rows, and share geometric candidate growth between npm and vendored discovery. tests/test_depindex.c covers INT_MAX discovery for both vendored Make dependencies and manifest-backed Python dependencies. CBM_ONLY_SUITE=depindex make -f Makefile.cbm test: 42 passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Move semantic_pair_classification and historical_speedup out of per-row loops in benchmarks/summarize_results.py, use pairwise for monotonic count validation, and make subprocess check behavior explicit. Use contextlib.suppress for expected score-conversion and process-group cleanup errors, and render capability differences and lifecycle rows without append-only transformation loops. Verified with Ruff E4/E7/E9/F/B/SIM/PERF/PLW (PERF401 excluded as a readability-neutral preference), Ruff format, and 204 passed, 1 skipped, 34 subtests across the five retained benchmark suites. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Exact incremental publication could retain lsp_constructor confidence 0.85 for a package re-export while a fresh rebuild emitted import_map confidence 0.95 for the same canonical target. Restrict calls_refresh_reexport_resolution in src/pipeline/pass_calls.c to store-backed exact publication and require cbm_pipeline_import_map_entry_is_reexport to prove the current IMPORTS edge re-exports the selected qualified name. Direct imports and different LSP targets retain existing precedence. Reuse import_map_source_file in src/pipeline/pass_pkgmap.c and centralize direct import-map strategy recognition in src/pipeline/registry.c. Configure pipeline_minhash_incremental_new_clone with incremental_derived_results_refresh=at_publish so its SIMILAR_TO freshness assertion explicitly requests derived-view regeneration. Verified: registry 61 passed; pipeline 385 passed; simhash 24 passed; incremental 164 passed; git diff --cached --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
cbm_cli_print_main_help omitted check_index_coverage even though classic tools/list and installed evidence guidance advertise it. Add the tool beside index_status and correct README.md classic-tool counts from 15 to 16. Verified: build/c/codebase-memory-mcp --help prints config preset <list|apply> and check_index_coverage; CLI suite 243 passed; git diff --cached --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
ha_load_guidance_config previously rendered any negative auto_dep_limit from a retained or manually edited _config.db as unlimited, while dependency indexing falls back to CBM_DEFAULT_AUTO_DEP_LIMIT. Reuse cbm_dep_normalize_configured_limit, document the configured and effective sentinel contracts in src/depindex/depindex.h, and consolidate the raw-config fixture in tests/test_helpers.h. Clarify in docs/BENCHMARK_EXPERIMENTS.md that retired script paths are load-time aliases rather than executable wrappers. Verified CBM_ONLY_SUITE=cli (243 passed), CBM_ONLY_SUITE=pagerank (60 passed), CBM_ONLY_SUITE=depindex (42 passed), changed-line clang-format checks, and git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
build_automatic_spec emitted optional-graph-disabled and minimal-indexing with identical candidate labels and capability manifests, repeating the same measurements under two names. Emit only minimal-indexing in new automatic plans. Keep optional_graph_disabled as a retained-plan loader alias backed by the shared MINIMAL_INDEXING_CAPABILITIES map, document that compatibility boundary, and assert candidate/capability signature uniqueness. Correct the auto_dep_limit effective-sentinel comment in src/depindex/depindex.c. Verified 207 passed, 1 skipped, and 34 subtests across the six benchmark Python suites; Ruff format/check and git diff --check passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
docs/BENCHMARK_EXPERIMENTS.md presented optional_graph_disabled as a current command even though new experiment plans emit only minimal_indexing. Document minimal_indexing as the active command and retain optional_graph_disabled solely as a historical-plan loader alias. Align README.md with the 16-tool classic contract asserted by tests/test_tool_consolidation.c, remove a stale source-line reference in tests/test_mcp.c, and use CBM_DEFAULT_AUTO_INDEX_LIMIT for CLI fallback assertions. Verification: CBM_ONLY_SUITE=cli make -f Makefile.cbm test (243 passed); CBM_ONLY_SUITE=mcp make -f Makefile.cbm test (273 passed); bash scripts/check-source-safety.sh. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Apply Ruff's canonical single-line formatting to the Path expressions that load benchmarks/fact_comparisons.py and benchmarks/summarize_results.py. Verification: uv run --with pytest python -m pytest tests/test_benchmark_fact_comparisons.py tests/test_summarize_benchmark_results.py -q (55 passed); ruff format --check and ruff check over benchmarks and benchmark tests. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/depindex/depindex.c now applies dep_max_files with CBM_MODE_DEP bounded discovery before parsing, skips oversized packages atomically, preserves the existing cbm_dep_auto_index_effective API as a wrapper, and records package/file-cap statistics. src/mcp/mcp.c returns dependency_auto_index fields and actionable index_dependencies recovery from index_repository. src/cli/hook_augment.c reports active auto_dep_limit and dep_max_files values through the shared normalizers; disabled and unlimited configurations omit inapplicable recovery text. README.md, docs/CONFIGURATION.md, and the cli.c registry document the enforced behavior. Tests replace the former 1000==1000 assertion with real skip/unlimited fixtures and cover serialized MCP feedback plus config-sensitive hook guidance. Verification: CLI 243 passed; MCP 274 passed; depindex 42 passed; tool_consolidation 113 passed; input_validation 56 passed; bash scripts/check-source-safety.sh; git diff --cached --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
CLAUDE.md previously described internal/cbm as a Go/CGO wrapper even though CONTRIBUTING.md identifies the project as a pure C binary and internal/cbm contains Tree-sitter extraction code. Name that actual ownership instead.\n\nCONTRIBUTING.md hardcoded 14 MCP tools while src/mcp/mcp.c now derives the canonical count from TOOLS. Remove the drift-prone count and retain the JSON-RPC transport description.\n\nVerification: git diff --check Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
A unique_name registry fallback could bind a C call such as format() to an unrelated Variable extracted from a JSON schema. Registry insertion order then changed the CALLS target between incremental and clean benchmark graphs. Add cbm_pipeline_should_suppress_weak_noncallable_call_target in src/pipeline/pipeline_internal.h and apply it in resolve_single_call and resolve_file_calls. Keep Function, Method, type-like constructors, exact LSP targets, and strong same_module/import_map matches so callable C variables remain supported. Add focused coverage in tests/test_pipeline.c and tests/test_lang_contract.c. Verified CBM_ONLY_SUITE=pipeline (386 passed) and CBM_ONLY_SUITE=lang_contract (39 passed). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Indent the inputSchema contract comment with the surrounding _hidden_tools object construction in cbm_mcp_tools_list_range at src/mcp/mcp.c:3158. This corrects session-introduced changed-line drift without reformatting pre-existing repository-wide violations.\n\nVerification: CBM_ONLY_SUITE=mcp make -f Makefile.cbm test (274 passed); bash scripts/check-source-safety.sh; git diff --cached --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
tests/test_matrix_known_classes.c previously treated any CALLS edge in the C++ operator[] fixture as proof of operator resolution. The edge was actually a weak match to a data field and disappeared when cbm_pipeline_should_suppress_weak_noncallable_call_target began enforcing callable targets.\n\nRename the case to state its contract, require zero false CALLS edges, retain the observed USAGE edge, and leave the missing operator[] desugaring documented as a known gap.\n\nVerification: CBM_ONLY_SUITE=matrix_known_classes make -f Makefile.cbm test (43 passed); git diff --cached --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
DeusData
added
bug
DeusData
added
parsing/quality
Owner
|
Thanks for this — there is clearly a lot of substantial work here, and I want to be straight with you about the review problem it creates rather than let it sit silently. As it stands the PR spans 17 top-level directories: The title promises incremental-indexing parity and a 3x edit-indexing win, and that is a change I am genuinely interested in. But it is currently bundled with a benchmark harness, documentation, packaging and tooling changes, and this project reviews atomically — one PR, one claim. A change of this size touching this many subsystems cannot be reviewed responsibly as a unit, and the incremental-indexing work deserves better than being approved or rejected on the strength of everything around it. Could you split it? The natural seam looks like: (1) the incremental-indexing correctness and performance change in Two mechanical notes: the branch is CONFLICTING against I realise splitting completed work is annoying, and I would rather ask than quietly let a 124k-line PR go stale. The incremental-indexing piece is the part I most want to see land. |
Contributor
Author
|
Thanks for the positive feedback about incremental indexing, and I totally understand the review friction. A diff touching 17 directories looks heavy, but I want to explain why splitting this work breaks functional validation and why trying it live is the lowest-friction path forward. First, I am in the middle of merging latest upstream main right now and hope to land that update later today. Because upstream main moves so fast, full merging and validation will likely trail slightly behind HEAD again. Merging non-trivial upstream changes and running full validation takes about two full days of compute and testing. That cycle includes the actual merging process and going through all the changes to repair previously repaired and new bugs, production builds, AddressSanitizer memory safety checks, and the 39-cell benchmark suite across 9 languages to ensure exact clean-rebuild graph parity. If this work is broken into three separate PRs, managing three parallel merge and 2-day validation loops against a fast-moving main becomes unmanageable. Landing the change cleanly may ultimately require a brief two-day window of minor merges upstream so a conflict-free, fully validated commit can land without main immediately jumping ahead. Second, the benchmark harness under benchmarks/ and the engine logic cannot be separated. The runner is not extra bloat; it directly toggles the configuration switches and preset flags added in this PR to run ablations and verify that incremental indexing does not silently drop graph edges. Likewise, the incremental engine depends on underlying fixes to storage locking, class-method edge linking, and Cypher query bounds. Separating the harness or sub-fixes from the engine removes the exact mechanism that proves correctness. I did very deep manual and automated validation to guarantee that net bugs drop significantly once this lands. Additionally, as noted in the original description, any default parameters, tool lists, or feature flags can be trivially tuned to match whatever preferences you want for upstream. The flag registry was built specifically so default behaviors are easy to adjust without touching core engine logic. Because reading a 124k-line diff line by line is hard, the lowest-cost option on your end is simply to check out the branch and try running it. You can execute Once my current merge and validation pass finishes landing, hopefully later today (though it might trail a bit again), please give the build and benchmark suite a quick run and try using it live for a bit. If you test it and still feel the scope cannot land as a single unit due to review rules, I understand completely, and I will simply continue maintaining these performance and quality standards, harness tests, and graph-parity guarantees on my existing fork. But please give it a quick test run before making a final call. |
Contributor
Author
|
Update: it is taking me a bit longer than I hoped but I'm now doing benchmarks on the merged version. I expect to be done today. Also it might be worth having benchmarks of reasonable duration be part of CI, to be able to compare and prevent performance regressions on an ongoing basis. Though GitHub CI might have some variations in overhead so that might be noisier than desired. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
True incremental indexing: clean-rebuild parity, 3× edit-indexing speedup, and measured search-quality gains
1. What does this PR do?
Hey, thanks again for the cool project! I've updated my earlier PR #151 addressing the feedback and fixing many bugs from upstream main, eg where incremental indexing after an edit returns a graph that differs from a clean rebuild. This PR makes incremental source-structure indexing exact; with derived results refreshed at publish or optional passes disabled, all results enabled by that configuration also equal an independent clean rebuild. In the measured
minimal-indexingconfiguration that reaches clean-rebuild parity, it indexes the large-codebase edit about 3× faster than upstream main.The PR adds optional indexing of installed dependency sources with package identity, optional graph-derived search ranking, and a shorter default MCP tool list, and it extends the read-only Cypher subset, Git-snapshot change detection, and semantic analysis.
This PR also connects classes to their member functions in the graph. On upstream main a class node can be stored with no edge to its methods, so classes can be absent from traversals and rankings entirely. PR #151 included this fix, and review flagged it as a newly introduced bug; the regression tests and the before/after graphs below show it repairs missing links for all indexed languages. Since findings like this are easy to misread from descriptions alone, I would appreciate review against the code, tests, and published run reports rather than summaries.
The full classic API, request schemas, and upstream 3D visualization are retained, and the upstream main base (
7d6cdb23ef5c) is merged. A newer local upstream snapshot (97ce23f98271) is not part of this PR and does not change the verified base. Every major capability has a one-command switch and defaults can be trivially revised if preferred.2. Capabilities and controls
Every setting is declared in one registry with its default, accepted values, and environment override:
CBM_CONFIG_REGISTRYinsrc/cli/cli.c:10157. Two commands control everything:The six preset bundles pair a tool surface (
streamlined-orclassic-) withautomatic-dependency-source-indexing-disabledor-enabled, plus therank-disabledandminimal-indexingbenchmark ablations (codebase-memory-mcp config preset listshows each name with its description; the apply transaction iscbm_config_apply_preset()incli.c:10087). When an environment override pins one of a preset's keys, apply returns status 1 rather than claiming an ineffective change.File names below are unique in the tree and live under
src/; each switch is the exact argument list forcodebase-memory-mcp, with the default in parentheses.cbm_pipeline_run_incremental()pipeline_incremental.c:2816config set incremental_reindex always(default
always;full_rebuildrebuilds atomically from scratch andfast_mode_indexes_onlyrestricts incremental reindexing to fast-mode indexes)DEFINES_METHODedges to their methods, aMEMBER_OFreverse edge, and a normalization pass that repairs missed links across languages, so classes participate in traversals and inherit rank from their members instead of being disconnectedprocess_def()pass_definitions.c:321-358repair:
pass_normalize.czz_order_core, moved from ninth to firstcbm_pagerank_compute()pagerank.c:303config set rank_enabled false(default
true)index_dependenciesindexes named packages on demand, and when installed packages exceedauto_dep_limit(default 20, 0 = unlimited, max 10,000) the most-imported packages are selected, ranked by project import referencescbm_dep_auto_index_effective()depindex.c:964ranking:
depindex.c:614config set auto_index_deps true(default
false)_hidden_tools; calling_hidden_toolsreveals the full classic tool list live, no restart (upstream advertises all tools and curates only statically via startup--profileflags)cbm_mcp_tools_list_range()mcp.c:3072config set tool_mode classic(default
streamlined)formatoverride and an actionable invalid-format errorcbm_mcp_response_format()mcp.c:2454config set default_response_format json(default
toon)auto_indexin
CBM_CONFIG_REGISTRYconfig set auto_index false(default
true; upstream documents false)WITH, and optional-match semantics; adds multi-key ordering, output caps, and schema-aware rejection of unsupported syntax; the tool description, parser, and executor share one definitioncbm_cypher_execute_impl()cypher.c:5863cbm_pipeline_pass_semantic_edges()pass_semantic_edges.c:1473config set semantic_edges_enabled falseconfig set similarity_enabled false(default
truefor both)cbm_git_snapshot_read()git_snapshot.c:361config set githistory_enabled false(default
true)7fa3acd0c6af), and fixes missing or misclassified HTTP caller-to-handler linkscbm_pipeline_pass_httplinks()pass_httplinks.c:1394config set httplinks_enabled false(default
true)at_publishrecomputes everything immediatelycbm_pipeline_run_incremental()pipeline_incremental.c:2816config set incremental_derived_results_refresh at_publishconfig set rank_refresh at_publish(default:
defer_all_incremental_reindexesfor both)3. Measured results
The headline numbers come from a 39-cell benchmark experiment run at the benchmark production commit: 13 configurations, each run 3 times in sequence and isolation (repetitions paired and interleaved across configurations to reduce timing drift), on production builds of upstream main and this PR. Every cell applies the same real C source edit to a large codebase, runs incremental indexing (
incremental_reindex always, this PR's default; upstream main has no such switch and always uses its incremental path), and compares the resulting graph with a separate clean rebuild after the same edit; every configuration row also passes 12 retrieval-evidence checks (4 post-edit retrieval probes, finding the changed file, its source text, its graph node, and its architecture evidence, across the 3 repetitions).Each focused experiment poses a fixed search task whose single correct answer is known in advance, and the retrieval metrics score the returned result list against that known answer: MRR (Mean Reciprocal Rank) rewards placing it near the top, Hit@1 and Hit@5 report whether it appears first or within the first five, and nDCG@5 (normalized Discounted Cumulative Gain) rewards correct ordering within the first five. Pair F1 is the F1 score over the explicitly judged
SEMANTICALLY_RELATEDpairs (pairs of code entities the semantic analysis links), penalizing both missing and spurious pairs. Task success means the task's named expected answer was returned.Repeated query is the median latency over the repeated read-only graph queries the retrieval probes issue. RSS is the highest peak resident memory observed across the 3 repetitions. For every quality metric 1.000 is best and 0 is worst. "Deferred" means the capability is enabled and the server explicitly reports that the named optional result awaits recomputation after the fast source-structure refresh.
minimal-indexingpreset‡ Refresh-at-publish recomputes every derived result; upstream computes none of them and returns a wrong graph, so this ratio compares unlike work.
Upstream main's post-edit reindex is labeled incremental but returns a graph that differs from the clean rebuild (speed bought with a wrong result), while this PR's default is both 2.3× faster (461 vs 1,044 ms) and correct on source structure. The upstream-equivalent row is the closest like-for-like comparison, matching upstream's feature set: it reindexes the edit 2.3× faster (460 vs 1,044 ms) at the cost of a 1.5× longer full index (6.907 vs 4.702 s). The minimal preset outputs a graph identical to the clean rebuild while indexing the edit 2.9× faster (357 vs 1,044 ms) and full-indexing 1.27× faster (3.693 vs 4.702 s) than upstream main. Upstream keeps an 88 MB (7%) lower peak RSS than minimal, the one metric this PR does not win.
The default configuration's higher full-index time (10.677 vs 4.702 s) is the cost of the optional capabilities it enables: disabling rank reduces the measured full index to 6.912 s, and disabling semantic edges reduces it to 7.847 s. It is not slower core indexing: with the optional capabilities off (minimal preset) this PR full-indexes faster than upstream. Automatic dependency indexing is off by default so its cost stays out of the default first index; enabling it adds an 18.132 s initial dependency index for this workload's six installed packages, which is work upstream cannot do, and after an edit, checking unchanged dependencies costs about 5 ms.
Refresh-at-publish recomputes every global derived result (rank, semantic edges, architecture, routes) immediately after the edit; at 5.968 s that is still 1.75× faster than the 10.428 s full rebuild, which is the only other way to get them all fresh. In the speedup column, the first value compares each configuration's post-edit reindex with its own fresh full rebuild after the identical edit, both measured in the same cell; the second compares it with upstream's post-edit reindex. Every configuration of this PR beats its own full rebuild, by 66.83× with dependency indexing enabled. Refresh-at-publish is opt-in; the default defers exactly that graph-wide work and explicitly reports the deferred results as stale until refreshed. These ratios apply to the named revisions, edit, and machine.
Result-quality contribution of the two new optional capabilities, isolated on one PR build and test project by repeating the same index-then-query measurement 5 times per state, alternating the capability between enabled and disabled:
zz_order_core) ahead of eight lexical-only decoyszz_order_corereturned 1st of 9; MRR, Hit@1, Hit@5, nDCG@5, task success all 1.000zz_order_core9th of 9; MRR 0.111, the other four metrics 0canonicalDependencyAPI) with its dependency, package, and read-only provenance on the same resultcanonicalDependencyAPIreturned 1st, with package identitycanonicalDependencyAPIabsentSemantic refresh quality was measured at revision
c9726c7fd9b7with 3 repetitions of the same edit-refresh-compare measurement: with refresh enabled, every judgedSEMANTICALLY_RELATEDpair matches the clean rebuild, with none missing and none spurious (Pair F1 and task success 1.000); with the capability disabled, the semantic task scores 0. Separately, the latest large-codebase benchmark at524b6b6f3ee7measures the refresh-at-publish configuration at 5.968 s per edit.4. Upstream main defects found and fixed
This PR corrects the following upstream main failures:
cbm_pipeline_run_incremental()pipeline_incremental.cprocess_def()pass_definitions.crepair:
pass_normalize.cleaks: 0 bytes)store_open_internal()store.ccbm_mcp_server_free()mcp.cresolve_store()mcp.cWITHstages, optional matches, and multi-keyORDER BYcompute correctlycbm_cypher_execute_impl()cypher.cWITHstages, and optional matches mis-computed or failed on advertised syntax; aWITHalias could replace a result's qualified name; secondaryORDER BYkeys were silently droppedLIMITlowers but cannot bypass the cap (regression test: a 65-match lookup upstream cut at 50)scan_pattern_nodes()cypher.cWHEREfilter rancbm_pipeline_pass_httplinks()pass_httplinks.c7fa3acd0c6af)auto_indexin
CBM_CONFIG_REGISTRYfalse, so indexing never startedcbm_pipeline_pass_configlink()pass_configlink.cindex_statusreports whether the working tree is dirty and whether the index matches itadd_git_context_json()mcp.cEvery row is regression-tested, with one exception: the crash fix's trigger, a small-stack background thread, is impractical to reproduce in a test, so that fix was verified by reproducing the crash before and after the change, and the rewritten pass has its own tests.
Defects introduced while developing this PR were fixed before release; each is documented in its commit message, and none remains open at final verified HEAD
be89d4969a77. Two behaviors are boundaries rather than defects: the Cypher subset does not claim full openCypher conformance, and optional derived results refresh on the documented deferred schedule. Productionsrc/contains nosystem()calls (allowlisted test helpers excluded), and the heap-leak gate reports 0 leaked bytes.5. Compatibility, breaking changes, and migration
config set tool_mode classicrestores the classic list on a running installation.--reset-indexesis passed; the verified system-wide upgrade preserved all 462 existing indexes across 17 detected clients, after an isolated install/uninstall smoke and source-safety check passed, andcodex doctorreports17 ok · 0 warn · 0 fail. Uninstall also removes the YAML editor's persistent lock sidecars beside managed configs.transportfield, the legacy JSON MCP"enabled": truemember (while"enabled": falseis still preserved as user-modified), and Hermes hook configs containing flow-style YAML such asplugins.enabled: []. It keeps user-owned TOML child tables when replacing a managed parent table and refuses to overwrite genuinely user-edited entries.6. How to run the tests
The full suite passed with zero failures at the final verified commit (7,235 passed, one Windows-only skip), and the heap-leak gate passed 1,530 tests with 0 leaked bytes. The suite includes generalized extraction and incremental tests across C, C++, Go, Python, JavaScript, TypeScript, Java, Kotlin, and Rust, plus mixed-language call resolution.
7. Evidence and revisions
7d6cdb23ef5c97ce23f98271524b6b6f3ee7be89d4969a7713a4679b27f6c9726c7fd9b7All experiments ran on production builds; the generated reports are published in a gist. Its current tree contains the 39-cell run set
be2097bdbf14and matching fact appendix; older focused ranking, dependency, and semantic reports remain in the gist revision history. The experiment root retains the input specification, per-run manifests, logs, build identities, and cleanup status, so the numbers are independently reproducible with the command above.Ranking follows PageRank (Brin and Page, 1998) and LinkRank (Kim, Son, and Jeong, 2010); nDCG (Järvelin and Kekäläinen, 2002) defines the ranking metric; RepoGraph (ICLR 2025) motivates repository graphs. The repository experiments, not the citations, are the evidence that this implementation works.
8. Checklist
git commit -s) (DCO, see CONTRIBUTING.md)make -f Makefile.cbm test)make -f Makefile.cbm lint-ci)