Skip to content

ci: measure AGENTS.md and CLAUDE.md token growth per PR - #3392

Open
Naeha-S wants to merge 2 commits into
ultraworkers:mainfrom
Naeha-S:ci/ctxcap-context-budget
Open

Naeha-S wants to merge 2 commits into
ultraworkers:mainfrom
Naeha-S:ci/ctxcap-context-budget

Conversation

@Naeha-S

@Naeha-S Naeha-S commented Nov 16, 2026

Copy link
Copy Markdown

Summary

Adds a workflow that measures the token cost of the harness context files (AGENTS.md, CLAUDE.md, and their rust/ counterparts) and fails a PR that grows the total. Current state on main: 1,893 tokens across the two root files.

Anti-slop triage

  • Classification: actionable-fix
  • Evidence: npx ctxcap@1.5.1 --root . prints the table below; the workflow in this PR is the repro.
  • Non-destructive review result: merge candidate

Verification

TOKENS  DELTA  FILE
1643    +1643  AGENTS.md
250     +250   CLAUDE.md
total 1893
  • Targeted tests/docs checks ran, or the gap is explicitly recorded.
  • git diff --check passes.
  • No live secrets, tokens, private logs, or unrelated generated churn.

Resolution gate

  • If this PR resolves an issue, the issue number and fix evidence are linked. (No issue; this is the kind of small CI addition CONTRIBUTING.md invites directly.)
  • If this PR should not merge, the rejection/defer rationale is evidence-backed and does not rely on vibes.
  • I did not merge/close remote PRs or issues from an automation lane without owner approval.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 13:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Naeha-S

Naeha-S commented Nov 16, 2026

Copy link
Copy Markdown
Author

Pushed a follow-up: .ctxcap.json, ctxcap-baseline.json, and a widened paths: filter.

Two things were wrong with the first version.

It only measured 2 of the 11 files. The default include set is root-level only, but this repo keeps per-crate instructions in rust/crates/*/AGENTS.md and src/AGENTS.md, which is what a harness working in those directories is actually given. .ctxcap.json switches the scope to **/AGENTS.md and **/CLAUDE.md, so the measured total is 9,008 tokens rather than 1,893, and the workflow triggers on a change to any of the eleven.

It would have failed the next context-file PR for the wrong reason. --baseline <merge-base> reads ctxcap-baseline.json at that ref; with no baseline file, the delta is the file's whole size instead of its growth. The baseline is committed here so growth is measured against it, and an intended increase is approved by reviewing the baseline diff.

Verified against main at the commit in this branch: delta 0, exit 0. With 30 lines added to rust/crates/api/AGENTS.md: +480 on that file, exit 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants