Skip to content

fix(web): reconcile subagents stuck at "running"#1969

Open
youngting520 wants to merge 1 commit into
MoonshotAI:mainfrom
youngting520:fix/1963-stale-running-subagents
Open

fix(web): reconcile subagents stuck at "running"#1969
youngting520 wants to merge 1 commit into
MoonshotAI:mainfrom
youngting520:fix/1963-stale-running-subagents

Conversation

@youngting520

@youngting520 youngting520 commented Jul 20, 2026

Copy link
Copy Markdown

Related Issue

Resolves #1963

Problem

Kimi Web can keep a foreground subagent row in running after the subagent has already finished. Reports show that this can happen both around interruption/reconnection and during an otherwise healthy session.

The Web task projection is driven by an incremental event stream, while the session snapshot provides the authoritative foreground-subagent roster. Several convergence gaps could leave the projected state stale:

  1. The client collapsed an omitted subagents field from an older server and an explicit empty roster into the same value, then treated an empty roster as a no-op. An authoritative snapshot therefore could not remove stale foreground rows.
  2. Foreground-roster reconciliation did not consistently distinguish roster-owned tasks from background, detached, and BTW side-channel agents. Applying an empty foreground roster could remove valid work or suppress later side-channel tool progress.
  3. The live projector did not preserve terminal or authoritative-absence state. A late or replayed progress frame (assistant.delta, tool.call.started, or tool.progress) could recreate a removed row as running, with no guaranteed later event to correct it.
  4. A turn-boundary reconcile could stop after stale snapshot responses, and the legacy quiet fallback could request the same reconcile again after the normal turn-end path.

The exact event sequence behind every reported occurrence has not been captured, but these paths share the same underlying problem: the Web projection lacked a reliable way to converge back to authoritative state without affecting tasks outside the foreground roster.

The TUI does not use this Web-side task projection path.

What changed

  • Preserve the distinction between an omitted roster from an older server and an authoritative empty roster. An omitted roster leaves reconciliation pending so a later authoritative snapshot can still remove stale rows.
  • Reconcile authoritative snapshots against live foreground subagents only. Preserve REST-managed background tasks, detached tasks, terminal task history, and the active BTW side-channel agent when they are absent from the foreground roster.
  • Treat fresh lifecycle events as newer evidence: explicit subagent.spawned, subagent.started, and background task.started events clear pending cleanup and intentionally reopen a previously removed row.
  • Preserve terminal metadata and authoritative-absence tombstones across reconnect/reset so late or replayed progress cannot downgrade a terminal row or recreate a roster-removed row as running.
  • Keep BTW side-channel agents outside foreground-roster ownership in the projector, snapshot merge, and turn-boundary reconcile path so their tool progress survives reset and an empty roster.
  • Request a coalesced snapshot at the main-turn boundary and on the lost-turn.ended quiet fallback whenever a live foreground subagent still needs reconciliation. A reconcile makes the initial request plus at most two serial retries when snapshots lag the local cursor, and stops early when the candidate settles. Both protocol branches require a previously active main turn, preventing duplicate requests.
  • Bound retained terminal metadata and absence tombstones to the most recent 256 entries per session.
  • Keep the projector's subagent metadata in step with the reducer's task rows across reconnects and resets.
  • Add mapping, merge, reconnect/resync, terminal-stickiness, in-flight seeding, cross-version roster, background/detached/side-channel, bounded stale-snapshot retry, duplicate-boundary, and retained-state-bound coverage.

This keeps the fix in Kimi Web and avoids changing the agent engine, WebSocket protocol, or server behavior.

Reviewer guide

  • Cross-version snapshot mapping: apps/kimi-web/src/api/daemon/client.ts and apps/kimi-web/src/api/types.ts
  • Lifecycle projection, reset reconciliation, tombstones, and retained-state bounds: apps/kimi-web/src/api/daemon/agentEventProjector.ts
  • Turn-boundary snapshot orchestration and bounded stale-snapshot retry: apps/kimi-web/src/composables/useKimiWebClient.ts
  • Foreground-roster ownership and merge semantics: apps/kimi-web/src/lib/taskMerge.ts

Validation

  • pnpm --filter @moonshot-ai/kimi-web test (684 tests)
  • pnpm --filter @moonshot-ai/kimi-web typecheck
  • pnpm --filter @moonshot-ai/kimi-web build
  • pnpm --filter @moonshot-ai/kimi-web check:style
  • pnpm lint (0 errors; existing warnings only)
  • git diff --check upstream/main...HEAD
  • pnpm changeset status --since=upstream/main

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked a related issue, or explained the problem above.
  • I have added tests that prove my feature works.
  • Ran gen-changesets skill, or this PR needs no changeset.
  • Ran gen-docs skill, or this PR needs no doc update.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46d65dc369

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/kimi-web/src/api/daemon/eventReducer.ts Outdated
Comment thread packages/agent-core-v2/src/session/subagent/mirrorAgentRun.ts Outdated
@youngting520
youngting520 force-pushed the fix/1963-stale-running-subagents branch from 46d65dc to 47d0370 Compare July 20, 2026 15:07
@changeset-bot

changeset-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 91c3b5b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@moonshot-ai/kimi-code Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@youngting520
youngting520 force-pushed the fix/1963-stale-running-subagents branch 5 times, most recently from 35916c4 to 328ce08 Compare July 25, 2026 12:49
@youngting520

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 328ce08b6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/kimi-web/src/api/daemon/agentEventProjector.ts Outdated
@youngting520 youngting520 changed the title fix(web): settle subagent entries stuck at "running" after interruption or disconnect fix(web): reconcile subagents stuck at "running" Jul 25, 2026
@youngting520
youngting520 force-pushed the fix/1963-stale-running-subagents branch from 328ce08 to 37ce1f0 Compare July 25, 2026 13:44
@youngting520

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37ce1f07f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/kimi-web/src/api/daemon/agentEventProjector.ts Outdated
@youngting520
youngting520 force-pushed the fix/1963-stale-running-subagents branch 3 times, most recently from acdaa62 to 8b41de7 Compare July 26, 2026 16:14
@youngting520

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b41de794d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +970 to +971
((appEvent.mainTurnActive === false && wasMainTurnActive) ||
(appEvent.mainTurnActive === undefined && !appEvent.busy)) &&
(appEvent.mainTurnActive === undefined && !appEvent.busy && wasMainTurnActive)) &&

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep clearing legacy quiet events without an active-turn marker

When an older server omits mainTurnActive and the client misses turn.started, wasMainTurnActive is false even though a locally submitted prompt may still have inFlightBySession set. Requiring that marker now skips clearWorkingFlags on the authoritative busy: false fallback, leaving the waiting indicator active and causing subsequent prompts to remain queued despite the session being idle. Keep the unconditional legacy quiet cleanup, and gate only the new subagent-reconciliation request on wasMainTurnActive if duplicate snapshots are the concern.

Useful? React with 👍 / 👎.

@youngting520
youngting520 force-pushed the fix/1963-stale-running-subagents branch from 8b41de7 to 3253cdf Compare July 26, 2026 16:38
@youngting520

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3253cdfe8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/kimi-web/src/api/daemon/agentEventProjector.ts
Comment thread apps/kimi-web/src/composables/useKimiWebClient.ts Outdated
Finished foreground subagents could stay 'running' in Kimi Web when the
terminal lifecycle event was missed or a late/replayed progress frame
re-opened the row.

- Preserve the distinction between an omitted subagent roster (older
  server) and an authoritative empty roster, and reconcile authoritative
  snapshots by removing stale live foreground rows while keeping
  REST-managed background tasks and terminal history.
- Request a coalesced snapshot at the turn boundary (turn.ended, plus
  the quiet sessionWorkChanged fallback for a lost turn.ended) whenever
  a live foreground subagent still needs reconciliation.
- Never let a late/replayed subagent progress frame downgrade a terminal
  row back to running; explicit lifecycle events (subagent.started on
  resume) still re-open it.
- Seed the projector's subagent meta from the same snapshot roster that
  seeds the reducer's task rows, so the terminal knowledge survives a
  resync reset.
@youngting520
youngting520 force-pushed the fix/1963-stale-running-subagents branch from 3253cdf to 91c3b5b Compare July 26, 2026 17:00
@youngting520

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91c3b5b8e0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1410 to +1414
) {
subagentReconcileStaleRetries.set(sessionId, reconcileRetryCount + 1);
snapshotSyncRunner.request(sessionId);
} else {
subagentReconcileStaleRetries.delete(sessionId);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep reconciliation pending after exhausting immediate retries

When background-agent, BTW, or other durable events advance the session cursor during all three snapshot requests, every response enters this stale-snapshot branch; the final attempt deletes the retry state without applying the authoritative roster or scheduling a later retry. Since the main turn has already ended, there may be no subsequent boundary to trigger reconciliation, so the foreground subagent remains permanently running—the new test at event-batcher.test.ts even demonstrates this terminal state. Use a delayed/backoff retry or reconcile the roster independently of the stale full-session payload rather than abandoning the candidate.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Web UI keeps showing finished subagents as running after reconnect or interruption

1 participant