Skip to content

Replace per-message heartbeat timeout with a watchdog#395

Merged
Monkatraz merged 2 commits into
mainfrom
brn-heartbeat-watchdog
Jul 27, 2026
Merged

Replace per-message heartbeat timeout with a watchdog#395
Monkatraz merged 2 commits into
mainfrom
brn-heartbeat-watchdog

Conversation

@Monkatraz

@Monkatraz Monkatraz commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Why

SessionConnected tracked its inactivity deadline by clearing and re-arming a setTimeout on every valid inbound frame, inside updateBookkeeping(). That means a timer and a closure allocated per message. On high-throughput browser sessions this is significant allocation churn, and each cleared timer becomes garbage whose callback captures this, briefly retaining the session graph. Browser profiling on a downstream consumer attributed hundreds of megabytes to this path.

Note that even an idle session churns here, since heartbeat traffic alone re-arms the timeout every heartbeatIntervalMs.

The per-message timeout dates to #302, which moved away from an interval plus a miss counter because browser background throttling made counting interval executions unreliable. This keeps that fix while dropping the allocation.

What changed

The session now records a lastInboundAt timestamp on each inbound frame and runs a single interval for its lifetime that closes the connection once Date.now() - lastInboundAt passes the deadline. Inbound messages no longer allocate any timer. Detection stays correct under browser throttling because it compares actual elapsed wall time instead of counting interval executions, so a delayed or suspended timer can only postpone noticing a dead connection, never invent a missed heartbeat.

Reviewer pointers:

  • startMissingHeartbeatTimeout() is renamed to startHeartbeatWatchdog(), since it no longer arms a timeout. Both call sites are in transitions.ts. Teardown moved to a private clearHeartbeatWatchdog(), mirroring the existing clearRehandshakeTimer(), and is still called from _handleStateExit().
  • The watchdog clears itself before calling conn.close(). An interval would otherwise keep ticking and re-closing while the connection's close event propagates asynchronously; the old timeout fired exactly once, and this preserves that.

One behavioral difference

Detection is now quantized to the check interval. A dead connection is noticed between deadlineMs and deadlineMs + heartbeatIntervalMs after the last frame, rather than exactly at deadlineMs. With the defaults (1s interval, 2 heartbeats until dead) that is 2s to 3s instead of exactly 2s. This is inherent to polling, and erring late is the safe direction.

If the extra timer per session is ever a concern at high session counts, the natural follow-up is one transport-level watchdog scanning every session's lastInboundAt. That was deliberately left out here to keep the change small.

Testing

The heartbeats describe block had no coverage of the death path itself, so this adds two tests: the connection closes after silence, and a message arriving just under the deadline keeps pushing the deadline out. I confirmed both are meaningful by removing the lastInboundAt update and watching them fail.

Full suite passes (770 passed, 1 skipped), and npm run check is clean. Since this changes timing behavior, I also ran the four timing-sensitive suites (disconnects, transport, stateMachine, cleanup) 12 times consecutively with no flakes.

The second commit fixes the two new tests, which failed on the Ubuntu runner in the first push. They originally asserted the session was still alive 1ms before the deadline. The suite runs with shouldAdvanceTime, so the fake clock also moves with real time, and on a contended runner that 1ms margin was crossed and the watchdog fired before the assertion. They now use the existing advanceFakeTimersByHeartbeat and advanceFakeTimersByDisconnectGrace helpers, which leave a full heartbeat interval of slack on either side of the deadline, matching how the rest of the suite handles heartbeat timing. This was purely a test bug; production behavior is unchanged, and both tests still fail if an inbound message stops refreshing the timestamp.

Versioning

  • Breaking protocol change
  • Breaking ts/js API change

No wire format change, so a client on either version interoperates with a server on the other. Bumped to 0.220.1.

Worth a second opinion on the API checkbox: SessionConnected is exported as a type from ./transport, so the startMissingHeartbeatTimeout() rename is technically visible to anyone who typed against it and called that method. Nothing outside the state machine has any reason to call it, so I left the box unchecked, but say the word and I will restore the old name or keep it as an alias.

Every inbound frame cleared and re-armed a setTimeout to track the
inactivity deadline, so a busy session allocated a timer and closure per
message. Track the last inbound timestamp instead and let a single
interval per session compare elapsed wall time against the deadline.

Comparing Date.now() rather than counting interval executions keeps this
safe under browser timer throttling: a delayed or suspended timer can
only postpone detection of a dead connection, never report a heartbeat as
missed while messages keep arriving.
@Monkatraz
Monkatraz requested a review from a team as a code owner July 26, 2026 23:47
@Monkatraz
Monkatraz requested review from poorvapotnis and removed request for a team July 26, 2026 23:47
The tests asserted the connection was still alive 1ms before the
inactivity deadline. The suite runs with shouldAdvanceTime, so the fake
clock also moves with real time, and on a contended runner it crossed
that 1ms margin and fired the watchdog early.

Use the existing heartbeat and disconnect grace helpers instead, which
leaves a full heartbeat interval of slack on either side of the deadline.
Both tests still fail if an inbound message stops refreshing the
timestamp.
@Monkatraz
Monkatraz merged commit e368b02 into main Jul 27, 2026
7 checks passed
@Monkatraz
Monkatraz deleted the brn-heartbeat-watchdog branch July 27, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants