Skip to content

fix(plugin): fire on_user_function_end with SUSPENDED on suspension - #648

Draft
wangyb-A wants to merge 1 commit into
fix/otel-balance-context-scopesfrom
fix/user-function-end-on-suspension
Draft

fix(plugin): fire on_user_function_end with SUSPENDED on suspension#648
wangyb-A wants to merge 1 commit into
fix/otel-balance-context-scopesfrom
fix/user-function-end-on-suspension

Conversation

@wangyb-A

Copy link
Copy Markdown
Contributor

Stacked on #647. Base is fix/otel-balance-context-scopes; review that first. It touches the same on_user_function_end functions, and landing the core change without the OTel side would export suspended attempt spans as if they had completed successfully.

Problem

wrap_user_function re-raised SuspendExecution without calling on_user_function_end (state.py:1171):

except SuspendExecution:
    raise
except Exception as e:
    self._plugin_executor.on_user_function_end(start_info, ErrorObject.from_exception(e))
    raise

This was deliberate — test_wrap_user_function_suspend_does_not_fire_end_hook pinned it, on the rationale that "the plugin observes it by absence (no end hook fires), with the instrumentation plugin's own per-invocation span sweep closing any open spans cleanly at invocation end."

That contract cannot be honoured for state that is thread-confined. The OTel plugins attach an opentelemetry.context token in on_user_function_start, and a token is only detachable in the contextvars.Context that created it — the user-code worker thread, not the handler thread the invocation hooks run on. So a suspended operation stranded its context scope with no hook able to release it: the invocation-end sweep runs on the wrong thread. The same applies to any plugin holding per-operation state — a timer, an open log group, a span.

Suspension is also not a rare path. It is the normal outcome of any child context whose inner operation is still pending, which is the reachable case here: SuspendExecution propagates out of the child context's user function, and the hook silently never fires.

Java already fires the hook here. BaseDurableOperation.runUserFunction catches Throwable — which covers SuspendExecutionException — and the javadoc gives the same reason this PR does:

onUserFunctionEnd fires for failures and suspensions alike so plugins (e.g. OTel) can end/clean up the attempt rather than leak state.

Changes

  • UserFunctionOutcome.SUSPENDED. Reported as its own outcome rather than reusing FAILED: nothing went wrong, and plugins that count failures or set an error span status must not treat it as one. Java models this as succeeded=false plus the suspend exception as the error, which reads as a failure to exactly those consumers.
  • Explicit outcome override on UserFunctionEndInfo.from_start_info and PluginExecutor.on_user_function_end, so the suspension path can report SUSPENDED with error=None rather than deriving the outcome from an absent error.
  • Fire the hook from wrap_user_function's SuspendExecution branch, then re-raise unchanged — durable control flow is untouched.
  • Both OTel plugins treat SUSPENDED as "release the scope, leave the span open". The attempt has not concluded, so it must not be ended with an outcome here; it is ended when the operation reaches a terminal status, matching how an operation that suspends mid-invocation is already handled.

Note SuspendExecution derives from BaseException, not Exception, so the existing except Exception clause never caught it — the gap was structural, not a missing branch. This PR keeps the catch narrow (SuspendExecution only); whether wrap_user_function should also fire the hook for other BaseExceptions such as KeyboardInterrupt, as Java's catch (Throwable) effectively does, is left as a separate question.

API impact

Adding an enum member is source-compatible, but plugin authors who exhaustively branch on UserFunctionOutcome will now see a third value. Anyone writing if outcome is FAILED: ... else: <treat as success> will classify a suspension as success, which is the intended reading for span status but may not be for metrics.

This also makes Python the first of the three SDKs with a third outcome. JS has no hook on this path at all — its attempt hooks cover steps and waitForCondition only, not child contexts, and its plugins scope context through wrapOperationAttemptFn/wrapChildContextFn, which restore on throw automatically. Java reports suspension through the existing boolean. Worth deciding whether JS and Java adopt SUSPENDED for parity; happy to split this into a discussion first if the team would rather agree the shape before the code lands.

Testing

3229 pass across the monorepo; hatch fmt --check and hatch run types:check clean.

  • test_wrap_user_function_suspend_does_not_fire_end_hook is inverted into test_wrap_user_function_suspend_fires_end_hook_with_suspended_outcome, asserting one end hook with SUSPENDED and error is None.
  • New end-to-end test drives a real run_in_child_context that suspends and asserts exactly one SUSPENDED end hook and no FAILED one, with the invocation still returning PENDING.
  • New OTel tests assert a suspended attempt releases its scope, restores the prior context, exports nothing, and is never marked ERROR (parametrised over both plugins).
  • The UserFunctionOutcome value-set test is updated.

Both halves mutation-tested: removing the hook call fails the two core tests; ignoring SUSPENDED in the plugins fails the four OTel tests.

@wangyb-A
wangyb-A force-pushed the fix/otel-balance-context-scopes branch from ceb4551 to c79cfda Compare August 14, 2026 22:12
@wangyb-A
wangyb-A force-pushed the fix/user-function-end-on-suspension branch from 3fe5c79 to d868e27 Compare August 14, 2026 22:19
@wangyb-A
wangyb-A force-pushed the fix/user-function-end-on-suspension branch from d868e27 to e1e03c7 Compare August 14, 2026 23:03
wrap_user_function re-raised SuspendExecution without calling
on_user_function_end, so a user function that stopped so the
execution could resume later never reported its end. Plugins were
expected to "observe it by absence" and clean up during their own
invocation-end sweep.

That contract cannot be honoured for state that is
thread-confined. The OTel plugins attach an opentelemetry.context
token in on_user_function_start, and a token is only detachable in
the contextvars.Context that created it -- the user-code worker
thread, not the handler thread the invocation hooks run on. A
suspended operation therefore stranded its context scope with no
hook able to release it. The same applies to any plugin holding
per-operation state: a timer, an open log group, a span.

The Java SDK already fires the end hook here.
BaseDurableOperation.runUserFunction catches Throwable -- which
covers SuspendExecutionException -- and its javadoc gives the same
reason: onUserFunctionEnd fires for failures and suspensions alike
so plugins can clean up the attempt rather than leak state.

Changes:

- Add UserFunctionOutcome.SUSPENDED. Suspension is its own
  outcome rather than reusing FAILED: nothing went wrong, and
  plugins that count failures or set an error status must not
  treat it as one. Java models this as succeeded=false plus the
  suspend exception as the error, which reads as a failure to
  exactly those consumers.
- Allow an explicit outcome on UserFunctionEndInfo.from_start_info
  and PluginExecutor.on_user_function_end, so the suspension path
  reports SUSPENDED with error=None instead of deriving the
  outcome from an absent error.
- Fire the hook from wrap_user_function's SuspendExecution branch
  and re-raise unchanged, so durable control flow is untouched.
- Teach both OTel plugins to treat SUSPENDED as "release the
  scope, leave the span open": the attempt has not concluded, so
  it must not be ended with an outcome here. It is ended when the
  operation reaches a terminal status, matching how an operation
  that suspends mid-invocation is already handled.

test_wrap_user_function_suspend_does_not_fire_end_hook pinned the
old behaviour and is inverted accordingly. Adds an end-to-end test
driving a real child context that suspends, and OTel tests
asserting a suspended attempt releases its scope, exports nothing,
and is never marked ERROR.

Note for reviewers: this makes Python the first of the three SDKs
with a third user-function outcome. JS has no hook on this path at
all, and Java reports suspension through the existing boolean. A
follow-up should decide whether JS and Java adopt SUSPENDED.
@wangyb-A
wangyb-A force-pushed the fix/user-function-end-on-suspension branch from e1e03c7 to 580da37 Compare August 14, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant