fix(plugin): fire on_user_function_end with SUSPENDED on suspension - #648
Draft
wangyb-A wants to merge 1 commit into
Draft
fix(plugin): fire on_user_function_end with SUSPENDED on suspension#648wangyb-A wants to merge 1 commit into
wangyb-A wants to merge 1 commit into
Conversation
wangyb-A
force-pushed
the
fix/otel-balance-context-scopes
branch
from
August 14, 2026 22:12
ceb4551 to
c79cfda
Compare
wangyb-A
force-pushed
the
fix/user-function-end-on-suspension
branch
from
August 14, 2026 22:19
3fe5c79 to
d868e27
Compare
wangyb-A
force-pushed
the
fix/user-function-end-on-suspension
branch
from
August 14, 2026 23:03
d868e27 to
e1e03c7
Compare
wrap_user_function re-raised SuspendExecution without calling on_user_function_end, so a user function that stopped so the execution could resume later never reported its end. Plugins were expected to "observe it by absence" and clean up during their own invocation-end sweep. That contract cannot be honoured for state that is thread-confined. The OTel plugins attach an opentelemetry.context token in on_user_function_start, and a token is only detachable in the contextvars.Context that created it -- the user-code worker thread, not the handler thread the invocation hooks run on. A suspended operation therefore stranded its context scope with no hook able to release it. The same applies to any plugin holding per-operation state: a timer, an open log group, a span. The Java SDK already fires the end hook here. BaseDurableOperation.runUserFunction catches Throwable -- which covers SuspendExecutionException -- and its javadoc gives the same reason: onUserFunctionEnd fires for failures and suspensions alike so plugins can clean up the attempt rather than leak state. Changes: - Add UserFunctionOutcome.SUSPENDED. Suspension is its own outcome rather than reusing FAILED: nothing went wrong, and plugins that count failures or set an error status must not treat it as one. Java models this as succeeded=false plus the suspend exception as the error, which reads as a failure to exactly those consumers. - Allow an explicit outcome on UserFunctionEndInfo.from_start_info and PluginExecutor.on_user_function_end, so the suspension path reports SUSPENDED with error=None instead of deriving the outcome from an absent error. - Fire the hook from wrap_user_function's SuspendExecution branch and re-raise unchanged, so durable control flow is untouched. - Teach both OTel plugins to treat SUSPENDED as "release the scope, leave the span open": the attempt has not concluded, so it must not be ended with an outcome here. It is ended when the operation reaches a terminal status, matching how an operation that suspends mid-invocation is already handled. test_wrap_user_function_suspend_does_not_fire_end_hook pinned the old behaviour and is inverted accordingly. Adds an end-to-end test driving a real child context that suspends, and OTel tests asserting a suspended attempt releases its scope, exports nothing, and is never marked ERROR. Note for reviewers: this makes Python the first of the three SDKs with a third user-function outcome. JS has no hook on this path at all, and Java reports suspension through the existing boolean. A follow-up should decide whether JS and Java adopt SUSPENDED.
wangyb-A
force-pushed
the
fix/user-function-end-on-suspension
branch
from
August 14, 2026 23:50
e1e03c7 to
580da37
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
wrap_user_functionre-raisedSuspendExecutionwithout callingon_user_function_end(state.py:1171):This was deliberate —
test_wrap_user_function_suspend_does_not_fire_end_hookpinned it, on the rationale that "the plugin observes it by absence (no end hook fires), with the instrumentation plugin's own per-invocation span sweep closing any open spans cleanly at invocation end."That contract cannot be honoured for state that is thread-confined. The OTel plugins attach an
opentelemetry.contexttoken inon_user_function_start, and a token is only detachable in thecontextvars.Contextthat created it — the user-code worker thread, not the handler thread the invocation hooks run on. So a suspended operation stranded its context scope with no hook able to release it: the invocation-end sweep runs on the wrong thread. The same applies to any plugin holding per-operation state — a timer, an open log group, a span.Suspension is also not a rare path. It is the normal outcome of any child context whose inner operation is still pending, which is the reachable case here:
SuspendExecutionpropagates out of the child context's user function, and the hook silently never fires.Java already fires the hook here.
BaseDurableOperation.runUserFunctioncatchesThrowable— which coversSuspendExecutionException— and the javadoc gives the same reason this PR does:Changes
UserFunctionOutcome.SUSPENDED. Reported as its own outcome rather than reusingFAILED: nothing went wrong, and plugins that count failures or set an error span status must not treat it as one. Java models this assucceeded=falseplus the suspend exception as the error, which reads as a failure to exactly those consumers.UserFunctionEndInfo.from_start_infoandPluginExecutor.on_user_function_end, so the suspension path can reportSUSPENDEDwitherror=Nonerather than deriving the outcome from an absent error.wrap_user_function'sSuspendExecutionbranch, then re-raise unchanged — durable control flow is untouched.SUSPENDEDas "release the scope, leave the span open". The attempt has not concluded, so it must not be ended with an outcome here; it is ended when the operation reaches a terminal status, matching how an operation that suspends mid-invocation is already handled.Note
SuspendExecutionderives fromBaseException, notException, so the existingexcept Exceptionclause never caught it — the gap was structural, not a missing branch. This PR keeps the catch narrow (SuspendExecutiononly); whetherwrap_user_functionshould also fire the hook for otherBaseExceptions such asKeyboardInterrupt, as Java'scatch (Throwable)effectively does, is left as a separate question.API impact
Adding an enum member is source-compatible, but plugin authors who exhaustively branch on
UserFunctionOutcomewill now see a third value. Anyone writingif outcome is FAILED: ... else: <treat as success>will classify a suspension as success, which is the intended reading for span status but may not be for metrics.This also makes Python the first of the three SDKs with a third outcome. JS has no hook on this path at all — its attempt hooks cover steps and
waitForConditiononly, not child contexts, and its plugins scope context throughwrapOperationAttemptFn/wrapChildContextFn, which restore on throw automatically. Java reports suspension through the existing boolean. Worth deciding whether JS and Java adoptSUSPENDEDfor parity; happy to split this into a discussion first if the team would rather agree the shape before the code lands.Testing
3229 pass across the monorepo;
hatch fmt --checkandhatch run types:checkclean.test_wrap_user_function_suspend_does_not_fire_end_hookis inverted intotest_wrap_user_function_suspend_fires_end_hook_with_suspended_outcome, asserting one end hook withSUSPENDEDanderror is None.run_in_child_contextthat suspends and asserts exactly oneSUSPENDEDend hook and noFAILEDone, with the invocation still returningPENDING.ERROR(parametrised over both plugins).UserFunctionOutcomevalue-set test is updated.Both halves mutation-tested: removing the hook call fails the two core tests; ignoring
SUSPENDEDin the plugins fails the four OTel tests.