feat(eval): add ondemand evaluate (synchronous, client-side) by jariy17 · Pull Request #1983 · aws/agentcore-cli

jariy17 · 2026-08-12T17:41:51Z

Summary

Adds agentcore eval ondemand evaluate — a client-side evaluation of existing sessions. On-demand gathers the sessions' traces from CloudWatch on the client, calls the Evaluate data-plane API directly, and prints scores.

	`batch-evaluation evaluate`	`ondemand evaluate` (this PR)
SDK call	`StartBatchEvaluation`	`Evaluate` (data plane)
Trace gathering	service-side	client-side (CloudWatch Logs Insights)
Returns	job id (poll with `get`)	scores, synchronously
Source arms	`--agent` / `--online-eval` / `--data-source-config`	`--agent` only

Usage

agentcore eval ondemand evaluate \
  --agent <harness-id|runtime-id> \
  --evaluator Builtin.Helpfulness \
  --session-ids <id...>            # or --lookback-days N, or --start-time/--end-time

Flags

--agent (required), --endpoint, --evaluator <ids...> (required)
time filter: --lookback-days N or --start-time/--end-time (ISO-8601, together)
--session-ids <ids...>, --trace-id <id> — independent, AND-ed fetch filters
--ground-truth <json> — inline / file:// / - → SDK-native EvaluationReferenceInput[]

Tests

Golden fixture suite (ondemand.fixture.test.tsx) — recorded GetAgentRuntime + Insights StartQuery/GetQueryResults (both log groups) + Evaluate fixtures, driven through the real root handler against a pinned window, diffed against evaluate.golden.json.
Command-flow suite (ondemand.test.tsx, TestCoreClient) — source-arm validation, getTracesForAgent → evaluate orchestration/order, --lookback-days window math, --trace-id, ground-truth passthrough.

tsc --noEmit, bun test src/ (1067 pass), oxlint, prettier — all clean.

codecov-commenter · 2026-08-12T17:43:41Z

Codecov Report

❌ Patch coverage is 96.41791% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (4d183dc) to head (9b57081).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	95.39%	10 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #1983      +/-   ##
============================================
- Coverage     96.96%   96.95%   -0.01%     
============================================
  Files           364      366       +2     
  Lines         20758    21093     +335     
============================================
+ Hits          20127    20450     +323     
- Misses          631      643      +12

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

aidandaly24 · 2026-08-13T16:55:55Z

+    const spanId = span.spanId;
+    if (typeof spanId !== "string" || spanId.length === 0) continue;
+    const attrs = span.attributes as Record<string, unknown> | undefined;
+    if (attrs?.["gen_ai.tool.name"] ?? attrs?.["tool.name"]) spanIds.push(spanId);


Should tool-call selection use the standard operation/kind markers rather than requiring a tool-name attribute? Current telemetry identifies tool spans using gen_ai.operation.name === "execute_tool", openinference.span.kind === "TOOL", or traceloop.span.kind === "tool". The name fields are not always present. I reproduced a valid LangGraph-style tool span producing an empty toolCallSpanIds, so a TOOL_CALL evaluator makes no Evaluate request. Could we use the same marker logic as the evaluation SDK?

I'll add support to detect these "gen_ai.operation.name === "execute_tool", openinference.span.kind === "TOOL", or traceloop.span.kind === "tool"

aidandaly24 · 2026-08-13T16:57:09Z

+    try {
+      const evaluator = await control.send(new GetEvaluatorCommand({ evaluatorId: id }));
+      levels.set(id, evaluator.level ?? "SESSION");
+    } catch {


Could evaluator lookup failures propagate instead of silently defaulting to SESSION? The level determines whether Evaluate receives traceIds, spanIds, or no target. I reproduced an AccessDeniedException here causing a trace evaluator to be submitted without an evaluation target, which can either evaluate the wrong scope or hide the actual permissions error. If a fallback is needed for compatibility, could it be limited to a missing level rather than every exception?

AWS SDK v3 responses have | undefined even if they are required. I'm just going to do ! because every evaluator must have a level.

aidandaly24 · 2026-08-13T17:01:17Z

+        }
+      }
+    }
+    return { sessionsEvaluated: input.traces.length, results };


Could sessionsEvaluated count sessions for which an Evaluate request was actually sent? TRACE and TOOL_CALL sessions with no IDs are skipped above but still included here. I reproduced sessionsEvaluated: 1 with zero API calls and zero results. Tracking submitted sessions, or naming this sessionsDiscovered, would make the output less misleading.

nborges-aws · 2026-08-13T17:51:15Z

+      // resolve inline / file:// / -, then hand the array to core verbatim — core
+      // groups it by session.
+      const resolver = new SourceResolver({ stdin: io.stdin });
+      const groundTruth = parseJsonFlag<EvaluationReferenceInput[]>(


Can we change this to use parseJsonArrayFlag helper instead? Since groundTruth has to be an array.

nborges-aws · 2026-08-13T17:51:43Z

Agree with Aidans findings + one additional comment

…d-evaluate-clean # Conflicts: # src/core/eval.tsx # src/handlers/eval/index.tsx # src/handlers/eval/types.tsx # src/testing/TestCoreClient.tsx

nborges-aws · 2026-08-13T20:43:42Z

+    // Sessions that actually produced Evaluate results — distinct from the sessions
+    // handed in, since a TRACE/TOOL_CALL session with no matching ids makes no call.
+    const evaluatedSessions = new Set<string>();
+    for (const evaluatorId of input.evaluatorIds) {


nit: should we use levels.keys() here instead of the raw id array? Consistent with rest of code + leverages the dedup logic in resolveEvaluatorLevels(). A follow up item if you think its worth it

Hweinstock · 2026-08-14T14:48:07Z

+    const logGroupName = runtimeLogGroup(runtimeId, qualifier);
+    const serviceName = runtimeServiceName(runtimeName, qualifier);
+
+    // CloudWatch Insights takes epoch seconds. Discovery defaults to now-7d when


nit: i feel like the code explains this already.

Removed in follow-up PR #2007: #2007

Hweinstock · 2026-08-14T14:50:07Z

+    const [runtimeRows, sharedRows] = await Promise.all([
+      runInsightsQuery(logs, [logGroupName], queryString, startSec, endSec).catch((error) => {
+        if (error instanceof ResourceNotFoundException) {
+          throw new InputValidationError(


should this be a different error type for telemetry? I wonder if it would be useful to distinguish invalid inputs from valid inputs without results.

I was thinking we create a new Exception called TracesNotFound once we add observability to the cli.

Addressed in follow-up PR #2007: missing runtime telemetry now throws ResourceNotFoundError instead of InputValidationError, while preserving the CloudWatch exception as its cause. #2007

Hweinstock · 2026-08-14T14:54:31Z

+        // (empty batch list); SESSION always makes one call with no target.
+        for (const target of targetBatches(level, trace)) {
+          const response = await data.send(
+            new EvaluateCommand({


q: if a single evaluate fails, do we want to fail the entire run?

On demand evaluate should be use for small evaluations (1-2 sessions with 2-3 evaluators) so this situation is unlikely to happen and also you could multiple evaluate calls per session if one of those fails, you have an incomplete evaluations

Yes, fail-fast is intentional for now. One session can require multiple Evaluate API calls across evaluators and target batches. Continuing after one fails could return incomplete results for that session while appearing successful. On-demand evaluation targets small synchronous runs; larger workloads should use batch evaluation. We can add explicit partial-failure handling later if customers need it.

Hweinstock · 2026-08-14T14:55:58Z

+    }
+    return {
+      sessionsRequested: input.traces.length,
+      sessionsEvaluated: evaluatedSessions.size,


q: I see we return the sessions evaluated and the results separately. Is there a use case for getting the results for a certain session or is it more useful in aggregate?

Results are already associated with sessions through context.spanContext.sessionId. Keeping them flat preserves the SDK response shape while still supporting per-session filtering and aggregate analysis. I also confirmed the session ID is present in a live on-demand evaluation response.

Hweinstock · 2026-08-14T14:58:06Z

+  return value.replace(/'/g, "");
+}
+
+// buildSpanQuery is the single-phase Insights query: scope to one runtime by its


nit: is there info in this comment not expressed by the code?

Removed in follow-up PR #2007: #2007

Hweinstock · 2026-08-14T16:09:20Z

+    try {
+      doc = JSON.parse(message) as SpanRecord;
+    } catch {
+      continue;


is it worth logging a warning here or would this be noisy?

Addressed in follow-up PR #2007 by passing the existing logger into the grouping helper and warning at most once when malformed telemetry records are skipped. #2007

Hweinstock · 2026-08-14T16:12:15Z

+  groundTruth?: EvaluationReferenceInput[];
+};
+
+// EvaluateResult returns the raw Evaluate API results across all evaluators and


nit: these comments describe usages of the type and feel like they could drift.

Removed the usage-oriented type comments in follow-up PR #2007: #2007

Hweinstock · 2026-08-14T16:22:56Z

+// with start before end. On-demand owns this rather than reusing batch's resolver:
+// batch has no --lookback-days and its window feeds a service-side data source, not
+// a client-side Insights query.
+function resolveWindow(flags: {


how does this differ from

agentcore-cli/src/handlers/eval/batch-evaluation/evaluate/index.tsx

Line 164 in ea3fa84

function resolveWindow(flags: DataSourceFlags): SessionWindow | undefined {

?

Batch's resolveWindow feeds the batch evaluation API's DataSourceConfig shape, which supports CloudWatch log groups and onlineEvaluationConfigArn.

Hweinstock · 2026-08-14T16:23:57Z

+
+// Record with: RECORD=1 bun test src/handlers/eval/ondemand/ondemand.fixture.test.tsx
+//
+// This exercises the real seam end to end: parsing → handler → CoreClient →


nit: this feels overly verbose.

Removed the verbose fixture comment block in follow-up PR #2007: #2007

Hweinstock · 2026-08-14T16:24:56Z

+    ).rejects.toThrow(/--agent/);
+  });
+
+  test("requires --evaluator", async () => {


would these make sense as a test.each pattern for the different set of flags that reject?

Converted the repeated validation cases to test.each in follow-up PR #2007: #2007

github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026

jariy17 force-pushed the feat/eval-ondemand-evaluate-clean branch 9 times, most recently from 23781e6 to 25228b3 Compare August 12, 2026 20:49

jariy17 marked this pull request as ready for review August 12, 2026 21:17

aidandaly24 reviewed Aug 13, 2026

View reviewed changes

nborges-aws reviewed Aug 13, 2026

View reviewed changes

feat(eval): add ondemand evaluate (synchronous, client-side)

59e81dd

jariy17 force-pushed the feat/eval-ondemand-evaluate-clean branch from d1c79cf to 59e81dd Compare August 13, 2026 18:58

Merge remote-tracking branch 'origin/refactor' into feat/eval-ondeman…

9b57081

…d-evaluate-clean # Conflicts: # src/core/eval.tsx # src/handlers/eval/index.tsx # src/handlers/eval/types.tsx # src/testing/TestCoreClient.tsx

nborges-aws reviewed Aug 13, 2026

View reviewed changes

nborges-aws approved these changes Aug 13, 2026

View reviewed changes

aidandaly24 approved these changes Aug 13, 2026

View reviewed changes

jariy17 mentioned this pull request Aug 13, 2026

feat(eval): add batch-evaluation simulate (dataset simulations) #2000

Draft

Hweinstock reviewed Aug 14, 2026

View reviewed changes

jariy17 merged commit a9d34be into aws:refactor Aug 14, 2026
7 of 13 checks passed

jariy17 mentioned this pull request Aug 14, 2026

refactor(eval): address on-demand review feedback #2007

Open

Uh oh!

Conversation

jariy17 commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

Flags

Tests

Uh oh!

codecov-commenter commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nborges-aws commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

jariy17 commented Aug 12, 2026 •

edited

Loading

codecov-commenter commented Aug 12, 2026 •

edited

Loading

nborges-aws commented Aug 13, 2026 •

edited

Loading