feat(tel): implement otel sink by Hweinstock · Pull Request #1888 · aws/agentcore-cli

Hweinstock · 2026-07-31T18:57:33Z

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

codecov-commenter · 2026-07-31T19:00:28Z

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           refactor    #1888   +/-   ##
=========================================
  Coverage     96.69%   96.70%           
=========================================
  Files           291      292    +1     
  Lines         16012    16068   +56     
=========================================
+ Hits          15483    15538   +55     
- Misses          529      530    +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock · 2026-07-31T19:02:35Z

^ the line missing coverage is getName() on the collector sink.

notgitika · 2026-07-31T21:02:45Z

+
+  async shutdown(): Promise<void> {
+    try {
+      await this.meterProvider.forceFlush({ timeoutMillis: this.flushTimeoutMs });


outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

This method provides a way for provider to do any cleanup required. Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible. Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out. Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable. Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

notgitika · 2026-07-31T21:06:32Z

+
+    if (globalConfig.telemetry.enabled)
+      metricSinks.push(
+        new OtelHistogramSink({


it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

notgitika

LGMT

AlexanderRichey · 2026-08-05T18:15:46Z

+    try {
+      await this.meterProvider.forceFlush({ timeoutMillis: this.flushTimeoutMs });
+    } catch (e) {
+      const error = e instanceof Error ? e : new Error(String(e));


Do we want to use our AgentCoreError.fromError() here instead?

Yeah I think that makes sense to ensure its consistent.

Hweinstock · 2026-08-10T15:49:27Z

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
  Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

aidandaly24 · 2026-08-12T16:14:04Z

+      resource: resourceFromAttributes(config.resourceAttributes),
+      readers: [
+        new PeriodicExportingMetricReader({
+          exporter: new OTLPMetricExporter({


I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

aidandaly24 · 2026-08-12T16:15:30Z

+      .child({ metricName, metricValue: value, metricAttributes: attributes })
+      .info(`sending telemetry metric to collector`);
+
+    this.getHistogram(metricName).record(value, attributes);


I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

good catch, I think removing is the right call to simplify.

notgitika · 2026-08-14T17:20:44Z

looks good to me!

Hweinstock added 3 commits July 31, 2026 16:38

feat(tel): implement otel sink

e2862d4

docs(tel): add docstring comments to otel sink

14a2603

test(tel): add a test of a local server for the collector sink

42bc9df

github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Jul 31, 2026

chore: merge in refactor-main changes

4b537a7

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Jul 31, 2026

chore: remove extra space from telemetry test file

da70cbb

docs: add note in comment about default value

ead49c9

Hweinstock changed the title ~~0feat(tel): implement otel sink~~ feat(tel): implement otel sink Jul 31, 2026

fix(tel): avoid letting flush failures stop shutdown

367dee5

Hweinstock marked this pull request as ready for review July 31, 2026 19:22

notgitika reviewed Jul 31, 2026

View reviewed changes

notgitika previously approved these changes Aug 1, 2026

View reviewed changes

AlexanderRichey reviewed Aug 5, 2026

View reviewed changes

feat(tel): swap to cli from error method

13e745a

Hweinstock dismissed notgitika’s stale review via 13e745a August 10, 2026 15:22

Hweinstock added 3 commits August 10, 2026 15:26

Merge branch 'refactor-main' into feat/otel-sink

b7d1947

Merge branch 'refactor-main' into feat/otel-sink

53b705b

Merge branch 'refactor-main' into feat/otel-sink

863c664

notgitika previously approved these changes Aug 12, 2026

View reviewed changes

aidandaly24 reviewed Aug 12, 2026

View reviewed changes

aidandaly24 previously approved these changes Aug 12, 2026

View reviewed changes

Hweinstock added 2 commits August 13, 2026 13:00

fix(tel): remove duplicate resource attributes

799dde6

feat(tel): add installationId header and temporarily preference

7e9a97e

Hweinstock dismissed stale reviews from aidandaly24 and notgitika via 7e9a97e August 13, 2026 13:05

test(tel): adjust test to use passed down resource attributes

6806436

aidandaly24 approved these changes Aug 13, 2026

View reviewed changes

notgitika approved these changes Aug 14, 2026

View reviewed changes

Hweinstock merged commit 01eb60b into aws:refactor Aug 14, 2026
8 of 11 checks passed

Hweinstock deleted the feat/otel-sink branch August 14, 2026 19:09

Uh oh!

Conversation

Hweinstock commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Testing

Uh oh!

codecov-commenter commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Hweinstock commented Jul 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notgitika left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hweinstock commented Aug 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notgitika commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Hweinstock commented Jul 31, 2026 •

edited

Loading

codecov-commenter commented Jul 31, 2026 •

edited

Loading