feat(profiling): Add OTLP profiles export with JFR conversion pipeline

[jbachorik]

What Does This Do

Adds OTLP profile export to the agent: a JFR-to-OTLP/P converter plus an uploader wired into the profiling subsystem, sharing the same OTLP transport the tracer already uses for traces/logs/metrics. Off by default.

• JFR-to-OTLP/P converter in profiling-otel — hand-coded protobuf encoder with dictionary-based compression (string/function/location/stack/attribute/link tables), no external dependencies. Supports datadog.ExecutionSample (CPU), datadog.MethodSample (wall clock), datadog.ObjectSample (allocation), jdk.JavaMonitorEnter/jdk.JavaMonitorWait (lock contention). Samples are emitted one-per-JFR-event (stacks, functions, locations and attributes are deduplicated via the dictionary tables); period/period_type are emitted only for the periodic CPU/wall profiles, omitted for the event-driven alloc/lock profiles; optional original-payload embedding; PROTO and JSON encodings produce identical content. convert-jfr.sh CLI wrapper; validated via profcheck and protoc under TestContainers; JMH benchmarks included.
• Extracted OtlpSender/OtlpHttpSender/OtlpGrpcSender and the request-body/payload helpers from dd-trace-core into a shared communication:otlp-exporter module; bundled once in the agent jar's shared/ section. Only HTTP_PROTOBUF and GRPC protocols are supported; anything else fails fast.
• OtlpProfileUploader wires the converter and sender into ProfilingAgent. The scrubber wraps the OTLP listener, so OTLP exports scrubbed recordings. A sender-construction failure degrades OTLP export to disabled instead of aborting the profiling subsystem. Embedded original payloads are capped at 64 MiB — oversized recordings are exported converted-only. Temp recordings are deleted on every exit path.
• RecordingData implements reference counting (base count 1, implemented in the OpenJDK/Oracle/ddprof controllers): retain() after full release throws, over-release is a silent no-op, doRelease() fires exactly once when the count reaches zero.
• Config: profiling.otlp.enabled (default false), profiling.otlp.mode = light | converted | full (default light). light ships the raw JFR as the original-payload blob without conversion, converted emits converted samples only, full emits both. LightweightOtlpEncoder covers the raw-blob path, streaming the payload from disk with the size re-stat'ed immediately before the stream opens. Registered DD_OTLP_PROFILES_{ENDPOINT,HEADERS,PROTOCOL,COMPRESSION,TIMEOUT} and DD_PROFILING_OTLP_{ENABLED,MODE} in supported-configurations.json, mirroring the traces/logs/metrics pattern.
• Reports JFR→OTLP conversion duration as telemetry gauges (avg/max per flush window, accumulated under the conversion lock so each window reports exactly its own conversions) plus profiles export counters; rejected queue offers are logged instead of dropped silently.
• CODEOWNERS: communication/otlp-exporter/ → apm-sdk-capabilities-java, docker/Dockerfile.profcheck → profiling-java.
• Bumps jafar to 0.26.2 (adds the jafar-parser and jmc-flightrecorder-writer dependencies) and the agent jar size budget to 35,200,000 bytes (~33.6 MiB).
• Bumps TempLocationManagerTest concurrent cleanup deadlines from 30s to 60s.

Motivation

Lets the agent export profiles in the OpenTelemetry profiles format over the same OTLP transport the tracer already uses for traces/logs/metrics.

Additional Notes

Upstream maturity: not for production use

The OTLP profiles signal is not stable upstream, so this export must not be used in production:

• profiles/* and collector/profiles/* are at Development maturity for both binary protobuf and JSON (opentelemetry-proto v1.11.1 maturity table).
• The profiles signal specification is at Alpha status (OpenTelemetry Profiles spec).
• Related telemetry metrics PR is opened

When OTLP profiles export is enabled, the uploader logs a startup warning that the protocol is at Development maturity and must not be used in production. The warning is also sent to instrumentation telemetry.

Performance

JDK 21, macOS Darwin 25.4.0:

Component	Result
protobuf encoder	6-56 Mops/s depending on operation
dictionary interning	28-103 Mops/s at 50% cache hit rate
jafar JFR reader (dominant cost)	13.2% + 5.9% + 2.9% CPU — not application-owned

ByteArrayOutputStream.write went from 13.7% to 0% after switching to a direct byte[] + cursor.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
  — the 7 new keys (DD_OTLP_PROFILES_{ENDPOINT,HEADERS,PROTOCOL,COMPRESSION,TIMEOUT}, DD_PROFILING_OTLP_ENABLED, DD_PROFILING_OTLP_MODE) are registered in the Feature Parity Dashboard, which feeds the public config registry the Java config page renders from (see PROF-15986)

Jira ticket: PROF-15986

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

[github-actions]

This pull request has been marked as stale because it has not had activity over the past quarter. It will be closed in 7 days if no further activity occurs. Feel free to reopen the PR if you are still working on it.

[pr-commenter]

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jb/rnd_otlp_profile
git_commit_date	1776844785	1777025973
git_commit_sha	c72f067	2ac3d50
release_version	1.62.0-SNAPSHOT~c72f06780f	1.62.0-SNAPSHOT~2ac3d50b80

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1777027838	1777027838
ci_job_id	1627595483	1627595483
ci_pipeline_id	109510136	109510136
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-hyayd6qs 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-hyayd6qs 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1057846
Total [baseline] (11.007 s) : 0, 11007201
Agent [candidate] (1.066 s) : 0, 1065583
Total [candidate] (11.189 s) : 0, 11188886
section appsec
Agent [baseline] (1.26 s) : 0, 1259761
Total [baseline] (11.001 s) : 0, 11001103
Agent [candidate] (1.265 s) : 0, 1265377
Total [candidate] (11.008 s) : 0, 11008154
section iast
Agent [baseline] (1.235 s) : 0, 1234875
Total [baseline] (11.328 s) : 0, 11328182
Agent [candidate] (1.234 s) : 0, 1234219
Total [candidate] (11.341 s) : 0, 11340962
section profiling
Agent [baseline] (1.189 s) : 0, 1189232
Total [baseline] (11.082 s) : 0, 11082069
Agent [candidate] (1.188 s) : 0, 1188392
Total [candidate] (11.112 s) : 0, 11112255

Loading

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	appsec	1.26 s	201.915 ms (19.1%)
Agent	iast	1.235 s	177.029 ms (16.7%)
Agent	profiling	1.189 s	131.386 ms (12.4%)
Total	tracing	11.007 s	-
Total	appsec	11.001 s	-6.098 ms (-0.1%)
Total	iast	11.328 s	320.981 ms (2.9%)
Total	profiling	11.082 s	74.868 ms (0.7%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.066 s	-
Agent	appsec	1.265 s	199.794 ms (18.7%)
Agent	iast	1.234 s	168.636 ms (15.8%)
Agent	profiling	1.188 s	122.809 ms (11.5%)
Total	tracing	11.189 s	-
Total	appsec	11.008 s	-180.733 ms (-1.6%)
Total	iast	11.341 s	152.075 ms (1.4%)
Total	profiling	11.112 s	-76.631 ms (-0.7%)

gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.245 ms) : 0, 1245
crashtracking [candidate] (1.237 ms) : 0, 1237
BytebuddyAgent [baseline] (632.627 ms) : 0, 632627
BytebuddyAgent [candidate] (637.115 ms) : 0, 637115
AgentMeter [baseline] (29.523 ms) : 0, 29523
AgentMeter [candidate] (29.812 ms) : 0, 29812
GlobalTracer [baseline] (249.021 ms) : 0, 249021
GlobalTracer [candidate] (250.429 ms) : 0, 250429
AppSec [baseline] (32.297 ms) : 0, 32297
AppSec [candidate] (32.625 ms) : 0, 32625
Debugger [baseline] (59.875 ms) : 0, 59875
Debugger [candidate] (60.195 ms) : 0, 60195
Remote Config [baseline] (625.989 µs) : 0, 626
Remote Config [candidate] (595.217 µs) : 0, 595
Telemetry [baseline] (8.012 ms) : 0, 8012
Telemetry [candidate] (8.823 ms) : 0, 8823
Flare Poller [baseline] (8.387 ms) : 0, 8387
Flare Poller [candidate] (8.424 ms) : 0, 8424
section appsec
crashtracking [baseline] (1.256 ms) : 0, 1256
crashtracking [candidate] (1.233 ms) : 0, 1233
BytebuddyAgent [baseline] (672.82 ms) : 0, 672820
BytebuddyAgent [candidate] (676.515 ms) : 0, 676515
AgentMeter [baseline] (12.138 ms) : 0, 12138
AgentMeter [candidate] (12.17 ms) : 0, 12170
GlobalTracer [baseline] (248.548 ms) : 0, 248548
GlobalTracer [candidate] (249.384 ms) : 0, 249384
AppSec [baseline] (186.48 ms) : 0, 186480
AppSec [candidate] (186.61 ms) : 0, 186610
Debugger [baseline] (65.891 ms) : 0, 65891
Debugger [candidate] (66.598 ms) : 0, 66598
Remote Config [baseline] (573.663 µs) : 0, 574
Remote Config [candidate] (579.897 µs) : 0, 580
Telemetry [baseline] (7.86 ms) : 0, 7860
Telemetry [candidate] (7.962 ms) : 0, 7962
Flare Poller [baseline] (3.493 ms) : 0, 3493
Flare Poller [candidate] (3.482 ms) : 0, 3482
IAST [baseline] (24.267 ms) : 0, 24267
IAST [candidate] (24.329 ms) : 0, 24329
section iast
crashtracking [baseline] (1.23 ms) : 0, 1230
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (810.22 ms) : 0, 810220
BytebuddyAgent [candidate] (810.38 ms) : 0, 810380
AgentMeter [baseline] (11.435 ms) : 0, 11435
AgentMeter [candidate] (11.437 ms) : 0, 11437
GlobalTracer [baseline] (239.944 ms) : 0, 239944
GlobalTracer [candidate] (240.13 ms) : 0, 240130
AppSec [baseline] (27.933 ms) : 0, 27933
AppSec [candidate] (28.481 ms) : 0, 28481
Debugger [baseline] (65.932 ms) : 0, 65932
Debugger [candidate] (64.829 ms) : 0, 64829
Remote Config [baseline] (546.729 µs) : 0, 547
Remote Config [candidate] (532.54 µs) : 0, 533
Telemetry [baseline] (7.888 ms) : 0, 7888
Telemetry [candidate] (7.803 ms) : 0, 7803
Flare Poller [baseline] (3.449 ms) : 0, 3449
Flare Poller [candidate] (3.399 ms) : 0, 3399
IAST [baseline] (30.107 ms) : 0, 30107
IAST [candidate] (29.874 ms) : 0, 29874
section profiling
ProfilingAgent [baseline] (94.328 ms) : 0, 94328
ProfilingAgent [candidate] (94.543 ms) : 0, 94543
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (694.146 ms) : 0, 694146
BytebuddyAgent [candidate] (693.801 ms) : 0, 693801
AgentMeter [baseline] (9.219 ms) : 0, 9219
AgentMeter [candidate] (9.246 ms) : 0, 9246
GlobalTracer [baseline] (207.463 ms) : 0, 207463
GlobalTracer [candidate] (207.346 ms) : 0, 207346
AppSec [baseline] (33.09 ms) : 0, 33090
AppSec [candidate] (32.946 ms) : 0, 32946
Debugger [baseline] (66.288 ms) : 0, 66288
Debugger [candidate] (65.923 ms) : 0, 65923
Remote Config [baseline] (591.996 µs) : 0, 592
Remote Config [candidate] (577.193 µs) : 0, 577
Telemetry [baseline] (7.83 ms) : 0, 7830
Telemetry [candidate] (7.804 ms) : 0, 7804
Flare Poller [baseline] (3.517 ms) : 0, 3517
Flare Poller [candidate] (3.595 ms) : 0, 3595
Profiling [baseline] (94.889 ms) : 0, 94889
Profiling [candidate] (95.091 ms) : 0, 95091

Loading

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054596
Total [baseline] (8.882 s) : 0, 8882385
Agent [candidate] (1.061 s) : 0, 1061090
Total [candidate] (8.87 s) : 0, 8870455
section iast
Agent [baseline] (1.243 s) : 0, 1242502
Total [baseline] (9.634 s) : 0, 9634118
Agent [candidate] (1.23 s) : 0, 1229987
Total [candidate] (9.576 s) : 0, 9576132

Loading

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	iast	1.243 s	187.907 ms (17.8%)
Total	tracing	8.882 s	-
Total	iast	9.634 s	751.733 ms (8.5%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.061 s	-
Agent	iast	1.23 s	168.897 ms (15.9%)
Total	tracing	8.87 s	-
Total	iast	9.576 s	705.676 ms (8.0%)

gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.227 ms) : 0, 1227
crashtracking [candidate] (1.236 ms) : 0, 1236
BytebuddyAgent [baseline] (631.427 ms) : 0, 631427
BytebuddyAgent [candidate] (635.978 ms) : 0, 635978
AgentMeter [baseline] (29.497 ms) : 0, 29497
AgentMeter [candidate] (29.91 ms) : 0, 29910
GlobalTracer [baseline] (247.81 ms) : 0, 247810
GlobalTracer [candidate] (250.618 ms) : 0, 250618
AppSec [baseline] (32.257 ms) : 0, 32257
AppSec [candidate] (32.498 ms) : 0, 32498
Debugger [baseline] (58.659 ms) : 0, 58659
Debugger [candidate] (59.361 ms) : 0, 59361
Remote Config [baseline] (593.41 µs) : 0, 593
Remote Config [candidate] (607.426 µs) : 0, 607
Telemetry [baseline] (7.979 ms) : 0, 7979
Telemetry [candidate] (8.791 ms) : 0, 8791
Flare Poller [baseline] (8.963 ms) : 0, 8963
Flare Poller [candidate] (5.903 ms) : 0, 5903
section iast
crashtracking [baseline] (1.247 ms) : 0, 1247
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (817.434 ms) : 0, 817434
BytebuddyAgent [candidate] (808.554 ms) : 0, 808554
AgentMeter [baseline] (11.551 ms) : 0, 11551
AgentMeter [candidate] (11.412 ms) : 0, 11412
GlobalTracer [baseline] (240.613 ms) : 0, 240613
GlobalTracer [candidate] (239.001 ms) : 0, 239001
IAST [baseline] (29.365 ms) : 0, 29365
IAST [candidate] (29.184 ms) : 0, 29184
AppSec [baseline] (29.748 ms) : 0, 29748
AppSec [candidate] (26.756 ms) : 0, 26756
Debugger [baseline] (64.452 ms) : 0, 64452
Debugger [candidate] (64.676 ms) : 0, 64676
Remote Config [baseline] (528.078 µs) : 0, 528
Remote Config [candidate] (539.326 µs) : 0, 539
Telemetry [baseline] (7.781 ms) : 0, 7781
Telemetry [candidate] (7.743 ms) : 0, 7743
Flare Poller [baseline] (3.434 ms) : 0, 3434
Flare Poller [candidate] (3.353 ms) : 0, 3353

Loading

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jb/rnd_otlp_profile
git_commit_date	1776844785	1777025973
git_commit_sha	c72f067	2ac3d50
release_version	1.62.0-SNAPSHOT~c72f06780f	1.62.0-SNAPSHOT~2ac3d50b80

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1777028372	1777028372
ci_job_id	1627595489	1627595489
ci_pipeline_id	109510136	109510136
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-snb1fluy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-snb1fluy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 3 performance regressions! Performance is the same for 15 metrics, 18 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:iast:high_load	worse [+93.903µs; +146.263µs] or [+3.625%; +5.646%]	same [-134.031µs; +240.303µs] or [-1.788%; +3.206%]	unstable [-179.991op/s; +107.429op/s] or [-13.052%; +7.790%]	2.710ms	7.549ms	1342.781op/s	2.590ms	7.496ms	1379.062op/s
scenario:load:insecure-bank:no_agent:high_load	worse [+24.931µs; +67.227µs] or [+2.443%; +6.589%]	unstable [-379.895µs; +57.373µs] or [-13.401%; +2.024%]	unstable [-424.923op/s; +376.361op/s] or [-12.032%; +10.657%]	1.066ms	2.673ms	3507.375op/s	1.020ms	2.835ms	3531.656op/s
scenario:load:petclinic:code_origins:high_load	worse [+0.488ms; +1.520ms] or [+2.819%; +8.775%]	unsure [+101.511µs; +1880.679µs] or [+0.355%; +6.583%]	unstable [-41.914op/s; +15.164op/s] or [-15.944%; +5.769%]	18.331ms	29.559ms	249.500op/s	17.327ms	28.568ms	262.875op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.25 ms) : 1237, 1263
.   : milestone, 1250,
iast (3.319 ms) : 3272, 3366
.   : milestone, 3319,
iast_FULL (5.999 ms) : 5938, 6059
.   : milestone, 5999,
iast_GLOBAL (3.665 ms) : 3602, 3727
.   : milestone, 3665,
profiling (2.1 ms) : 2081, 2120
.   : milestone, 2100,
tracing (1.961 ms) : 1944, 1978
.   : milestone, 1961,
section candidate
no_agent (1.261 ms) : 1249, 1272
.   : milestone, 1261,
iast (3.411 ms) : 3365, 3457
.   : milestone, 3411,
iast_FULL (5.965 ms) : 5906, 6024
.   : milestone, 5965,
iast_GLOBAL (3.669 ms) : 3615, 3722
.   : milestone, 3669,
profiling (357.437 µs) : 350, 365
.   : milestone, 357,
tracing (1.974 ms) : 1957, 1991
.   : milestone, 1974,

Loading

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.25 ms [1.237 ms, 1.263 ms]	-
iast	3.319 ms [3.272 ms, 3.366 ms]	2.069 ms (165.5%)
iast_FULL	5.999 ms [5.938 ms, 6.059 ms]	4.748 ms (379.8%)
iast_GLOBAL	3.665 ms [3.602 ms, 3.727 ms]	2.414 ms (193.1%)
profiling	2.1 ms [2.081 ms, 2.12 ms]	850.315 µs (68.0%)
tracing	1.961 ms [1.944 ms, 1.978 ms]	710.684 µs (56.8%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.261 ms [1.249 ms, 1.272 ms]	-
iast	3.411 ms [3.365 ms, 3.457 ms]	2.15 ms (170.5%)
iast_FULL	5.965 ms [5.906 ms, 6.024 ms]	4.704 ms (373.1%)
iast_GLOBAL	3.669 ms [3.615 ms, 3.722 ms]	2.408 ms (191.0%)
profiling	357.437 µs [350.176 µs, 364.697 µs]	-903.3 µs (-71.6%)
tracing	1.974 ms [1.957 ms, 1.991 ms]	713.286 µs (56.6%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.229 ms) : 18044, 18414
.   : milestone, 18229,
appsec (18.781 ms) : 18593, 18969
.   : milestone, 18781,
code_origins (17.752 ms) : 17575, 17929
.   : milestone, 17752,
iast (18.05 ms) : 17871, 18229
.   : milestone, 18050,
profiling (19.227 ms) : 19037, 19418
.   : milestone, 19227,
tracing (17.669 ms) : 17494, 17843
.   : milestone, 17669,
section candidate
no_agent (18.958 ms) : 18769, 19147
.   : milestone, 18958,
appsec (18.49 ms) : 18310, 18671
.   : milestone, 18490,
code_origins (18.707 ms) : 18523, 18890
.   : milestone, 18707,
iast (17.856 ms) : 17684, 18028
.   : milestone, 17856,
profiling (255.456 µs) : 245, 266
.   : milestone, 255,
tracing (17.603 ms) : 17434, 17773
.   : milestone, 17603,

Loading

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.229 ms [18.044 ms, 18.414 ms]	-
appsec	18.781 ms [18.593 ms, 18.969 ms]	551.989 µs (3.0%)
code_origins	17.752 ms [17.575 ms, 17.929 ms]	-476.925 µs (-2.6%)
iast	18.05 ms [17.871 ms, 18.229 ms]	-178.544 µs (-1.0%)
profiling	19.227 ms [19.037 ms, 19.418 ms]	998.501 µs (5.5%)
tracing	17.669 ms [17.494 ms, 17.843 ms]	-560.279 µs (-3.1%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.958 ms [18.769 ms, 19.147 ms]	-
appsec	18.49 ms [18.31 ms, 18.671 ms]	-467.524 µs (-2.5%)
code_origins	18.707 ms [18.523 ms, 18.89 ms]	-251.394 µs (-1.3%)
iast	17.856 ms [17.684 ms, 18.028 ms]	-1.102 ms (-5.8%)
profiling	255.456 µs [244.602 µs, 266.31 µs]	-18.702 ms (-98.7%)
tracing	17.603 ms [17.434 ms, 17.773 ms]	-1.355 ms (-7.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jb/rnd_otlp_profile
git_commit_date	1776844785	1777025973
git_commit_sha	c72f067	2ac3d50
release_version	1.62.0-SNAPSHOT~c72f06780f	1.62.0-SNAPSHOT~2ac3d50b80

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1777028278	1777028278
ci_job_id	1627595491	1627595491
ci_pipeline_id	109510136	109510136
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-8sidintu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-8sidintu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.491 ms) : 1479, 1503
.   : milestone, 1491,
appsec (3.844 ms) : 3622, 4066
.   : milestone, 3844,
iast (2.286 ms) : 2216, 2357
.   : milestone, 2286,
iast_GLOBAL (2.321 ms) : 2251, 2392
.   : milestone, 2321,
profiling (2.101 ms) : 2046, 2156
.   : milestone, 2101,
tracing (2.097 ms) : 2043, 2151
.   : milestone, 2097,
section candidate
no_agent (1.489 ms) : 1478, 1501
.   : milestone, 1489,
appsec (3.851 ms) : 3625, 4077
.   : milestone, 3851,
iast (2.282 ms) : 2212, 2352
.   : milestone, 2282,
iast_GLOBAL (2.331 ms) : 2260, 2401
.   : milestone, 2331,
profiling (2.109 ms) : 2053, 2164
.   : milestone, 2109,
tracing (2.1 ms) : 2046, 2154
.   : milestone, 2100,

Loading

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.491 ms [1.479 ms, 1.503 ms]	-
appsec	3.844 ms [3.622 ms, 4.066 ms]	2.353 ms (157.8%)
iast	2.286 ms [2.216 ms, 2.357 ms]	795.282 µs (53.3%)
iast_GLOBAL	2.321 ms [2.251 ms, 2.392 ms]	830.134 µs (55.7%)
profiling	2.101 ms [2.046 ms, 2.156 ms]	610.035 µs (40.9%)
tracing	2.097 ms [2.043 ms, 2.151 ms]	605.947 µs (40.6%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.489 ms [1.478 ms, 1.501 ms]	-
appsec	3.851 ms [3.625 ms, 4.077 ms]	2.362 ms (158.6%)
iast	2.282 ms [2.212 ms, 2.352 ms]	792.479 µs (53.2%)
iast_GLOBAL	2.331 ms [2.26 ms, 2.401 ms]	841.228 µs (56.5%)
profiling	2.109 ms [2.053 ms, 2.164 ms]	619.25 µs (41.6%)
tracing	2.1 ms [2.046 ms, 2.154 ms]	610.292 µs (41.0%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~2ac3d50b80, baseline=1.62.0-SNAPSHOT~c72f06780f
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.618 s) : 15618000, 15618000
.   : milestone, 15618000,
appsec (14.91 s) : 14910000, 14910000
.   : milestone, 14910000,
iast (18.599 s) : 18599000, 18599000
.   : milestone, 18599000,
iast_GLOBAL (18.106 s) : 18106000, 18106000
.   : milestone, 18106000,
profiling (14.75 s) : 14750000, 14750000
.   : milestone, 14750000,
tracing (14.973 s) : 14973000, 14973000
.   : milestone, 14973000,
section candidate
no_agent (15.067 s) : 15067000, 15067000
.   : milestone, 15067000,
appsec (14.752 s) : 14752000, 14752000
.   : milestone, 14752000,
iast (18.773 s) : 18773000, 18773000
.   : milestone, 18773000,
iast_GLOBAL (18.118 s) : 18118000, 18118000
.   : milestone, 18118000,
profiling (14.772 s) : 14772000, 14772000
.   : milestone, 14772000,
tracing (15.189 s) : 15189000, 15189000
.   : milestone, 15189000,

Loading

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.618 s [15.618 s, 15.618 s]	-
appsec	14.91 s [14.91 s, 14.91 s]	-708.0 ms (-4.5%)
iast	18.599 s [18.599 s, 18.599 s]	2.981 s (19.1%)
iast_GLOBAL	18.106 s [18.106 s, 18.106 s]	2.488 s (15.9%)
profiling	14.75 s [14.75 s, 14.75 s]	-868.0 ms (-5.6%)
tracing	14.973 s [14.973 s, 14.973 s]	-645.0 ms (-4.1%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.067 s [15.067 s, 15.067 s]	-
appsec	14.752 s [14.752 s, 14.752 s]	-315.0 ms (-2.1%)
iast	18.773 s [18.773 s, 18.773 s]	3.706 s (24.6%)
iast_GLOBAL	18.118 s [18.118 s, 18.118 s]	3.051 s (20.2%)
profiling	14.772 s [14.772 s, 14.772 s]	-295.0 ms (-2.0%)
tracing	15.189 s [15.189 s, 15.189 s]	122.0 ms (0.8%)

[github-actions]

This pull request has been marked as stale because it has not had activity over the past quarter. It will be closed in 7 days if no further activity occurs. Feel free to reopen the PR if you are still working on it.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.99 s	14.01 s	[-1.0%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent	12.96 s	12.98 s	[-0.8%; +0.5%] (no difference)
startup:petclinic:appsec:Agent	17.03 s	16.88 s	[-0.1%; +1.9%] (no difference)
startup:petclinic:iast:Agent	16.90 s	16.96 s	[-1.2%; +0.5%] (no difference)
startup:petclinic:profiling:Agent	16.11 s	16.69 s	[-7.8%; +0.9%] (no difference)
startup:petclinic:sca:Agent	16.99 s	16.33 s	[-0.4%; +8.5%] (no difference)
startup:petclinic:tracing:Agent	16.10 s	16.12 s	[-1.1%; +0.7%] (no difference)

Commit: 0ad43fb0 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-29T14:10:29.367437Z	207f9f5	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-29T14:10:46.055982Z	207f9f5	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 207f9f5a6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-datadog-prod-us1-2]

Standard-JFR environments silently lose CPU samples, while LIGHT and FULL payload construction can amplify large recordings enough to exhaust the agent heap.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 207f9f5 · @DataDog review to ask questions}

[jbachorik]

🗿 🤖 🔴

This is the verdict/bundle summary of the Bits review, not an independent finding: it names no file, line, or symbol beyond the two themes already covered by items 4134449271/4134541667 (JDK sampling loss) and 4134449264/4134541682/4134541696 (LIGHT/FULL heap amplification), all of which are verdicts needs_fix in this evaluation. The summary itself requires no code change; fixing it separately would double-count the inline items.
Also, the reasoning is completely wrong. JFR environments never silently lose CPU samples for heap pressure. Also, payload construction is restricted, the JFR parser does minimal allocations and the only memory cost is the OTELP aggregated data storage, which I haven't seen to cross 100-150kB yet, so there is very little chance that this would be the thing that would exhaust agent heap.

[mhlidd]

FWIW some of the shared OTLP mechanisms should probably be refactored by the Capabilities team.

Some design questions about Config handling and Resource Attributes

[mhlidd]

From Claude:

Skewed conversion telemetry. onProfilesConversion runs before the null check. It is also recorded in LIGHT mode, which is the default and converts nothing. Result: otel.profiles_conversion_ms and otel.profiles_conversion_max_ms mostly measure file-stat or encode time. Record only in CONVERTED/FULL mode and only when a payload was produced. Also confirm that all five new tracers metrics are in the backend telemetry metric registry: the two gauges plus otel.profiles_export_{attempts,successes,failures}. Serializing doubles as gauges is fine.

[jbachorik]

Fixed he the skew. I need to register the metrics still