Add config source reading from Test Optimization remote environment #9701
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikita-tkachenko-datadog
added
type: enhancement
|
🎯 Code Coverage 🔗 Commit SHA: 6c45053 | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 5 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.017 s) : 0, 1016759
Total [baseline] (10.597 s) : 0, 10597183
Agent [candidate] (1.032 s) : 0, 1031728
Total [candidate] (10.783 s) : 0, 10782502
section appsec
Agent [baseline] (1.193 s) : 0, 1192663
Total [baseline] (11.021 s) : 0, 11021278
Agent [candidate] (1.2 s) : 0, 1200048
Total [candidate] (10.941 s) : 0, 10940823
section iast
Agent [baseline] (1.149 s) : 0, 1149232
Total [baseline] (10.982 s) : 0, 10982413
Agent [candidate] (1.159 s) : 0, 1158589
Total [candidate] (10.977 s) : 0, 10977435
section profiling
Agent [baseline] (1.168 s) : 0, 1168342
Total [baseline] (11.047 s) : 0, 11046735
Agent [candidate] (1.162 s) : 0, 1162159
Total [candidate] (11.045 s) : 0, 11045286
gantt
title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.468 ms) : 0, 1468
crashtracking [candidate] (1.475 ms) : 0, 1475
BytebuddyAgent [baseline] (693.01 ms) : 0, 693010
BytebuddyAgent [candidate] (703.187 ms) : 0, 703187
GlobalTracer [baseline] (241.641 ms) : 0, 241641
GlobalTracer [candidate] (243.537 ms) : 0, 243537
AppSec [baseline] (32.352 ms) : 0, 32352
AppSec [candidate] (33.664 ms) : 0, 33664
Debugger [baseline] (6.409 ms) : 0, 6409
Debugger [candidate] (6.432 ms) : 0, 6432
Remote Config [baseline] (694.977 µs) : 0, 695
Remote Config [candidate] (698.897 µs) : 0, 699
Telemetry [baseline] (9.177 ms) : 0, 9177
Telemetry [candidate] (9.516 ms) : 0, 9516
Flare Poller [baseline] (10.805 ms) : 0, 10805
Flare Poller [candidate] (11.838 ms) : 0, 11838
section appsec
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.477 ms) : 0, 1477
BytebuddyAgent [baseline] (717.996 ms) : 0, 717996
BytebuddyAgent [candidate] (722.129 ms) : 0, 722129
GlobalTracer [baseline] (233.934 ms) : 0, 233934
GlobalTracer [candidate] (235.13 ms) : 0, 235130
AppSec [baseline] (174.348 ms) : 0, 174348
AppSec [candidate] (175.807 ms) : 0, 175807
Debugger [baseline] (6.101 ms) : 0, 6101
Debugger [candidate] (6.186 ms) : 0, 6186
Remote Config [baseline] (627.275 µs) : 0, 627
Remote Config [candidate] (654.121 µs) : 0, 654
Telemetry [baseline] (8.431 ms) : 0, 8431
Telemetry [candidate] (8.595 ms) : 0, 8595
Flare Poller [baseline] (3.968 ms) : 0, 3968
Flare Poller [candidate] (3.942 ms) : 0, 3942
IAST [baseline] (24.742 ms) : 0, 24742
IAST [candidate] (24.927 ms) : 0, 24927
section iast
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.473 ms) : 0, 1473
BytebuddyAgent [baseline] (814.121 ms) : 0, 814121
BytebuddyAgent [candidate] (821.026 ms) : 0, 821026
GlobalTracer [baseline] (231.344 ms) : 0, 231344
GlobalTracer [candidate] (232.201 ms) : 0, 232201
AppSec [baseline] (34.882 ms) : 0, 34882
AppSec [candidate] (35.994 ms) : 0, 35994
Debugger [baseline] (6.189 ms) : 0, 6189
Debugger [candidate] (6.198 ms) : 0, 6198
Remote Config [baseline] (610.446 µs) : 0, 610
Remote Config [candidate] (605.293 µs) : 0, 605
Telemetry [baseline] (8.603 ms) : 0, 8603
Telemetry [candidate] (8.746 ms) : 0, 8746
Flare Poller [baseline] (4.157 ms) : 0, 4157
Flare Poller [candidate] (4.18 ms) : 0, 4180
IAST [baseline] (26.522 ms) : 0, 26522
IAST [candidate] (26.957 ms) : 0, 26957
section profiling
ProfilingAgent [baseline] (108.044 ms) : 0, 108044
ProfilingAgent [candidate] (108.434 ms) : 0, 108434
crashtracking [baseline] (1.446 ms) : 0, 1446
crashtracking [candidate] (1.443 ms) : 0, 1443
BytebuddyAgent [baseline] (726.071 ms) : 0, 726071
BytebuddyAgent [candidate] (720.582 ms) : 0, 720582
GlobalTracer [baseline] (218.146 ms) : 0, 218146
GlobalTracer [candidate] (216.994 ms) : 0, 216994
AppSec [baseline] (33.038 ms) : 0, 33038
AppSec [candidate] (33.023 ms) : 0, 33023
Debugger [baseline] (7.347 ms) : 0, 7347
Debugger [candidate] (6.4 ms) : 0, 6400
Remote Config [baseline] (692.503 µs) : 0, 693
Remote Config [candidate] (697.968 µs) : 0, 698
Telemetry [baseline] (14.975 ms) : 0, 14975
Telemetry [candidate] (16.797 ms) : 0, 16797
Flare Poller [baseline] (4.956 ms) : 0, 4956
Flare Poller [candidate] (4.14 ms) : 0, 4140
Profiling [baseline] (109.347 ms) : 0, 109347
Profiling [candidate] (109.002 ms) : 0, 109002
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.02 s) : 0, 1020256
Total [baseline] (8.671 s) : 0, 8670942
Agent [candidate] (1.024 s) : 0, 1023678
Total [candidate] (8.637 s) : 0, 8637135
section iast
Agent [baseline] (1.149 s) : 0, 1148746
Total [baseline] (9.289 s) : 0, 9289338
Agent [candidate] (1.153 s) : 0, 1152732
Total [candidate] (9.24 s) : 0, 9239685
gantt
title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.461 ms) : 0, 1461
crashtracking [candidate] (1.482 ms) : 0, 1482
BytebuddyAgent [baseline] (694.068 ms) : 0, 694068
BytebuddyAgent [candidate] (699.336 ms) : 0, 699336
GlobalTracer [baseline] (242.42 ms) : 0, 242420
GlobalTracer [candidate] (241.19 ms) : 0, 241190
AppSec [baseline] (32.861 ms) : 0, 32861
AppSec [candidate] (33.089 ms) : 0, 33089
Debugger [baseline] (6.475 ms) : 0, 6475
Debugger [candidate] (6.333 ms) : 0, 6333
Remote Config [baseline] (692.277 µs) : 0, 692
Remote Config [candidate] (685.045 µs) : 0, 685
Telemetry [baseline] (9.383 ms) : 0, 9383
Telemetry [candidate] (9.473 ms) : 0, 9473
Flare Poller [baseline] (11.842 ms) : 0, 11842
Flare Poller [candidate] (10.924 ms) : 0, 10924
section iast
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.461 ms) : 0, 1461
BytebuddyAgent [baseline] (814.28 ms) : 0, 814280
BytebuddyAgent [candidate] (816.984 ms) : 0, 816984
GlobalTracer [baseline] (231.039 ms) : 0, 231039
GlobalTracer [candidate] (231.177 ms) : 0, 231177
IAST [baseline] (26.359 ms) : 0, 26359
IAST [candidate] (26.84 ms) : 0, 26840
AppSec [baseline] (34.856 ms) : 0, 34856
AppSec [candidate] (35.473 ms) : 0, 35473
Debugger [baseline] (6.149 ms) : 0, 6149
Debugger [candidate] (6.198 ms) : 0, 6198
Remote Config [baseline] (612.33 µs) : 0, 612
Remote Config [candidate] (590.574 µs) : 0, 591
Telemetry [baseline] (8.646 ms) : 0, 8646
Telemetry [candidate] (8.762 ms) : 0, 8762
Flare Poller [baseline] (4.14 ms) : 0, 4140
Flare Poller [candidate] (4.197 ms) : 0, 4197
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 3 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section baseline
no_agent (36.929 ms) : 36630, 37228
. : milestone, 36929,
appsec (46.681 ms) : 46275, 47088
. : milestone, 46681,
code_origins (45.278 ms) : 44890, 45667
. : milestone, 45278,
iast (45.034 ms) : 44652, 45417
. : milestone, 45034,
profiling (50.784 ms) : 50327, 51242
. : milestone, 50784,
tracing (44.81 ms) : 44415, 45204
. : milestone, 44810,
section candidate
no_agent (38.371 ms) : 38055, 38687
. : milestone, 38371,
appsec (49.475 ms) : 49017, 49934
. : milestone, 49475,
code_origins (44.187 ms) : 43802, 44571
. : milestone, 44187,
iast (45.729 ms) : 45328, 46130
. : milestone, 45729,
profiling (51.968 ms) : 51501, 52436
. : milestone, 51968,
tracing (44.697 ms) : 44305, 45089
. : milestone, 44697,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section baseline
no_agent (4.292 ms) : 4231, 4353
. : milestone, 4292,
iast (10.145 ms) : 9972, 10318
. : milestone, 10145,
iast_FULL (14.625 ms) : 14331, 14920
. : milestone, 14625,
iast_GLOBAL (11.183 ms) : 10982, 11383
. : milestone, 11183,
profiling (8.826 ms) : 8692, 8960
. : milestone, 8826,
tracing (7.741 ms) : 7622, 7860
. : milestone, 7741,
section candidate
no_agent (4.373 ms) : 4322, 4425
. : milestone, 4373,
iast (10.022 ms) : 9855, 10189
. : milestone, 10022,
iast_FULL (14.016 ms) : 13735, 14297
. : milestone, 14016,
iast_GLOBAL (10.821 ms) : 10624, 11018
. : milestone, 10821,
profiling (8.804 ms) : 8664, 8945
. : milestone, 8804,
tracing (8.127 ms) : 8001, 8253
. : milestone, 8127,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section baseline
no_agent (14.937 s) : 14937000, 14937000
. : milestone, 14937000,
appsec (15.003 s) : 15003000, 15003000
. : milestone, 15003000,
iast (18.652 s) : 18652000, 18652000
. : milestone, 18652000,
iast_GLOBAL (18.061 s) : 18061000, 18061000
. : milestone, 18061000,
profiling (14.935 s) : 14935000, 14935000
. : milestone, 14935000,
tracing (14.987 s) : 14987000, 14987000
. : milestone, 14987000,
section candidate
no_agent (14.944 s) : 14944000, 14944000
. : milestone, 14944000,
appsec (15.016 s) : 15016000, 15016000
. : milestone, 15016000,
iast (18.427 s) : 18427000, 18427000
. : milestone, 18427000,
iast_GLOBAL (18.048 s) : 18048000, 18048000
. : milestone, 18048000,
profiling (15.838 s) : 15838000, 15838000
. : milestone, 15838000,
tracing (15.159 s) : 15159000, 15159000
. : milestone, 15159000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~6c450537c3, baseline=1.55.0-SNAPSHOT~295f7bf5b4
dateFormat X
axisFormat %s
section baseline
no_agent (1.473 ms) : 1461, 1484
. : milestone, 1473,
appsec (2.452 ms) : 2401, 2503
. : milestone, 2452,
iast (2.196 ms) : 2133, 2260
. : milestone, 2196,
iast_GLOBAL (2.252 ms) : 2188, 2315
. : milestone, 2252,
profiling (2.044 ms) : 1993, 2095
. : milestone, 2044,
tracing (2.029 ms) : 1979, 2078
. : milestone, 2029,
section candidate
no_agent (1.47 ms) : 1459, 1482
. : milestone, 1470,
appsec (3.731 ms) : 3513, 3950
. : milestone, 3731,
iast (2.203 ms) : 2140, 2267
. : milestone, 2203,
iast_GLOBAL (2.247 ms) : 2183, 2311
. : milestone, 2247,
profiling (2.067 ms) : 2014, 2119
. : milestone, 2067,
tracing (2.017 ms) : 1967, 2066
. : milestone, 2017,
|
nikita-tkachenko-datadog
requested review from
Mariovido and
PerfectSlayer
and removed request for
a team
daniel-mohedano
approved these changes
Oct 10, 2025
There was a problem hiding this comment.
💭 thought: I have few concerns about this change.
First, it introduce both product -- CI Visibility -- and feature -- Config -- into a platform component. It also introduces a dependency to a logger.
I would very like to keep the platform component as light and "single-concerned" as possible.
🎯 suggestion: As an alternative, what about implementing an EnvironmentVariablesProvider with your logic, and apply is as source for the EnvironmentVariable?
If it works, that would leave both :environment and :config-utils untouched.
because Bazel runs tests in a sandbox where environment variables from the "outside" are not available by default.
❔ question: Just to be sure I get it right, it concerns cases where the dd-java-agent runs with the bazel tools when users runs their tests, right?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds a new config source that reads from Test Optimization remote environment.
The tracer will only use the new config source when explicitly configured to do so.
Motivation
For Test Optimization runs with Bazel we cannot read environment variables as usual, because Bazel runs tests in a sandbox where environment variables from the "outside" are not available by default.
Any environment variable exposed to the tests messes with Bazel's cache, as all env vars are considered as test tasks input, so if any of them changes, the previously cached outcome of the task cannot be used anymore.
Some environment vars read by the tracer (such as git commit SHA or CI job ID) change constantly making Bazel cache virtually unusable.
The temporary solution for now is to use a service that the tracer can query over HTTP to get the environment variables, which allows to circumvent Bazel's isolation without breaking caching.
Additional Notes
The solution described above is already being used for the env vars that the tracer reads from the CI provider.
This PR extends it to also be used for the env vars that control config values.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: LANGTOOLS-3400