-
Notifications
You must be signed in to change notification settings - Fork 319
Add Logs Capture feature #10156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
jpbempel
wants to merge
1
commit into
master
Choose a base branch
from
jpbempel/logs-capture
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Add Logs Capture feature #10156
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
0a5090f to
e60662e
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 2 performance improvements and 1 performance regressions! Performance is the same for 55 metrics, 7 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.079 s) : 0, 1078942
Total [baseline] (8.751 s) : 0, 8750621
Agent [candidate] (1.08 s) : 0, 1079867
Total [candidate] (8.768 s) : 0, 8767660
section iast
Agent [baseline] (1.217 s) : 0, 1217039
Total [baseline] (9.486 s) : 0, 9486242
Agent [candidate] (1.228 s) : 0, 1228370
Total [candidate] (9.478 s) : 0, 9477721
gantt
title insecure-bank - break down per module: candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (647.456 ms) : 0, 647456
BytebuddyAgent [candidate] (663.695 ms) : 0, 663695
GlobalTracer [baseline] (281.508 ms) : 0, 281508
GlobalTracer [candidate] (265.371 ms) : 0, 265371
AppSec [baseline] (32.239 ms) : 0, 32239
AppSec [candidate] (32.341 ms) : 0, 32341
Debugger [baseline] (67.54 ms) : 0, 67540
Debugger [candidate] (68.059 ms) : 0, 68059
Remote Config [baseline] (659.214 µs) : 0, 659
Remote Config [candidate] (670.348 µs) : 0, 670
Telemetry [baseline] (9.037 ms) : 0, 9037
Telemetry [candidate] (9.107 ms) : 0, 9107
Flare Poller [baseline] (3.763 ms) : 0, 3763
Flare Poller [candidate] (3.84 ms) : 0, 3840
section iast
crashtracking [baseline] (1.205 ms) : 0, 1205
crashtracking [candidate] (1.211 ms) : 0, 1211
BytebuddyAgent [baseline] (786.838 ms) : 0, 786838
BytebuddyAgent [candidate] (794.448 ms) : 0, 794448
GlobalTracer [baseline] (254.156 ms) : 0, 254156
GlobalTracer [candidate] (257.482 ms) : 0, 257482
AppSec [baseline] (35.619 ms) : 0, 35619
AppSec [candidate] (34.501 ms) : 0, 34501
Debugger [baseline] (64.553 ms) : 0, 64553
Debugger [candidate] (65.731 ms) : 0, 65731
Remote Config [baseline] (556.148 µs) : 0, 556
Remote Config [candidate] (563.703 µs) : 0, 564
Telemetry [baseline] (8.435 ms) : 0, 8435
Telemetry [candidate] (8.466 ms) : 0, 8466
Flare Poller [baseline] (3.428 ms) : 0, 3428
Flare Poller [candidate] (3.511 ms) : 0, 3511
IAST [baseline] (26.776 ms) : 0, 26776
IAST [candidate] (27.006 ms) : 0, 27006
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.08 s) : 0, 1080275
Total [baseline] (10.866 s) : 0, 10866223
Agent [candidate] (1.088 s) : 0, 1088330
Total [candidate] (10.81 s) : 0, 10809517
section appsec
Agent [baseline] (1.264 s) : 0, 1263845
Total [baseline] (11.201 s) : 0, 11200705
Agent [candidate] (1.266 s) : 0, 1266384
Total [candidate] (11.031 s) : 0, 11030742
section iast
Agent [baseline] (1.229 s) : 0, 1228987
Total [baseline] (11.194 s) : 0, 11193968
Agent [candidate] (1.231 s) : 0, 1230561
Total [candidate] (11.201 s) : 0, 11200994
section profiling
Agent [baseline] (1.212 s) : 0, 1212416
Total [baseline] (10.967 s) : 0, 10966965
Agent [candidate] (1.211 s) : 0, 1211469
Total [candidate] (11.133 s) : 0, 11133110
gantt
title petclinic - break down per module: candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.213 ms) : 0, 1213
BytebuddyAgent [baseline] (647.605 ms) : 0, 647605
BytebuddyAgent [candidate] (668.753 ms) : 0, 668753
GlobalTracer [baseline] (281.491 ms) : 0, 281491
GlobalTracer [candidate] (267.638 ms) : 0, 267638
AppSec [baseline] (32.317 ms) : 0, 32317
AppSec [candidate] (32.644 ms) : 0, 32644
Debugger [baseline] (68.73 ms) : 0, 68730
Debugger [candidate] (68.329 ms) : 0, 68329
Remote Config [baseline] (660.984 µs) : 0, 661
Remote Config [candidate] (647.768 µs) : 0, 648
Telemetry [baseline] (9.007 ms) : 0, 9007
Telemetry [candidate] (9.783 ms) : 0, 9783
Flare Poller [baseline] (3.728 ms) : 0, 3728
Flare Poller [candidate] (3.78 ms) : 0, 3780
section appsec
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.205 ms) : 0, 1205
BytebuddyAgent [baseline] (687.385 ms) : 0, 687385
BytebuddyAgent [candidate] (690.039 ms) : 0, 690039
GlobalTracer [baseline] (258.749 ms) : 0, 258749
GlobalTracer [candidate] (259.106 ms) : 0, 259106
AppSec [baseline] (175.32 ms) : 0, 175320
AppSec [candidate] (174.683 ms) : 0, 174683
Debugger [baseline] (67.216 ms) : 0, 67216
Debugger [candidate] (67.341 ms) : 0, 67341
Remote Config [baseline] (752.44 µs) : 0, 752
Remote Config [candidate] (707.825 µs) : 0, 708
Telemetry [baseline] (9.179 ms) : 0, 9179
Telemetry [candidate] (9.13 ms) : 0, 9130
Flare Poller [baseline] (3.888 ms) : 0, 3888
Flare Poller [candidate] (3.995 ms) : 0, 3995
IAST [baseline] (24.666 ms) : 0, 24666
IAST [candidate] (24.651 ms) : 0, 24651
section iast
crashtracking [baseline] (1.199 ms) : 0, 1199
crashtracking [candidate] (1.227 ms) : 0, 1227
BytebuddyAgent [baseline] (794.769 ms) : 0, 794769
BytebuddyAgent [candidate] (796.91 ms) : 0, 796910
GlobalTracer [baseline] (256.59 ms) : 0, 256590
GlobalTracer [candidate] (255.893 ms) : 0, 255893
AppSec [baseline] (35.583 ms) : 0, 35583
AppSec [candidate] (35.479 ms) : 0, 35479
Debugger [baseline] (65.804 ms) : 0, 65804
Debugger [candidate] (66.144 ms) : 0, 66144
Remote Config [baseline] (561.562 µs) : 0, 562
Remote Config [candidate] (554.921 µs) : 0, 555
Telemetry [baseline] (8.399 ms) : 0, 8399
Telemetry [candidate] (8.491 ms) : 0, 8491
Flare Poller [baseline] (3.462 ms) : 0, 3462
Flare Poller [candidate] (3.487 ms) : 0, 3487
IAST [baseline] (27.074 ms) : 0, 27074
IAST [candidate] (26.962 ms) : 0, 26962
section profiling
crashtracking [baseline] (1.206 ms) : 0, 1206
crashtracking [candidate] (1.201 ms) : 0, 1201
BytebuddyAgent [baseline] (705.863 ms) : 0, 705863
BytebuddyAgent [candidate] (705.373 ms) : 0, 705373
GlobalTracer [baseline] (221.752 ms) : 0, 221752
GlobalTracer [candidate] (221.86 ms) : 0, 221860
AppSec [baseline] (32.497 ms) : 0, 32497
AppSec [candidate] (32.468 ms) : 0, 32468
Debugger [baseline] (68.69 ms) : 0, 68690
Debugger [candidate] (68.909 ms) : 0, 68909
Remote Config [baseline] (670.186 µs) : 0, 670
Remote Config [candidate] (621.871 µs) : 0, 622
Telemetry [baseline] (9.054 ms) : 0, 9054
Telemetry [candidate] (9.058 ms) : 0, 9058
Flare Poller [baseline] (3.841 ms) : 0, 3841
Flare Poller [candidate] (3.798 ms) : 0, 3798
ProfilingAgent [baseline] (98.741 ms) : 0, 98741
ProfilingAgent [candidate] (98.225 ms) : 0, 98225
Profiling [baseline] (99.333 ms) : 0, 99333
Profiling [candidate] (98.806 ms) : 0, 98806
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 17 metrics, 19 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section baseline
no_agent (1.193 ms) : 1181, 1205
. : milestone, 1193,
iast (3.255 ms) : 3209, 3301
. : milestone, 3255,
iast_FULL (5.724 ms) : 5667, 5780
. : milestone, 5724,
iast_GLOBAL (3.598 ms) : 3546, 3651
. : milestone, 3598,
profiling (2.017 ms) : 2000, 2035
. : milestone, 2017,
tracing (1.871 ms) : 1854, 1887
. : milestone, 1871,
section candidate
no_agent (1.213 ms) : 1201, 1224
. : milestone, 1213,
iast (3.311 ms) : 3265, 3358
. : milestone, 3311,
iast_FULL (6.023 ms) : 5962, 6085
. : milestone, 6023,
iast_GLOBAL (3.597 ms) : 3544, 3650
. : milestone, 3597,
profiling (2.194 ms) : 2172, 2217
. : milestone, 2194,
tracing (1.836 ms) : 1820, 1852
. : milestone, 1836,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section baseline
no_agent (17.964 ms) : 17784, 18143
. : milestone, 17964,
appsec (18.464 ms) : 18279, 18649
. : milestone, 18464,
code_origins (17.572 ms) : 17400, 17745
. : milestone, 17572,
iast (17.987 ms) : 17807, 18168
. : milestone, 17987,
profiling (18.537 ms) : 18354, 18720
. : milestone, 18537,
tracing (17.545 ms) : 17375, 17716
. : milestone, 17545,
section candidate
no_agent (18.004 ms) : 17816, 18192
. : milestone, 18004,
appsec (18.555 ms) : 18366, 18744
. : milestone, 18555,
code_origins (17.834 ms) : 17657, 18010
. : milestone, 17834,
iast (17.591 ms) : 17417, 17766
. : milestone, 17591,
profiling (18.584 ms) : 18397, 18772
. : milestone, 18584,
tracing (17.753 ms) : 17574, 17931
. : milestone, 17753,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section baseline
no_agent (14.949 s) : 14949000, 14949000
. : milestone, 14949000,
appsec (14.985 s) : 14985000, 14985000
. : milestone, 14985000,
iast (18.007 s) : 18007000, 18007000
. : milestone, 18007000,
iast_GLOBAL (17.743 s) : 17743000, 17743000
. : milestone, 17743000,
profiling (14.626 s) : 14626000, 14626000
. : milestone, 14626000,
tracing (14.603 s) : 14603000, 14603000
. : milestone, 14603000,
section candidate
no_agent (14.924 s) : 14924000, 14924000
. : milestone, 14924000,
appsec (14.798 s) : 14798000, 14798000
. : milestone, 14798000,
iast (17.962 s) : 17962000, 17962000
. : milestone, 17962000,
iast_GLOBAL (17.925 s) : 17925000, 17925000
. : milestone, 17925000,
profiling (14.788 s) : 14788000, 14788000
. : milestone, 14788000,
tracing (14.675 s) : 14675000, 14675000
. : milestone, 14675000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~3d4d2ca562, baseline=1.57.0-SNAPSHOT~f38cc2ee7a
dateFormat X
axisFormat %s
section baseline
no_agent (1.477 ms) : 1465, 1488
. : milestone, 1477,
appsec (2.517 ms) : 2463, 2572
. : milestone, 2517,
iast (2.22 ms) : 2156, 2284
. : milestone, 2220,
iast_GLOBAL (2.265 ms) : 2200, 2330
. : milestone, 2265,
profiling (2.456 ms) : 2242, 2670
. : milestone, 2456,
tracing (2.052 ms) : 2001, 2103
. : milestone, 2052,
section candidate
no_agent (1.479 ms) : 1468, 1491
. : milestone, 1479,
appsec (2.456 ms) : 2405, 2508
. : milestone, 2456,
iast (2.223 ms) : 2159, 2288
. : milestone, 2223,
iast_GLOBAL (2.259 ms) : 2194, 2324
. : milestone, 2259,
profiling (2.099 ms) : 2044, 2153
. : milestone, 2099,
tracing (2.051 ms) : 2000, 2102
. : milestone, 2051,
|
e60662e to
9470cec
Compare
9470cec to
6ef9061
Compare
Add logs capture based on log framework instrumentation that hooks when log event are generated to be written. Log events are sent to Logs backend through Datadog agent's Event Platform (Evp) proxy. We are reusing a previous infrastructure used for uploading logs for CI Visibility (see #7082). We are adding support for Logback. This feature is opt-in through DD_LOGS_CAPTURE_ENABLED config. Logs are sent as structured log layout and allow to add any additional attributes. For now we are adding directly trace correlation ids (traceId and spanId) and give the logs injection feature for free and without any setup on the log framework side. Smoke tests are based on Logs Injection tests.
6ef9061 to
3d4d2ca
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Add logs capture based on log framework instrumentation that hooks when log event are generated to be written. Log events are sent to Logs backend through Datadog agent's Event Platform (Evp) proxy. We are reusing a previous infrastructure used for uploading logs for CI Visibility (see #7082).
We are adding support for Logback.
This feature is opt-in through
DD_LOGS_CAPTURE_ENABLEDconfig. Logs are sent as structured log layout and allow to add any additional attributes. For now we are adding directly trace correlation ids (traceId and spanId) and give the logs injection feature for free and without any setup on the log framework side.Smoke tests are based on Logs Injection tests.
Motivation
On-boarding Logs easily through tracer that are already installed through SSI
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]