Skip to content

Conversation

@NachoEchevarria
Copy link
Collaborator

Summary of changes

We found the following error in the CI:

2025-11-25T19:47:40.3540090Z [ERR] CheckBuildLogsForErr: Found errors in log file '/project/artifacts/build_data/logs/ProbesTestsLogs/RunMethodProbeTestsoptimized_True_type_LineProbesWithRevertTest/dotnet-tracer-managed-dotnet-2991.log':
2025-11-25T19:47:40.3540643Z [ERR] CheckBuildLogsForErr: 11/25/2025 19:40:39 +00:00 [Error] Probe Status Poller: Resuming the poller has failed. System.ObjectDisposedException: Cannot access a disposed object.
2025-11-25T19:47:40.3541006Z    at System.Threading.TimerQueueTimer.Change(UInt32 dueTime, UInt32 period)
2025-11-25T19:47:40.3541345Z    at System.Threading.Timer.Change(Int64 dueTime, Int64 period)
2025-11-25T19:47:40.3542243Z    at System.Threading.Timer.Change(TimeSpan dueTime, TimeSpan period)
2025-11-25T19:47:40.3542851Z    at Datadog.Trace.Debugger.ProbeStatuses.ProbeStatusPoller.ResumePollerTimer() in /project/tracer/src/Datadog.Trace/Debugger/ProbeStatuses/ProbeStatusPoller.cs:line 99
2025-11-25T19:47:40.3543348Z  { MachineName: ".", Process: "[2991 dotnet]", AppDomain: "[1 Samples.Probes]", AssemblyLoadContext: "\"\" Datadog.Trace.ClrProfiler.Managed.Loader.ManagedProfilerAssemblyLoadContext #1", TracerVersion: "3.33.0.0" }

This PR fixes a race condition in ProbeStatusPoller that caused flaky test failures when the poller is disposed while a timer callback is executing.

The problem is:

  1. The timer callback acquires _locker via TryAcquireLock()
  2. Inside the callback, it calls PausePollerTimer() then ResumePollerTimer()
  3. Dispose() doesn't acquire _locker, so it can dispose the timer while the callback is between pause and resume
  4. When ResumePollerTimer() tries to call _pollerTimer.Change(), it throws ObjectDisposedException

The fix ensures proper synchronization between timer callbacks and disposal:

  1. Timer callbacks acquire _locker before calling pause/resume operations
  2. Dispose() now also acquires _locker before setting _disposed and disposing the timer
  3. All timer operations check _disposed before attempting to use the timer
  4. This prevents the race where disposal happens between pause and resume operations

Reason for change

Implementation details

Test coverage

Other details

@github-actions github-actions bot added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:debugger labels Nov 26, 2025
@dd-trace-dotnet-ci-bot
Copy link

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (7866) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration74.69 ± (74.81 - 75.59) ms75.01 ± (75.09 - 75.82) ms+0.4%✅⬆️
.NET Framework 4.8 - Bailout
duration78.38 ± (78.41 - 79.08) ms78.90 ± (78.82 - 79.48) ms+0.7%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1046.28 ± (1051.76 - 1062.63) ms1061.34 ± (1065.91 - 1076.52) ms+1.4%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms22.92 ± (22.84 - 22.99) ms22.85 ± (22.77 - 22.92) ms-0.3%
process.time_to_main_ms86.86 ± (86.54 - 87.18) ms86.94 ± (86.57 - 87.31) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.92 ± (10.91 - 10.92) MB10.91 ± (10.90 - 10.91) MB-0.1%
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%
.NET Core 3.1 - Bailout
process.internal_duration_ms22.72 ± (22.67 - 22.78) ms22.98 ± (22.91 - 23.05) ms+1.1%✅⬆️
process.time_to_main_ms88.47 ± (88.12 - 88.82) ms89.41 ± (89.02 - 89.80) ms+1.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.96 ± (10.95 - 10.96) MB10.94 ± (10.94 - 10.95) MB-0.1%
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms220.36 ± (218.79 - 221.92) ms220.80 ± (219.08 - 222.52) ms+0.2%✅⬆️
process.time_to_main_ms498.07 ± (496.84 - 499.30) ms499.20 ± (498.07 - 500.32) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed47.76 ± (47.74 - 47.77) MB47.74 ± (47.72 - 47.77) MB-0.0%
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.5%
.NET 6 - Baseline
process.internal_duration_ms21.71 ± (21.64 - 21.78) ms21.88 ± (21.82 - 21.95) ms+0.8%✅⬆️
process.time_to_main_ms75.32 ± (75.02 - 75.63) ms77.25 ± (76.89 - 77.61) ms+2.6%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.59 ± (10.59 - 10.60) MB10.63 ± (10.63 - 10.63) MB+0.3%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 6 - Bailout
process.internal_duration_ms21.69 ± (21.63 - 21.76) ms21.90 ± (21.83 - 21.97) ms+0.9%✅⬆️
process.time_to_main_ms77.36 ± (77.01 - 77.70) ms78.34 ± (77.96 - 78.71) ms+1.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.66 ± (10.66 - 10.67) MB10.73 ± (10.72 - 10.73) MB+0.6%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms214.25 ± (211.44 - 217.07) ms209.38 ± (208.05 - 210.72) ms-2.3%
process.time_to_main_ms463.89 ± (462.89 - 464.88) ms466.95 ± (466.07 - 467.84) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed48.22 ± (48.20 - 48.24) MB48.13 ± (48.11 - 48.16) MB-0.2%
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms19.72 ± (19.66 - 19.78) ms20.17 ± (20.09 - 20.24) ms+2.3%✅⬆️
process.time_to_main_ms73.98 ± (73.65 - 74.32) ms76.15 ± (75.77 - 76.53) ms+2.9%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.64 ± (7.64 - 7.65) MB7.65 ± (7.64 - 7.66) MB+0.1%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 8 - Bailout
process.internal_duration_ms19.91 ± (19.85 - 19.98) ms19.80 ± (19.74 - 19.85) ms-0.6%
process.time_to_main_ms76.99 ± (76.65 - 77.32) ms76.62 ± (76.33 - 76.90) ms-0.5%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.71 ± (7.70 - 7.72) MB7.70 ± (7.70 - 7.71) MB-0.1%
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms191.09 ± (190.22 - 191.97) ms190.00 ± (189.06 - 190.93) ms-0.6%
process.time_to_main_ms454.27 ± (453.50 - 455.04) ms456.44 ± (455.39 - 457.50) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed36.44 ± (36.40 - 36.47) MB36.38 ± (36.35 - 36.41) MB-0.1%
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)-0.4%

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration192.23 ± (192.47 - 193.22) ms193.87 ± (193.86 - 194.72) ms+0.9%✅⬆️
.NET Framework 4.8 - Bailout
duration196.68 ± (196.49 - 197.08) ms195.93 ± (195.73 - 196.16) ms-0.4%
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1100.69 ± (1105.51 - 1115.38) ms1105.30 ± (1112.35 - 1122.49) ms+0.4%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms187.92 ± (187.62 - 188.22) ms187.82 ± (187.50 - 188.14) ms-0.1%
process.time_to_main_ms80.61 ± (80.40 - 80.82) ms80.86 ± (80.64 - 81.07) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.09 ± (16.07 - 16.11) MB16.17 ± (16.13 - 16.20) MB+0.5%✅⬆️
runtime.dotnet.threads.count20 ± (19 - 20)20 ± (19 - 20)+0.1%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms187.77 ± (187.44 - 188.09) ms187.96 ± (187.63 - 188.29) ms+0.1%✅⬆️
process.time_to_main_ms81.97 ± (81.82 - 82.12) ms82.16 ± (82.02 - 82.31) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.16 ± (16.13 - 16.20) MB16.23 ± (16.20 - 16.26) MB+0.4%✅⬆️
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (21 - 21)-0.1%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms408.61 ± (405.19 - 412.03) ms406.27 ± (403.11 - 409.42) ms-0.6%
process.time_to_main_ms470.17 ± (469.45 - 470.89) ms471.41 ± (470.70 - 472.12) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed58.25 ± (58.11 - 58.39) MB58.69 ± (58.57 - 58.81) MB+0.8%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)30 ± (29 - 30)+0.2%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms192.45 ± (192.07 - 192.82) ms192.31 ± (191.91 - 192.70) ms-0.1%
process.time_to_main_ms70.03 ± (69.86 - 70.21) ms70.16 ± (70.01 - 70.31) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.32 ± (16.22 - 16.42) MB16.31 ± (16.21 - 16.42) MB-0.1%
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)-0.1%
.NET 6 - Bailout
process.internal_duration_ms191.04 ± (190.79 - 191.30) ms190.93 ± (190.71 - 191.15) ms-0.1%
process.time_to_main_ms70.77 ± (70.70 - 70.84) ms70.82 ± (70.76 - 70.89) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.07 ± (15.92 - 16.23) MB16.30 ± (16.19 - 16.42) MB+1.4%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)20 ± (20 - 20)+2.7%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms415.18 ± (412.10 - 418.25) ms411.33 ± (408.47 - 414.19) ms-0.9%
process.time_to_main_ms442.10 ± (441.46 - 442.73) ms441.07 ± (440.41 - 441.72) ms-0.2%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed58.96 ± (58.82 - 59.09) MB58.77 ± (58.62 - 58.91) MB-0.3%
runtime.dotnet.threads.count30 ± (29 - 30)30 ± (29 - 30)+0.0%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms190.77 ± (190.34 - 191.19) ms190.16 ± (189.82 - 190.50) ms-0.3%
process.time_to_main_ms69.69 ± (69.48 - 69.89) ms69.59 ± (69.40 - 69.78) ms-0.1%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.75 ± (11.73 - 11.78) MB11.75 ± (11.72 - 11.77) MB-0.0%
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)+0.3%✅⬆️
.NET 8 - Bailout
process.internal_duration_ms190.54 ± (190.07 - 191.00) ms189.99 ± (189.69 - 190.30) ms-0.3%
process.time_to_main_ms70.73 ± (70.57 - 70.90) ms70.50 ± (70.41 - 70.60) ms-0.3%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.84 ± (11.81 - 11.87) MB11.86 ± (11.81 - 11.91) MB+0.2%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)-2.1%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms367.29 ± (365.95 - 368.62) ms365.25 ± (363.64 - 366.86) ms-0.6%
process.time_to_main_ms428.88 ± (428.34 - 429.41) ms430.12 ± (429.54 - 430.71) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed47.87 ± (47.84 - 47.91) MB47.90 ± (47.87 - 47.93) MB+0.1%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+0.5%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (75ms)  : 70, 81
    master - mean (75ms)  : 69, 81

    section Bailout
    This PR (7866) - mean (79ms)  : 74, 84
    master - mean (79ms)  : 74, 83

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (1,071ms)  : 996, 1146
    master - mean (1,057ms)  : 980, 1134

Loading
FakeDbCommand (.NET Core 3.1)
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (118ms)  : 109, 126
    master - mean (117ms)  : 110, 125

    section Bailout
    This PR (7866) - mean (120ms)  : 113, 127
    master - mean (119ms)  : 111, 126

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (758ms)  : 722, 795
    master - mean (755ms)  : 711, 799

Loading
FakeDbCommand (.NET 6)
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (106ms)  : 100, 113
    master - mean (104ms)  : 98, 110

    section Bailout
    This PR (7866) - mean (108ms)  : 99, 116
    master - mean (106ms)  : 100, 112

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (708ms)  : 666, 749
    master - mean (706ms)  : 661, 751

Loading
FakeDbCommand (.NET 8)
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (105ms)  : 98, 112
    master - mean (102ms)  : 95, 109

    section Bailout
    This PR (7866) - mean (105ms)  : 100, 110
    master - mean (106ms)  : 99, 112

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (680ms)  : 652, 708
    master - mean (678ms)  : 653, 703

Loading
HttpMessageHandler (.NET Framework 4.8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (194ms)  : 189, 199
    master - mean (193ms)  : 189, 197

    section Bailout
    This PR (7866) - mean (196ms)  : 194, 198
    master - mean (197ms)  : 194, 200

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (1,117ms)  : 1039, 1196
    master - mean (1,110ms)  : 1038, 1183

Loading
HttpMessageHandler (.NET Core 3.1)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (277ms)  : 272, 282
    master - mean (277ms)  : 272, 281

    section Bailout
    This PR (7866) - mean (278ms)  : 273, 284
    master - mean (278ms)  : 274, 282

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (909ms)  : 853, 965
    master - mean (909ms)  : 848, 969

Loading
HttpMessageHandler (.NET 6)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (270ms)  : 265, 276
    master - mean (270ms)  : 266, 275

    section Bailout
    This PR (7866) - mean (270ms)  : 267, 273
    master - mean (270ms)  : 267, 273

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (885ms)  : 829, 942
    master - mean (889ms)  : 832, 946

Loading
HttpMessageHandler (.NET 8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7866) - mean (270ms)  : 265, 275
    master - mean (270ms)  : 263, 278

    section Bailout
    This PR (7866) - mean (270ms)  : 266, 274
    master - mean (271ms)  : 264, 278

    section CallTarget+Inlining+NGEN
    This PR (7866) - mean (827ms)  : 808, 845
    master - mean (828ms)  : 809, 847

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:debugger area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants