Capture SP task dumps in support bundles #8177

wfchandler · 2025-05-16T14:24:51Z

Update the support bundle collector to capture task dumps from the SPs.

wfchandler · 2025-05-20T20:49:40Z

Tested on Dublin:

# Create a task dump
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -a /tmp/build-gimlet-image-e.zip --ip fe80::aa40:25ff:fe04:604%gimlet15 dump --task control_plane_agent
humility: connecting to fe80::aa40:25ff:fe04:604%51
humility: using UDP dump agent
humility: pulled 10.12KB in 0 seconds
humility: dumping to "hubris.core.control_plane_agent.0"
humility: dumped 661.38KB in 0 seconds

# Confirm the dump is present on the SP
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -a /tmp/build-gimlet-image-e.zip --ip fe80::aa40:25ff:fe04:604%gimlet15 dump -l
humility: connecting to fe80::aa40:25ff:fe04:604%51
humility: using UDP dump agent
AREA TASK                  TIME       SIZE
   0 control_plane_agent   12942213   10368

# Contents of original task dump
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -d hubris.core.control_plane_agent.0 tasks
humility: attached to dump
system time = 12942213
ID TASK                       GEN PRI STATE    
19 control_plane_agent          0   7 recv, notif: usart-irq(irq37) socket timer

# Create a support bundle
root@oxz_switch0:/tmp# omdb -w nexus sb create
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:103::4]:12221
created support bundle: 39bff2f6-15c2-4f6e-93ba-c832380816e6

# Download the dehydrated task dump from the bundle
root@oxz_switch0:/tmp# omdb nexus sb get-file 39bff2f6-15c2-4f6e-93ba-c832380816e6 sp_task_dumps/sled_15/dump-0.zip > /tmp/dump-0.zip

# Rehydrate the dump from the archive
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless --archive /tmp/build-gimlet-image-e.zip hydrate dump-0.zip 
humility: read dehydrated crash dump
humility:   task index: 19
humility:   crash time: 12942213
humility:   archive id: [d5, 3a, 8f, a4, c9, b5, 6f, 91]
humility:   board:      gimlet-e
humility:   git commit: 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab
humility:   version:    1.0.37
humility:   2 memory regions:
humility:     0x24001108: 168 bytes
humility:     0x24030000: 32768 bytes
humility: dumping to "hubris.core.control_plane_agent.1"
humility: dumped 661.38KB in 0 seconds

# Confirm we have the same tasks
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -d hubris.core.control_plane_agent.1 tasks
humility: attached to dump
system time = 12942213
ID TASK                       GEN PRI STATE    
19 control_plane_agent          0   7 recv, notif: usart-irq(irq37) socket timer

# Rehydrated file matches the original
root@oxz_switch0:/tmp# shasum hubris.core.control_plane_agent.*
00a066ce8b911b09579648847650b096dd594831  hubris.core.control_plane_agent.0
00a066ce8b911b09579648847650b096dd594831  hubris.core.control_plane_agent.1

wfchandler · 2025-05-20T20:54:07Z

nexus/src/app/background/init.rs

@@ -505,6 +505,7 @@ impl BackgroundTasksInitializer {
            task_impl: Box::new(
                support_bundle_collector::SupportBundleCollector::new(
                    datastore.clone(),
+                    resolver.clone(),


Maybe this isn't the best approach, given the conversation in the control plane huddle today. On the other hand, we're very unlikely to be collecting bundles fast enough to have a real impact.

I'm fine to punt on this until we sort out the progenitor integration with qorb more holistically

smklein · 2025-05-20T21:00:31Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+        tokio::fs::create_dir_all(&sp_dumps_dir).await.with_context(|| {
+            format!("failed to create SP task dump directory {sp_dumps_dir}")
+        })?;
+        let sp_dumps_fut =


We should strongly consider modifying SupportBundleCollectionReport in nexus/types/src/internal_api/background.rs to identify whether or not collection of the SP dumps was successful or not.

Good call, added.

smklein · 2025-05-20T21:01:08Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+        let sp_dumps_dir = dir.path().join("sp_task_dumps");
+        tokio::fs::create_dir_all(&sp_dumps_dir).await.with_context(|| {
+            format!("failed to create SP task dump directory {sp_dumps_dir}")
+        })?;


This looks replicated from above? We should define this variable / create_dir_all in one spot, perhaps?

Whoops, forgot to remove this one when I moved the start above the sled tasks.

smklein · 2025-05-20T21:01:28Z

nexus/src/app/background/init.rs

@@ -505,6 +505,7 @@ impl BackgroundTasksInitializer {
            task_impl: Box::new(
                support_bundle_collector::SupportBundleCollector::new(
                    datastore.clone(),
+                    resolver.clone(),


I'm fine to punt on this until we sort out the progenitor integration with qorb more holistically

smklein · 2025-05-20T21:03:02Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+        .context("failed to get list of SPs from MGS")?
+        .into_inner();
+
+    let mut futures = futures::stream::iter(all_sps.into_iter())


I recommend checking out parallel-task-set, which we added relatively recently, to perform this task-saving operation in parallel.

(It should be quite similar to using buffer_unordered, but it'll actually spawn tokio tasks for each SP - but we can still set a limit on the maximum amount of parallelism)

Great, that's very nice. I went with the default parallelism since the requests are very light.

smklein · 2025-05-20T21:06:19Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+    sp_dumps_dir: &Utf8Path,
+) -> anyhow::Result<()> {
+    let dump_count = mgs_client
+        .sp_task_dump_count(sp.type_, sp.slot)


When / how are these task dumps deleted? Just trying to understand if there's a TOCTTOU issue between "get # of tasks" vs "iterate over them"

AFAIK they're never removed during standard operation. So we may miss a dump created after fetching the count, but would never have one removed out from under us.

Humility can clear them out with --initialize-dump-agent, and I think upgrades will as well.

smklein · 2025-05-22T18:08:18Z

nexus/src/app/background/tasks/support_bundle_collector.rs

@@ -593,6 +611,12 @@ impl BundleCollection {
            }
        }

+        if let Err(e) = sp_dumps_fut.await {


Most minor of nit; still seems weird this is "sandwiching" the sled-querying code. Why not immediately use sp_dumps_fut next to where it is declared?

My initial thinking was to allow both the sled and SP tasks to run concurrently, but the SP calls should be very quick so I don't think we're really getting any benefit. Moved the await up top as you've suggested.

Semantic merge conflict between #8177 and #8183. The test appears to pass locally.

wfchandler force-pushed the wc/bundle-sp-dumps branch from 7380a0e to a4b0acf Compare May 16, 2025 14:49

Capture SP task dumps in support bundles

dbb6669

Update the support bundle collector to capture task dumps from the SPs.

wfchandler force-pushed the wc/bundle-sp-dumps branch from a4b0acf to dbb6669 Compare May 18, 2025 23:05

wfchandler commented May 20, 2025

View reviewed changes

wfchandler marked this pull request as ready for review May 20, 2025 20:55

wfchandler requested review from papertigers and smklein and removed request for papertigers May 20, 2025 20:55

smklein reviewed May 20, 2025

View reviewed changes

wfchandler added 4 commits May 21, 2025 10:16

Merge branch 'main' into wc/bundle-sp-dumps

50457f7

Switch to ParallelTaskSet and show SPs in report

fc341e1

Assert sp dump dir is created

132e7bb

Fix name assertion

6b2892d

wfchandler requested a review from smklein May 21, 2025 23:33

smklein approved these changes May 22, 2025

View reviewed changes

Immediately await the SP dump task

7cb6043

wfchandler merged commit 068bf5d into main May 23, 2025
16 checks passed

wfchandler deleted the wc/bundle-sp-dumps branch May 23, 2025 14:40

sunshowers mentioned this pull request May 23, 2025

fix broken main #8209

Merged

sunshowers added a commit that referenced this pull request May 23, 2025

fix broken main (#8209)

710abbd

Semantic merge conflict between #8177 and #8183. The test appears to pass locally.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Capture SP task dumps in support bundles #8177

Capture SP task dumps in support bundles #8177

Uh oh!

wfchandler commented May 16, 2025

Uh oh!

wfchandler commented May 20, 2025

Uh oh!

wfchandler May 20, 2025 •

edited

Loading

Uh oh!

smklein May 20, 2025

Uh oh!

smklein May 20, 2025

Uh oh!

wfchandler May 21, 2025

Uh oh!

smklein May 20, 2025

Uh oh!

wfchandler May 21, 2025

Uh oh!

smklein May 20, 2025

Uh oh!

smklein May 20, 2025

Uh oh!

wfchandler May 21, 2025

Uh oh!

smklein May 20, 2025

Uh oh!

wfchandler May 21, 2025

Uh oh!

smklein May 22, 2025

Uh oh!

wfchandler May 22, 2025

Uh oh!

Uh oh!

Uh oh!

Capture SP task dumps in support bundles #8177

Capture SP task dumps in support bundles #8177

Uh oh!

Conversation

wfchandler commented May 16, 2025

Uh oh!

wfchandler commented May 20, 2025

Uh oh!

wfchandler May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wfchandler May 20, 2025 •

edited

Loading