Assets-controller instrumentation is climbing daily under six new, unsuppressed names — 10%+ of volume and rising

[MajorLift]

Parent Epic: #43370 - Re-architect assets-controller AssetsDataSourceTiming / AssetsUpdatePipeline instrumentation to be quota-safe before re-enabling traces
Blocks: #43410 - Sentry Quota Breach — Extension Telemetry (May 2026 – ongoing) — see "Why this blocks the epic" below
Related: #43211 - Assets Controller Sentry Instrumentation exceeding quota
File: app/scripts/lib/sentry-traces-sampler.ts (build-time per-name rates) and shared/lib/sentry-remote-rates.ts (remote-flag override layer) — unreleased, see "Deployment state" below
Priority: P0 | Size: M | Hours: ~1 for the code change itself; the real cost is release timing, not engineering hours — unreleased, see "Deployment state" below

---

Deployment state of the suppression mechanism

The suppression this ticket concerns lives in app/scripts/lib/sentry-traces-sampler.ts — DEFAULT_TRANSACTION_SAMPLE_RATES = { AssetsDataSourceTiming: 0, AssetsUpdatePipeline: 0 }, a frozen build-time constant — with a genuine LaunchDarkly-backed remote-flag layer alongside it in shared/lib/sentry-remote-rates.ts (sentry.transactionSampleRates) that can push new per-name rates to already-installed clients without a new build, once the code that reads it has shipped.

Neither file has shipped to any release currently in the field:

File	v13.40.0	v13.41.0	v13.42.0	v13.43.0
app/scripts/lib/sentry-traces-sampler.ts	404	404	404	404
shared/lib/sentry-remote-rates.ts	404	404	404	404

Both exist only on main (commit 563d83e, 2026-08-17). The near-zero AssetsDataSourceTiming/AssetsUpdatePipeline reading in current production is therefore not this sampler suppressing anything — that code isn't running on any installed client yet. The old call sites simply stopped being invoked when the six new names replaced them. Fixing the ramp below requires the same release-then-adoption path as every other fix in this incident, not a bypass of it — there is no live mechanism to flip today.

Problem

#43211 (the original quota-breach ticket) suppressed two transaction names — AssetsDataSourceTiming and AssetsUpdatePipeline — via a DEFAULT_TRANSACTION_SAMPLE_RATES override of 0 (unreleased — see "Deployment state" below; the original names read near-zero today because their call sites were replaced, not because this sampler is running). The code has since started emitting the same instrumentation area under six different names, none of which are in that override map. This is not a stable side effect — it is a live ramp, measured daily and still climbing on the last complete day measured. Live in Sentry — daily breakdown, Aug 11–18:

Combined volume, six new Assets* names, project `metamask` (273505), production
2026-08-11        989
2026-08-12     59,943
2026-08-13    195,181
2026-08-14    481,067
2026-08-15  1,414,469
2026-08-16  2,899,994   (last complete day)
2026-08-17    870,239   (partial day, in progress at measurement time)

Direct query against the May 19–23 window (not a top-N cutoff artifact — the query returns zero rows) confirms these six names emitted nothing then. Aug 11→12 (989 → 59,943, 60.6x) is a launch artifact — zero to nonzero — not a rate, and is excluded below. The four subsequent day-over-day multiples are 3.26x, 2.46x, 2.94x, 2.05x (geomean ~2.64x/day) — roughly tripling day over day. The most recent transition (2.05x) is the smallest of the four; four points is too few to call that a deceleration, but it's noted rather than smoothed over.

Severity in quota terms (corrected 2026-08-17 — see note). On the clean post-filter window (15–16 Aug) these six names run 2,160,568/day extrapolated, which converts to roughly 17.0M/month billed — about 57% of the metamask project's 30M/month allocation, or 17% of the org-wide 100M.

Correction. An earlier revision of this ticket stated "~87M/month, or ~87% of the entire 100M monthly transaction quota". That figure took an extrapolated count() from Sentry's spans dataset (which reports a sampled estimate, not billed events) and compared it against a billed quota. Those extrapolated counts run roughly 3.82× the billed figure on these days, so the real figure is ~23% of the org quota, not 87%. The conclusion that this is the largest single instrumentation item is unchanged; the magnitude was overstated about fourfold.

Root cause identified (2026-08-17)

Established in the incident channel with the owning team.

• Change: MetaMask/core#9672 — fix: nest AssetsController Sentry sub spans under cold-start parents, merged 2026-07-29, in packages/assets-controller.
• Gate: the assets-unify-state LaunchDarkly flag (project metamask-client-config-api-extension).
• Release: shipped in 13.43.0, independently confirmed by the owning team — matching the measurement below, where 99.99% of this volume carries the 13.43.0 release tag.

The change was intended to reduce Sentry volume, not increase it. Its stated purpose is to avoid rate limits by nesting pipeline timings as subspans and emitting them only on the first unlock-session fetch. The observed outcome is the opposite: six new transaction families at ~17M/month billed.

Working hypothesis for why the intent inverted — stated as a hypothesis, not a conclusion, and worth confirming before any redesign. packages/assets-controller/src/utils/trace.ts documents its contract as "Pass parentContext to nest as a subspan", so nesting is conditional on that context resolving to a live parent span. In the extension's background/service-worker process a cross-process trace context cannot resolve to a live parent, and the call falls through to a continuation path that produces a root span — and a root span is a billed transaction. If that is what is happening here, every intended "subspan" is instead a new billed transaction, which would explain both the six new names and the volume. This is the same failure mode already documented for background RPC calls elsewhere in this incident.

If confirmed, the fix is not to remove the instrumentation but to make the parent resolvable in the background process — the nesting design is sound, the parenting is what fails.

Mitigation is available today via the product flag

The assets-unify-state flag gates this instrumentation and can be switched off without a client release. The owning team has offered to disable it. That is a materially better lever than either option previously named here:

• it is not the unreleased sentry.transactionSampleRates sampler (404 on every shipped tag through 13.43.0, so unavailable);
• it does not require the blunt release inbound filter, which is all-or-nothing per release and would discard ~65% of production telemetry.

Recommended sequence: disable the flag to stop the bleeding, then confirm the parenting hypothesis above and re-enable behind a corrected implementation, so the diagnostic value the instrumentation was built for is not permanently lost.

A residual question remains for releases already in the field: disabling the flag stops new emission, but clients on affected versions may continue emitting until they update. Sentry-side release inbound filters for prior releases are the complementary lever there, and that decision is separate from the flag.

The ramp is release adoption, not unbounded instrumentation growth (2026-08-17)

99.99% of this volume comes from a single release, metamask-extension@13.43.0 (5,315,933 of 5,316,224 over 15–17 Aug; the remaining 291 are 13.44.0). The apparent exponential curve is that release rolling out, not instrumentation growing without bound — the six names hold at a roughly constant share of 13.43.0's own traffic while its adoption climbs:

	Aug 11	Aug 13	Aug 15	Aug 16
13.43.0 total transactions	3,292	1,877,345	10,126,008	13,727,187
the six Assets* names	989	195,181	1,414,469	2,899,994
Assets as share of 13.43.0	30.0%	10.4%	14.0%	21.1%

This bounds the problem, and the bound is above where it sits today. On Aug 16, 13.43.0 was 64.9% of all production transaction volume (13,727,187 of 21,140,534), with ~35% of the population still to migrate onto it. Holding the observed ratio, full adoption projects to roughly 4.47M/day extrapolated ≈ 26M/month billed — about 88% of the metamask project's 30M/month allocation, or 26% of the org-wide 100M.

Two things temper that. It assumes the ratio holds as adoption completes, which is an extrapolation rather than a measurement. And it is a pre-fix figure: PR #45581 removes the 58% of this family that is phantom-parent — a span that names a parent Sentry never received, so it is billed as a root volume, so the post-fix plateau is nearer 11M/month billed, ~36% of the project allocation.

So the growth does stop, but it stops well above the contract, not below it. Any projection that assumes continued day-over-day compounding overstates the rate; any projection that assumes today's flat rate understates the ceiling.

This makes one same-day mitigation available that the Solution section previously lacked

Because the volume is release-scoped, a Sentry release inbound filter on 13.43.0 would stop it server-side, today, with no client release and no dependence on the unreleased sampler. This is the same mechanism that resolved the August AggregatedBalanceSelector incident, and it is version-independent by construction.

The cost is that release inbound filters are all-or-nothing per release: filtering 13.43.0 discards all telemetry from it, currently 64.9% of production volume, not just these six names. That is a serious observability loss and should not be done casually — but it is a real lever that exists now, and the trade should be made deliberately rather than left unnamed.

By share of spike-excluded transaction volume, 15–17 Aug: 10.18%, up from 0% — breakdown scoped to exactly these six names, live in Sentry (the 10.18% is these six as a share of all transactions; the all-transaction denominator is visible here for cross-checking the ratio, not for finding the six rows in it).

15-17 Aug, share of transaction volume
  5.63%  AssetsUpdateEnrichment
  1.79%  AssetsStateSize
  1.64%  AssetsFetchPipeline
  1.08%  AssetsBackgroundFetch
  0.02%  AssetsFullFetch
  (+ AssetsControllerFirstInitFetch, smaller)

Volume is returning to the same controller that caused the original incident, under names the suppression override does not reach, at a rate that has not stopped climbing, and nobody made a decision about it as volume — it happened as a side effect of instrumentation work, most likely related to #43370 (this ticket's parent, the assets-controller re-architecture).

More than half of this volume is the shared tracing defect, not the instrumentation

Splitting the six names by whether they carry a parent id (15–16 Aug):

	Extrapolated/day	~billed M/mo	% of 30M project
All six, today	2,160,568	16.97	56.6%
Carrying a parent id — removed by #7569	1,257,348	9.87	32.9%
True roots — remain after that fix	903,220	7.09	23.6%

58% of this family is parent-declaring volume, well above the ~44.6% estate average. So the shared-layer fix in PR #45581 cuts this family from ~17M to ~7M/month billed on its own, without changing a line of assets-controller code.

That reorders the remedies. Switching off assets-unify-state remains the correct immediate lever, because it acts today and the nesting fix has not shipped. But the durable fix is the tracing-layer one, and the instrumentation may well be affordable once it lands — which is an argument for re-measuring before deciding to remove or permanently downsample this instrumentation.

Why this blocks the epic

#43410 (the Sentry Quota Breach epic)'s closing claim is that the incident's drivers are mitigated. The #43211 (the original quota-breach ticket) fix was a name-keyed suppression — unreleased as of this writing, see "Deployment state" above — and the same controller is now emitting an unrelated-by-name but same-origin volume that has grown from zero to 10%+ of transactions in six days with no sign of leveling off — and at ~17M/month billed it is ~57% of the metamask project's own allocation. That is a live, worsening condition in the exact subsystem the incident was about — closer to the original incident's shape than any other open item, and the one candidate in this update with no declining trend to point to.

Why this may not be a defect at the code level

The instrumentation change itself could be entirely legitimate: coarse, high-cardinality timers replaced by several narrower, better-scoped ones is a reasonable re-architecture outcome, and may well be better instrumentation than what #43211 (the original quota-breach ticket) suppressed. What is not legitimate by default is 10%+ of transaction volume, still climbing, going live with no sample-rate decision and no suppression coverage — regardless of whether the underlying instrumentation is good. The sampler being built to suppress this class of problem (see "Deployment state" above) does not include these names, and even once it ships, someone still has to add them.

Why it matters beyond this one controller

The same name-keyed weakness applies to detection, not only suppression: the per-name anomaly alerting proposed in #43411 watches names too. A family that renames or re-splits itself — as this one did, and as the service-worker.js rename at 13.32.0 also did — passes underneath any purely name-based rule on both sides: suppression and alerting.

---

Solution

• Available today, no release required: apply a Sentry release inbound filter on 13.43.0 to stop the volume server-side (see the release-adoption section above). All-or-nothing per release — it discards all 13.43.0 telemetry, ~65% of production volume — so this is a deliberate trade, not a free win. It is nonetheless the only lever that acts before a release ships, and it is the mechanism that resolved the August incident.
• First decision, today, not code: given the severity (see above) and that the correct fix needs a release to reach anyone, whoever owns release scheduling should decide now whether this qualifies for an expedited/hotfix train rather than the normal cadence — that decision, not the code change itself, is the actual time-critical path.
• Add the six names (or a prefix/attribute match covering the Assets* family) to DEFAULT_TRANSACTION_SAMPLE_RATES in sentry-traces-sampler.ts alongside the existing two — small, mechanical, but only takes effect once shipped and adopted, same as every other fix in this incident (see 13.41.0→13.42.0 in #45574 for how long that takes in practice).
• Once sentry-remote-rates.ts itself has shipped and reached meaningful adoption, the same six names can be pushed via the sentry.transactionSampleRates remote flag instead — instant, no further release needed. Not available for this incident, but worth prioritizing getting that groundwork shipped precisely so the next one doesn't need a release either.
• Identify what is driving the ramp — a rollout percentage increasing, a new call site, a retry loop — since "still climbing with no plateau" is itself informative about the cause
• Measure whether the six names, combined, represent a volume increase, a like-for-like replacement, or a reduction versus the original pre-suppression AssetsDataSourceTiming/AssetsUpdatePipeline baseline, once the ramp has stabilized enough to measure
• Make an explicit call: keep at current granularity, consolidate, or extend the sample-rate override to cover the new names — as a decision, not a default
• Prefer a match that survives a rename (prefix or attribute-based) over enumerating exact names again, so the same gap doesn't reopen under the next rename

---

Acceptance Criteria

• Release-train decision made and stated explicitly: expedited or normal cadence, with the reasoning
• DEFAULT_TRANSACTION_SAMPLE_RATES updated to include the six names (or a surviving-rename match), shipped
• Cause of the day-over-day ramp identified
• Combined volume of the six new Assets* names measured against the pre-suppression AssetsDataSourceTiming/AssetsUpdatePipeline baseline
• Explicit decision recorded: acceptable as-is / needs sampling / needs consolidation
• If sampling is applied, verified to survive a future name change (prefix/attribute match, not another exact-name list)
• sentry-remote-rates.ts shipping and reaching adoption tracked as its own follow-up — it's the thing that makes the next version of this problem not need a release at all
• Finding referenced from #43370 (the assets-controller re-architecture, parent) so it isn't lost inside the wider re-architecture scope

---

Labels

team-extension-platform, area-sentry

---

Dependencies

Related:
#43370 (assets-controller re-architecture, parent) ·
#43410 (Sentry Quota Breach epic) ·
#43211 (original quota-breach ticket) ·
#43411 (per-name anomaly alerting) ·
#45574 (Provider Create Accounts v2 fan-out — release-attrition precedent cited above)

[metamask-ci]

Triage Agent -- Classification

Severity: Sev2-normal (confidence: 0.88) ✅

This is a monitoring/observability gap where ~10% of Sentry transaction volume has returned under new instrumentation names that bypass the existing suppression list. It's not causing user-facing breakage or data loss, but it represents uncontrolled telemetry volume that could impact Sentry quotas/costs and masks the effectiveness of a previous mitigation. It's a process/infrastructure issue rather than a user-facing bug.

Size: size-S (confidence: 0.82) ⚠️ Advisory

The fix involves adding the six new transaction names to the suppression/sampling rate configuration, and ideally implementing a more robust suppression mechanism (e.g., prefix-based matching). The core change is likely a configuration update in the Sentry sampling rates file, plus potentially a small refactor to make suppression more resilient to renames.
Not auto-applied -- confidence below threshold.

Team: team-assets ✅

Matched via CODEOWNERS: @MetaMask/metamask-assets

---

Root Cause Analysis

Affected component: Sentry sampling/tracing configuration and AssetsController instrumentation

The original incident caused by high Sentry transaction volume from AssetsController was mitigated by adding 'AssetsDataSourceTiming' and 'AssetsUpdatePipeline' to the transactionSampleRates suppression list (likely in the Sentry remote rates or wrapper-sampling configuration). Subsequently, the AssetsController instrumentation was refactored/split into six new trace names (AssetsUpdateEnrichment, AssetsStateSize, AssetsFetchPipeline, AssetsBackgroundFetch, AssetsFullFetch, AssetsControllerFirstInitFetch). Since the suppression mechanism is name-keyed (exact string match), these new names bypass the suppression entirely. The root cause is twofold: (1) the suppression list was not updated when instrumentation names changed, and (2) the suppression mechanism itself is fragile — it relies on exact name matching rather than a more robust pattern like prefix matching (e.g., 'Assets*').

Evidence:

• Issue explicitly names six new Sentry transaction names that are not on the suppression list
• KB context confirms sentry-remote-rates.ts manages remote sampling rate configuration and wrapper-sampling.ts controls sampling rates
• The issue states the original mitigation for AssetsDataSourceTiming and AssetsUpdatePipeline is working, confirming name-keyed suppression
• ~10% volume return indicates the new instrumentation is actively emitting at non-trivial rates

Suggested Fix

Confidence: medium: likely but needs investigation - the exact file paths for the suppression config are not in the KB context, but the KB confirms sentry-remote-rates.ts and wrapper-sampling.ts are the relevant files for sampling rate management

Add the six new transaction names to the Sentry transaction sample rates suppression list, and consider refactoring the suppression mechanism to use prefix-based matching for more resilient coverage against future renames.

Target files: app/scripts/lib/sentry-remote-rates.ts, app/scripts/lib/wrapper-sampling.ts
Target functions: transactionSampleRates configuration, sampling rate lookup/matching logic

Approach:

1. Locate the transactionSampleRates configuration (likely in sentry-remote-rates.ts or a related config file) where 'AssetsDataSourceTiming' and 'AssetsUpdatePipeline' are suppressed (set to 0 or very low rate). 2. Add the six new names to the same suppression list: 'AssetsUpdateEnrichment', 'AssetsStateSize', 'AssetsFetchPipeline', 'AssetsBackgroundFetch', 'AssetsFullFetch', 'AssetsControllerFirstInitFetch'. 3. For a more robust long-term fix, refactor the sampling rate lookup in wrapper-sampling.ts to support prefix/glob matching (e.g., any transaction name starting with 'Assets' gets the suppressed rate), so future renames or splits are automatically covered. 4. Add a test that verifies any transaction name matching the 'Assets*' pattern gets the expected sample rate. 5. Consider adding an alerting mechanism or CI check that detects new Sentry transaction names not covered by the sampling configuration.

---

Confidence threshold: 0.85. Labels marked ✅ were auto-applied. Labels marked ⚠️ are advisory only (below threshold).
To re-triage, remove the ta-bot-triage label and re-apply ta-needs-triage.
⚠️ Some dimensions could not be confidently classified. An engineer should review this issue.