Skip to content

A few fixes to gc-stats and a cram test for validate-gc-stats - #167

Open
klakplok wants to merge 5 commits into
tarides:mainfrom
klakplok:fix-gc-stats-ring-reuse
Open

klakplok wants to merge 5 commits into
tarides:mainfrom
klakplok:fix-gc-stats-ring-reuse

Conversation

@klakplok

Copy link
Copy Markdown

While writing #166, I read the JSON and FXT traces of a multicore program (to exercise the FXT thread id allocation).

I also read the gc-stats output to cross check things, and detected a few issues :

  • the per domain GC overhead was giving 2500% on my test sample code (should be < 100% as its not really an overhead but a share) ;
  • the domain stats and domain_alloc_stats tables were inconsistent (the second was missing a domain).

While investigating I also found that the validate-gc-stats was not exercised in the test base, and was showing the first issue above, but also two others. One was a false detection, for which a fix is included in this branch (it made any execution of validate-gc-stats that doesn't trigger a >10s GC pause fail).

In this PR

This PR includes 2 fixes to gc_stats and 2 fixes to its checker, and a cram test to serve as regression test for these fixes, and to have at least one test exercizing validate-gc-stats,

See the commit messages for detailed explanations.

Not in this PR

Even after the fixes, validate-gc-stats still fails on the test sample, complaining that "0 <= promoted_pct <= 100: 115.636415 ∉ [0., 100.]". It says that a domain promoted more words to the major heap than it allocated in its own minor heap. I think the checker is fine, no typo here, but the assumption could be wrong : I believe a domain can promote values outside of its own minor heap if they are reachable from its roots. The documentation for EV_C_MINOR_PROMOTED in the OCaml manual seems to say otherwise but it is very terse so it may just be a leap.

The stats and traces call domain what is actually a ring id. If the monitored program spawns domains sequentially, they will end up reusing the same ring id, and end up in the same slot in gc-stats tables, and in the same track in perfetto.
We could decide to have domain ids independant from ring ids, by having a ring id -> domain id table updated at EV_DOMAIN_SPAWN time (we cannot reconstruct the original ids because for some reason, OCaml sends the pid of the process as the value of EV_DOMAIN_SPAWN instead of a domain id). I think this is somehow related to #124. It's also not so simple if we miss these lifecycle events.

The reconciliation of domain indexed tables fix is not tested in the cram test because I its easy to write in jq but I didn't know if jq would work in all CIs. It was tested manually though.

The gc_stats checker has some checks for long GC pauses (outliers).
These checks are done even if there wasn't a long GC pause,
in which case the latency records are at their initial values
of 0 and mean nothing.

The gc_stats results were thus wrongly flagged by the checker.

This patch just skips the tests if there was no long GC pause.
The JSON format still contains the useless zeroes, I think this is
fine and does not motivate a version change.
@klakplok
klakplok marked this pull request as draft November 18, 2026 17:55
The computation was
  "cumulative gc time for all domains sharing ring id X"
  /
  "running time of the last domain who used ring id X".
On a program that launches domains sequentially in a loop, the stats
were thus wrong, and could even quickly get above 100% GC cpu usage.

This was visible in the gc_stats output (textual and JSON).
This was also detected by the `validate-gc-stats` tool, but the tool
was never called in tests.

This patch introduces a new table to store cumulative running time for
each ring.

IMO it may not be the right long term solution, as it continues to mix
the notions of domain and ring id, but at least the number is
meaningful now (proportion of GC time for all domains that shared the
given ring id).
The issue from the previous patch showed up only on the
  "gc_time <= wall_time"
check because the per domain
  "0 <= gc_overhead <= 100"
had a typo.
The gc_stats output has tables "domain_stats" and "domain_alloc_stats".
The second is filtered to drop non allocating records.

There was two issues with the filtering:
  1. the filter was on major words, excluding domains that allocate
     short lived objects from the stats ;
  2. the filter was renumbering the fields, so identifiers in the two
     tables were inconsistent.

This patches uses minor words as a discriminant for the filter, and
restores the identifiers to always keep the ring id.
This test includes a regression test for the previous fix patches
 - invalid gc-stats with reused ring ids
 - validate-gc-stats false detection of bad outlyer computations

The test uses grep instead of expecting full validate-gc-stats success,
because some checks still fail at this point.
@klakplok
klakplok force-pushed the fix-gc-stats-ring-reuse branch from 6f222b8 to 169be19 Compare November 18, 2026 18:30
@klakplok
klakplok marked this pull request as ready for review November 18, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant