Skip to content

Custom FXT fuchsia library focused on write performance annd associated fixes - #166

Open
klakplok wants to merge 11 commits into
tarides:mainfrom
klakplok:adhoc-fxt-lib
Open

klakplok wants to merge 11 commits into
tarides:mainfrom
klakplok:adhoc-fxt-lib

Conversation

@klakplok

@klakplok klakplok commented Nov 17, 2026 •

Copy link
Copy Markdown

This PR extracts the Fuchsia library part of PR #89 by @tmcgilchrist and adds some related tests and fixes.
The patch was initially written in a performance improvement PR, but the current motivation is to make olly more standalone by dropping the external dependencies on trace and trace-fuchsia.

Edits

  • Added a patch "Fix FXT library on big endian architectures" to make CI greenify the CI.

Short summary of the patches

  • Change emit from records to fields.
  • Added emit_counter to the Format module type and both backends

two preliminary patches to cherry-pick the main patch with less changes

  • Add a cram test of the FXT trace backend

A cram test that generates a trace with the external FXT library mocking the FXT features and format used by olly, printed in a normalized form, to serve as validation for the new implem / regression tests.

  • Custom FXT fuchsia library focused on write performance.

The main patch by @tmcgilchrist

  • Fix the FXT initialization record header

The second record header output by the custom FXT lib (to set the timestamp unit) was not of the expected form, and the produced traces did not display properly in Perfetto. This patch fixes the record header.

  • Cleaner handling of max domains
  • Bump [max_domains] to 4096

The original PR was bumping the max_domain supported by olly to meet the OCaml value increased by 5.3 (128 --> 4096), since the new FXT library makes the change easier.
This PR extracts this change in two patches : one to fail more gracefully in case the monitored program has an unsupported value, and another that just bumps the value.

  • Translate the FXT externals back to OCaml

This patch is OPTIONAL, the new lib was using C stubs to avoid int64 boxing, and I was curious to see if OCaml was really that bad at unboxing. Turns out OCaml can producted non allocating code (see APPENDIX), but it's a case of "it works on my computer" (tested on 5.5.2, with and without flambds).

  • Sync FXT trace with JSON trace output

This is not strictly related, but I think it's worth putting in that batch of patches. The FXT trace for the same program is not as readable as the JSON one in Perfetto because some tracks are squashed. This patch aligns the FXT choices with the JSON ones. Maybe there was other reasons leading to this choice.

  • Add doc strings to fxt library

One of the original patches I forgot and

Remarks

  • I did not cherry-pick b8cc0a (Remove runtime_events_tools_bare) because the no bare version still declares a dependency on hdr_histogram, which pulls more system dependencies than fuchsia was. I wonder if the right move isn't to move fuchsia into the bare version instead ? Or maybe it's because the dependencies weren't exact ? I didn't investigate.
    - the FXT library defines external put_raw_64_le : Bytes.t -> int -> int64 -> unit = "%caml_bytes_set64u" which is misleading, i think this intrinsic uses native endianness, also we could rename it "unsafe", it's more accurate than "raw".
  • I reviewed the FTX lib which seems safe and sufficient for our purposes
  • I have doubts about the zero allocation in steady state claim, for instance emit and counter call the following function, and I doubt its result allocation can be optimized.
let thr_ref_lookup threads v =
  match Hashtbl.find_opt threads.thr_index (v.pid, v.tid) with
  | Some i -> Thr_ref i
  | None -> Thr_inline { pid = v.pid; tid = v.tid }

AI disclosure

The FXT dump tool was written by Claude, then fully reviewed against the official format doc at fuchsia.dev, tweaked and commented by me.
I also used Claude's help for the dune file updates and the Unix commands in the cram tests.
All the english is my own broken english.

APPENDIX

	.align	16
	.globl	camlFxt_buf.put_event_header_509
camlFxt_buf.put_event_header_509:
	.loc	1	48	30
	.cfi_startproc
	.loc	1	60	32
.L121:
	.loc	1	60	32
	sarq	$1, %r8
	.loc	1	60	20
	salq	$48, %r8
	.loc	1	59	32
	sarq	$1, %rcx
	.loc	1	59	20
	salq	$24, %rcx
	.loc	1	58	17
	orq	%r8, %rcx
	.loc	1	57	29
	sarq	$1, %rdx
	.loc	1	57	17
	salq	$20, %rdx
	.loc	1	56	14
	orq	%rcx, %rdx
	.loc	1	55	26
	sarq	$1, %rsi
	.loc	1	55	14
	salq	$16, %rsi
	.loc	1	54	11
	orq	%rdx, %rsi
	.loc	1	53	23
	sarq	$1, %rdi
	.loc	1	53	11
	salq	$4, %rdi
	.loc	1	52	8
	orq	%rsi, %rdi
	.loc	1	51	6
	orq	$4, %rdi
	.loc	1	49	2
	sarq	$1, %rbx
	.loc	1	49	2
	movq	%rdi, (%rax,%rbx)
	movl	$1, %eax
	ret
	.cfi_endproc
	.type camlFxt_buf.put_event_header_509,@function
	.size camlFxt_buf.put_event_header_509,. - camlFxt_buf.put_event_header_509
	.text
	.align	16
	.globl	camlFxt_buf.put_arg_header_i32_573
camlFxt_buf.put_arg_header_i32_573:
	.loc	1	71	32
	.cfi_startproc
	.loc	1	79	26
.L122:
	.loc	1	79	26
	sarq	$1, %rdx
	.loc	1	79	14
	salq	$32, %rdx
	.loc	1	78	26
	sarq	$1, %rsi
	.loc	1	78	14
	salq	$16, %rsi
	.loc	1	77	11
	orq	%rdx, %rsi
	.loc	1	76	23
	sarq	$1, %rdi
	.loc	1	76	11
	salq	$4, %rdi
	.loc	1	75	8
	orq	%rsi, %rdi
	.loc	1	74	6
	orq	$1, %rdi
	.loc	1	72	2
	sarq	$1, %rbx
	.loc	1	72	2
	movq	%rdi, (%rax,%rbx)
	movl	$1, %eax
	ret
	.cfi_endproc
	.type camlFxt_buf.put_arg_header_i32_573,@function
	.size camlFxt_buf.put_arg_header_i32_573,. - camlFxt_buf.put_arg_header_i32_573
	.text
	.align	16
	.globl	camlFxt_buf.put_arg_header_i64_580
camlFxt_buf.put_arg_header_i64_580:
	.loc	1	88	32
	.cfi_startproc
	.loc	1	94	23
.L123:
	.loc	1	94	23
	sarq	$1, %rsi
	.loc	1	94	11
	salq	$16, %rsi
	.loc	1	93	23
	sarq	$1, %rdi
	.loc	1	93	11
	salq	$4, %rdi
	.loc	1	92	8
	orq	%rsi, %rdi
	.loc	1	91	6
	orq	$3, %rdi
	.loc	1	89	2
	sarq	$1, %rbx
	.loc	1	89	2
	movq	%rdi, (%rax,%rbx)
	movl	$1, %eax
	ret
	.cfi_endproc

tmcgilchrist and others added 10 commits November 17, 2026 17:42
Changed emit from trace -> Event.t -> unit to trace -> ~ring_id:int ->
~ts:int64 -> ~name:string -> ~kind:Event.kind -> unit, passing fields
directly instead of boxing them into a record on every event. This
should avoid one Event.t allocation per event. Event.t type is
preserved since kind is still needed as an extensible type.
Provides a direct path for counter events that takes ~value:int
instead of ~kind:(Counter value).
This is a preliminary patch for a library change to come, to ensure
that the emitted traces have the same information, and that the new
library works properly with the kind of traces that olly emits.

The test calls Olly_format_fuchsia directly with every kind of event
olly emits, and decodes the resulting trace with a small FXT reader
(test/dump_fxt_trace.ml). The FXT reader normalizes the trace to be
agnostic to the library's table allocation policy (each event is fully
printed with referentes resolved).

The resulting trace is committed as a cram test.
Key design choices:
  - C stubs (fxt_put_event_header, fxt_put_arg_header_i32/i64) pack multi-field headers into int64
  words using C bit manipulation — zero Int64 boxing
  - String interning (32K table) — repeated phase names like "major", "minor" are emitted once as
  string records, then referenced by index. Makes traces ~50% smaller and avoids redundant string
  writes
  - Thread interning (256 table) — domain thread refs registered once, referenced by index
  - Single Bytes.t buffer (64KB) flushed to out_channel — no Buf_chain, no locks, no pool (olly is
  single-threaded)
  - %caml_bytes_set64u compiler intrinsic for timestamp writes — single instruction, no allocation
Defines a global [max_domains] constant, set to the previously value
of 128 that was hardcoded in two places (gc-stats, trace backends).

Olly now fails earlier, at events file loading instead of on the
first domain index over the limit encountered.
This value is consistent with the new hard limit of OCaml since 5.3,
and now possible with the new FXT lib.

The footprint of olly is increased, mostly in the tables allocated at
loadctime by gc-stats (domain_times, domain_gc_times,
domain_minor_words, domain_promoted_words, domain_major_words) and by
the FXT backend, accounting for an extra ~500 KB, and at final display
time (print_percentiles), with a few MB of short-lived allocations for
the text and JSON reports. The final print allocations could be dropped
using a high watermark of used domain indices.
Tested in both plain and flambda ocaml 5.5.2, the ocaml assembly is
simple and does not allocate. OCaml appears to have good enough
unboxing for these patterns (simple expressions including calls to
of_int as arg to the %caml_bytes_set64u primitive).
In perfetto, events have a pid and a tid. Spans are associated to a
pair (tid, pid), and counters are associated pid, disregarding thetid.

The JSON  trace backend outputs (tid = pid = ring_id) on all events,
resulting in a pid track section for each ring_id, each containing a
track for this ring_id's spans and a track for each counter emitted
from this ring_id.

The FXT backend outputs (pid = 0, tid = ring_id). resulting in a single
pid section, with one track per thread for the spans, and a single
track for each counter where the same counters from different ring_ids
are merged. Domain local counters get all smashed on a single track,
resulting in a meaningless plot (for intance, major_heap_words looks
like a square signal, at each slice, the last domain to emit its
counter event "wins" the race for the value of the plot until the next
slice).

This patch gives the behaviour of JSON to FXT.
let emit_counter trace ~ring_id ~ts ~name ~value =
let t_ref = trace.doms.(ring_id) and time_ns = ts in
Trace.Event.Counter.encode trace.buf ~t_ref ~name ~time_ns
~args:[ ("v", A_int value) ]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the remark in the PR summary that the changes might not lead to zero allocations: Initially I thought, here's about 3 allocations ([], (), A_int) that are traded for one (Counter).

However, the call is changed to drop the new allocations, two commits later.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. If we need to enforce zero alloc, we should have a principled approach to this, for instance with a test that preheats the lib and then does a series of calls, checking the GC allocated words before and after.

Comment thread lib/fxt/fxt_buf.ml

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!

Comment thread lib/fxt/fxt_buf.mli Outdated
[@@noalloc]

external put_event_header :
Bytes.t -> int -> int -> int -> int -> int -> int -> unit

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Labels would be great here, to distinguish all the ints. Ditto for the next 2 functions.

Comment thread test/cram/fuchsia.t

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

The library is packing headers in 64-bit words and then writing them
with intrinsic "%caml_bytes_set64u". This intrinsic write in platform
native byte order, not little endian as expected by the format.

This patch swaps the byte order using intrinsic "%bswap_int64" before
the write (as the stdlib does for the safe version).

Should make the FXT trace check tests pass on s390x.
The generated code for header writing functions is still zero-alloc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants