[PoC] Remote captive Core service: stream ledgers to RPC over the network

[marwen-abid]

TL;DR

Prototype a standalone Go service that owns captive Core on its own hardware and streams ledgers to RPC over WebSocket or gRPC. Measure whether a network stream can feed ingest as fast as the local subprocess pipe does today. If it can, Core's RAM, CPU, and disk load leave the RPC box entirely, and one Core service can feed multiple RPC instances.

Why

Captive Core runs as a subprocess on the RPC box. It consumes RAM and writes to disk on the same hardware that has to ingest each ledger inside the Phase 3 budget (80 ms). Current benchmarks are replaying synthetic ledgers, this PoC is done in anticipation of the risk of core contending for hardware with RPC.

How RPC consumes Core today

The full-history path consumes Core through ledgerbackend.LedgerStream — one method, RawLedgers(ctx, Range, ...), yielding raw XDR bytes per ledger. The production implementation opens a local captive Core subprocess (cmd/stellar-rpc/internal/fullhistory/daemon.go:470, ledgerbackend.NewCaptiveCoreStream); the stream owns Core's lifecycle. This interface is the seam: a remote implementation of LedgerStream slots in without touching ingest.

The service

A Go server that owns captive Core:

• Starts and supervises captive Core on its own hardware with its own NVMe.
• Maintains a retention buffer of ~10,000 ledgers, written to disk.
• Streams new ledgers to every subscriber over WebSocket or gRPC. A subscriber asks for the tip, or for a specific ledger if it fell behind. Too far behind returns an error — with the retention window sized right this does not happen, since RPC v2 runs near the tip.

The assumption the PoC exists to test: the network roundtrip may not cost what it looks like. With a streaming transport the server writes bytes as Core produces the ledger, so the consumer does not wait for the whole ledger to materialize before it starts ingesting. If per-ledger delivery latency stays within the local pipe's envelope, the architecture works.

What it buys beyond latency

• Removes Core from RPC's RAM and CPU budget entirely — the OOM class of failure leaves the box.
• One Core service feeds multiple RPC instances; fewer Core instances lower the fleet cost.

Residual risk

The query path stays on the RPC box. Moving Core removes Core's resource pressure, but query load still competes with ingest — that side is the companion status-query PoC.

Done when

A prototype service streams ledgers to an RPC instance implementing LedgerStream over the network, a Phase 3-style run and the results support a go/no-go recommendation on the architecture.

Related

• Investigate and fix the issues surfaced by the O3 ingestion load test #800 — fixes to the existing ingest path (latency tail, ingest memory, RocksDB config); complementary, no code overlap.
• Run the Phase 3 benchmark campaign — ingest and reads at 600 ms blocks #874 — Phase 3 campaign; source of the ingest numbers and the run conditions to reproduce.
• Companion PoC: transaction-status query path (ledger hint / WebSocket push) — same Phase 3 gap, read side.
• Prior art: go-stellar-sdk #2841 (design), #4823/#4826 (removal rationale).

[marwen-abid]

PoC complete — recommendation: GO

The prototype is public at stellar-experimental/remote-core-poc.

What was built

• corestreamd runs captive stellar-core and streams each ledger’s raw XDR to subscribers over WebSocket. It keeps the last 10,000 ledgers on disk so reconnecting subscribers can replay recent history.
• remoteledger implements ledgerbackend.LedgerStream, the same Go interface that the RPC v2 ingestion loop uses to read ledgers. The ingestion code therefore handles remote and local ledger streams in the same way.
• benchrunner measures delivery latency, throughput, and inter-arrival time.
• The documented RPC changes are in docs/rpc-wiring.md. They add one configuration key and one small component, with no changes to ingestion itself.

How it was measured

The benchmark ran on two development machines on 2026-08-18.

• Server: corestreamd and captive stellar-core 27.1.1 (8612ca32) on a c6id.8xlarge, with retention stored on local NVMe.
• Client: benchrunner using remoteledger on an m6id.2xlarge, the epic’s “minimum requirement” hardware size.
• Real-ledger test: 1,001 live pubnet ledgers, sequences 63,999,663–64,000,663, averaging 2.13 MiB each.
• Synthetic-load test: 2.93 MiB ledgers produced every 600 ms, matching the epic’s Phase 3 load profile.
• Local baseline: captive stellar-core ran in the reading process on the same server machine and delivered the same 1,001 ledgers through a local pipe.
• Both machines had NTP-synchronized clocks.
• Delivery latency measures the time from when the server stamped a ledger as it left stellar-core until the client received it.

Results

Test	Delivery p50	Delivery p90	Delivery p99	Delivery max	Throughput and timing
Remote, 1,001 real pubnet ledgers	2.45 ms	3.03 ms	4.54 ms	11.37 ms	8.2 MiB/s; total wall time 4m22.1s
Remote, synthetic 2.93 MiB ledgers produced every 600 ms	3.24 ms	3.42 ms	3.64 ms	254.75 ms¹	Inter-arrival p50 604.7 ms; max 606.2 ms
Local pipe, same 1,001 ledgers on the server machine	—	—	—	—	8.2 MiB/s; total wall time 4m19.6s

¹ The 254.75 ms maximum occurred for one ledger out of 962 measured. Another 38 ledgers were replayed from the disk buffer and excluded because replayed ledgers do not carry a send timestamp.

What the results mean

• For real pubnet ledgers, remote delivery added 4.54 ms at p99. This is under 6% of the 80 ms per-ledger ingestion budget.
• The pod’s metrics document also sets a 100 ms ingestion-p99 target. The measured network cost is under 5% of that target.
• Remote and local delivery both reached 8.2 MiB/s.
• Remote delivery increased total wall time from 4m19.6s to 4m22.1s, about 1.0%.
• With 2.93 MiB ledgers produced every 600 ms, delivery latency was 3.64 ms at p99, while inter-arrival time was 604.7 ms at p50 and 606.2 ms at maximum.

Recommendation

GO. stellar-core can run on a separate machine and stream ledgers to RPC within the measured latency budget.

This moves stellar-core’s RAM, CPU, and disk usage off the RPC machine. One core service can also feed several RPC instances.

Production requirements

The repository lists the following production work, which was outside this PoC:

• Add TLS and authentication to the stream.
• Define how clients recover after falling behind the 10,000-ledger retention window. The client library already returns a typed error for this condition.
• Move stellar-core’s HTTP query endpoints to the core machine and update deployments that use them.

[marwen-abid]

Rerun with the heaviest real synthetic ledgers

We reran with the heaviest ledgers the benchmark datasets contain, in place of the uniform synthetic payloads above. This run streamed real LedgerCloseMeta XDR from the sac-6000 apply-load dataset — the heaviest profile in the epic's frozen dataset set — at the Phase 3 cadence of 600 ms.

How it was measured

• A dump tool extracted the raw XDR of 1,200 consecutive ledgers (sequences 10,002–11,201) from the sac-6000 frozen ledger pack.
• corestreamd received a --source file mode (repository) that replays a dumped ledger file at a fixed cadence. The payloads and their per-ledger sizes are exactly what the pack contains; the wire protocol is unchanged.
• Same two machines and roles as the first run: server on the c6id.8xlarge with the replay file on local NVMe, client on the m6id.2xlarge. Both machines had NTP-synchronized clocks.
• The client measured 1,001 ledgers (sequences 10,152–11,152). All 1,001 were delivered live with send timestamps; none were replayed from the retention buffer.

Ledger sizes

The decoded sac-6000 ledgers are near-constant in size: minimum 13.60 MiB, mean 14.49 MiB, maximum 14.49 MiB of raw XDR per ledger. This is 5× the 2.93 MiB synthetic leg of the first run, and 7.4× the dataset's compressed on-disk size — compressed pack sizes understate the XDR volume that ingestion actually receives. At 600 ms cadence this is a sustained wire rate of 23.6 MiB/s.

Results

Test	Delivery p50	Delivery p90	Delivery p99	Delivery max	Throughput and timing
Remote, 1,001 real sac-6000 ledgers (14.49 MiB mean) at 600 ms cadence	18.98 ms	19.34 ms	19.89 ms	35.27 ms	23.6 MiB/s; inter-arrival p50 615.1 ms, max 621.8 ms

The emitted cadence measured 615 ms rather than 600 ms: the file source paces each ledger as the 600 ms interval plus the ~15 ms read of a 14.5 MiB payload from disk. This does not affect the delivery metric, which measures from the server's send stamp to client receipt. The inter-arrival spread held within ±7 ms.

What the results mean

• Delivery p99 is 19.89 ms — 25% of the 80 ms per-ledger ingestion budget, and 20% of the pod's 100 ms ingestion-p99 target, on ledgers 5× heavier than the first run's load leg.
• Latency scaled linearly with payload size (5× the bytes, 5.5× the p99). This is wire serialization time, not queueing: the maximum was 35 ms, with no outliers.
• The network hop leaves more than 75% of the per-ledger budget for ingestion work at the heaviest load profile the benchmark datasets model.

[marwen-abid]

Run 3: chunked streaming on two 8x dev-boxes

Ran the two-box tests on two c6id.8xlarge instances using remote-core-poc PR #2 at 5b8d8af.

The boxes were in the same Availability Zone but were not in an EC2 placement group. The first received ledger in each test was excluded as warmup.

Test	What it measures	delivery p50	p99	max	n
Controlled workload	Repeatable 14.48 MiB synthetic ledgers, emitted over about 8 ms every 600 ms	16.60 ms	17.26 ms	17.36 ms	1,000
Controlled workload, reverse direction	Same test with the server and client roles exchanged	16.70 ms	17.33 ms	17.38 ms	1,000
Store-and-forward baseline	Previous implementation at 7d74d10, before chunked streaming	30.70 ms	31.18 ms	43.60 ms	1,000
Real Core	Real SAC-6000 LedgerCloseMeta XDR from stellar-core apply-load	16.54 ms	18.56 ms	18.56 ms	99

Real-Core result

The real-Core test is the most representative result. It runs a real stellar-core process and streams the XDR that Core writes to its metadata pipe.

Its median delivery time was 16.54 ms, and its worst warmup-excluded sample was 18.56 ms. This did not meet the 4 ms p99 target.

The sample is small: with 99 measured ledgers, p99 is also the single worst remaining sample. However, the median closely matches the 16.60 ms controlled result, which shows that both tests reached the same network limit.

Why the test missed the target

The transport uses one WebSocket, which means it uses one TCP flow.

EC2 limits one flow to approximately 5 Gbit/s when the instances are not in the same cluster placement group. A raw TCP test on these boxes confirmed this:

• One flow: 4.96 Gbit/s
• Four flows together: 12.26 Gbit/s

At 5 Gbit/s, a 14.48 MiB ledger takes about 24.3 ms to transfer. Chunked streaming starts sending before Core finishes producing the ledger, so approximately 7.5 ms of that transfer overlaps with Core’s output:

24.3 ms - 7.5 ms = 16.8 ms

This closely matches the measured median of 16.60 ms.

The four-flow result shows that the instances had enough total network capacity. The limit was the bandwidth available to the protocol’s single connection, not the total NIC capacity.

Checks against other causes

The reverse-direction test produced almost the same result: 16.70 ms instead of 16.60 ms. The difference indicates approximately 50 µs of combined clock and direction asymmetry, which is negligible relative to the measured latency.

The ENA bandwidth-allowance counters stayed at zero, so the instances were not being throttled at their aggregate network limit. The narrow difference between p50 and p99 also shows a stable network result rather than a large interrupt or scheduling tail.

What the PR improved

Although the test missed the 4 ms target, it confirms that chunked streaming works as designed:

• Chunked streaming reduced median delivery from 30.70 ms to 16.60 ms on the same connection, a 1.85× improvement over store-and-forward.
• No client fell back from a partial live delivery to the retention ring in approximately 2,100 measured ledger deliveries.
• Real Core had a median emission window of 7.28 ms, which confirms the 7.5 ms model used by the controlled test.
• The controlled and real-Core medians agreed within 0.1 ms.

The earlier network emulation allowed one connection to use the instances’ full 12.5 Gbit/s aggregate capacity. The real EC2 network limits one connection to approximately 5 Gbit/s outside a placement group. That incorrect assumption explains the difference between the emulated and two-box results.

[marwen-abid]

Run 4: compression on two 8x dev-boxes

Reran the two-box tests on the same c6id.8xlarge pair using remote-core-poc PR #3 at 530f67c4.

The boxes were not in an EC2 placement group. The WebSocket therefore remained limited to approximately 5 Gbit/s. The first received ledger in each test was excluded as warmup.

Test	What it measures	delivery p50	p90	p99	max	n
Controlled workload, compressed	Repeatable 14.48 MiB synthetic ledgers, emitted over about 8 ms every 600 ms	0.240 ms	0.321 ms	0.421 ms	0.588 ms	1,000
Controlled workload, compressed, reverse direction	Same test with the server and client roles exchanged	0.314 ms	0.371 ms	0.448 ms	0.569 ms	999
Controlled workload, uncompressed	Network control with compression disabled	16.51 ms	17.18 ms	17.31 ms	17.37 ms	1,000
Real Core, compressed	Real SAC-6000 LedgerCloseMeta XDR from stellar-core apply-load	1.56 ms	2.27 ms	3.99 ms	3.99 ms	98
Real Core, uncompressed	Same real-Core workload with compression disabled	16.42 ms	16.96 ms	18.64 ms	18.64 ms	99

Real-Core result

The real-Core test is the most representative result. It runs a real stellar-core process and streams the XDR that Core writes to its metadata pipe.

With compression enabled, median delivery fell from 16.42 ms to 1.56 ms. The worst warmup-excluded sample was 3.99 ms, which meets the 4 ms target.

The sample is small: with 98 measured ledgers, p99 is the single worst sample. The result passes, but it has almost no p99 margin and should not be read as a precise estimate of the production tail.

Why the controlled workload is faster

The controlled test isolates the transport under repeatable conditions. Its synthetic payload has the same size and approximate compressibility as the SAC-6000 workload, but it emits data evenly:

• The controlled source produced 58 chunks of approximately 256 KiB per ledger.
• Real Core wrote through the pipe in smaller bursts, producing about 234 chunks per ledger.
• The larger number of chunks adds compression-frame, WebSocket, decompression, scheduling, and assembly work.
• Real ledger XDR also has less uniform compression behavior than the calibrated synthetic payload.

This explains the difference between 0.240 ms for the controlled test and 1.56 ms for real Core. The controlled result proves that the transport mechanism has substantial margin under ideal emission. The real-Core result is the better reference for expected behavior.

What changed from Run 3

At 5 Gbit/s, an uncompressed 14.48 MiB ledger takes about 24.3 ms to transfer. Only about 8 ms can overlap with Core’s emission, which leaves approximately 16 ms after emission. This matches both uncompressed control results.

Compression reduces the ledger to approximately 2 MiB, or about 3.2 ms of transfer time. The transfer can then fit inside Core’s emission window. Most bytes are already at the client when Core finishes emitting the ledger.

The uncompressed controlled result reproduced Run 3: 17.31 ms p99, compared with 17.26 ms previously. This confirms that the network conditions did not materially change and that compression caused the improvement.

The reverse-direction controlled test also passed. Its results indicate approximately 37 µs of combined clock and direction asymmetry, which is small relative to the 4 ms target.

Client-side retention-ring fallbacks were zero in every test. The controlled tests also finished with zero raw compression fallbacks. During real Core’s initial setup burst, the server recorded 18 chunks sent without compression because all encoder workers were busy. The final server counter was not captured after Core exited, so the real-Core run does not have a complete end-of-run raw-fallback check.

[marwen-abid]

Write-up about the full Remote PoC experiment: https://marwen-abid.github.io/docs/rpc/remote-core-poc-results.html