A reliable UDP multicast transport in C++, modelled on NASDAQ's MoldUDP64. Data is published once to a multicast group; receivers detect gaps from sequence numbers and request retransmission over unicast. Recovery is NACK-driven, there is no congestion control, and the receive path is split across two threads connected by a lock-free SPSC queue.
[TODO]
TCP provides reliability as a bundle: in-order delivery, flow control, congestion control, and retransmission, all mandatory. Several parts of that bundle are actively harmful for market data on a controlled network.
Head-of-line blocking is the main one. If segment N is lost, TCP buffers N+1, N+2 and delivers nothing to the application until N is retransmitted. A message that arrived on time is withheld for a round trip because of an unrelated earlier loss. Congestion control is the second. It reads loss as a signal that the network is saturated and throttles accordingly, which is the wrong inference on a dedicated segment and adds latency variance.
MoldCast implements the parts that are needed (sequencing, gap detection, retransmission) and omits the rest. Ordering becomes the application's choice rather than a transport guarantee, so a lost message delays only itself.
sender receiver
------ --------
publish loop --- multicast ------------> RX thread
| |
retransmit ring SPSC queue
| |
NACK handler <--- unicast -------------- consume thread
(parse, gap detection,
NACK, delivery)
The sender publishes on a timer, stores every message it sends in a fixed-size retransmit ring keyed by sequence number, and answers NACKs by unicasting the requested messages back to the requester. It holds no per-receiver state; the requester's address is read from the incoming datagram.
The receiver splits the work. The RX thread does nothing but drain the socket and push raw packets into the SPSC queue. All protocol state (sequence tracking, the gap map, the NACK retry sweep) lives on the consume thread. Two reasons for this. The socket is always being drained, so the kernel receive buffer cannot overflow and cause loss that is invisible to the application. And only one thread touches protocol state, so no mutex is required anywhere in the gap logic. The SPSC queue is the only shared object, and it synchronises itself with acquire/release ordering on its head and tail indices.
A 20-byte header followed by length-prefixed message blocks:
offset size field type
0 10 session name ASCII, space-padded
10 8 sequence uint64, big-endian
18 2 message count uint16, big-endian
20 2 message length uint16, big-endian \ repeated
22 n message payload bytes / message_count times
Sequence numbers count messages, not packets. A packet carrying sequence 100 with a count of 5 contains messages 100 through 104; the next packet begins at 105. This is what makes a retransmission request for a single message unambiguous, and it allows a receiver to deliver message 103 while 102 is still outstanding.
Integer fields are big-endian per the MoldUDP64 specification. The session name is treated as text and is not byte-swapped. Payloads are capped so that one datagram fits in one Ethernet frame (1500 MTU less 20 bytes IP and 8 bytes UDP), avoiding IP fragmentation, where the loss of any single fragment discards the entire datagram.
A NACK is a bare header: session, first missing sequence, and count.
NACK rather than ACK. With one sender and many receivers, per-packet acknowledgement does not scale: the sender would be drowned by ACK traffic proportional to the number of subscribers. NACKs cost nothing when nothing is lost. The trade-off is that a receiver cannot report a gap it does not know exists, so trailing loss at the end of a burst is invisible without a periodic heartbeat.
Retransmissions are unicast, data is multicast. Multicasting a retransmission would re-deliver the data to every receiver that already has it and force all of them to run duplicate suppression. The requester's address is taken from the source of the NACK datagram, so the sender needs no registry of receivers.
No congestion control. Congestion control assumes that bandwidth is shared and that loss indicates saturation. Neither holds on a dedicated segment. Flow control is provided instead by the SPSC queue: if the consume thread falls behind, the queue fills and the RX thread discards packets and increments a counter. The loss is visible and countable, and the NACK machinery recovers it. Blocking instead would stall the drain and move the loss into the kernel buffer, where it is silent.
Bounded retransmit ring. The sender remembers a fixed window of recent messages. A request for a sequence that has aged out is answered with whatever remains contiguous; the receiver's retry budget then expires and the gap is reported as unrecoverable rather than stalling indefinitely.
64-bit sequence numbers starting at 1. No wraparound handling is required. Unlike TCP, initial sequence numbers are not randomised. Randomisation solves two problems that do not arise here: stale segments from a previous connection using the same four-tuple, and blind injection by an off-path attacker. A session identifier in the header distinguishes a restarted sender from a sequence gap.
Retransmission is handled by the sender rather than a separate re-request server.
There is no snapshot channel; a receiver that falls beyond the retransmit window reports unrecoverable loss instead of resynchronising.
Scope is a single LAN segment. Multicast TTL is 1, so IGMP is sufficient and no inter-router multicast routing (PIM) is involved.
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
./build/receiver 100000 # bind, join the group, deliver N messages
./build/sender 100000 # publish N packets
The receiver's NACK destination is currently a fixed address rather than the source of the data stream, so cross-machine operation requires configuration.
Gap filling handles a retransmission fully contained within a known gap; a partially overlapping range is not merged.
No heartbeat, so loss of the final messages in a stream is not detected.
Benchmarks are over loopback and therefore exclude NIC and wire latency; they measure protocol and pipeline overhead only.