SHM create race: attachers mmap a size-0 segment ("cannot mmap an empty file")

[mustafab0]

Symptom: consumers crash attaching to a SHM segment with ValueError: cannot mmap an empty file (seen on real runs, July sprint).

Race: stdlib SharedMemory(create=True, size=N) does shm_open then ftruncate. An attacher that opens in that window sees size 0 and mmaps nothing. _open_shm_with_retry (ipc_factory.py:49) only catches FileNotFoundError, so it never covered this. Affected paths are the name-derived symmetric ones: _create_or_open (:145) attach arm, plus direct SharedMemory(create=True, ...) sites (~:174).

Fix: explicit readiness, readers now wait until the segment actually has a size, with a 10s cap and a clear error if it never shows up. No sleep ladder, no blind retries.

Blast radius: shared transport infra — all robots, sim modules, cameras, every SHM pubsub user.

[TomCC7]

Solution:

1. Default to zenoh
2. Configure zenoh shm
3. Remove the inhouse shm transport :)

[mustafab0]

Repro (deterministic — injects a delay into the shm_open→ftruncate window; the real window is microseconds so it will not reproduce on demand without it). Save as repro_shm_race.py at the repo root, python repro_shm_race.py. Exits 0 when it reproduces.

"""Repro for the SHM create race: attachers mmap a size-0 segment.

  python repro_shm_race.py            # deterministic (delay injected in the window)
  python repro_shm_race.py --natural  # no injection; races on real timing, may need luck

stdlib SharedMemory(create=True, size=N) does shm_open() THEN ftruncate(). An
attacher that opens between them sees a 0-byte file and mmap() fails.
"""

import os
import sys
import threading
import time
from multiprocessing.shared_memory import SharedMemory

from dimos.protocol.pubsub.shm.ipc_factory import _open_shm_with_retry

NAME = f"dimos_race_{os.getpid()}"
SIZE = 1 << 20
NATURAL = "--natural" in sys.argv

if not NATURAL:
    _real_ftruncate = os.ftruncate

    def _slow_ftruncate(fd, length):  # widen the window to make it deterministic
        time.sleep(0.05)
        return _real_ftruncate(fd, length)

    os.ftruncate = _slow_ftruncate

hits = {"value_error": 0, "not_found": 0, "ok": 0, "retry_helper": None}
stop = threading.Event()


def attacher():
    while not stop.is_set():
        try:
            SharedMemory(name=NAME).close()
            hits["ok"] += 1
        except FileNotFoundError:
            hits["not_found"] += 1
        except ValueError as e:
            hits["value_error"] += 1
            print(f"  ValueError: {e}")
            # the backoff helper is supposed to be the mitigation -- is it?
            if hits["retry_helper"] is None:
                try:
                    _open_shm_with_retry(NAME).close()
                    hits["retry_helper"] = "survived"
                except Exception as e2:
                    hits["retry_helper"] = f"{type(e2).__name__}: {e2}"
            stop.set()


t = threading.Thread(target=attacher, daemon=True)
t.start()
time.sleep(0.01)

shm = SharedMemory(create=True, size=SIZE, name=NAME)
stop.set()
t.join(timeout=2)
shm.close()
shm.unlink()

print(f"\nattach attempts: ok={hits['ok']} FileNotFoundError={hits['not_found']} ValueError={hits['value_error']}")
print(f"_open_shm_with_retry on the same window: {hits['retry_helper']}")
sys.exit(0 if hits["value_error"] else 1)

Output on main:

  ValueError: cannot mmap an empty file

attach attempts: ok=0 FileNotFoundError=2540 ValueError=1
_open_shm_with_retry on the same window: ValueError: cannot mmap an empty file

Confirms finding (b): the backoff helper propagates the ValueError instead of retrying it, so it never covered this race. --natural (no injection) runs the same loop on real timing and normally does not hit it, which is why the regression test needs the injected window.

[mustafab0]

Solution:

Default to zenoh
Configure zenoh shm

This alone does not work because mujoco_shm has the same bug and that does not even check what transport is running. It is working with whatever SHM protocol we have