Self-supervised Pretraining In Neutrino Experiments. The repo produces pretrained backbones (encoder checkpoints) that downstream supervised benchmarks fine-tune. First pretrain: CURTAIN (occupancy / light-front forecast); built so new SSL methods are small plugins.
A run is: Data → Backbone → Pretrain(head + targets + loss), wired by an
Engine, named by a Config. The only thing you write to add a method is a
pretrain/ plugin — data, backbone, and engine are reused unchanged.
| block | responsibility |
|---|---|
data/ |
geometry asset + sensor-key lookup, FeatureScaler scaling, datamodule (reader- & selection-agnostic) |
backbones/ |
encoder interface (swappable; graphnet-free; DeepIce impl in integrations/spine_graphnet/) |
pretrain/ |
pretrain-task interface + curtain/ (sampler, head, objectives, task, val callbacks) |
ssl_module.py |
Lightning module; optimizer/scheduler injected as factories (transfer export in utils.py) |
configs/ + train.py |
Hydra groups compose a run (examples/train_curtain.py); fit() assembles |
Backbone.encode(batch) -> EncodedEvent(tokens, token_mask, cls)— swap architectures without touching pretrain/engine.PretrainTask—make_sample(CPU: mask/target),collate,build_head,loss. A task carries a list of weightedObjectives, each an abstract class owning its own head (build_head) andloss, over one sample.
sampler.py computes query_dt for every event; v1 scores
[OccupancyObjective], v2 adds DtObjective(weight). Each objective owns its head (off a shared
per-query embedding) and its loss + masking, so v2 is task/objectives=v2
— no forked
model/dataset/train files.
- Core has zero graphnet dependency.
src/spine/imports no graphnet; the reference DeepIce backbone and the readers live inexamples/as the graphnet integration. The backbone drivesDeepIce's submodules directly over the collated jagged nested tensor (to_padded_tensorin the backbone) — noarray_to_sequence, no torch_geometric — so theBackbonecontract is a plain nested tensor, model-agnostic (padding or varlen backbones both project from it). The finetuning bench loadsckpt["backbone"]into graphnet DeepIce, so the exported state_dict must stay compatible — keep DeepIce, or vendor a state-dict-identical encoder later (examples/deepice_backbone.pyTODO). - Data layer. Pretrain needs raw pulses (the Δt reference is
charge-weighted-mean-time on raw values), so standardization runs at the model
boundary after the split, not in the source. LMDB is welcome for speed but
as the low-level read utilities behind the read
Dataset(raw pulses; identity detector; no truth/labels) — not the fullLMDBDataset(which drags the graph pipeline back into the data layer). SPINE ships no reader — it consumes any Dataset meeting the RawPulseDataset contract (canonical statement:spine/data/datamodule.py), and readers must emit per-pulsesensor_keyidentity. Profile the loader before converting. - Batch = jagged nested tensor.
collateemits the pulses and the query fields astorch.nestedjagged NJTs (no padding baked in); each backbone projects — DeepIce callsto_padded_tensor, a varlen backbone would readoffsets. Verified to survive the multiprocessing DataLoader and Lightning's GPU batch transfer (pin_memory +move_data_to_device) on torch 2.6. This is NJT-as-transport only: NJT-in-attention stays off-limits on this torch (the rel-spacetime kernel bugs). - Selections are the caller's. SPINE owns no split:
fittakes train/val readers, and the caller supplies disjoint train/val event lists (and keeps the test set off-limits). A leak inflates every pretrained-vs-scratch number, so that hygiene lives with whoever builds the selection — filtered withsampler.can_always_split, the predicate stating exactly which events the sampler is guaranteed to handle. - Sensor identity is data-carried. Readers emit an integer
sensor_keyper pulse (multi-level string/module/PMT IDs composed by the reader; single-PMT detectors use 1 for the missing level);load_geometry(sensor_key=...)resolves keys to rows through a per-row array verified offline at asset-build time. Reconstructing identity from coordinates is not supported: float matching collapses near-duplicate positions and near-tie times at the margins, exactly where training then crashes. - Transfer invariant: extend the encoder, don't wrap it. The checkpoint's
backboneentry is the backbone's own state_dict and must load directly into the downstream encoder — a wrapper would prefix every key and silently break that load. HenceDeepIceBackbone(DeepIce, Backbone)subclasses the encoder. - Epoch-global metrics are callbacks. Rank-based metrics (the occupancy AUCs) cannot flow through per-batch log averaging; callbacks cache per batch and reduce once per epoch, keeping task and engine interfaces closed.
- Optimizer/scheduler are injected factories.
SSLModuletakesparams -> Optimizer/Optimizer -> schedulercallables (Hydra_partial_configs); swapping optimization is a config override, not a code change. - DDP correctness baked in.
sync_dist=Trueon the val metric (else ReduceLROnPlateau desyncs replica LRs); the data path is fail-loud — the caller pre-filters andmake_sampleraises on any event it cannot split (never returns None, never skips), so a batch is never empty (empty batch → a rank skips its step → NCCL deadlock); and multi-GPU runs should setTORCH_NCCL_ENABLE_MONITORING=0— PyTorch's NCCL heartbeat monitor can abort a healthy run on a GIL-starved false positive during heavy compute.
utils.py's TransferCheckpoint writes {backbone, full_state, config, step, val_loss} on
best val (rank-0 only). Downstream loads ckpt["backbone"]. Finetuning/eval
stays in the existing bench — this repo emits encoders, nothing more.
New folder under pretrain/, point a task/<name>.yaml _target_ at the
new PretrainTask:
- MAE:
make_samplemasks pulses; head = decoder; loss = reconstruct. - Contrastive:
make_sample= two views; head = projection oncls; loss = NT-Xent. Data, backbone, engine unchanged.
- Wire end-to-end and reproduce v1 (occupancy) ✓ — validated against the reference pretraining (best val loss within noise, AUCs within 7e-4).
- Reproduce v2 by config (
task/objectives=v2exists; revalidation open). - Config system (hydra) ✓ — profile loader remains. Deferred: other backbones, other pretraining tasks, multi-detector, in-repo eval.
- graphnet DeepIce behind the interface vs vendor a standalone encoder.
- Repo scope: pretraining-only (recommended) vs pull in a finetuning harness.
- Multi-detector: single-detector MVP (geometry-parametrized) vs multi from day one.