`tensogram-szip` decoder disagrees with libaec on real GRIB2 CCSDS payloads (off-by-one in preprocessed samples)

[johncarmack1984]

Hi, and thanks for publishing this — a pure-Rust AEC implementation with a libaec parity suite is exactly what downstream GRIB2 tooling needs.

I was wiring tensogram-szip into the grib crate as an optional CCSDS backend (it already has libaec-sys and one other pure-Rust backend) and validated it against libaec on real ECMWF data before proposing it. It does not currently match.

Symptom

Decoding GRIB2 data representation template 5.42 payloads, a small number of samples differ from libaec by one or two quantization steps, in either direction.

On the grib crate's own ECMWF fixture (ECMWF_REALTIME_OPER_FC_89):

• 69 of 1,038,240 samples differ (~0.007%)
• error distribution (tensogram − libaec): {-2: 2, -1: 24, +1: 41, +2: 1, +3: 1}
• affected samples are scattered across 15 of 253 RSI blocks
• no differing sample sits at offset 0 within its 32-sample block — i.e. never at a block's reference sample

Parameters for that payload: bits_per_sample=16, block_size=32, rsi=128, flags=14 (AEC_DATA_3BYTE | AEC_DATA_MSB | AEC_DATA_PREPROCESS).

That the reference sample of each block is always correct, while scattered later samples in the block drift by ±1, suggests the issue is in the preprocessor's inverse delta/mapping step rather than in the entropy decoding itself.

Reproduced on live operational data

Independently reproduced against five fields from a live ECMWF open-data operational forecast (20260720/00z/ifs/0p25/oper, step 12), decoding each through the same code path with libaec and with tensogram-szip and comparing outputs:

field	points differing	max deviation
2t	8,188 / 1,038,240 (0.79%)	0.03 K
msl	12,271 / 1,038,240 (1.18%)	4 Pa
10u	5 / 1,038,240 (0.0005%)	0.016 m/s
100u	0	—
ssrd	0	—

The magnitudes are physically negligible — at or below the packing's own quantization step — so this would not matter for most consumers. It matters for mine because the output feeds published verification statistics that need to be reproducible against the reference implementation.

Possible reason it survived the existing suite

tests/libaec_parity.rs and tests/ffi_crosscheck.rs look like they compress with libaec (or with tensogram) and round-trip, over generated inputs. If the generated inputs do not reach the same preprocessor states that real ECMWF GRIB2 payloads do, a divergence like this can pass. Adding a fixture of real template-5.42 section-7 bytes decoded against libaec would close that gap.

Environment

• tensogram-szip 0.23.0
• compared against libaec-sys 0.1.3 (libaec 1.1.4)
• aarch64-apple-darwin, rustc stable

Happy to share the exact payloads and the comparison harness, and happy to test a fix — I have the integration branch ready and would like to land it once decoding matches.