Package Arrow conversion helpers (row-based path + ANDData bridge); bump 0.51.1

[atalyaalon]

Why

Services like the author-disambiguation service (ADS) must produce the Arrow IPC artifacts that Clusterer.predict_from_arrow_paths consumes. The conversion helpers lived in scripts/arrow_conversion_helpers.py, but scripts/ is not part of the installed s2and package (packages.find include = ["s2and*"]), so a pip-installed service cannot import them.

What

• New packaged module s2and/arrow_service_io.py.
• write_feature_block_arrow_from_rows — build the FeatureBlock Arrow tables directly from service-native row dicts, no ANDData. This is the preferred production path (per review): the caller groups pairwise cluster-seeds into require (signature_id -> cluster_id) + disallow pairs and passes rows in.
• write_feature_block_arrow_from_anddata (+ feature_block_from_anddata) — compatibility bridge for callers that already hold an ANDData. scripts/arrow_conversion_helpers.py re-exports the public helpers.
• Bump 0.51.0 -> 0.51.1 (synced across VERSION / pyproject / Cargo via the git hooks).
• Test fixtures derive the supported Rust extension version from the runtime guard (runtime.min_supported_rust_extension_version_string()) instead of hardcoding it, so a version bump can't leave a stale fixture tripping the lockstep version check.

Verification

• tests/test_feature_block.py: 78 passed (row path + from_anddata path).
• Row helper round-trips Arrow tables + batch indexes from real service rows (validated locally; Rust predict is exercised in CI).

Stacked on codex/anddata-featureblock-plan. Draft — paired with the ADS arrow-ingest integration (direct-from-rows).

Columnar row→Arrow build (last commit)

The earlier write_feature_block_arrow_from_rows built one FeatureBlockSignature/Paper/PaperAuthor dataclass per row, then called FeatureBlock.to_arrow_tables(). On large blocks that per-row object materialization dominates the conversion (profile on l_li, 251k papers / 7.1M paper_authors: ~54s of dataclass construction).

The last commit makes the row path columnar: it builds the Arrow columns directly from the request rows — no per-row dataclass objects — for a ~4.5× speedup on l_li (~54s → ~11s), with peak RSS held flat (validator consumes a one-shot generator rather than materializing the position keys).

To avoid a second, drifting copy of the Arrow schema, both entry points now funnel through one shared schema builder and one shared validator:

• build_feature_block_arrow_tables(...) — single source of truth for the Arrow schema. Both FeatureBlock.to_arrow_tables (columns extracted from row objects) and the columnar service path feed columns through it, so the pa.array(..., type=...) definitions live in exactly one place.
• validate_feature_block_columns(...) — single source of truth for the structural FeatureBlock invariants (unique signature_id / paper_id, no duplicate (paper_id, position), query_signature_ids subset, cluster-seed require checks, non-empty ids). Both FeatureBlock.__post_init__ and the columnar path call it, so the columnar path rejects exactly the same malformed inputs the dataclass path always did (including empty-string ids, which the columnar path would otherwise have let through).

[sergeyf]

Wait, can't we call rust directly without ever going through ANDData?

[sergeyf]

I think this packaging move makes sense as a compatibility bridge for callers that already have an ANDData object in memory, but I want to call out the ideal ADS production path so we do not accidentally make ANDData the service boundary.

If ADS can stay on the Arrow/Rust path, it should produce the feature-block Arrow artifacts directly from its service-native rows and call Clusterer.predict_from_arrow_paths(...) without constructing ANDData:

pred_clusters, pred_distance_matrices = clusterer.predict_from_arrow_paths(
    block_dict,
    {
        signatures: .../signatures.arrow,
        papers: .../papers.arrow,
        paper_authors: .../paper_authors.arrow,
        signatures_batch_index: .../signatures.signatures_batch_index.bin,
        papers_batch_index: .../papers.papers_batch_index.bin,
        paper_authors_batch_index: .../paper_authors.paper_authors_batch_index.bin,
        # Required when the loaded model uses embeddings:
        specter: .../specter2.arrow,
        specter_batch_index: .../specter2.specter_batch_index.bin,
        # Required when the loaded model uses name-count features:
        name_counts_index: .../name_counts_index,
        # Optional for seeded/incremental requests:
        # cluster_seeds: .../cluster_seeds.arrow,
        # cluster_seed_disallows: .../cluster_seed_disallows.arrow,
    },
    load_name_counts=True,
    name_tuples=filtered,
)

So I would frame s2and.arrow_service_io.write_feature_block_arrow_from_anddata(...) as the installable escape hatch for legacy/compatibility services that still own ANDData, not as the preferred ADS runtime contract. The preferred contract is still service rows -> Arrow IPC + batch indexes -> predict_from_arrow_paths.