Skip to content

fix: reject nulls at load time in every schema validator#26

Merged
michael-denyer merged 1 commit into
mainfrom
fix/strict-null-checks-at-load
Jul 21, 2026
Merged

fix: reject nulls at load time in every schema validator#26
michael-denyer merged 1 commit into
mainfrom
fix/strict-null-checks-at-load

Conversation

@michael-denyer

Copy link
Copy Markdown
Owner

Closes the one follow-up left open by #25.

validate_gwas_dataframe already required non-null positions and p-values.
validate_eqtl_dataframe, validate_finemapping_dataframe, and
validate_genes_dataframe did not, so a file with a blank p_value, pip,
start, or end passed load validation and silently lost those rows later,
when plot-time intake filtered them with a warning the user was unlikely to
connect back to the file.

That gap was an artefact of the old hand-rolled implementation rather than a
decision. #25 preserved it deliberately so the refactor stayed
behaviour-neutral; this is the labelled behaviour change that closes it.

Change

All four schema validators now reject nulls in the numeric columns they already
range-check, naming the offending column:

Validator Columns newly checked
validate_eqtl_dataframe pos, p_value
validate_finemapping_dataframe pos, pip
validate_genes_dataframe start, end

validate_gwas_dataframe is unchanged; it was already strict.

What is deliberately not changed

Plot-time intake stays lenient and still filters nulls with a warning
(prepare_pvalue_data, prepare_eqtl_for_plotting). That split is the point: a
frame reaching the plotter may have been assembled by the caller rather than
read from a file, so the strict gate belongs at the file boundary. Recorded in
CONTEXT.md next to the other load-bearing intake distinctions, and in the
contract test's module docstring.

Verification

Five new contract cases were red before the change and green after, and two
cases that asserted the old tolerance are flipped to rejections.

End to end through a real loader, not just the validator:

clean SuSiE file: 2 rows loaded
null pip file: rejected -> Fine-mapping validation failed:
  - Column 'pip' has 1 null values

Full suite: 1195 passed, 1 skipped. ruff check and ruff format clean.

Release note

2.0.0 is merged but not yet tagged, so this lands in the existing 2.0.0
CHANGELOG entry rather than a patch bump.

validate_gwas_dataframe already required non-null positions and p-values.
validate_eqtl_dataframe, validate_finemapping_dataframe, and
validate_genes_dataframe did not, so a file with a blank p_value, pip, start, or
end passed load validation and silently lost those rows later, when plot-time
intake filtered them with a warning the user was unlikely to connect to the
file. The gap was an artefact of the old hand-rolled implementation rather than
a decision; the 2.0 refactor preserved it deliberately to stay
behaviour-neutral, and this is the follow-up that closes it.

All four validators now reject nulls in the numeric columns they range-check,
naming the offending column. Verified end to end through load_susie: a clean
file loads, one with a blank pip fails with "Column 'pip' has 1 null values".

Plot-time intake is untouched and still filters nulls with a warning. That split
is the point: a frame reaching the plotter may have been assembled by the caller
rather than read from a file. Recorded in CONTEXT.md alongside the other
load-bearing intake distinctions.
@michael-denyer
michael-denyer merged commit f4aa549 into main Jul 21, 2026
6 checks passed
@michael-denyer
michael-denyer deleted the fix/strict-null-checks-at-load branch July 21, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant