Problem
Whether internal columns appear in output is only controllable on to_pandas/show. The other exporters (to_parquet, to_csv, to_records, to_json/jsonl, to_columnar_data_with_names, to_database) have no toggle.
Two different internal concerns are involved and need separate flags:
| Flag |
Controls |
Nature |
Default |
include_hidden |
File-style hidden fields (source/etag/version/is_latest/last_modified/location) |
real fields, essential to re-open/identify a File |
True |
include_type_tag |
the _type_tag discriminator for Optional[DataModel] (and future Union) |
synthetic; only needed for the rare present-all-None round-trip edge |
False |
They're decoupled because dropping File-hidden makes a File unreadable (presentation/essential), while keeping _type_tag only buys exactness on a rare edge (fidelity).
What to build
- Add
include_hidden (default True) and include_type_tag (default False) to all exporters, threaded into _leaf_values / get_headers_with_length.
include_hidden=False → drop File hidden fields (clean; the re-read File can't be opened).
include_type_tag=True → keep _type_tag for a lossless round-trip (default False drops it; read falls back to the all-none heuristic, lossy only on a present Optional[DataModel] whose every field is None).
show() keeps include_hidden=False.
Per-format notes
- JSON/JSONL: never carry a literal
_type_tag (absence is structural null); include_type_tag is a no-op there. include_hidden still controls File fields (via model_dump(exclude=…)).
to_database: already omits _type_tag (one-way export); include_type_tag=False is its current behavior.
Backward-compat
Defaults (include_hidden=True, include_type_tag=False, show()=False) = exactly today's behavior. include_hidden=False and include_type_tag=True are purely additive opt-ins.
Out of scope: Optional[list/dict] on ClickHouse; the broader nullability strategy (#1815).
Problem
Whether internal columns appear in output is only controllable on
to_pandas/show. The other exporters (to_parquet,to_csv,to_records,to_json/jsonl,to_columnar_data_with_names,to_database) have no toggle.Two different internal concerns are involved and need separate flags:
include_hiddenFile-style hidden fields (source/etag/version/is_latest/last_modified/location)Trueinclude_type_tag_type_tagdiscriminator forOptional[DataModel](and futureUnion)FalseThey're decoupled because dropping File-hidden makes a File unreadable (presentation/essential), while keeping
_type_tagonly buys exactness on a rare edge (fidelity).What to build
include_hidden(defaultTrue) andinclude_type_tag(defaultFalse) to all exporters, threaded into_leaf_values/get_headers_with_length.include_hidden=False→ drop File hidden fields (clean; the re-read File can't be opened).include_type_tag=True→ keep_type_tagfor a lossless round-trip (defaultFalsedrops it; read falls back to the all-none heuristic, lossy only on a presentOptional[DataModel]whose every field isNone).show()keepsinclude_hidden=False.Per-format notes
_type_tag(absence is structuralnull);include_type_tagis a no-op there.include_hiddenstill controls File fields (viamodel_dump(exclude=…)).to_database: already omits_type_tag(one-way export);include_type_tag=Falseis its current behavior.Backward-compat
Defaults (
include_hidden=True,include_type_tag=False,show()=False) = exactly today's behavior.include_hidden=Falseandinclude_type_tag=Trueare purely additive opt-ins.Out of scope:
Optional[list/dict]on ClickHouse; the broader nullability strategy (#1815).