Skip to content

Expose include_hidden and include_type_tag on all export methods #1813

@dmpetrov

Description

@dmpetrov

Problem

Whether internal columns appear in output is only controllable on to_pandas/show. The other exporters (to_parquet, to_csv, to_records, to_json/jsonl, to_columnar_data_with_names, to_database) have no toggle.

Two different internal concerns are involved and need separate flags:

Flag Controls Nature Default
include_hidden File-style hidden fields (source/etag/version/is_latest/last_modified/location) real fields, essential to re-open/identify a File True
include_type_tag the _type_tag discriminator for Optional[DataModel] (and future Union) synthetic; only needed for the rare present-all-None round-trip edge False

They're decoupled because dropping File-hidden makes a File unreadable (presentation/essential), while keeping _type_tag only buys exactness on a rare edge (fidelity).

What to build

  • Add include_hidden (default True) and include_type_tag (default False) to all exporters, threaded into _leaf_values / get_headers_with_length.
  • include_hidden=False → drop File hidden fields (clean; the re-read File can't be opened).
  • include_type_tag=Truekeep _type_tag for a lossless round-trip (default False drops it; read falls back to the all-none heuristic, lossy only on a present Optional[DataModel] whose every field is None).
  • show() keeps include_hidden=False.

Per-format notes

  • JSON/JSONL: never carry a literal _type_tag (absence is structural null); include_type_tag is a no-op there. include_hidden still controls File fields (via model_dump(exclude=…)).
  • to_database: already omits _type_tag (one-way export); include_type_tag=False is its current behavior.

Backward-compat

Defaults (include_hidden=True, include_type_tag=False, show()=False) = exactly today's behavior. include_hidden=False and include_type_tag=True are purely additive opt-ins.

Out of scope: Optional[list/dict] on ClickHouse; the broader nullability strategy (#1815).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions