Skip to content

fix: Materialize zone-aware nanosecond timestamps on DuckDB 2 - #7097

Open
Nikhi00718 wants to merge 2 commits into
feast-dev:masterfrom
Nikhi00718:fix/ibis-materialize-duckdb-ns-timestamps
Open

Nikhi00718 wants to merge 2 commits into
feast-dev:masterfrom
Nikhi00718:fix/ibis-materialize-duckdb-ns-timestamps

Conversation

@Nikhi00718

@Nikhi00718 Nikhi00718 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

feast materialize on the DuckDB offline store crashes on DuckDB 2.0 when a FileSource holds zone-aware nanosecond timestamps, which is what pandas writes by default for a datetime64[ns, UTC] column:

duckdb.ConversionException: Conversion Error: Unimplemented type for cast (TIMESTAMPTZ_NS -> TIMESTAMP) when casting from source column event_timestamp

The duckdb extra is ibis-framework[duckdb]>=10.0.0, and ibis 12 only requires duckdb>=0.10.3, so a fresh install today gets DuckDB 2.0.0, released 2026-10-21. Feast's CI pins 1.5.6, which is why CI doesn't see this. get_historical_features keeps working, so the failure first shows up at materialization.

pull_latest_from_table_or_query_ibis and pull_all_from_table_or_query_ibis filter on table[timestamp_field].cast("timestamp"). DuckDB 2 won't cast TIMESTAMPTZ_NS to a zone-less TIMESTAMP. ibis 12 also reports that column's type as unknown, so the code can't read its zone.

The filter now goes through _comparable_timestamp:

  • A zone-aware column is cast to a zone-aware timestamp at the default precision.
  • A column ibis reports as unknown is cast to a UTC timestamp.
  • Everything else is still cast to timestamp, as before, including strings and zone-less columns.

Which issue(s) this PR fixes:

None filed. I found no existing issue or PR for it. ibis-project/ibis#12206, "support duckdb 2.0", is open and may teach ibis this type; the unknown-type branch is harmless if it lands.

Testing

Materialize with a DuckDB-store FeatureStore, on a host at UTC+05:30, 24 hourly rows, a window that should select 4 of them, reading back through the SQLite online store:

Parquet timestamp DuckDB 2.0.0, before DuckDB 2.0.0, after DuckDB 1.5.6, after
timestamp[ns, tz=UTC] (pandas default) ConversionException correct 4 rows correct 4 rows
timestamp[us, tz=UTC] correct 4 rows correct 4 rows correct 4 rows
timestamp[us] correct 4 rows correct 4 rows correct 4 rows

New test in test_ibis.py, across both pull functions and 4 timestamp types:

  • DuckDB 2.0.0: 2 failed, 8 passed before the fix, with both failures on datetime64[ns, UTC]; 10 passed after.
  • DuckDB 1.5.6: 10 passed before and after, so this test won't catch a regression until CI moves to DuckDB 2.

ruff check and ruff format pass on both changed files.

On DuckDB 2.0.0 with this change, a point-in-time join against a brute-force reference also matched on all 130 entity rows, with and without a TTL, on generated data with same-timestamp rows.

Checks

  • I've made sure the tests are passing. (Only test_ibis.py, from a scratch copy; see Limitations.)
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Limitations

  • I ran test_ibis.py from a scratch copy, because the repository's conftest.py needs the GCP extras, which I didn't install. I did not run the rest of the unit suite or the integration suite.
  • Windows, Python 3.12, ibis 12.0.0. Not tested on other ibis backends. The zone-aware cast is the form point_in_time_join already uses.
  • I didn't add an upper bound on duckdb or change the CI pin. Moving CI to DuckDB 2 is a separate decision for you.

The ibis pull functions cast the timestamp column to a zone-less timestamp for the window filter. DuckDB 2 refuses that cast for TIMESTAMPTZ_NS, which pandas writes by default, and ibis reports the type as unknown. Keep the zone when casting.

Signed-off-by: NIKHIL <atnikhil06@gmail.com>
@Nikhi00718
Nikhi00718 requested a review from a team as a code owner November 3, 2026 21:39
@codecov-commenter

codecov-commenter commented Nov 3, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.33%. Comparing base (bbaca76) to head (2f46278).

Files with missing lines Patch % Lines
sdk/python/feast/infra/offline_stores/ibis.py 66.66% 1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7097      +/-   ##
==========================================
+ Coverage   51.32%   51.33%   +0.01%     
==========================================
  Files         458      458              
  Lines       57441    57447       +6     
  Branches     8421     8422       +1     
==========================================
+ Hits        29482    29493      +11     
+ Misses      25950    25939      -11     
- Partials     2009     2015       +6     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 52.81% <66.66%> (+0.01%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/infra/offline_stores/ibis.py 68.75% <66.66%> (+3.61%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbaca76...2f46278. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…e-duckdb-ns-timestamps

Signed-off-by: NIKHIL <atnikhil06@gmail.com>
@Nikhi00718
Nikhi00718 force-pushed the fix/ibis-materialize-duckdb-ns-timestamps branch from 87bfb0e to 2f46278 Compare November 7, 2026 10:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants