Repository navigation
fix: Materialize zone-aware nanosecond timestamps on DuckDB 2 - #7097
Open
Nikhi00718 wants to merge 2 commits into
Open
Nikhi00718 wants to merge 2 commits into
Nikhi00718 wants to merge 2 commits into
Conversation
The ibis pull functions cast the timestamp column to a zone-less timestamp for the window filter. DuckDB 2 refuses that cast for TIMESTAMPTZ_NS, which pandas writes by default, and ibis reports the type as unknown. Keep the zone when casting. Signed-off-by: NIKHIL <atnikhil06@gmail.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7097 +/- ##
==========================================
+ Coverage 51.32% 51.33% +0.01%
==========================================
Files 458 458
Lines 57441 57447 +6
Branches 8421 8422 +1
==========================================
+ Hits 29482 29493 +11
+ Misses 25950 25939 -11
- Partials 2009 2015 +6
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
…e-duckdb-ns-timestamps Signed-off-by: NIKHIL <atnikhil06@gmail.com>
Nikhi00718
force-pushed
the
fix/ibis-materialize-duckdb-ns-timestamps
branch
from
November 7, 2026 10:10
87bfb0e to
2f46278
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
feast materializeon the DuckDB offline store crashes on DuckDB 2.0 when aFileSourceholds zone-aware nanosecond timestamps, which is what pandas writes by default for adatetime64[ns, UTC]column:The
duckdbextra isibis-framework[duckdb]>=10.0.0, and ibis 12 only requiresduckdb>=0.10.3, so a fresh install today gets DuckDB 2.0.0, released 2026-10-21. Feast's CI pins 1.5.6, which is why CI doesn't see this.get_historical_featureskeeps working, so the failure first shows up at materialization.pull_latest_from_table_or_query_ibisandpull_all_from_table_or_query_ibisfilter ontable[timestamp_field].cast("timestamp"). DuckDB 2 won't castTIMESTAMPTZ_NSto a zone-lessTIMESTAMP. ibis 12 also reports that column's type as unknown, so the code can't read its zone.The filter now goes through
_comparable_timestamp:timestamp, as before, including strings and zone-less columns.Which issue(s) this PR fixes:
None filed. I found no existing issue or PR for it. ibis-project/ibis#12206, "support duckdb 2.0", is open and may teach ibis this type; the unknown-type branch is harmless if it lands.
Testing
Materialize with a DuckDB-store
FeatureStore, on a host at UTC+05:30, 24 hourly rows, a window that should select 4 of them, reading back through the SQLite online store:timestamp[ns, tz=UTC](pandas default)ConversionExceptiontimestamp[us, tz=UTC]timestamp[us]New test in
test_ibis.py, across both pull functions and 4 timestamp types:datetime64[ns, UTC]; 10 passed after.ruff checkandruff formatpass on both changed files.On DuckDB 2.0.0 with this change, a point-in-time join against a brute-force reference also matched on all 130 entity rows, with and without a TTL, on generated data with same-timestamp rows.
Checks
test_ibis.py, from a scratch copy; see Limitations.)git commit -s)Testing Strategy
Limitations
test_ibis.pyfrom a scratch copy, because the repository'sconftest.pyneeds the GCP extras, which I didn't install. I did not run the rest of the unit suite or the integration suite.point_in_time_joinalready uses.duckdbor change the CI pin. Moving CI to DuckDB 2 is a separate decision for you.