Relax nanosecond datetime restriction in CF time decoding #9618
Conversation
|
Nice, mypy 1.12 is out and breaks our typing, 😭. |
Can we pin it in the CI temporarily? |
Yes, 1.11.2 was the last version. |
kmuehlbauer
force-pushed
the
any-time-resolution-2
branch
from
ca5050d to
f7396cf
Compare
|
This is now ready for a first round of review. I think this is already in a quite usable state. But no rush, this should be thoroughly tested. |
kmuehlbauer
added
topic-CF conventions
topic-cftime
run-upstream
|
Sounds good @kmuehlbauer! I’ll try and take an initial look this weekend. |
…ore/variable.py to use any-precision datetime/timedelta with autmatic inferring of resolution
…ocessing, raise now early
…t resolution, fix code and tests to allow this
|
I went over the docs once again, also added whats-new.rst entry. Some more cosmetic changes. I'd say this is finished. Please let me know, if there is something missing. |
There was a problem hiding this comment.
Thank you for all of your hard work on this @kmuehlbauer 🙏.
I think we are basically there! I just want to make sure we are not committing anything that we know is a breaking change or otherwise does not work out of the box. I'm approving, but take a look at kmuehlbauer#3, which fixes a couple issues related to timedelta decoding / encoding.
xarray/tests/test_coding_times.py
Outdated
| @@ -595,8 +624,9 @@ def test_infer_cftime_datetime_units(calendar, date_args, expected) -> None: | |||
| ], | |||
| ) | |||
| def test_cf_timedelta(timedeltas, units, numbers) -> None: | |||
| # todo: check, if this test is OK | |||
There was a problem hiding this comment.
Almost—I realize my comments about enabling non-nanosecond timedelta decoding, but defaulting to nanosecond resolution decoding could only be satisfied by implementing a time_unit option in encode_cf_timedelta...I gave it a try in kmuehlbauer#3.
Let me know if that looks OK. Your refactoring of the timedelta handling in 0308672 really does the heavy lifting 👍.
Co-authored-by: Spencer Clark <[email protected]>
* Fix timedelta encoding overflow issue; always decode to ns resolution * Implement time_unit for decode_cf_timedelta * Reduce diff
|
Thanks again @spencerkclark! I've merged your kmuehlbauer#3 as it is essential for this PR to keep consistency and backwards compatibility. |
Co-authored-by: Kai Mühlbauer <[email protected]>
|
Failing CI looks unrelated. Decision in the dev meeting was to merge fast to have a bit of time to test before release later this month. |
|
Thanks All! 🚀 |
|
Awesome, congrats @kmuehlbauer—thank you! |
|
Yes @kmuehlbauer - monumental effort!!! |
|
Amazing! 👏 👏 👏 |
whats-new.rstThis is another attempt to resolve #7493. This goes a step further than #9580.
The idea of this PR is to automatically infer the needed resolutions for decoding/
encodingand only keep the constraints pandas imposes ("s" - lowest resolution, "ns" - highest resolution). There is still the idea of adefault resolution, but this should only take precedence if it doesn't clash with the automatic inference. This can be discussed, though. Update: I've implementedtime-unit-kwarga first try to have default resolutionon decode, which will override the current inferred resolution only to higher resolution (eg.'s'->'ns'). To work towards #4490 the time decoding options (decode_timeanduse_cftimeare bundled withinCFDatetimeCoderwhich is distributed viadecode_timeskwarg.use_cftime-kwarg is deprecated.For sanity checking, and also for my own good, I've created a documentation page on time-coding in the internal dev section. Any suggestions (especially grammar) or ideas for enhancements are much appreciated.
There still might be room for consolidation of functions/methods (mostly in coding/times.py), but I have to leave it alone for some days. I went down that rabbit hole and need to relax, too 😬.
Looking forward to get your insights here, @spencerkclark, @ChrisBarker-NOAA, @pydata/xarray.
Todo:
time_units(where appropriate)CFDatetimeCoderas input fordecode_timeskwarg