Skip to content

Commit 3633da0

Browse files
cjdoriscjdoris
authored
NumpyDates (#666)
* add NumpyDates module with DateTime64 and TimeDelta64 types, plus conversion support * feat(numpydates): add constructors and conversions for delta64 types Implement constructors, accessors, and display for TimeDelta64 and InlineTimeDelta64. Add conversions from Dates.Period and NaT strings. Unify NaT handling via NAT sentinel. Replace unit access with unitpair and store units as (Unit, Cint). Provide defaultunit and a showvalue helper; update isnan to use NAT. Changing units between instances is not yet implemented (except for NaT). BREAKING CHANGE: TimeDelta64 fields changed from unit_base/unit_scale to unit::Tuple{Unit,Cint}. The unit(...) accessor is removed; use unitpair(d) instead. The generic Dates.value(::AbstractTimeDelta64) method was removed; use Dates.value on concrete types. * fix(numpydates): correct year/month extraction in DateTime64(Date) Use Dates.year(d) and Dates.month(d) in the Date-based constructor instead of value(Dates.Year(d)) and value(Dates.Month(d)). The previous approach could error or yield wrong values when computing YEAR/MONTH offsets relative to 1970, affecting YEARS and MONTHS units. Also includes minor whitespace cleanup. * test(numpydates): add tests for Date to DateTime64 unit mapping Add NumpyDates tests verifying Date conversions to DateTime64 and InlineDateTime64 across units Y, M, D, h, m, s, ms, us, ns, comparing against expected int values. Covers: - DateTime64(Date, Symbol) - InlineDateTime64{U}(Date) - InlineDateTime64(Date, Symbol) and checks Dates.value and unitpair correctness. Add test/scripts/np_dates.py to generate NumPy-based ground-truth int64 values for selected dates. * fix(numpydates): use floor div and fix atto scale - Replace ÷ with fld in conversions to apply floor semantics for negative values, matching NumPy datetime64 behavior - Correct ATTOSECONDS multiplier to 1e15 - Add comprehensive DateTime -> DateTime64/InlineDateTime64 tests, including pre-epoch cases, and refine existing Date tests - Add script to generate NumPy reference values for tests * test(numpydates): test more constructors * test(numpydates): add test for show(datetime64) * fix(numpydates): resolve method ambiguity by splitting unions * test(numpydates): add tests for timedelta64 * test(numpydates): add test for conversion to Date/DateTime * test(numpydates): add tests for defaultunit and add more methods for period types * test(numpydates): code shuffle * test(numpydates): add tests for isnan * docs(numpydates): add comprehensive docstrings for NumpyDates types Add module docstring and detailed docs for Unit enum and datetime/ timedelta types (DateTime64, InlineDateTime64, TimeDelta64, InlineTimeDelta64), including accepted constructor inputs and unit references. Document supertypes AbstractDateTime64 and AbstractTimeDelta64. No functional changes. * docs(numpydates): add NumpyDates reference and clarify unit docs - Add NumpyDates section to pythoncall-reference with API docs - Update release notes with NumpyDates overview - Clarify docstrings: runtime vs inline units and memory layout * docs(numpydates): add numpy.datetime64 and numpy.timedelta64 conversions Add mapping entries for numpy.datetime64 and numpy.timedelta64 in conversion-to-julia.md. Document conversion to NumpyDates types and inline/Dates fallbacks to clarify behavior when passing numpy date/time objects from Python. * refactor(numpydates): simplify date and time delta conversions with rescale function - Add pyconvert rules for InlineDateTime64 and InlineTimeDelta64 types - Refactor DateTime64 and TimeDelta64 constructors to use rescale for unit conversions - Introduce unitscale and rescale functions in Unit.jl for efficient scaling between units - Add convert methods for DatesInstant and DatesPeriod types - Update common.jl with new type aliases and helper functions - Enhance tests for new conversion capabilities and error handling These changes streamline the conversion logic, reduce code duplication, and improve performance by centralizing unit scaling operations. No breaking changes to public APIs. * refactor(numpydates): add rescale support and expand convert methods for datetime64 and timedelta64 - Implement rescale function to handle unit changes in DateTime64 and TimeDelta64 instead of throwing errors - Expand Base.convert methods for InlineDateTime64 and InlineTimeDelta64 to support more types - Add new pyconvert rules for NumpyDates.DatesInstant and NumpyDates.DatesPeriod in numpy.jl This enhances type conversions and removes previous limitations on unit scaling. * refactor(numpydates): simplify accessors and add inline datetime/timedelta support - Simplified accessor functions in DateTime64, InlineDateTime64, InlineTimeDelta64, and TimeDelta64 by removing unnecessary function wrappers - Added support for NumpyDates.InlineDateTime64 and InlineTimeDelta64 in pyjlarray_isarrayabletype and pytypestrdescr - Refactored pytypestrdescr to use if-elseif structure for better readability - Updated PyArray.jl to handle unbound units in type conversion This enhances numpy date type integration while cleaning up code structure. * refactor(numpydates): add concrete type check for inline datetime/timedelta types Ensure that type T is concrete before checking if it subtypes the union of InlineDateTime64 and InlineTimeDelta64, preventing potential issues with abstract types in pytypestrdescr function. * docs(numpydates): update release notes with NumpyDates features * refactor(numpydates): remove unbound units export * test(numpydates): fixes for julia 1.9 --------- Co-authored-by: Christopher Doris <github.com/cjdoris>
1 parent f7fa4c2 commit 3633da0

23 files changed

+2305
-26
lines changed

‎docs/src/conversion-to-julia.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@ From Python, the arguments to a Julia function will be converted according to th
3131
| `datetime.date`/`datetime.time`/`datetime.datetime` | `Date`/`Time`/`DateTime` |
3232
| `datetime.timedelta` | `Microsecond` (or `Millisecond` or `Second` on overflow) |
3333
| `numpy.intXX`/`numpy.uintXX`/`numpy.floatXX` | `IntXX`/`UIntXX`/`FloatXX` |
34+
| `numpy.datetime64` | `NumpyDates.DateTime64` |
35+
| `numpy.timedelta64` | `NumpyDates.TimeDelta64` |
3436
| **Standard priority (other reasonable conversions).** | |
3537
| `None` | `Missing` |
3638
| `bytes` | `Vector{UInt8}`, `Vector{Int8}`, `String` |
@@ -48,6 +50,8 @@ From Python, the arguments to a Julia function will be converted according to th
4850
| `ctypes.c_char_p` | `Cstring`, `Ptr{Cchar}`, `Ptr` |
4951
| `ctypes.c_wchar_p` | `Cwstring`, `Ptr{Cwchar}`, `Ptr` |
5052
| `numpy.bool_`/`numpy.intXX`/`numpy.uintXX`/`numpy.floatXX` | `Bool`, `Integer`, `Rational`, `Real`, `Number` |
53+
| `numpy.datetime64` | `NumpyDates.InlineDateTime64`, `Dates.DateTime` |
54+
| `numpy.timedelta64` | `NumpyDates.InlineTimeDelta64`, `Dates.Period` |
5155
| Objects satisfying the buffer or array interface | `Array`, `AbstractArray` |
5256
| **Low priority (fallback to `Py`).** | |
5357
| Anything | `Py` |

‎docs/src/pythoncall-reference.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,3 +255,18 @@ PythonCall.getptr
255255
PythonCall.pydel!
256256
PythonCall.unsafe_pynext
257257
```
258+
259+
## NumpyDates
260+
261+
The submodule `PythonCall.NumpyDates` provides types corresponding to Numpy's `datetime64` and `timedelta64` types. Enables conversion of these Numpy types (either as scalars or in arrays) to native Julia types.
262+
263+
```@docs
264+
PythonCall.NumpyDates
265+
PythonCall.NumpyDates.AbstractDateTime64
266+
PythonCall.NumpyDates.InlineDateTime64
267+
PythonCall.NumpyDates.DateTime64
268+
PythonCall.NumpyDates.AbstractTimeDelta64
269+
PythonCall.NumpyDates.InlineTimeDelta64
270+
PythonCall.NumpyDates.TimeDelta64
271+
PythonCall.NumpyDates.Unit
272+
```

‎docs/src/releasenotes.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
# Release Notes
22

33
## Unreleased
4+
* Added `NumpyDates`: NumPy-compatible DateTime64/TimeDelta64 types and units.
5+
* Added `pyconvert` rules for NumpyDates types.
6+
* Added `PyArray` support for NumPy arrays of `datetime64` and `timedelta64`.
7+
* Added `juliacall.ArrayValue` support for Julia arrays of `InlineDateTime64` and `InlineTimeDelta64`.
48
* Bug fixes.
59
* Internal: switch from Requires.jl to package extensions.
610

‎src/Convert/Convert.jl‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ using ..PythonCall
99
using ..Utils
1010
using ..C
1111
using ..Core
12+
using ..NumpyDates
1213

1314
using Dates: Date, Time, DateTime, Second, Millisecond, Microsecond, Nanosecond
1415

@@ -20,8 +21,7 @@ import ..PythonCall:
2021
pyconvert,
2122
PyConvertPriority
2223

23-
export
24-
pyconvert_isunconverted,
24+
export pyconvert_isunconverted,
2525
pyconvert_result,
2626
pyconvert_result,
2727
pyconvert_tryconvert,

‎src/Convert/numpy.jl‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,76 @@ function (::pyconvert_rule_numpysimplevalue{R,SAFE})(::Type{T}, x::Py) where {R,
99
end
1010
end
1111

12+
function pyconvert_rule_datetime64(::Type{DateTime64}, x::Py)
13+
pyconvert_return(C.PySimpleObject_GetValue(DateTime64, x))
14+
end
15+
16+
function pyconvert_rule_datetime64(::Type{T}, x::Py) where {T<:InlineDateTime64}
17+
pyconvert_tryconvert(T, C.PySimpleObject_GetValue(DateTime64, x))
18+
end
19+
20+
function pyconvert_rule_datetime64(::Type{T}, x::Py) where {T<:NumpyDates.DatesInstant}
21+
d = C.PySimpleObject_GetValue(DateTime64, x)
22+
if isnan(d)
23+
pyconvert_unconverted()
24+
else
25+
pyconvert_tryconvert(T, d)
26+
end
27+
end
28+
29+
function pyconvert_rule_datetime64(::Type{Missing}, x::Py)
30+
d = C.PySimpleObject_GetValue(DateTime64, x)
31+
if isnan(d)
32+
pyconvert_return(missing)
33+
else
34+
pyconvert_unconverted()
35+
end
36+
end
37+
38+
function pyconvert_rule_datetime64(::Type{Nothing}, x::Py)
39+
d = C.PySimpleObject_GetValue(DateTime64, x)
40+
if isnan(d)
41+
pyconvert_return(nothing)
42+
else
43+
pyconvert_unconverted()
44+
end
45+
end
46+
47+
function pyconvert_rule_timedelta64(::Type{TimeDelta64}, x::Py)
48+
pyconvert_return(C.PySimpleObject_GetValue(TimeDelta64, x))
49+
end
50+
51+
function pyconvert_rule_timedelta64(::Type{T}, x::Py) where {T<:InlineTimeDelta64}
52+
pyconvert_tryconvert(T, C.PySimpleObject_GetValue(TimeDelta64, x))
53+
end
54+
55+
function pyconvert_rule_timedelta64(::Type{T}, x::Py) where {T<:NumpyDates.DatesPeriod}
56+
d = C.PySimpleObject_GetValue(TimeDelta64, x)
57+
if isnan(d)
58+
pyconvert_unconverted()
59+
else
60+
pyconvert_tryconvert(T, d)
61+
end
62+
end
63+
64+
function pyconvert_rule_timedelta64(::Type{Missing}, x::Py)
65+
d = C.PySimpleObject_GetValue(TimeDelta64, x)
66+
if isnan(d)
67+
pyconvert_return(missing)
68+
else
69+
pyconvert_unconverted()
70+
end
71+
end
72+
73+
function pyconvert_rule_timedelta64(::Type{Nothing}, x::Py)
74+
d = C.PySimpleObject_GetValue(TimeDelta64, x)
75+
if isnan(d)
76+
pyconvert_return(missing)
77+
else
78+
pyconvert_unconverted()
79+
end
80+
end
81+
1282
const NUMPY_SIMPLE_TYPES = [
1383
("bool_", Bool),
1484
("int8", Int8),
@@ -28,6 +98,7 @@ const NUMPY_SIMPLE_TYPES = [
2898
]
2999

30100
function init_numpy()
101+
# simple numeric scalar types
31102
for (t, T) in NUMPY_SIMPLE_TYPES
32103
isbool = occursin("bool", t)
33104
isint = occursin("int", t) || isbool
@@ -54,4 +125,36 @@ function init_numpy()
54125
iscomplex && pyconvert_add_rule(name, Complex, rule)
55126
isnumber && pyconvert_add_rule(name, Number, rule)
56127
end
128+
129+
# datetime64
130+
pyconvert_add_rule(
131+
"numpy:datetime64",
132+
DateTime64,
133+
pyconvert_rule_datetime64,
134+
PYCONVERT_PRIORITY_ARRAY,
135+
)
136+
pyconvert_add_rule("numpy:datetime64", InlineDateTime64, pyconvert_rule_datetime64)
137+
pyconvert_add_rule(
138+
"numpy:datetime64",
139+
NumpyDates.DatesInstant,
140+
pyconvert_rule_datetime64,
141+
)
142+
pyconvert_add_rule("numpy:datetime64", Missing, pyconvert_rule_datetime64)
143+
pyconvert_add_rule("numpy:datetime64", Nothing, pyconvert_rule_datetime64)
144+
145+
# timedelta64
146+
pyconvert_add_rule(
147+
"numpy:timedelta64",
148+
TimeDelta64,
149+
pyconvert_rule_timedelta64,
150+
PYCONVERT_PRIORITY_ARRAY,
151+
)
152+
pyconvert_add_rule("numpy:timedelta64", InlineTimeDelta64, pyconvert_rule_timedelta64)
153+
pyconvert_add_rule(
154+
"numpy:timedelta64",
155+
NumpyDates.DatesPeriod,
156+
pyconvert_rule_timedelta64,
157+
)
158+
pyconvert_add_rule("numpy:timedelta64", Missing, pyconvert_rule_timedelta64)
159+
pyconvert_add_rule("numpy:timedelta64", Nothing, pyconvert_rule_timedelta64)
57160
end

‎src/JlWrap/JlWrap.jl‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ module JlWrap
77

88
using ..PythonCall
99
using ..Utils
10+
using ..NumpyDates: NumpyDates
1011
using ..C
1112
using ..Core
1213
using ..Convert

‎src/JlWrap/array.jl‎

Lines changed: 41 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,8 @@ pyjlarray_isarrayabletype(::Type{T}) where {T} = T in (
223223
Complex{Float32},
224224
Complex{Float64},
225225
)
226+
pyjlarray_isarrayabletype(::Type{NumpyDates.InlineDateTime64{U}}) where {U} = true
227+
pyjlarray_isarrayabletype(::Type{NumpyDates.InlineTimeDelta64{U}}) where {U} = true
226228
pyjlarray_isarrayabletype(::Type{T}) where {T<:Tuple} =
227229
isconcretetype(T) &&
228230
Base.allocatedinline(T) &&
@@ -235,22 +237,45 @@ const PYTYPESTRDESCR = IdDict{Type,Tuple{String,Py}}()
235237
pytypestrdescr(::Type{T}) where {T} =
236238
get!(PYTYPESTRDESCR, T) do
237239
c = Utils.islittleendian() ? '<' : '>'
238-
T == Bool ? ("$(c)b$(sizeof(Bool))", PyNULL) :
239-
T == Int8 ? ("$(c)i1", PyNULL) :
240-
T == UInt8 ? ("$(c)u1", PyNULL) :
241-
T == Int16 ? ("$(c)i2", PyNULL) :
242-
T == UInt16 ? ("$(c)u2", PyNULL) :
243-
T == Int32 ? ("$(c)i4", PyNULL) :
244-
T == UInt32 ? ("$(c)u4", PyNULL) :
245-
T == Int64 ? ("$(c)i8", PyNULL) :
246-
T == UInt64 ? ("$(c)u8", PyNULL) :
247-
T == Float16 ? ("$(c)f2", PyNULL) :
248-
T == Float32 ? ("$(c)f4", PyNULL) :
249-
T == Float64 ? ("$(c)f8", PyNULL) :
250-
T == Complex{Float16} ? ("$(c)c4", PyNULL) :
251-
T == Complex{Float32} ? ("$(c)c8", PyNULL) :
252-
T == Complex{Float64} ? ("$(c)c16", PyNULL) :
253-
if isstructtype(T) && isconcretetype(T) && Base.allocatedinline(T)
240+
if T == Bool
241+
("$(c)b$(sizeof(Bool))", PyNULL)
242+
elseif T == Int8
243+
("$(c)i1", PyNULL)
244+
elseif T == UInt8
245+
("$(c)u1", PyNULL)
246+
elseif T == Int16
247+
("$(c)i2", PyNULL)
248+
elseif T == UInt16
249+
("$(c)u2", PyNULL)
250+
elseif T == Int32
251+
("$(c)i4", PyNULL)
252+
elseif T == UInt32
253+
("$(c)u4", PyNULL)
254+
elseif T == Int64
255+
("$(c)i8", PyNULL)
256+
elseif T == UInt64
257+
("$(c)u8", PyNULL)
258+
elseif T == Float16
259+
("$(c)f2", PyNULL)
260+
elseif T == Float32
261+
("$(c)f4", PyNULL)
262+
elseif T == Float64
263+
("$(c)f8", PyNULL)
264+
elseif T == Complex{Float16}
265+
("$(c)c4", PyNULL)
266+
elseif T == Complex{Float32}
267+
("$(c)c8", PyNULL)
268+
elseif T == Complex{Float64}
269+
("$(c)c16", PyNULL)
270+
elseif isconcretetype(T) &&
271+
T <: Union{NumpyDates.InlineDateTime64,NumpyDates.InlineTimeDelta64}
272+
u, m = NumpyDates.unitpair(T)
273+
tc = T <: NumpyDates.InlineDateTime64 ? 'M' : 'm'
274+
us =
275+
u == NumpyDates.UNBOUND_UNITS ? "" :
276+
m == 1 ? "[$(Symbol(u))]" : "[$(m)$(Symbol(u))]"
277+
("$(c)$(tc)8$(us)", PyNULL)
278+
elseif isstructtype(T) && isconcretetype(T) && Base.allocatedinline(T)
254279
n = fieldcount(T)
255280
flds = []
256281
for i = 1:n

‎src/NumpyDates/AbstractDateTime64.jl‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
"""
2+
abstract type AbstractDateTime64 <: Dates.TimeType
3+
4+
Supertype for [`DateTime64`](@ref) and [`InlineDateTime64`](@ref).
5+
"""
6+
abstract type AbstractDateTime64 <: Dates.TimeType end
7+
8+
function Dates.DateTime(d::AbstractDateTime64)
9+
isnan(d) && error("Cannot convert NaT to DateTime")
10+
v = value(d)
11+
u, _ = unit = unitpair(d)
12+
b = Dates.DateTime(1970)
13+
if u > MONTHS
14+
v, _ = rescale(v, unit, MILLISECONDS)
15+
b + Dates.Millisecond(v)
16+
else
17+
v, _ = rescale(v, unit, MONTHS)
18+
b + Dates.Month(v)
19+
end
20+
end
21+
22+
function Dates.Date(d::AbstractDateTime64)
23+
isnan(d) && error("Cannot convert NaT to Date")
24+
Dates.Date(Dates.DateTime(d))
25+
end
26+
27+
Base.convert(::Type{Dates.DateTime}, d::AbstractDateTime64) = Dates.DateTime(d)
28+
Base.convert(::Type{Dates.Date}, d::AbstractDateTime64) = Dates.Date(d)
29+
30+
function Base.isnan(d::AbstractDateTime64)
31+
value(d) == typemin(Int64)
32+
end
33+
34+
function showvalue(io::IO, d::AbstractDateTime64)
35+
u, m = unit = unitpair(d)
36+
if isnan(d)
37+
show(io, "NaT")
38+
elseif u ≤ DAYS
39+
d2 = Dates.Date(d)
40+
if value(DateTime64(d2, unit)) == value(d)
41+
show(io, string(d2))
42+
else
43+
show(io, value(d))
44+
end
45+
else
46+
d2 = Dates.DateTime(d)
47+
if value(DateTime64(d2, unit)) == value(d)
48+
show(io, string(d2))
49+
else
50+
show(io, value(d))
51+
end
52+
end
53+
nothing
54+
end
55+
56+
function defaultunit(d::AbstractDateTime64)
57+
unitpair(d)
58+
end

‎src/NumpyDates/AbstractTimeDelta64.jl‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
"""
2+
abstract type AbstractTimeDelta64 <: Dates.Period
3+
4+
Supertype for [`TimeDelta64`](@ref) and [`InlineTimeDelta64`](@ref).
5+
"""
6+
abstract type AbstractTimeDelta64 <: Dates.Period end
7+
8+
function construct(::Type{T}, d::AbstractTimeDelta64) where {T<:DatesPeriod}
9+
v, r = rescale(value(d), unitpair(d), Unit(T))
10+
iszero(r) || throw(InexactError(:convert, T, d))
11+
T(v)
12+
end
13+
14+
Dates.Year(d::AbstractTimeDelta64) = construct(Dates.Year, d)
15+
Dates.Month(d::AbstractTimeDelta64) = construct(Dates.Month, d)
16+
Dates.Day(d::AbstractTimeDelta64) = construct(Dates.Day, d)
17+
Dates.Hour(d::AbstractTimeDelta64) = construct(Dates.Hour, d)
18+
Dates.Minute(d::AbstractTimeDelta64) = construct(Dates.Minute, d)
19+
Dates.Second(d::AbstractTimeDelta64) = construct(Dates.Second, d)
20+
Dates.Millisecond(d::AbstractTimeDelta64) = construct(Dates.Millisecond, d)
21+
Dates.Microsecond(d::AbstractTimeDelta64) = construct(Dates.Microsecond, d)
22+
Dates.Nanosecond(d::AbstractTimeDelta64) = construct(Dates.Nanosecond, d)
23+
24+
Base.convert(::Type{Dates.Year}, d::AbstractTimeDelta64) = Dates.Year(d)
25+
Base.convert(::Type{Dates.Month}, d::AbstractTimeDelta64) = Dates.Month(d)
26+
Base.convert(::Type{Dates.Day}, d::AbstractTimeDelta64) = Dates.Day(d)
27+
Base.convert(::Type{Dates.Hour}, d::AbstractTimeDelta64) = Dates.Hour(d)
28+
Base.convert(::Type{Dates.Minute}, d::AbstractTimeDelta64) = Dates.Minute(d)
29+
Base.convert(::Type{Dates.Second}, d::AbstractTimeDelta64) = Dates.Second(d)
30+
Base.convert(::Type{Dates.Millisecond}, d::AbstractTimeDelta64) = Dates.Millisecond(d)
31+
Base.convert(::Type{Dates.Microsecond}, d::AbstractTimeDelta64) = Dates.Microsecond(d)
32+
Base.convert(::Type{Dates.Nanosecond}, d::AbstractTimeDelta64) = Dates.Nanosecond(d)
33+
34+
function Base.isnan(d::AbstractTimeDelta64)
35+
value(d) == NAT
36+
end
37+
38+
function showvalue(io::IO, d::AbstractTimeDelta64)
39+
u, m = unitpair(d)
40+
if isnan(d)
41+
show(io, "NaT")
42+
else
43+
show(io, value(d))
44+
end
45+
nothing
46+
end
47+
48+
function defaultunit(d::AbstractTimeDelta64)
49+
unitpair(d)
50+
end

0 commit comments

Comments
 (0)