Skip to content

Commit 098174c

Browse files
authored
Fix for ns timestamp non utc (#3142)
Closes #3127 # Rationale for this change `pa.timestamp("ns", tz="US/Pacific") converts to TimestampNanoType()` (tz-naive) instead of raising `TypeError`. The `us` path correctly rejects non-UTC timezones. raises TypeError `visit_pyarrow(pa.timestamp("us", tz="US/Pacific"), _ConvertToIceberg())` silently drops timezone `visit_pyarrow(pa.timestamp("ns", tz="US/Pacific"), _ConvertToIceberg(format_version=3))` Fixes this by adding the correct elif clause ## Are these changes tested? Yes - added a couple of tests in the pyarrow visitor ## Are there any user-facing changes? It fixes the behavior that was a bug, so I think it would be.
1 parent fcfaaf6 commit 098174c

File tree

2 files changed

+14
-1
lines changed

2 files changed

+14
-1
lines changed

pyiceberg/io/pyarrow.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1451,7 +1451,7 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType:
14511451
elif self._format_version >= 3:
14521452
if primitive.tz in UTC_ALIASES:
14531453
return TimestamptzNanoType()
1454-
else:
1454+
elif primitive.tz is None:
14551455
return TimestampNanoType()
14561456
else:
14571457
raise TypeError(

tests/io/test_pyarrow_visitor.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@
6363
NestedField,
6464
StringType,
6565
StructType,
66+
TimestampNanoType,
6667
TimestampType,
6768
TimestamptzType,
6869
TimeType,
@@ -226,6 +227,18 @@ def test_pyarrow_timestamp_tz_invalid_tz() -> None:
226227
visit_pyarrow(pyarrow_type, _ConvertToIceberg())
227228

228229

230+
def test_pyarrow_timestamp_ns_tz_invalid_tz() -> None:
231+
pyarrow_type = pa.timestamp(unit="ns", tz="US/Pacific")
232+
with pytest.raises(TypeError, match=re.escape("Unsupported type: timestamp[ns, tz=US/Pacific]")):
233+
visit_pyarrow(pyarrow_type, _ConvertToIceberg(format_version=3))
234+
235+
236+
def test_pyarrow_timestamp_ns_no_tz_accepted() -> None:
237+
pyarrow_type = pa.timestamp(unit="ns")
238+
converted = visit_pyarrow(pyarrow_type, _ConvertToIceberg(format_version=3))
239+
assert converted == TimestampNanoType()
240+
241+
229242
@pytest.mark.parametrize("pyarrow_type", [pa.string(), pa.large_string(), pa.string_view()])
230243
def test_pyarrow_string_to_iceberg(pyarrow_type: pa.DataType) -> None:
231244
converted_iceberg_type = visit_pyarrow(pyarrow_type, _ConvertToIceberg())

0 commit comments

Comments
 (0)