[Data] PyArrow 20.0.0 Backward Incompatability (`unexpected keyword argument 'maps_as_pydicts'`)

[ivanthewebber]

What happened + What you expected to happen

A recent update to PyArrow introduced a backwards incompatibility (apache/arrow#45471 (comment)). This causes unexpected keyword argument 'maps_as_pydicts'.

ds.take_all()
...

  File "pyarrow/array.pxi", line 987, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 5174, in pyarrow.lib.Table._to_pandas
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/pyarrow/pandas_compat.py", line 825, in table_to_dataframe
    _reconstruct_block(item, column_names, ext_columns_dtypes)
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/pyarrow/pandas_compat.py", line 765, in _reconstruct_block
    arr = pandas_dtype.__from_arrow__(arr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/ray/air/util/object_extensions/pandas.py", line 146, in __from_arrow__
    return PythonObjectArray(array.to_pylist())
                             ^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1380, in pyarrow.lib.ChunkedArray.to_pylist
  File "pyarrow/array.pxi", line 1775, in pyarrow.lib.Array.to_pylist
TypeError: ArrowPythonObjectScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'

Versions / Dependencies

Ray and PyArrow latest.

Reproduction script

I'm in a rush, but this should do it:

ray.data.from_pandas(pd.DataFrame([{'id': i} for i in range(10)])).take_all()

Issue Severity

Low: It annoys or frustrates me.