Skip to content

Commit 59c7acb

Browse files
committed
Make pyarrow optional, and show an error if not installed when trying to use parquet files.
Example Error ``` $ uv run python test_pyarrow.py Expected ImportError: pyarrow is not installed and is required to use parquet files. Please install it via `pip install together[pyarrow]` ``` Confirmed installing resolves issue: ``` uv pip install "dist/together-1.5.0-py3-none-any.whl[pyarrow]" Resolved 33 packages in 394ms Installed 1 package in 30ms + pyarrow==20.0.0 ```
1 parent 10a1e74 commit 59c7acb

File tree

2 files changed

+10
-3
lines changed

2 files changed

+10
-3
lines changed

pyproject.toml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,16 @@ aiohttp = "^3.9.3"
3939
filelock = "^3.13.1"
4040
eval-type-backport = ">=0.1.3,<0.3.0"
4141
click = "^8.1.7"
42-
pyarrow = ">=10.0.1"
42+
pyarrow = { version = ">=10.0.1", optional = true }
4343
numpy = [
4444
{ version = ">=1.23.5", python = "<3.12" },
4545
{ version = ">=1.26.0", python = ">=3.12" },
4646
]
4747
pillow = "^11.1.0"
4848

49+
[tool.poetry.extras]
50+
pyarrow = ["pyarrow"]
51+
4952
[tool.poetry.group.quality]
5053
optional = true
5154

src/together/utils/files.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -371,8 +371,12 @@ def _check_jsonl(file: Path) -> Dict[str, Any]:
371371

372372

373373
def _check_parquet(file: Path) -> Dict[str, Any]:
374-
# in method import - this allows client to exclude the pyarrow dep if they don't need it. Saved ~80MB and more compatible with older systems.
375-
from pyarrow import ArrowInvalid, parquet
374+
try:
375+
# Pyarrow is optional as it's large (~80MB) and isn't compatible with older systems.
376+
from pyarrow import ArrowInvalid, parquet
377+
except ImportError:
378+
raise ImportError("pyarrow is not installed and is required to use parquet files. Please install it via `pip install together[pyarrow]`")
379+
376380
report_dict: Dict[str, Any] = {}
377381

378382
try:

0 commit comments

Comments
 (0)