opt(rfdetr): build Pillow-SIMD beside Pillow in the x86-64 CPU and GPU images

[leeclemnet]

What does this PR do?

Builds Pillow-SIMD v12.2.0.post0 (MIT-CMU license) into the x86-64 CPU and GPU images beside the Pillow wheel, and lets the RF-DETR preprocessor use it for one call: the PIL resize that matches training. Pillow-SIMD is the Pillow resample algorithm with SSE4 kernels.

Pillow-SIMD tracks Pillow 12.2.0, which carries 13 advisories that 12.3.0 fixed (heap out-of-bounds writes in paste/crop, ImageCms, RankFilter; decompression-bomb bypasses in the GD, BDF and PCF loaders; DoS in PDF, EPS and JPEG2000). So it must not replace the wheel. The Pillow floor stays at >=12.3.0, PIL stays the wheel, and only Image.fromarray plus resize on an already decoded array of known shape run on the SIMD build. No decoder, filter, font or PDF path ever reaches it.

Changes:

• Dockerfile.onnx.cpu, .cpu.dev: on linux/amd64, pip install --no-deps --target /opt/pillow_simd from the pinned commit, then docker/scripts/verify_pillow_simd.py.
• Dockerfile.onnx.gpu, .gpu.dev, .cu13.gpu: the same on amd64; the multi-stage images copy /opt/pillow_simd into the runtime stage. On other arches the directory exists and is empty.
• docker/scripts/verify_pillow_simd.py: asserts that the standard wheel is at least 12.3, that the SIMD build loads under an alias with its own extension module, and that a resize agrees with the wheel within one gray level. Standalone, because the CPU image installs inference-models from PyPI.
• inference_models/models/common/pillow_simd.py: importing it loads the build at INFERENCE_MODELS_PILLOW_SIMD_PATH (default /opt/pillow_simd) as package PILSIMD and exposes its Image module. The import raises ModuleNotFoundError when the directory is absent or the variable is empty, and ImportError when the build does not load.
• RF-DETR numpy preprocessor: try: from ...pillow_simd import Image / except ImportError: from PIL import Image, then fromarray and resize as before. The resize is called on the image, not through torchvision, because a PILSIMD image is not a PIL.Image.Image instance. Same PIL bilinear resize either way.
• No SIMD compile flag, so the build is SSE4. AVX2 is 30% faster on the resize but faults on hosts without it, including Rosetta.
• arm64 CPU images, Jetson images, and the slim, parallel, udp and 3d variants are untouched.

Output differs from the wheel by at most one gray level on about 0.1% of pixels on downscales, and is bit-exact on a no-op resize. With the directory absent or the variable empty the output is bit-exact with today's code.

Measured on lee-t4-dev-cu128 (T4, Xeon 2.3GHz), rfdetr-small TRT, 2048×1371 chess image, pre_process median of 25, mean of two rounds in one session:

	pre_process raw	pre_process 512
main	38.7ms	3.9ms
this PR	24.7ms	3.8ms
#2988	21.5ms	2.6ms
this PR + #2988	7.9ms	2.5ms

With the directory absent, or INFERENCE_MODELS_PILLOW_SIMD_PATH empty, the output is bit-exact with main and the time matches main.

Combined effect with #2988. That PR moves the channel swap after the resize, which removes a strided full-size copy that sits in front of the resize this PR speeds up. Each alone saves 14ms and 17ms of pre_process on this image. Together they save 31ms, from 38.7ms to 7.9ms, and the server time on raw images drops from about 67ms to 35ms. The two PRs are independent and merge in either order; both touch the tail of _pre_process_numpy, so the second to merge takes a small conflict.

Deploy note. The GPU images build inference_models from the repo, so they get the loader with the image. The CPU image installs inference-models from PyPI, so its RF-DETR path keeps the wheel until the next inference-models release carries the loader. The directory is harmless in the meantime.

Related Issue(s): #2988

Type of Change

• Other: performance, output within one gray level of the wheel on the one resize; no dependency change

Testing

• I have tested this change locally
• I have added/updated tests for this change no test update needed

Test details: see T4 latency measurement above

Workflow runs dispatched on this branch at 206d8cdbb (they run only on main otherwise):

• CPU image build + server integration tests: https://github.com/roboflow/inference/actions/runs/34974576267 (passed, USE_INFERENCE_MODELS true and false)
• GPU image build + T4 integration tests: https://github.com/roboflow/inference/actions/runs/34974579377 (passed, USE_INFERENCE_MODELS true and false)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

Unit tests in tests/unit_tests/models/common/test_pillow_simd.py cover the import errors when the path is absent, empty, or fails to load, the alias load beside standard Pillow, and that the RF-DETR preprocessor binds to the build and falls back with equal output. They use a fake package, so they run on any CI box. tests/unit_tests/conftest.py pins the path to empty, so the RF-DETR exactness tests compare against standard Pillow on a host that has a build under /opt.

The fork has no 12.3-based commit: simd/master is 12 commits on top of upstream 12.2.0 and 343 behind 12.3.0, and its 12.3.0 tag is upstream Pillow's tag with no SIMD code. When the fork releases a 12.3-based build, the alias stays valid and the pin moves.

🤖 Generated with Claude Code

[github-actions]

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

• 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
• 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

• 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
• ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
• 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
• 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

• Prefer to skip automated review entirely? Add the skip-claude-review label.

[github-actions]

🤖 Claude review started at commit d94870b7e70618d1f649716199a85cf013510c55.

New commits are not auto-reviewed. Add the claude-review label to request a re-review — the label is consumed when the review starts, so just add it again next time.

[github-actions]

Medium (supply-chain / reproducibility): this installs and compiles third-party source into the official published images from a mutable git tag (@v12.2.0.post0). A tag can be force-moved and an upstream account/repo can be compromised, so a future build of the same Dockerfile could pull different code and bake it into a shipped Roboflow image — with no checksum or immutable ref to detect it. This also makes the build non-reproducible.

The rest of this repo already avoids exactly this: Dockerfile.onnx.gpu pins its from-source builds by immutable ref + checksum (GSTREAMER_COMMIT=88e3121…, FFMPEG_SHA256, OPENCV_SHA256). Please pin pillow-simd to the immutable commit SHA the tag currently points to (keep the tag in a comment for readability), e.g. pillow-simd @ git+https://github.com/uploadcare/pillow-simd.git@<commit-sha>.

Same line repeats in Dockerfile.onnx.cpu.dev, Dockerfile.onnx.gpu, Dockerfile.onnx.gpu.dev, and Dockerfile.onnx.cu13.gpu — please update all five.

Reviewed at HEAD: d94870b

[leeclemnet]

Pinned commit instead of tag in fbb023f.

Note: new pypi releases have been uploaded in the last 24h. 12.2.0.post0 is not there yet (12.1.1post0 is) but will continue to monitor and switch to standard pypi install once available. https://pypi.org/project/pillow-simd/#history

[github-actions]

@leeclemnet

⏳ This PR is on hold pending your answers — the review will not advance to sign-off until the IMPORTANT question below is answered.

IMPORTANT question

1. Is anything (notably a security fix) lost by dropping the Pillow floor from >=12.3.0 to >=12.2.0 and shipping pillow-simd v12.2.0.post0 (Pillow 12.2 base) in the x86-64 images?
   The floor was deliberately >=12.3.0 in requirements/_requirements.txt, requirements/requirements.sdk.http.txt, and inference_models/pyproject.toml before this PR. The fork newest release tracks Pillow 12.2, so the published x86-64 images will now run the 12.2 code line rather than 12.3.x.
   What would change the outcome: if Pillow 12.3.0 contained a security fix (or a decode/correctness fix) absent from the 12.2 line, these images regress it — a blocker needing mitigation (cherry-pick, or scoping the SIMD swap so non-SIMD paths keep the 12.3 floor). If 12.3.0 carried no such fix, this is fine as-is.

Please also confirm the >=12.3.0 to >=12.2.0 relaxation on the non-SIMD surfaces (arm64 CPU, Jetson, slim/parallel/udp/3d variants, and the published inference-sdk wheel via requirements.sdk.http.txt) is intended — those paths keep the stock Pillow wheel and do not need the fork, so the floor drop widens their accepted range without benefit.

Unanswered questions may keep this PR out of a release.

Re-review is not automatic: new commits are not auto-reviewed — add the claude-review label to request another pass (it is consumed when the review starts, so just add it again each time).

Reviewed at HEAD: d94870b

[github-actions]

Skills: review-packaging-ci, review-topic-backward-compat-and-versioning

Review summary. Replaces the Pillow wheel with pillow-simd v12.2.0.post0 (built from source, SSE4) as the last pip step of the five x86-64 CPU/GPU images, guarded on amd64, and relaxes the Pillow floor to >=12.2.0. I verified: the step is the last pip step in every touched Dockerfile (nothing reinstalls the wheel after it); the GPU images run it in the builder stage and copy /usr/local/lib/python3.1x into runtime; cu13 relies on PIP_BREAK_SYSTEM_PACKAGES=1 (set earlier) so the un-flagged pip uninstall/install still work under PEP 668; builder stages carry a C toolchain + libjpeg/zlib headers (via libvips-dev) so the required-feature source build compiles, and the runtime stages pull the matching .sos transitively through libvips42 — so no silent JPEG-less build. uv.lock was regenerated alongside the pyproject.toml specifier change.

Open items (see the pinned action-item and the inline comment):

• Medium — pillow-simd is installed from a mutable git tag; pin the immutable commit SHA to match this repo other SHA/checksum-pinned source builds (inline on Dockerfile.onnx.cpu:67).
• IMPORTANT question — whether the drop below the deliberate >=12.3.0 floor loses a fix; this blocks sign-off until answered.

Maintainer notice (non-blocking): this touches inference_models/pyproject.toml, so inference-models will need a version bump at release time (contributors never bump it). A one-line ## Unreleased entry in inference_models/docs/changelog.md noting the Pillow floor change would be nice-to-have but is not required for a pin relaxation.

Minor doubt (non-blocking): runtime shared-lib linkage for the source-built extension depends on libvips42 continuing to drag in libfreetype6/libwebp/liblcms2/libopenjp2; the CI image builds + T4/CPU integration runs are the real proof here since it cannot be confirmed statically.

Reviewed at HEAD: d94870b

[leeclemnet]

IMPORTANT question — whether the drop below the deliberate >=12.3.0 floor loses a fix; this blocks sign-off until answered.

The pillow version floor was raised from 12.2.0 to 12.3.0 in #2679. Pillow 12.3.0 fixed 13 security advisories that all affect 12.2.0, and pillow-simd 12.2.0.post0 carries the 12.2.0 code. As of today there is no pillow-simd 12.3.0 fork.

Options:

1. Wait for pillow-simd 12.3.0post0, unknown release schedule
2. Keep pillow floor at 12.3.0, install pillow-simd 12.2.0post0 under an alias, and use it for selected operations in rfdetr preprocessing. None of the 13 advisories touches fromarray or resize, and no decoder, paste, crop, filter, font, or PDF path ever runs on the SIMD copy.

Implemented option 2. Upgrade to pillow-simd 12.3.0post0 when available.

[github-actions]

🤖 Claude review started at commit 206d8cdbb03bd918b9a325e4c3351020c7a15b31.

New commits are not auto-reviewed. Add the claude-review label to request a re-review — the label is consumed when the review starts, so just add it again next time.

[github-actions]

Skills: review-packaging-ci, review-inference-models-pkg, review-topic-prediction-integrity, review-topic-backward-compat-and-versioning, review-topic-test-hygiene

Re-review at HEAD 206d8cd (redesign of the prior approach). Both previously-raised blockers are resolved in code:

• Mutable git tag → pinned commit SHA. All five x86-64 Dockerfiles now pin pillow-simd @ ...@633de896bd7b493e03396d31e1a30f04a7a5ea80.
• Pillow floor drop → withdrawn. requirements/_requirements.txt, requirements/requirements.sdk.http.txt, and inference_models/pyproject.toml all keep pillow>=12.3.0,<13.0.0 (unchanged in this diff). Standard Pillow stays the PIL wheel; Pillow-SIMD is loaded beside it as PILSIMD and reaches only Image.fromarray + resize in the RF-DETR numpy preprocessor. This directly implements the answer given to the earlier IMPORTANT question, verified against the diff.

Verified: the except ImportError fallback catches the loader's ModuleNotFoundError; the fallback resize is byte-for-byte equivalent to the previous TF.resize(..., antialias=True) (torchvision maps that to pil.resize((W,H), BILINEAR)), so arm64 / CPU-from-PyPI / empty-/opt/pillow_simd surfaces are unchanged; every Dockerfile has the repo in the build context at the verify step and persists (CPU/gpu.dev) or copies (gpu/cu13) /opt/pillow_simd into the runtime stage; the SIMD path's ≤1 gray-level drift is disclosed in inference_models/docs/changelog.md.

Maintainer notice (non-blocking). This is a functional change under inference_models/inference_models/** + inference_models/pyproject.toml, so inference-models will need a version bump at release time (contributors never bump it). The ## Unreleased changelog entry is already present.

Reviewed at HEAD: 206d8cd

[github-actions]

😎 PR passes the vibe-check and trust-me-bro verification.