Local evals tutorial fails with generic 'Engine core initialization failed' when Python dev headers are missing

[kamran-rapidfireAI]

Summary

Running tutorial_notebooks/rag-contexteng/rf-tutorial-rag-fiqa.ipynb locally (Linux, non-Colab) fails at experiment.run_evals(...) with:

• RayTaskError(RuntimeError)
• Failed to initialize pipeline: RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The top-level error is not actionable and hides the real root cause.

Repro

1. Open and run tutorial_notebooks/rag-contexteng/rf-tutorial-rag-fiqa.ipynb locally.
2. Execute:

results = experiment.run_evals(
    config_group=config_group,
    dataset=fiqa_dataset,
    num_actors=1,
    num_shards=4,
    seed=42,
)

3. Observe actor init failure with generic engine-core message.

Actual Root Cause (from Ray worker logs)

In Ray worker stderr, the underlying failure is:

fatal error: Python.h: No such file or directory

and then Triton/vLLM fails while compiling runtime CUDA helper code:

CalledProcessError ... /usr/bin/gcc ... -I/usr/include/python3.12 ... returned non-zero exit status 1

Machine was missing Python dev headers package (python3.12-dev / python3-dev).

Why this happens

vllm + torch._inductor + Triton can compile native/CUDA helper modules at runtime. On Linux, this requires Python C headers (Python.h) and compiler toolchain. If headers are missing, model engine init fails.

Confirmed Fix

Installing Python dev headers resolved the issue:

sudo apt-get update
sudo apt-get install -y python3.12-dev

(Equivalent distro package names apply, e.g. python3-dev.)

Requested Improvements

1. Preflight dependency checks before actor/model init in evals mode:
   • verify Python.h exists (sysconfig.get_paths()["include"]/Python.h)
   • verify compiler exists (gcc/cc)
   • fail fast with actionable install instructions.
2. Improve surfaced error message in QueryProcessingActor.initialize_for_pipeline path:
   • include root cause snippet from worker stderr (not only generic “Engine core initialization failed”).
3. Docs update for local/tutorial setup (non-Colab):
   • Linux prerequisites should explicitly include Python dev headers and build essentials.
4. Add CI smoke test for missing headers scenario:
   • assert user-facing error is clear and actionable.

Environment

• OS: Linux
• Python: 3.12
• vLLM: 0.10.2
• GPU: NVIDIA L4

User Impact

This is a first-run blocker for local users and can be mistaken for GPU/vLLM incompatibility. Better guardrails and messaging would significantly improve onboarding and support load.

[kamran-rapidfireAI]

Adding a deterministic repro checklist so anyone on the team can quickly validate.

Repro Checklist (Linux)

1. Base environment details

cat /etc/os-release
uname -a
python -V

2. Dependency snapshot (from the same venv used for notebook)

pip freeze | grep -E "rapidfireai|vllm|torch|triton|ray"

3. Precondition: verify missing headers (expected failure state)

ls /usr/include/python3.12/Python.h
# expected before fix: No such file or directory

4. Minimal non-notebook repro (optional but faster than full tutorial)

python - <<'PY'
from vllm import LLM
llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct", dtype="half", tensor_parallel_size=1)
print("LLM init ok")
PY

Expected in failing state: error chain includes fatal error: Python.h: No such file or directory and gcc compile failure.

5. Notebook repro (current tutorial path)
   Run tutorial_notebooks/rag-contexteng/rf-tutorial-rag-fiqa.ipynb and execute:

results = experiment.run_evals(
    config_group=config_group,
    dataset=fiqa_dataset,
    num_actors=1,
    num_shards=4,
    seed=42,
)

6. Capture root-cause logs from Ray

# most recent Ray session
RAY_LOG_DIR=$(ls -td /tmp/ray/session_*/logs | head -1)

# largest worker stderr often contains root cause
ls -S "$RAY_LOG_DIR"/worker-*.err | head -3

tail -n 200 $(ls -S "$RAY_LOG_DIR"/worker-*.err | head -1)

Please capture both lines if present:

• fatal error: Python.h: No such file or directory
• failing gcc ... -I/usr/include/python3.12 ...

7. Apply fix and verify

sudo apt-get update
sudo apt-get install -y python3.12-dev
ls /usr/include/python3.12/Python.h

Then rerun step (4) and/or step (5). Expected: engine init succeeds.

---

If we implement a preflight check, this scenario should fail fast with an actionable message before Ray actor init.