Add RAPIDS-based devcontainer for velox/cuDF/presto development#250
Open
bdice wants to merge 32 commits intorapidsai:mainfrom
Open
Add RAPIDS-based devcontainer for velox/cuDF/presto development#250bdice wants to merge 32 commits intorapidsai:mainfrom
bdice wants to merge 32 commits intorapidsai:mainfrom
Conversation
bdice
force-pushed
the
devcontainer
branch
2 times, most recently
from
00f1a8c to
284ac0c
Compare
Contributor
|
This PR looks good to me. Let’s get it merged. |
Avinash-Raj
approved these changes
Mar 26, 2026
Contributor
|
It would be better if we had a dev container script to start Presto and run Presto integration tests. Maybe we could add those on a follow up PR. |
devavret
requested changes
Apr 1, 2026
scripts/devcontainer/build-velox
Outdated
| BUILD_TYPE="${BUILD_TYPE:-release}" | ||
| NUM_THREADS="${NUM_THREADS:-$(nproc --all --ignore=1)}" | ||
| BUILD_BASE_DIR="${BUILD_BASE_DIR:-/opt/velox-build}" | ||
| CUDA_ARCH="${CUDA_ARCH:-}" |
Contributor
There was a problem hiding this comment.
I tried setting this to a specific arch and got an error:
Unknown option: --cuda-arch
Usage: configure-velox [OPTIONS] [-D<VAR>=<VALUE>]...
Run CMake configuration for velox without building.
Useful for IDE integration (CMake Tools, clangd, etc.)
Any -D flags are passed through to CMake, e.g.:
configure-velox -DVELOX_ENABLE_BENCHMARKS=ON
Options:
--release Configure for release build (default)
--debug Configure for debug build
-h, --help Show this help message
Environment variables:
BUILD_TYPE release or debug
BUILD_BASE_DIR Base directory for build output (default: /opt/velox-build)
CUDAARCHS CUDA architectures (default: RAPIDS)
I'm not sure if related but the env var to set cuda arch is named differently in build-velox and configure-velox
Adds devcontainer configurations for CUDA 12.9 and 13.1 based on RAPIDS devcontainers infrastructure. Includes helper scripts for building velox, presto, and RAPIDS libraries (rmm, cudf, kvikio). Key features: - Based on rapidsai/devcontainers with UCX and OpenMPI - Integrates rapids-build-utils for build-rmm, build-cudf, etc. - Auto-detects RAPIDS library paths for velox cmake configuration - sccache with S3 remote caching and sccache-dist support - Persisted venvs, pip cache, and bash history across rebuilds - Codespaces support
VELOX_DEPENDENCY_SOURCE is read from env by CMake, but individual overrides like GTest_SOURCE, cudf_SOURCE, duckdb_SOURCE, faiss_SOURCE must be passed as -D flags to CMake.
- Fix DuckDB_SOURCE case sensitivity (DuckDB vs duckdb) - Resolve CUDAARCHS from RAPIDS to actual architecture list for CUDA 13.x - Use prebuilt cudf (SYSTEM) when available, fall back to BUNDLED - Add BUNDLED sources: simdjson, FastFloat, folly, absl, gRPC, xsimd, Arrow, geos - Add system packages: libfmt-dev, libprotobuf-dev, libprotoc-dev, libc-ares-dev, libstemmer-dev, libboost-all-dev, bison, flex, protobuf-compiler
Standalone velox now bundles all dependencies (folly, xsimd, etc.) with no prerequisites. FB OSS deps (folly, fbthrift, proxygen) are built automatically inside build-presto on first run and cached at /opt/fb-deps. - Extract shared constants and RAPIDS detection to _common.sh - Fix xsimd_SOURCE override bug in configure-velox (BUNDLED always won) - Fix CUDAARCHS not propagating to presto build (was setting wrong var) - Fix test-presto pointing at nonexistent release/ subdirectory - Add ldconfig for /opt/fb-deps/lib so presto tests find shared libs - Add libxxhash-dev and gperf to Dockerfile (fbthrift build deps) - Add -no-pie linker flag and -Wno-error=nonnull for presto link - Add devcontainer README
Replace === heading === banners with bold-cyan/bold-green ANSI headings matching the build-rmm / build-cudf pattern: echo -e "\033[1;36m<Section>\033[0m" # cyan heading echo -e "\033[1;32m<Completion>\033[0m" # green completion
…TINES Dep-skip guard now also tests for libproxygen.so so a partial install (folly present, proxygen absent) correctly triggers a full dep rebuild. Pass -UFOLLY_CFG_NO_COROUTINES in CMAKE_CXX_FLAGS to undo the global add_compile_definitions in presto's CMakeLists.txt. That define forces FOLLY_HAS_COROUTINES=0, which breaks fbthrift headers (BiDiStream.h etc.) that unconditionally reference folly::coro::AsyncGenerator. Velox has no folly::coro usage so removing the define for all TUs is safe.
This reverts commit d72679f.
Rename CUDA_ARCH to CUDAARCHS in build-velox to match the CMake standard env var used by configure-velox. Add --cuda-arch CLI flag to configure-velox so it can be called directly or forwarded from build-velox.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
build-all,clean-all,configure-all) that orchestrate RAPIDS builds alongside velox and presto-native-execution