Skip to content

feat(cudf): Support map construction and element lookup - #61

Open
winningsix wants to merge 1 commit into
HighPerfDataAccelerator:devfrom
winningsix:feat/cudf-map-access
Open

feat(cudf): Support map construction and element lookup#61
winningsix wants to merge 1 commit into
HighPerfDataAccelerator:devfrom
winningsix:feat/cudf-map-access

Conversation

@winningsix

Copy link
Copy Markdown
Collaborator

Summary

  • add cuDF support for map construction from alternating key/value expressions
  • represent map columns as LIST<STRUCT<K,V>> and preserve sliced offsets and parent validity
  • route Spark and Presto element_at map lookups through cuDF
  • support literal and column keys, return null for missing keys, and preserve null map/key semantics
  • keep unsupported key/value shapes fail closed

Testing

  • added GPU selection coverage for literal and column map keys
  • added Spark expression parity coverage for map construction, present/missing keys, null values, and dynamic keys
  • clang-format 18.1.8 validation

@github-actions github-actions Bot added the cudf label Aug 17, 2026
@winningsix winningsix changed the title feat(cudf): support map construction and element lookup feat(cudf): Support map construction and element lookup Aug 17, 2026
@sperlingxx

Copy link
Copy Markdown
Collaborator

Plz fix the format issues


Run pre-commit run --show-diff-on-failure --color=always --all-files
2m 19s
Run pre-commit run --show-diff-on-failure --color=always --all-files
    pre-commit run --show-diff-on-failure --color=always --all-files
    shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  [INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
  [INFO] Initializing environment for local:clang-tidy==18.1.8.
  [INFO] Initializing environment for local.
  [INFO] Initializing environment for local:regex.
  [INFO] Initializing environment for https://github.com/BlankSpruce/gersemi.
  [INFO] Initializing environment for https://github.com/pre-commit/mirrors-clang-format.
  [INFO] Initializing environment for https://github.com/astral-sh/ruff-pre-commit.
  [INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
  [INFO] Initializing environment for https://github.com/scop/pre-commit-shfmt.
  [INFO] Initializing environment for https://github.com/adrienverge/yamllint.git.
  [INFO] Initializing environment for https://github.com/google/yamlfmt.
  [INFO] Initializing environment for https://github.com/zizmorcore/zizmor-pre-commit.
  [INFO] Initializing environment for https://github.com/mpalmer/action-validator.
  [INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for local.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for local.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for local.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/BlankSpruce/gersemi.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/pre-commit/mirrors-clang-format.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/astral-sh/ruff-pre-commit.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/shellcheck-py/shellcheck-py.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/scop/pre-commit-shfmt.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/adrienverge/yamllint.git.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/google/yamlfmt.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/zizmorcore/zizmor-pre-commit.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  [INFO] Installing environment for https://github.com/mpalmer/action-validator.
  [INFO] Once installed this environment will be reused.
  [INFO] This may take a few minutes...
  Check hooks apply to the repository......................................Passed
  Check for useless excludes...............................................Passed
  trim trailing whitespace.................................................Passed
  fix end of files.........................................................Passed
  check for added large files..............................................Passed
  check that executables have shebangs.....................................Passed
  check that scripts with shebangs are executable..........................Failed
  - hook id: check-shebang-scripts-are-executable
  - exit code: 1
  
  scripts/ci/Jenkinsfile.premerge: has a shebang but is not marked executable!
    If it is supposed to be executable, try: `chmod +x scripts/ci/Jenkinsfile.premerge`
    If on Windows, you may also need to: `git add --chmod=+x scripts/ci/Jenkinsfile.premerge`
    If it not supposed to be executable, double-check its shebang is wanted.
  
  clang-tidy...............................................................Passed
  check-header-ownership...................................................Passed
  check-readme-blogs.......................................................Failed
  - hook id: check-readme-blogs
  - exit code: 1
  - files were modified by this hook
  
  Updated recent blog posts in README.md
  
  check-workflow-images....................................................Passed
  license-header...........................................................Passed
  CMake formatter..........................................................Failed
  - hook id: gersemi
  - files were modified by this hook
  
  Warning: unknown command 'velox_add_grouped_tests' used at:
  /home/runner/work/velox/velox/velox/common/caching/tests/CMakeLists.txt:46:1
  
  Warning: unknown command 'velox_configure_clang_atomic_linker_flags' used at:
  /home/runner/work/velox/velox/CMakeLists.txt:550:1
  
  Warning: unknown command 'velox_add_grouped_tests' used at:
  /home/runner/work/velox/velox/velox/exec/tests/CMakeLists.txt:199:1
  /home/runner/work/velox/velox/velox/exec/tests/CMakeLists.txt:302:1
  
  Warning: unknown command 'velox_add_grouped_tests' used at:
  /home/runner/work/velox/velox/velox/functions/prestosql/aggregates/tests/CMakeLists.txt:89:1
  
  Warning: unknown command 'velox_add_grouped_tests' used at:
  /home/runner/work/velox/velox/velox/serializers/tests/CMakeLists.txt:58:1
  
  clang-format.............................................................Failed
  - hook id: clang-format
  - files were modified by this hook
  ruff.....................................................................Passed
  ruff-format..............................................................Passed
  shellcheck...............................................................Passed
  shfmt....................................................................Passed
  yamllint.................................................................Failed
  - hook id: yamllint
  - exit code: 1
  
  .github/workflows/blossom-ci.yml:23:22: [error] string value is redundantly quoted with any quotes (quoted-strings)
  .github/workflows/blossom-ci.yml:26:22: [error] string value is redundantly quoted with any quotes (quoted-strings)
  .github/workflows/blossom-ci.yml:59:22: [error] string value is redundantly quoted with any quotes (quoted-strings)
  .github/workflows/blossom-ci.yml:107:22: [error] string value is redundantly quoted with any quotes (quoted-strings)
  .github/workflows/blossom-ci.yml:119:22: [error] string value is redundantly quoted with any quotes (quoted-strings)
  
  yamlfmt..................................................................Failed
  - hook id: yamlfmt
  - files were modified by this hook
  zizmor...................................................................Failed
  - hook id: zizmor
  - exit code: 14

VELOX_CHECK_EQ(nextInputColumnIndex, inputColumns.size());

auto keys =
cudf::interleave_columns(cudf::table_view(keyViews), stream, mr);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this preserves duplicate keys, while the Spark CPU MapFunction either applies LAST_WIN or throws when throw_exception_on_duplicate_map_keys is enabled. Since the new map lookup uses FIND_FIRST, element_at(map("k", c0, "k", c1), "k") returns c0 on GPU but c1 on CPU. Please implement the same duplicate-key policy (including the exception mode), or fail GPU selection when uniqueness cannot be guaranteed, and add literal/dynamic duplicate-key parity tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants