Skip to content

Commit dbfb7c2

Browse files
Merge branch 'develop' into adamg/bitpack-compare
2 parents 9177626 + ac8c751 commit dbfb7c2

219 files changed

Lines changed: 6988 additions & 4932 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/ISSUE_TEMPLATE/epic.md

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
name: Epic
3+
about: Public roadmap umbrella for a major Vortex initiative.
4+
title: "Epic: "
5+
labels: epic
6+
---
7+
8+
<!--
9+
An Epic is a public roadmap umbrella for a major Vortex initiative. It should
10+
explain the goal, why it matters, what is out of scope, and where concrete work
11+
is tracked.
12+
13+
Attach related tracking issues with GitHub's "Create sub-issue" or "Add
14+
existing sub-issue" button. The sub-issue panel tracks progress, so this issue
15+
does not need a checklist.
16+
17+
Keep Epic issues readable. Route discussion elsewhere:
18+
19+
- Open-ended scope or product questions: open a GitHub Discussion and link it
20+
here.
21+
- Design debate scoped to one piece of work: open a tracking issue as a
22+
sub-issue and discuss it there.
23+
- Bug reports: file a bug issue and reference this Epic.
24+
25+
Short clarifying comments are fine. Long threads should be redirected.
26+
-->
27+
28+
This Epic is for ...
29+
30+
<!--
31+
One or two sentences. State the initiative and link any roadmap, design, or
32+
external context that helps readers understand the scope.
33+
-->
34+
35+
## Status
36+
37+
<!--
38+
Proposed / Active / Blocked / Complete. Briefly explain the current state.
39+
-->
40+
41+
## Goal
42+
43+
<!--
44+
One short paragraph covering both:
45+
46+
- What "done" looks like for a user or downstream consumer. Name the concrete
47+
API, format, encoding, or benchmark.
48+
- What people might reasonably expect this Epic to cover but it will not.
49+
Non-goals stop scope creep.
50+
-->
51+
52+
## Motivation
53+
54+
<!--
55+
Why this, why now? Cover:
56+
57+
- The concrete problem and who/what hits it. Examples: a downstream integration
58+
(DataFusion, DuckDB, Python, Java), a benchmark gap (TPC-H, ClickBench,
59+
vortex-bench), or a file format or encoding limitation.
60+
- What is currently blocked, slower, or larger than it should be. Link the
61+
benchmark numbers, profile output, or related issues.
62+
63+
If the answer is "it would be nice to have", the Epic is not ready.
64+
-->
65+
66+
## Unresolved questions
67+
68+
<!--
69+
Scope-level questions that must be answered before this Epic closes.
70+
Implementation questions belong in the relevant tracking issue.
71+
72+
Each entry should be specific enough that a future reader can tell whether it
73+
has been answered. If a question grows large, please create a new tracking
74+
issue or Discussion and link it.
75+
-->
76+
77+
- [ ] None yet.
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
---
2+
name: Tracking Issue
3+
about: Implementation context for work likely to span multiple PRs.
4+
title: "Tracking Issue: "
5+
labels: tracking-issue
6+
---
7+
8+
<!--
9+
Tracking issues are for work that needs shared implementation context and is
10+
likely to span multiple PRs. Use this template when the issue should track
11+
planning, implementation, stabilization, and follow-through.
12+
13+
Route discussion elsewhere:
14+
15+
- New ideas, fit questions, or early design feedback: start with a GitHub
16+
Discussion.
17+
- Bug reports surfaced while implementing: file a bug issue and reference this
18+
tracking issue.
19+
- A design thread big enough to need its own page: spawn a tracking issue (as
20+
a sub-issue of this one if it is narrow) and discuss it there.
21+
22+
Tracking issues should be understandable a year from creation. Short clarifying
23+
comments are fine. Long threads should be redirected.
24+
-->
25+
26+
This is a tracking issue for ...
27+
28+
<!--
29+
One or two sentences. State what the feature is and where it lives in the
30+
codebase (crate, module, file format, encoding). Link the parent Epic, design
31+
doc, prototype PR, or external reference.
32+
-->
33+
34+
## Design
35+
36+
<!--
37+
Optional but recommended. Size to fit the work:
38+
39+
- API additions: a stripped-down signature block. Rust traits or functions,
40+
file-format struct, or wire-format message. Drop doc comments and bodies.
41+
- Behavioral changes: a short before / after description, ideally with a small
42+
example.
43+
- Architectural changes: a paragraph explaining the new shape, or a link to a
44+
design doc.
45+
46+
Skip only if the change is fully obvious from the title. The Design section is
47+
what lets a reviewer judge whether the Steps below are the right ones.
48+
-->
49+
50+
## Steps
51+
52+
<!--
53+
Major checkpoints required to call this feature done. A handful of milestones,
54+
not every PR. Tick steps as they land; do not delete them.
55+
56+
Steps are milestones, not PRs (initial implementation, documentation,
57+
stabilization). If a step is itself a separable shippable unit, promote it to
58+
a sub-issue.
59+
-->
60+
61+
- [ ] Initial implementation
62+
- [ ] Documentation
63+
- [ ] Public API stabilization
64+
65+
## Unresolved questions
66+
67+
<!--
68+
Open design or implementation questions blocking progress. Link discussions
69+
and conclusions so a future reader can see how each question was settled.
70+
71+
If a question is large enough to need its own thread, spawn a tracking issue
72+
or Discussion and link it.
73+
-->
74+
75+
- [ ] None yet.
76+
77+
## Implementation history
78+
79+
<!--
80+
A running log of every PR that touched this feature: initial implementation,
81+
follow-ups, fixes, reverts. Grows continuously and is never ticked off. The
82+
archaeology trail for someone reading this issue a year later.
83+
84+
Add PRs as they merge, in chronological order. A one-line description per PR
85+
is helpful but not required.
86+
-->

.github/actions/alert-incident-io/action.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ inputs:
2727
runs:
2828
using: "composite"
2929
steps:
30-
- uses: incident-io/github-action@v1
30+
- uses: incident-io/github-action@7aa5f85e67679cd8fdf2a19aed3d5450335a004b # v1
3131
with:
3232
api-key: ${{ inputs.api-key }}
3333
alert-source-id: 01KH4EYTH3HA4PDZPRAPEV1Q10

.github/actions/setup-rust/action.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ runs:
3636

3737
- name: Install Mold
3838
if: runner.os == 'Linux'
39-
uses: rui314/setup-mold@v1
39+
uses: rui314/setup-mold@9c9c13bf4c3f1adef0cc596abc155580bcb04444 # v1
4040

4141
- name: Check for rustup
4242
id: check-rustup
@@ -45,7 +45,7 @@ runs:
4545

4646
- name: Rust Toolchain
4747
id: rust-toolchain
48-
uses: dtolnay/rust-toolchain@stable
48+
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
4949
if: steps.check-rustup.outputs.exists != 'true'
5050
with:
5151
toolchain: "${{ steps.toolchain-config.outputs.toolchain }}"
@@ -73,7 +73,7 @@ runs:
7373
7474
- name: Rust Compile Cache
7575
if: inputs.enable-sccache == 'true'
76-
uses: mozilla-actions/sccache-action@v0.0.9
76+
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
7777

7878
- name: Pre-start sccache server
7979
if: inputs.enable-sccache == 'true'

.github/runs-on.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,6 @@
11
_extends: .github-private
2+
images:
3+
ubuntu24-full-arm64-pre:
4+
platform: "linux"
5+
arch: "arm64"
6+
ami: "ami-08f97c6362847dc5d"

.github/workflows/bench-dispatch.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121
timeout-minutes: 10
2222
if: github.event.label.name == 'action/benchmark'
2323
steps:
24-
- uses: actions-ecosystem/action-remove-labels@v1
24+
- uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1
2525
if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex'
2626
with:
2727
labels: action/benchmark
@@ -37,7 +37,7 @@ jobs:
3737
timeout-minutes: 10
3838
if: github.event.label.name == 'action/benchmark-sql'
3939
steps:
40-
- uses: actions-ecosystem/action-remove-labels@v1
40+
- uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1
4141
if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex'
4242
with:
4343
labels: action/benchmark-sql

.github/workflows/bench-pr.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
if: github.event.pull_request.head.repo.fork == false
3939
with:
4040
sccache: s3
41-
- uses: actions/checkout@v6
41+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
4242
with:
4343
ref: ${{ github.event.pull_request.head.sha }}
4444
- name: Setup benchmark environment
@@ -64,7 +64,7 @@ jobs:
6464
6565
- name: Setup Polar Signals
6666
if: github.event.pull_request.head.repo.fork == false
67-
uses: polarsignals/gh-actions-ps-profiling@v0.8.1
67+
uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1
6868
with:
6969
polarsignals_cloud_token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
7070
labels: "branch=${{ github.ref_name }};gh_run_id=${{ github.run_id }};benchmark=${{ matrix.benchmark.id }}"
@@ -83,13 +83,13 @@ jobs:
8383
8484
- name: Setup AWS CLI
8585
if: github.event.pull_request.head.repo.fork == false
86-
uses: aws-actions/configure-aws-credentials@v6
86+
uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6
8787
with:
8888
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
8989
aws-region: us-east-1
9090

9191
- name: Install uv
92-
uses: spiraldb/actions/.github/actions/setup-uv@0.18.5
92+
uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6
9393
with:
9494
sync: false
9595

@@ -117,14 +117,14 @@ jobs:
117117
118118
- name: Comment PR
119119
if: github.event.pull_request.head.repo.fork == false
120-
uses: thollander/actions-comment-pull-request@v3
120+
uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3
121121
with:
122122
file-path: comment.md
123123
comment-tag: bench-pr-comment-${{ matrix.benchmark.id }}
124124

125125
- name: Comment PR on failure
126126
if: failure() && github.event.pull_request.head.repo.fork == false
127-
uses: thollander/actions-comment-pull-request@v3
127+
uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3
128128
with:
129129
message: |
130130
# BENCHMARK FAILED

.github/workflows/bench.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,9 @@ jobs:
1616
runs-on: ubuntu-latest
1717
timeout-minutes: 10
1818
steps:
19-
- uses: actions/checkout@v6
19+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
2020
- name: Setup AWS CLI
21-
uses: aws-actions/configure-aws-credentials@v6
21+
uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6
2222
with:
2323
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
2424
aws-region: us-east-1
@@ -54,7 +54,7 @@ jobs:
5454
if: github.repository == 'vortex-data/vortex'
5555
with:
5656
sccache: s3
57-
- uses: actions/checkout@v6
57+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
5858
- name: Setup benchmark environment
5959
run: sudo bash scripts/setup-benchmark.sh
6060
- uses: ./.github/actions/setup-rust
@@ -77,7 +77,7 @@ jobs:
7777
cargo build --bin ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} --features unstable_encodings
7878
7979
- name: Setup Polar Signals
80-
uses: polarsignals/gh-actions-ps-profiling@v0.8.1
80+
uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1
8181
with:
8282
polarsignals_cloud_token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
8383
labels: "branch=${{ github.ref_name }};gh_run_id=${{ github.run_id }};benchmark=${{ matrix.benchmark.id }}"
@@ -95,7 +95,7 @@ jobs:
9595
bash scripts/bench-taskset.sh target/release_debug/${{ matrix.benchmark.id }} --formats ${{ matrix.benchmark.formats }} -d gh-json -o results.json
9696
9797
- name: Setup AWS CLI
98-
uses: aws-actions/configure-aws-credentials@v6
98+
uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6
9999
with:
100100
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
101101
aws-region: us-east-1

0 commit comments

Comments
 (0)