Repository navigation

String expressions on the compute engine, and utf8() to query parity - #686

Merged

FrancescAlted merged 86 commits into

mainfrom

dsl-string-support

Jul 30, 2026

Merged

String expressions on the compute engine, and utf8() to query parity#686
FrancescAlted merged 86 commits into
mainfrom
dsl-string-support

Commits on Jul 26, 2026
- Phase 1f-1g: size string results from miniexpr, not from NumPy
  
  FrancescAlted
  and
  claude
  committed
  4598b43
- Phase 1h (blosc2 side): lower Python string syntax to the DSL grammar
  
  FrancescAlted
  and
  claude
  committed
  9a310e6
- Size string DSL kernel output from miniexpr, and check it
  
  FrancescAlted
  and
  claude
  committed
  0f92c73
- docs: release note for string expressions and DSL kernels
  
  FrancescAlted
  and
  claude
  committed
  9d5ec14
- Phase 2 (blosc2 side): route bytes ('S') arrays to miniexpr
  
  FrancescAlted
  and
  claude
  committed
  6f5c9fe
- Compile row["colname"] kernels that branch, and let strings through
  
  FrancescAlted
  and
  claude
  committed
  729a208
Commits on Jul 27, 2026
- Fix miniexpr prefilter reads for operands wider than 255 bytes
  
  FrancescAlted
  and
  claude
  committed
  cbcb15b
- Phase 3: string expressions over utf8 columns, plus utf8_array()
  
  FrancescAlted
  and
  claude
  committed
  cd6317d
- docs: utf8() no longer disclaims string-expression filters
  
  FrancescAlted
  and
  claude
  committed
  332ac40
- Rename the utf8_array module to _utf8_array
  
  FrancescAlted
  and
  claude
  committed
  2638a0d
- Phase 4: answer utf8 scalar predicates with a raw-byte scan
  
  FrancescAlted
  and
  claude
  committed
  47b033c
- docs: release note for the utf8 scalar-predicate fast path
  
  FrancescAlted
  and
  claude
  committed
  cfd3898
- Fix string indexes returning zero rows at the default column width
  
  FrancescAlted
  and
  claude
  committed
  0546293
- docs: release note for the string-index zero-rows fix
  
  FrancescAlted
  and
  claude
  committed
  c8fd3f5
- Sweep the capped-typesize family: one helper, plus SChunk.get_slice
  
  FrancescAlted
  and
  claude
  committed
  393a200
- docs: release note for the wide-typesize SChunk slice fix
  
  FrancescAlted
  and
  claude
  committed
  ff63d7a
- Link the SChunk wide-typesize workaround to Blosc/c-blosc2#796
  
  FrancescAlted
  and
  claude
  committed
  6edf058
- Drop the wide-typesize get_slice() workaround, fixed upstream
  
  FrancescAlted
  and
  claude
  committed
  9bb345d
- Read chunk sub-ranges via blosc2_getitem_bytes_ctx()
  
  FrancescAlted
  and
  claude
  committed
  7cbd48b
- Bump internal C-Blosc2 and miniexpr deps
  FrancescAlted
  committed
  e3cd570
- utf8 span driver: return a Utf8Array for string results
  
  FrancescAlted
  and
  claude
  committed
  9cb490a
- Bump miniexpr to 9e9b8d9 (varlen string output)
  
  FrancescAlted
  and
  claude
  committed
  25d9e64
- Add compute_varlen(): Arrow varlen results from string expressions
  
  FrancescAlted
  and
  claude
  committed
  f6b0643
- Revert "Add compute_varlen(): Arrow varlen results from string expressions"
  FrancescAlted
  committed
  4e36490
- Restore the string benchmark dropped by the compute_varlen revert
  
  FrancescAlted
  and
  claude
  committed
  615700d
- Bump miniexpr to 5a7de4f (width-preserving upper/lower)
  
  FrancescAlted
  and
  claude
  committed
  7ec1751
- Stop expression results from losing the code-unit shuffle width
  
  FrancescAlted
  and
  claude
  committed
  bbb94f4
- bench: size the string benchmark's blocks for the result, not the operands
  
  FrancescAlted
  and
  claude
  committed
  f78f657
- bench: refresh the string-ops results after the three fixes
  
  FrancescAlted
  and
  claude
  committed
  df04137
- bench: use LZ4-5 for the blosc2 string rows
  
  FrancescAlted
  and
  claude
  committed
  ef9a6cf
- bench: final string-ops plot (24.3M rows, LZ4-5)
  
  FrancescAlted
  and
  claude
  committed
  2835d60
- bench: drop SHUFFLE for the string rows too
  
  FrancescAlted
  and
  claude
  committed
  eb2ff11
- bench: present the filter choice as the tradeoff it is
  
  FrancescAlted
  and
  claude
  committed
  56ed4ff
- bench: keep SHUFFLE at the UCS4 width, take throughput from the codec
  
  FrancescAlted
  and
  claude
  committed
  0ea4720
- bench: regenerate the plot from a full all-engines run
  
  FrancescAlted
  and
  claude
  committed
  a1473b4
- Add the regression test for the 4096-element eval-block bug
  
  FrancescAlted
  and
  claude
  committed
  c5dc1d0
- Repoint miniexpr at bd2c602
  
  FrancescAlted
  and
  claude
  committed
  bf9b484
Commits on Jul 28, 2026
- Cache the dictionary code->value map instead of decoding per row
  
  FrancescAlted
  and
  claude
  committed
  a944684
- Stop bucket indexes from costing more than the scan they replace
  
  FrancescAlted
  and
  claude
  committed
  92c39aa
- Assess the string column flavours against measurements
  
  FrancescAlted
  and
  claude
  committed
  12c5629
- Update bucket index tests for the block-selectivity gate
  
  FrancescAlted
  and
  claude
  committed
  64d4a76
- Give Utf8Array real comparison operators
  
  FrancescAlted
  and
  claude
  committed
  3692673
- Record the Utf8Array comparison fix in the assessment
  
  FrancescAlted
  and
  claude
  committed
  174eb7c
- Route bare Utf8Array expressions through the span driver
  
  FrancescAlted
  and
  claude
  committed
  0b486b0
- Record the G4 fix in the assessment
  
  FrancescAlted
  and
  claude
  committed
  d04badc
- Give dictionary and varlen scalar columns element-wise comparisons
  
  FrancescAlted
  and
  claude
  committed
  486be88
- Record the dictionary/vlstring comparison fix in the assessment
  
  FrancescAlted
  and
  claude
  committed
  9a9c176
Commits on Jul 29, 2026
- Support create_index on utf8 columns via alphabetical ranks
  
  FrancescAlted
  and
  claude
  committed
  b1bbc54
- Record the utf8 rank index in the assessment
  
  FrancescAlted
  and
  claude
  committed
  3fac526
- Answer utf8 scalar predicates from the rank index
  
  FrancescAlted
  and
  claude
  committed
  f132d6d
- Record indexed utf8 predicates in the assessment
  
  FrancescAlted
  and
  claude
  committed
  93b45ac
- Serve dictionary equality from the rank index, and fix != while there
  
  FrancescAlted
  and
  claude
  committed
  1c22fdf
- Record dictionary index predicates in the assessment
  
  FrancescAlted
  and
  claude
  committed
  7174109
- Update the user-facing docs for the string index work
  
  FrancescAlted
  and
  claude
  committed
  e707c91
- Mark the doc update done in the assessment
  
  FrancescAlted
  and
  claude
  committed
  e273ef1
- Let add_column() fill a new column from values=
  
  FrancescAlted
  and
  claude
  committed
  f8af071
- Publish the utf8 <-> fixed-width conversion pair
  
  FrancescAlted
  and
  claude
  committed
  5b31abe
- Record the conversion pair in the assessment
  
  FrancescAlted
  and
  claude
  committed
  2b23b07
- Fix index-summary min()/max(), and restrict rank indexes to kind=FULL
  
  FrancescAlted
  and
  claude
  committed
  9b5eaba
- Make the utf8 compute refusals route instead of just refusing
  
  FrancescAlted
  and
  claude
  committed
  8e3868b
- Record the error-message routing in the assessment
  
  FrancescAlted
  and
  claude
  committed
  c712c95
- Refresh the capability matrix for the routed errors
  
  FrancescAlted
  and
  claude
  committed
  93cea90
- Dispatch the array constructors on NumPy's StringDType
  
  FrancescAlted
  and
  claude
  committed
  0ed3823
- Record the StringDType dispatch decision in the assessment
  
  FrancescAlted
  and
  claude
  committed
  246dcf5
- Rename Utf8Array to UTF8Array
  
  FrancescAlted
  and
  claude
  committed
  cacac07
- Rename Utf8Spec to UTF8Spec, keeping the old name as an alias
  
  FrancescAlted
  and
  claude
  committed
  8d0eca3
- Keep the string tests running on NumPy 1.26
  
  FrancescAlted
  and
  claude
  committed
  7eca6f6
- Let nested (dotted) utf8 leaves be filtered
  
  FrancescAlted
  and
  claude
  committed
  86d466f
- Record the G2/G3/G4 withdrawal in the utf8 parity plan
  
  FrancescAlted
  and
  claude
  committed
  eaeaaed
- Close item 5 of the assessment, and refile nested leaves
  
  FrancescAlted
  and
  claude
  committed
  ccd82c4
- Sort dictionary columns by rank instead of decoded strings
  
  FrancescAlted
  and
  claude
  committed
  d41d92e
- Record the dictionary rank sort in the assessment
  
  FrancescAlted
  and
  claude
  committed
  f5838d5
- Address the Copilot review on PR #684
  
  FrancescAlted
  and
  claude
  committed
  7a322dd
- Keep test names under 50 characters
  
  FrancescAlted
  and
  claude
  committed
  de99ca8
- Rewrite the string-type decision path around the real trade-off
  
  FrancescAlted
  and
  claude
  committed
  cb28624
- Correct what an over-long max_length actually does
  
  FrancescAlted
  and
  claude
  committed
  26c7052
- Check add_column(values=) against the declared constraints
  
  FrancescAlted
  and
  claude
  committed
  44fda19
- Address the Copilot review on PR #686
  
  FrancescAlted
  and
  claude
  committed
  d673c80
- Size the rank indexes by physical extent, not the live row count
  
  FrancescAlted
  and
  claude
  committed
  e517bd7
Commits on Jul 30, 2026
- Count blocks, not chunks, when a bucket is as wide as a block
  
  FrancescAlted
  and
  claude
  committed
  2ac13b1
- Attach jit hints as notes instead of rebuilding the exception
  
  FrancescAlted
  and
  claude
  committed
  9a3efa6
- Key the string-width cache on the operand dtypes too
  
  FrancescAlted
  and
  claude
  committed
  cf45079
- Drop the stale nested-utf8-leaf disclaimer from utf8()
  
  FrancescAlted
  and
  claude
  committed
  672c102
- Assert the string fill is flat in n, not under a fixed ceiling
  
  FrancescAlted
  and
  claude
  committed
  92788a6
- Stand the min/max shortcut down on any hole, not just fresh ones
  
  FrancescAlted
  and
  claude
  committed
  e7b336e
- Make a string result refuse clashing utf8 null sentinels
  
  FrancescAlted
  and
  claude
  committed
  c5f5823

Commits

Commits on Jul 26, 2026

Commits on Jul 27, 2026

Commits on Jul 28, 2026

Commits on Jul 29, 2026

Commits on Jul 30, 2026