.writeEmptyParquet()/.writeParquetDataset()hardcodedcompression = "zstd", which errors on a "minimal"arrowbuild without zstd codec support (seeDuckDBDataFrame#1). Now useDuckDBDataFrame::resolveParquetCompression(), which falls back tosnappy(with no compression level, since a level paired with a non-zstd codec errors on both DuckDB andarrow) when zstd is unavailable, and still prefers zstd otherwise.
- Fixed two Bioconductor
nebbiolo2build-reportNOTEs. Vendored a local.extract_Nindex_from_syscall()helper (used by the[methods forDuckDBArraySeed,DuckDBArray, andDuckDBMatrix) instead of calling unexportedS4Arrays:::extract_Nindex_from_syscall(). Replaced unanchored\linkS4class{ArrayGrid}/\link{DuckDBTable-class}Rd cross-references (to classes defined inS4ArraysandDuckDBDataFramerespectively) with explicit\link[pkg]{...}package anchors.
.check_gram_size()(DuckDBMatrix-utils.R), the size tripwire run before acrossprod(x)/tcrossprod(x)self-join, callednzcount(x@seed)to bound the self-join unconditionally. Ifxwas itself the lazy, uncollected result of a priorcrossprod()/tcrossprod()call (these deliberately return a still-lazyDuckDBMatrix),nzcount()re-executed that entire upstream join + GROUP BY, embedded as a subquery, just to answer a count, e.g.G <- tcrossprod(pqmat); tcrossprod(G)ranpqmat's self-join twice..check_gram_size()now first checks a matrix's shape (prod(dim(x))), which is always a valid, zero-query upper bound on its true nonzero count, and only falls back to the realnzcount()query when that cheap bound alone cannot already prove the operation is within budget. On a chained lazy Gram matrix built from a 600 x 100, 20%-dense fixture, this measures a 580x-840x median wall-clock speedup for the size check alone across repeated runs (15 reps each; seetest-DuckDBMatrix.R); an earlier pass had reported 205x from a single run. Re-measurement (with the fixture rewritten under a density-awarewriteCoordArray()block configuration, per the methodology note on the.marginVars()entry below) found the new path's cost is a pure shape/metadata check with no query at all (~0.14 ms, flat across every run and both block configurations), so the fix is structural (1 query -> 0 queries) rather than throughput-per-row and is insensitive to Parquet partition/row-group layout: writing the fixture under corrected vs. uncorrected block settings changed the measured speedup only from 813x to 839x, well within run-to-run noise (both writes landed in a single partition at this fixture size, so this particular case never actually exercised the block-shape distortion either way). What did move the number run-to-run was ambient DuckDB thread count / machine load, which changes how expensive the eliminated old-path self-join happens to be (580x-840x across sessions with the new path pinned at ~0.14 ms throughout); 205x sits below that range but is the same phenomenon, not evidence of a smaller real effect.
.duckdb_matmult_copy()(DuckDBMatrix-utils.R), used by the%*%/crossprod(x, v)-style matrix-vector product pushdown to register the vector operand as a temp table before joining it against the underlying data, calledcopy_to()with its defaults (analyze = TRUE, in_transaction = TRUE), adding an unconditional BEGIN/CREATE TEMP TABLE/ ANALYZE/COMMIT sequence for a table that is joined exactly once and immediately discarded. It now passesanalyze = FALSE, in_transaction = FALSE(both plain pass-through argumentscopy_to()/db_copy_to.DBIConnection()already accept, not a new dependency). On the fixture sizes measured (a 500 x 400 and a 20000 x 5000 sparse matrix), this overhead was a small fraction of totalcopy_to()cost (measured 1.00-1.04x), so this is primarily a hygiene fix (removing DDL/transaction overhead that buys nothing for a single-use table) rather than a measurable win at these scales; the benefit should grow with call frequency and under lock contention from concurrent matrix-vector products, which were not measured here.dbplyr::copy_inline()was considered as an alternative (avoiding the round trip entirely) and rejected per an earlier evaluation: DuckDB's SQL parser re-parsing inlined literals was found to cost more than its native zero-copy dataframe registration viacopy_to(), making it slower overall despite fewer round trips..marginVars()'sfill != 0branch (DuckDBTable-matrixStats.R, used byrowVars/colVars/rowSds/colSdson a non-zero-filledDuckDBArray/DuckDBMatrix) was a genuine two-pass query: agroup_by/summarizeto compute each group's mean, aleft_joinback to the underlying data, and a secondgroup_by/summarizeon squared deviations, scanning the underlying data twice. It has been generalized to a single query, reusing the samen = COUNT(*),sum_y = SUM(y),S2 = COALESCE(VAR_SAMP(y), 0)native aggregates already used by the (already single-pass)fill == 0case, and folding in the implicit fill-valued elements analytically via the law of total variance.VAR_SAMP(DuckDB's numerically stable aggregate) is still the sole source of the stored-value variance; no hand-rolled sum-of-squares is introduced. On a 2000 x 300, ~60%-dense fixture (fill = 7,rowVars), this measures a 1.6x-1.7x median wall-clock speedup over the old two-pass query (15-25 reps; seetest-DuckDBTable-matrixStats.R); on a larger, more realistic single-cell- like 20000 x 5000, 2%-dense fixture, 1.8x-1.85x. An earlier pass had reported 1.57x from the smaller fixture alone.
- Documented a real, previously-unstated inconsistency: a
DuckDBArraySeed'skeycolsare implicitly expected to uniquely identify each row (each(i, j)key combination appears at most once), but this was not written down anywhere, and the invariant is not enforced at construction time. If it is violated, every materialization path behaves differently: thedgCMatrixcoercion (DuckDBArray-class.R'ssetAs(..., "dgCMatrix", ...), viaMatrix::sparseMatrix(..., repr = "C")) silently SUMS duplicate(i, j)entries, whileas.matrix(),as(x, "COO_SparseArray"), andas(x, "SparseArray")silently keep the LAST one. Added a "Keycol uniqueness" section to?DuckDBArrayand?DuckDBArraySeed, plus comments at.collectCOO()and thedgCMatrixcoercion itself. This is documentation only:sparseMatrix(..., use.last.ij = TRUE)(to make thedgCMatrixpath match the others) was measured 7-10x slower on the normal, invariant-holding case, and an eager duplicate-key guard was measured to cost 159% of the construction it would protect; both are an unacceptable cost to guard against a case that is supposed to be structurally impossible.
rowDeviances(x, family = "poisson")'s sparse path (.poissonDevianceBlock,rowDeviances.R) computedv * log(v / sz[j])directly on adgCMatrix's stored values with no zero guard, unlike the dense path three lines above it (ratio[x == 0] <- 1 # so log(1) = 0). AdgCMatrixis not required to bedrop0()'d; an explicitly stored zero hit this unguarded branch aslog(0) = -Inf, so0 * -Inf = NaN, which then propagated throughrowSumsinto that gene's deviance and was silently zeroed by the function's owndev[is.na(dev)] <- 0cleanup, reporting a plausible-looking0instead of the correct value. Found while checking BiocDuckDB against an independently discovered deviance bug of the same shape in an unrelated codebase (stored zeros/saturated counts silently returning 0 instead of the correct likelihood). The productionDuckDBMatrixmethod was never affected: its generated SQL already guards both families withif (value > 0) ... else 0, so it never evaluateslog(0). Only an in-memorydgCMatrixwith an explicitly stored zero (never produced byas(x, "dgCMatrix")from a dense matrix, but not excluded by the class itself) could trigger this. A swept check of every other sparse/dense statistical computation in the BiocDuckDB suite (scran.R,scuttle.R, the matrix-stats files) found no other instance of this pattern; it is an isolated case, not a systemic one, sincerowDeviances.Ris the only file in the suite that does manual arithmetic on adgCMatrix's stored-value slot rather than pushing the computation down as SQL.
Ops(DuckDBArray, atomic)/Ops(atomic, DuckDBArray)(DuckDBArray-utils.R) only ever supported a length-1atomicoperand; anything longer fell through toOps(DuckDBTable, atomic)'s scalar-only check and errored with "can only perform binary operations with a scalar value". This was latent but harmless as long as every caller realized a%*%/crossprodresult (itself a lazyDuckDBMatrixby design, see 0.99.9) before combining it with a plain vector. CRAN'sirlba2.4.1 (2026-10-05) rewrote its Lanczos loop and, while doing so, dropped thedrop()call that used to wrapmult()'s result before the subsequentF <- F - subcentering step; for ordinary dense/sparse matrices this is invisible, since%*%already returns a realized object, but it leftFas a lazyDuckDBMatrixforDuckDBMatrixinput, so centered SVDs (BiocSingular::runSVD(center = TRUE), and anything built on it:scran::fixedPCA(),scater::runPCA()) started failing. Found via BiocDuckDB'sDuckDBIrlbaParam/SVD composition tests immediately after the 2.4.1 release. Fixed by having these twoOpsmethods realizee1/e2viaas.array()and hand off to base R whenever the atomic operand isn't a true scalar, since elementwise arithmetic between two plain vectors is never something base R itself defers.
- Documented
writeCoordArray()'s partition sizing for large or sparse arrays: thegrid = NULLdefault (defaultAutoGrid(COO_SparseArray(dim(x)))) budgets partitions fromdim(x)alone, with no sparsity awareness; the density correction viasetAutoBlockSize()/setAutoBlockShape("scale"); and the hard ceiling onblock.length(.Machine$integer.max) beyond which no such tuning can reach a given partition-count target.createDimTables()cross-references this. Added a matching vignette section with a worked example.
%*%/crossprod/tcrossprod's vector-multiply helpers (.matmult_DuckDBMatrix_vector/.matmult_vector_DuckDBMatrix,DuckDBMatrix-utils.R) named theirleft_join(..., copy = TRUE)temp table viadplyr's default,dbplyr::unique_table_name(), which draws its "random" 10-character suffix from R's global RNG stream, not an independent source. Under a caller-seeded RNG, e.g. any reproducible analysis that pinsset.seed()before an iterative solver likeirlba(needed for deterministic SVD results, sinceirlbaalso draws its starting vector from the global RNG when none is supplied), repeated%*%calls within one Lanczos iteration, or separate calls across a session that reset to the same seed, could regenerate the exact same "random" table name and collide with a still-live temp table: none of these helpers ever drop theirs (matching every other lazy pushdown in this file, the result stays an un-executed lazy query, so dropping the source temp table on function return would break it before anyone materializes the result). Found via BiocDuckDB's SVD composition tests, which pinset.seed()for reproducibility and hit exactly this collision under a full test run. Fixed by copying the small in-memory vector explicitly with a monotonic, RNG-independent name (.duckdb_matmult_copy()) instead of relying oncopy = TRUE's naming.
- Constructing a
DuckDBArray/DuckDBArraySeed/DuckDBMatrixover a genuinelyBIGINT/HUGEINT-typed value column no longer crashes withvector: cannot make a vector of mode 'integer64'.coltypes()reports"integer64"for such columns, and 9 call sites acrossDuckDBArraySeed-class.R/DuckDBArraySeed-matrixStats.Rbuilt the seed's structural fill value viavector(coltypes(...), 1L), which errors because"integer64"is abit64package class name, not a validvector()mode. Replaced with a shared.coltypeFillVector()helper. - Fixed a second, related bug this surfaced:
as.matrix()/extract_array()on a successfully-constructedinteger64-valuedDuckDBArray(the standard construction path opens its DuckDB connection withbigint = "integer64", seeacquireDuckDBConn()) silently returned garbage values (e.g.4.94e-324instead of1) rather than erroring.array()and[<-are class-blind and operate on the underlyingdoublestorage only, so they never invokebit64'sas.double.integer64()conversion; a genuinelyinteger64-classed value copied into a plain array reinterprets its raw 64-bit pattern as a double instead of converting it.rowSums()/rowCounts()were unaffected (computed via SQL, not R-side materialization).DuckDBArrayalready treatsinteger64as double-precision fidelity everywhere else, soextract_array(),extract_sparse_array(), and the COO fast path (.collectCOO()) now normalize anyinteger64-classed datacol value to plaindoubleat the point it's materialized from the database, via a new.dropInteger64()helper, instead of trying to keep a realbit64-classed value alive througharray()/[<-/COO_SparseArray(). - The
DuckDBArraySeedfillslot is now"ANY"(was"atomic"), with validity enforced viais.atomic()(TRUEforinteger64values too) rather than the slot's formal class, so a genuineinteger64fill value is accepted.
writeCoordArray()'sarrowtypeandmax_dimdocs now correctly describe theDuckDBArrayfast-path method: unlike theANYmethod, which infers the narrowest type from the data (or fromdim(x)for index columns) when these areNULL, theDuckDBArraymethod does not materialize values to compute this; it takes the source's existing declared column type as-is. Passarrowtype/max_dimexplicitly to narrow the on-disk type for aDuckDBArraysource.
- Added
scanKeycols(path, keycols), a convenience function that scans a COO Parquet file/dataset for the distinct, sorted values of the named key columns and returns them already in the list shape expected by thekeycolsargument ofDuckDBArray()/DuckDBMatrix(). Introduced in response to Bioconductor review feedback on the introspection example in the vignette, which required repeating a manualdistinct()/arrange()/collect()scan by hand for each key column; the sort is pushed down into the query plan rather than done in R after collecting. The vignette's "Introspecting a Parquet file you did not write" section now uses it.
- Added
skip_if_not_installed("sparseMatrixStats")to therowNnzs()tests that exercise adgCMatrix(test-rowNnzs.R), which otherwise error rather than skip whensparseMatrixStats/DelayedMatrixStatsaren't installed, sinceMatrixGenerics::rowCounts()on adgCMatrixhas no method without one of them.
- Removed two duplicate
switch()case labels ("integer", "double") in.duckdbTypeToArrow()(writeCoordArray-types.R); R'sswitch()only ever matches the first occurrence of a label, so the later entries were dead code. Both duplicates mapped to the same value as their first occurrence, so this is not a behavior change. - Removed the unused internal helper
.buildTempTableFilter(). sapply()calls inR/DuckDBTable-matrixStats.Rare replaced withlapply()(where the result's names already came from a named input) orsetNames(lapply(...), ...)(wheresapply()'s auto-naming of an unnamed character-vector input was load-bearing), resolving the correspondingBiocCheckNOTE with no behavior change.- The multi-line
paste0()-built format string in.check_gram_size()'s error message (R/DuckDBMatrix-utils.R) is replaced with twosprintf()/ literal arguments passed directly tostop(), which concatenates its...arguments natively; this resolves the "avoid paste in condition signals" NOTE and produces byte-identical message text (verified).
- Benchmarking vignette: the
rowVars()comparison against a plaindgCMatrix(which dispatches throughMatrixGenericsto a method supplied bysparseMatrixStatsorDelayedMatrixStats) is now gated on one of those optional packages being installed, so the vignette no longer errors when neither is available.rowDeviances(), DuckDBArray's own generic, is unaffected and always runs. - Introduction vignette: added a section on introspecting a Parquet file's
columns and key-column value ranges with
arrow::open_dataset()before constructing aDuckDBArray/DuckDBMatrix, for the case where the file was produced by another tool and the key structure isn't already known.
- Loading the
airwaySuggests package intests/testthat/setup.Ris now conditional, and thetest_that()blocks that useairway_counts/airway_counts_pathcallskip_if_not_installed("airway"). Previously the unconditionaldata(airway, ...)call insetup.Rwould fail and abort the entire test suite (not just the airway-dependent tests) whenairwaywas not installed.
- Replaced em dashes with commas or colons in the vignettes.
- Added
URLandBugReportsfields to DESCRIPTION. - Added a package-level man page (
?DuckDBArray). - Removed the redundant
library(BiocStyle)call from the vignettes (theBiocStyle::html_documentoutput and::-qualified helpers apply the style without loading the package). Applied suite-wide for consistency with the Bioconductor review of DuckDBDataFrame.
writeCoordArray()/writeParquet()forDuckDBArraygain acluster_byargument: an optional within-partition row ordering (aDuckDBDataFrame::zorder()/hilbert()spec, or a character vector of index columns) lowered to theCOPY TOORDER BYviaDuckDBDataFrame::clusterOrderSQL(). Clustering the COO on a chosen axis (e.g.cluster_by = "<sample index>") tightens that axis's row-group zonemaps so a range or point query on it prunes row groups, a faithful reorder (the COO is order-agnostic on read; no schema, contract, or reader change). DefaultNULLkeeps the existing index ordering, so output is byte-identical and existing goldens are unaffected. Reuses DuckDBDataFrame's single Morton/Hilbert generator rather than re-deriving it.
- The DuckDB fast-path append now writes new parts into an existing
hive-partitioned dataset (via the DuckDB
APPENDcopy option) instead of failing with "Directory ... is not empty". A secondwriteCoordArray(..., append = TRUE)slab is preserved alongside the first, and the combined dataset round-trips; previously the fast path supported only a fresh write.
-
The DuckDB fast write path now types the coordinate-index columns from the pre-computed,
max_dim-awareidxtypes(.buildIndexMappings()) instead of inferring them from the remapped indices. Previously a> 2^31along-axis offset made the remapped indices a double, which inferredfloat64and degraded to a DuckDBINTEGERtemp column that overflowed; and an append part could narrow differently from part 0. The index type now matches the R write path and is consistent across parts. -
writeCoordArray()accepts a whole-numbermax_dimbeyond the 32-bit integer range (a coordinate axis larger than ~2.1e9). The validator previously coercedmax_dimviaas.integer(), overflowing such a dimension toNA; it now validates integrality without coercion and keeps within-32-bit dims as integer.
- Fast one-query coercions to concrete sparse-matrix targets:
as(x, "dgCMatrix")/as(x, "CsparseMatrix")/as(x, "COO_SparseMatrix")/as(x, "COO_SparseArray")on aDuckDBArray/DuckDBMatrixnow fetch the whole COO in a single query and build the target directly, skipping theCOO_SparseArray -> SVT_SparseArrayround-trip the default DelayedArray coercions pay for (measured ~1.3-1.7x on a cell x gene realize, byte-identical to the default). They fall back to the default path when the array is not zero-filled, dropped, the wrong rank, or (fordgCMatrix) non-numeric, so correctness is never traded for speed. Denseas.matrix/as.arraystay on the existing (already single-query) path: a direct dense build measured only ~1.1x, not worth a separate code path.
- The test harness pins DuckDB to a single thread
(
options(DuckDBDataFrame.threads = 1L)insetup.R) so parallel floating-point reductions (sum/var_samp) accumulate in a fixed order and tight-tolerance expectations stay reproducible run-to-run. This is a test-only change; real sessions use all cores.
- Relicensed under the MIT License.
crossprod()andtcrossprod()on aDuckDBMatrix(the SQL self-join form, with no second operand) now raise a clear error before the self-join can exhaust memory, instead of silently OOMing on a large matrix. The self-join emits roughlynnz^2 / contracted_dimintermediate row-pairs before theGROUP BYcan reduce them — quadratic and un-spillable at scale — so a size estimate now trips astop()with guidance (subset to highly variable genes, or materialize withas.matrix()) above a safeguard of 5e8 pairs. The threshold is overridable withoptions(DuckDBArray.gram_pair_limit = )on a machine with more memory.
- The
DuckDBTablemargin statistics (rowSums/colSums,rowMeans/colMeans,rowVars/colVars,rowSds/colSds,rowMaxs/colMaxs,rowMins/colMins,rowCounts/colCounts) now honor theirna.rmargument. Previouslyna.rmwas accepted but ignored — the statistic always droppedNA/NULL, diverging from theMatrixGenericsdefault ofna.rm = FALSE. Because SQL aggregate functions inherently dropNULL,na.rm = FALSEis emulated with a per-group guard: a group containing anyNULLyieldsNA. The commonna.rm = TRUEpath (and any group withoutNULLs) is unchanged.
- Restructured the vignettes into a user-first set: Introduction to DuckDBArray (overview and usage), Benchmarking DuckDBArray (a best-effort comparison against in-memory, HDF5Array, and TileDBArray), and Implementing the DuckDBArray backend (the DelayedArray seed contract and SQL translation, for developers).
- The benchmarking vignette renders precomputed results (produced offline by
inst/scripts/run_vignette_benchmarks.R) for both a single-threaded and a best-effort-parallel regime, with each backend configured at its best and the configuration recorded alongside the numbers. - Rewrote the README.
- Added
\valuesections to all exported-object man pages. - Manual-page examples build objects via their constructors/accessors instead of
accessing S4 slots with
@.