Python SDK v1.2.0

Major Changes

• License: Migrated from the Databricks License to the Apache License 2.0

New Features and Improvements

• Arrow Flight Support (Experimental): Added support for ingesting pyarrow.RecordBatch and pyarrow.Table objects via Arrow Flight protocol
  • Note: Arrow Flight is not yet supported by default from the Zerobus server side.
  • New ZerobusArrowStream class (sync in zerobus.sdk.sync, async in zerobus.sdk.aio) with ingest_batch(), wait_for_offset(), flush(), close(), get_unacked_batches() methods
  • New ArrowStreamConfigurationOptions for configuring Arrow streams (max inflight batches, recovery, timeouts)
  • New create_arrow_stream() and recreate_arrow_stream() methods on both sync and async ZerobusSdk
  • Accepts both pyarrow.RecordBatch and pyarrow.Table (Tables are combined to a single batch internally)
  • Arrow is opt-in: install via pip install databricks-zerobus-ingest-sdk[arrow] (requires pyarrow>=14.0.0)
  • Arrow types gated behind _core.arrow submodule — not loaded unless pyarrow is installed
  • Available from both zerobus.sdk.sync and zerobus.sdk.aio, and re-exported from top-level zerobus package

Bug Fixes

• Fixed proto generation tool to skip reserved field numbers 19000-19999 for tables with more than 19000 columns

Internal Changes

• Bumped Rust SDK dependency to v1.0.1 with arrow-flight feature
• Added arrow-ipc, arrow-schema, arrow-array (v56.2.0) Rust dependencies for IPC serialization
• Added PyO3 arrow module (arrow.rs) with ArrowStreamConfigurationOptions, ZerobusArrowStream, AsyncZerobusArrowStream pyclasses
• Added Python-side serialization helpers in zerobus.sdk.shared.arrow (_serialize_schema, _serialize_batch, _deserialize_batch)

API Changes

• Added create_arrow_stream(table_name, schema, client_id, client_secret, options=None, headers_provider=None) to sync and async ZerobusSdk
• Added recreate_arrow_stream(old_stream) to sync and async ZerobusSdk
• Added ZerobusArrowStream class (sync and async variants) with methods: ingest_batch(), wait_for_offset(), flush(), close(), get_unacked_batches(), properties: is_closed, table_name
• Added ArrowStreamConfigurationOptions class with fields: max_inflight_batches, recovery, recovery_timeout_ms, recovery_backoff_ms, recovery_retries, server_lack_of_ack_timeout_ms, flush_timeout_ms, connection_timeout_ms
• Added optional dependency: pyarrow>=14.0.0 via pip install databricks-zerobus-ingest-sdk[arrow]

TypeScript SDK v1.0.2

Bug Fixes

• Split platform-specific native binaries into separate npm packages (@databricks/zerobus-ingest-sdk-linux-x64-gnu, -linux-arm64-gnu, -win32-x64-msvc). npm now auto-installs only the binary matching the user's OS/arch via optionalDependencies, reducing download size from ~15MB to ~5MB.

Java SDK v1.1.1

Bug Fixes

• Fixed isClosed() returning false after the stream internally closed due to non-retryable errors (e.g. server rejection, auth failure). The JNI layer now checks both the wrapper state and the Rust stream's internal closed flag. Affects both proto/JSON streams and Arrow streams.

Java SDK v1.1.0

Major Changes

• License: Migrated from the Databricks License to the Apache License 2.0

New Features and Improvements

• Arrow Flight Support (Experimental): Added support for ingesting Apache Arrow VectorSchemaRoot batches via Arrow Flight protocol
  • Note: Arrow Flight is not yet supported by default from the Zerobus server side.
  • New ZerobusArrowStream class with ingestBatch(), waitForOffset(), flush(), close(), getUnackedBatches() methods
  • New ArrowStreamConfigurationOptions for configuring Arrow streams (max inflight batches, recovery, timeouts, IPC compression)
  • Configurable IPC compression via ArrowStreamConfigurationOptions.setIpcCompression() (supports LZ4_FRAME and ZSTD)
  • New createArrowStream() and recreateArrowStream() methods on ZerobusSdk
  • Accepts VectorSchemaRoot directly via ingestBatch() (IPC serialization handled internally)
  • Arrow is opt-in: add arrow-vector and arrow-memory-netty as dependencies (provided scope, >= 15.0.0)

Bug Fixes

• Classloader Isolation Compatibility: Fixed NoClassDefFoundError when using the SDK inside Spring Boot. JNI class references are now cached as GlobalRefs during JNI_OnLoad, so native daemon threads no longer rely on FindClass through the system classloader.
• Fixed proto generation tool to skip reserved field numbers 19000-19999 for tables with more than 19000 columns

Internal Changes

• Added arrow-vector 17.0.0 as provided dependency for Arrow Flight support
• Added arrow-memory-netty 17.0.0 as test dependency for integration tests
• Uses existing JNI Arrow Flight bindings from Rust SDK (nativeCreateArrowStream, nativeIngestBatch, etc.)

API Changes

• Added createArrowStream(String tableName, Schema schema, String clientId, String clientSecret) to ZerobusSdk
• Added createArrowStream(String tableName, Schema schema, String clientId, String clientSecret, ArrowStreamConfigurationOptions options) to ZerobusSdk
• Added recreateArrowStream(ZerobusArrowStream closedStream) to ZerobusSdk
• Added ZerobusArrowStream class with methods: ingestBatch(), waitForOffset(), flush(), close(), getUnackedBatches(), isClosed(), getTableName(), getOptions()
• Added ArrowStreamConfigurationOptions class with fields: maxInflightBatches, recovery, recoveryTimeoutMs, recoveryBackoffMs, recoveryRetries, serverLackOfAckTimeoutMs, flushTimeoutMs, connectionTimeoutMs, ipcCompression
• Added IPCCompressionType enum with values: NONE, LZ4_FRAME, ZSTD
• Added optional dependency: org.apache.arrow:arrow-vector >= 15.0.0 (provided scope)

Go SDK v1.1.1

Re-release of v1.1.0 with pre-built FFI libraries included. v1.1.0 is retracted due to missing native libraries.

Major Changes

• License: Migrated from the Databricks License to the Apache License 2.0

New Features and Improvements

[Experimental] Arrow Flight Ingestion: Added experimental Arrow Flight support for high-throughput Apache Arrow RecordBatch ingestion

• New CreateArrowStream and CreateArrowStreamWithHeadersProvider methods on ZerobusSdk
• New ZerobusArrowStream type with IngestBatch, WaitForOffset, Flush, Close, and GetUnackedBatches methods
• Configurable IPC compression via ArrowStreamConfigurationOptions.IpcCompression (supports LZ4Frame and Zstd)

TypeScript SDK v1.0.1

Bug Fixes

• Fixed npm packaging: v1.0.0 was published without the napi-rs generated index.js loader and index.d.ts type declarations, causing MODULE_NOT_FOUND on require('@databricks/zerobus-ingest-sdk'). The platform-specific native binary packages (e.g. @databricks/zerobus-ingest-sdk-linux-x64-gnu) were also missing from npm. This release includes all generated files and platform packages.

FFI Libraries v1.1.0

Major Changes

• License: Migrated from the Databricks License to the Apache License 2.0
• Removed macOS x86_64 and macOS aarch64 support.

New Features and Improvements

• Added dynamic library (.so / .dylib / .dll) output alongside static library

Rust SDK v1.2.0

Major Changes

• License: Migrated from the Databricks License to the Apache License 2.0

New Features and Improvements

• Added the schema module with descriptor_from_uc_columns /
  descriptor_from_uc_schema, which convert a Unity Catalog table schema
  (including nested STRUCT, ARRAY, and MAP columns via type_json) into
  a prost_types::DescriptorProto that can be passed to
  TableProperties::descriptor_proto. Enables building descriptors at runtime
  without pre-generating .proto files.

Internal Changes

• The generate_files CLI tool now delegates schema → descriptor conversion
  to the SDK's new schema module instead of its own hand-rolled DDL-string
  parser, and renders the resulting DescriptorProto back to proto2 text.

Breaking Changes

• generate_files: the emitted .proto files have changed shape for
  non-trivial schemas. Consumers regenerating existing files should expect:
  • Field numbers now follow Unity Catalog's position + 1 (so gaps from
    DROP COLUMN under Delta column-mapping are preserved) instead of the
    previous 1,2,3… sequential numbering with a 19000-range skip.
  • Nested struct messages use path-based names (e.g. OuterInner instead of
    Inner) and are emitted hierarchically inside their parent message.
  • Struct field nullability now honors Unity Catalog's nullable flag
    instead of being forced to optional.

Rust SDK v1.1.0

[Rust] Release 1.1.0 (#191)

Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>

Rust SDK v1.0.1

Bug Fixes

• Fixed TLS certificate validation failure when behind corporate VPN/proxy with MITM certificates (e.g., GlobalProtect). Changed reqwest TLS configuration from rustls-tls to rustls-tls-native-roots + rustls-tls-webpki-roots, so the SDK now loads CA certificates from the OS native trust store (respecting SSL_CERT_FILE and system certificate stores) while keeping bundled Mozilla roots as a fallback for minimal environments.

New Features and Improvements

• Exported OAuthHeadersProvider in the public API, allowing clients to directly construct and use the built-in OAuth 2.0 headers provider.