From d2dd3571fe870fcc7eefabc03097c9f476850570 Mon Sep 17 00:00:00 2001 From: Alex Gaetano Padula Date: Sun, 4 Oct 2026 14:20:04 -0400 Subject: [PATCH] pin the tidesdb major at build time, add --mysql-src, drop --s3 cmake reads TIDESDB_VERSION from the installed tidesdb/tidesdb_version.h and refuses to configure against a library whose major is not the one this plugin major pairs with.the linker cannot catch it: the soname carries the major, so a mismatch surfaces at load as a missing library, or at run time as a format the engine misreads.a missing version header, an unparseable version and a wrong major each say so separately rather than failing the same way. install.sh grows --mysql-src, to build against a mysql source tree the caller already has instead of cloning one.it validates the path is a mysql tree, and prepare_mysql no longer deletes or clones over a tree it was handed.every path that named the cloned tree now goes through it, so --rebuild-plugin works against an external checkout. --s3 is removed.TIDESDB_WITH_S3 is read by no cmakelists and no source in either the plugin or the library, so the flag passed a define nothing acted on; the rebuild path re-ran cmake only to sync it, and no longer re-runs cmake at all. the paired library moves to tidesdb v10.1.1, in VERSIONING.md, the format table, manual.json, the preface and the compatibility page. install.sh pins the paired tidesdb release (v10.1.1) instead of resolving to whatever is newest, so a fresh install cannot drift onto a library this release was never tested against.--tidesdb-version still overrides, and cmake still refuses a different major --- VERSIONING.md | 15 ++++-- doc/00-preface/preface.md | 2 +- doc/01-getting-started/install.md | 19 +++++-- doc/06-appendix/compatibility.md | 2 +- doc/manual.json | 2 +- install.sh | 90 ++++++++++++++++--------------- tidesdb/CMakeLists.txt | 37 +++++++++++++ 7 files changed, 114 insertions(+), 53 deletions(-) diff --git a/VERSIONING.md b/VERSIONING.md index 6feb8fd..b77ebc5 100644 --- a/VERSIONING.md +++ b/VERSIONING.md @@ -17,10 +17,17 @@ than in engine code. The capabilities MySQL does and does not offer an engine are listed in **What MySQL offers the engine** below, and a change to that list versions like any other change to the public surface. -## TideSQL 2.0.0 pairs with TidesDB v10.0.1 +## TideSQL 2.0.0 pairs with TidesDB v10.1.1 + +The pairing is enforced at build time, not left to the operator: the plugin's +CMake reads `TIDESDB_VERSION` from the installed `tidesdb/tidesdb_version.h` and +refuses to configure against a library whose major is not the one this TideSQL +major pairs with. The linker would not catch it -- the soname carries the major, +so a mismatch surfaces later as a missing library at load, or as a format the +engine misreads. Each TideSQL release links exactly one TidesDB release and stores data in that -library's on-disk format. TideSQL 2.0.0 links TidesDB v10.0.1 and writes the +library's on-disk format. TideSQL 2.0.0 links TidesDB v10.1.1 and writes the v10 format line. The plugin version and the library version keep their own cadence, so the pairing is recorded here and surfaced at runtime through the `tidesdb_version` status variable for the plugin and `tidesdb_library_version` @@ -89,9 +96,9 @@ format line. | TideSQL | TidesDB library | On-disk format | Rollback boundary | |---------|-----------------|----------------|------------------------------| -| 2.0.0 | 10.0.1 | 10 | any TideSQL linking format 10 | +| 2.0.0 | 10.1.1 | 10 | any TideSQL linking format 10 | -- **TideSQL 2.0.0 writes the v10 format line** by linking TidesDB v10.0.1. It +- **TideSQL 2.0.0 writes the v10 format line** by linking TidesDB v10.1.1. It reads only the v10 format, so a database created by an earlier release line does not open in place and no in-place migration ships for that step. A database from an earlier release moves across by dumping with `mysqldump` and diff --git a/doc/00-preface/preface.md b/doc/00-preface/preface.md index b1a2c53..db73c72 100644 --- a/doc/00-preface/preface.md +++ b/doc/00-preface/preface.md @@ -10,7 +10,7 @@ library. It lets a table keep its data in a TidesDB LSM B+tree instead of InnoDB ordinary SQL. Switching a table from InnoDB to TidesDB is a change to the `ENGINE` clause and nothing more. -This is the TideSQL for MySQL version 2.x manual, the storage engine for TidesDB v10.1.0, and the companion to the +This is the TideSQL for MySQL version 2.x manual, the storage engine for TidesDB v10.1.1, and the companion to the [TidesDB library manual](/preface). It tracks the 2.x line as a whole, so a minor or patch release extends this manual rather than starting a separate one. Where this book says "the library" it means TidesDB, and where it needs to explain a storage behavior in depth it points at the library manual rather than diff --git a/doc/01-getting-started/install.md b/doc/01-getting-started/install.md index 1a2b690..8eb159b 100644 --- a/doc/01-getting-started/install.md +++ b/doc/01-getting-started/install.md @@ -35,12 +35,12 @@ The flags worth knowing: | `--mysql-version VERSION` | MySQL branch or tag; defaults to the latest on GitHub | | `--tidesdb-prefix DIR`, `--mysql-prefix DIR` | Where each is installed | | `--build-dir DIR` | Working directory for the build | +| `--mysql-src DIR` | Build against a MySQL source tree you already have, instead of cloning one | | `--jobs N` | Parallel build jobs; auto-detected otherwise | | `--skip-deps`, `--skip-tidesdb` | Skip dependency installation, or the library build if it is already installed | | `--skip-engines ENGINES` | Comma-separated engines to leave out of the server; `--list-engines` shows what can be skipped | | `--rebuild-plugin` | Rebuild only the plugin against an existing server build, for a fast edit-build-test cycle | | `--allocator NAME` | Allocator for `libtidesdb`: `system` (default), `jemalloc`, `mimalloc` or `tcmalloc`. See the loading note below | -| `--s3` | Build the library's S3 object-store connector; needs libcurl | | `--pgo` | Three-phase profile-guided build: instrument, train on the test suite, rebuild optimised | ```bash @@ -50,8 +50,21 @@ The flags worth knowing: ``` `--rebuild-plugin` needs a full run to have happened first; it reuses that build tree rather than -configuring a new one. It does not rebuild `libtidesdb`, so a changed `--allocator` or `--s3` needs -a full run to take effect. +configuring a new one. It does not rebuild `libtidesdb`, so a changed `--allocator` needs a full run +to take effect. + +`--mysql-src` points the build at a MySQL source tree you already have rather than cloning one, and +combines with `--rebuild-plugin` to build only the engine against it. The tree has to be the same +version as the server the plugin will be loaded into. A storage engine records the +`MYSQL_VERSION_ID` it was compiled against, and the server loads only a plugin whose id matches its +own exactly -- patch releases included, so a plugin built against 26.7.0 is refused by 26.7.1 with +`API version for STORAGE ENGINE plugin is too different`. There is no storage-engine ABI across +server versions to rely on. + +```bash +./install.sh --rebuild-plugin --skip-deps --skip-tidesdb \ + --mysql-src /path/to/mysql-server --mysql-prefix /usr/local/mysql +``` The rest of this page is the manual path, which is what to read if you are building against a server tree you already have. diff --git a/doc/06-appendix/compatibility.md b/doc/06-appendix/compatibility.md index 5811378..4e17cb7 100644 --- a/doc/06-appendix/compatibility.md +++ b/doc/06-appendix/compatibility.md @@ -16,7 +16,7 @@ that server version. | MySQL 26.7.0 | 2.0.0 | Yes | This is the TideSQL 2.x manual and it pairs with TidesDB v10. The current pinned release is 2.0.0, -linking TidesDB v10.1.0, at **beta** maturity. A 2.x minor or patch links a TidesDB v10 release and +linking TidesDB v10.1.1, at **beta** maturity. A 2.x minor or patch links a TidesDB v10 release and extends this same manual, so only a new major opens a new manual. See [Versioning](https://github.com/tidesdb/tidesql/blob/master/VERSIONING.md) for how the plugin and the library versions relate. diff --git a/doc/manual.json b/doc/manual.json index bca3248..9b389ba 100644 --- a/doc/manual.json +++ b/doc/manual.json @@ -1,7 +1,7 @@ { "$comment": "Table of contents for the TideSQL manual, in reading order. TideSQL is the MySQL storage engine that embeds the TidesDB library, and this manual is the companion to the TidesDB library manual. It documents the TideSQL 2.x line, which pairs with TidesDB v10. Minor and patch releases extend this same manual rather than starting a new one. The exact pinned release comes from the website's versions.js and is not repeated here.", "title": "TideSQL Manual", - "tidesdb": "10.0.1", + "tidesdb": "10.1.1", "parts": [ { "id": "preface", diff --git a/install.sh b/install.sh index 6335e7b..268e52e 100755 --- a/install.sh +++ b/install.sh @@ -40,11 +40,15 @@ # ./install.sh [OPTIONS] # # Options: -# --tidesdb-version VERSION TidesDB release tag (default: latest from GitHub) +# --tidesdb-version VERSION TidesDB release tag (default: the paired release) # --mysql-version VERSION MySQL branch or tag (default: latest from GitHub) # --tidesdb-prefix DIR TidesDB install prefix (default: platform-dependent) # --mysql-prefix DIR MySQL install prefix (default: platform-dependent) # --build-dir DIR Working directory (default: platform-dependent) +# --mysql-src DIR Use this MySQL source tree instead of cloning one. The tree must be +# the same version as the server the plugin will load into: a storage +# engine records the server's MYSQL_VERSION_ID and the server refuses +# a plugin whose id differs, patch releases included. # --jobs N Parallel build jobs (default: auto-detected) # --skip-deps Skip system dependency installation # --skip-tidesdb Skip TidesDB library build (use if already installed) @@ -52,7 +56,6 @@ # --list-engines List storage engines that can be skipped and exit # --rebuild-plugin Rebuild only the TidesDB plugin (fast dev cycle) # --pgo Enable Profile-Guided Optimization (3-phase build) -# --s3 Build TidesDB with S3 object store connector (requires libcurl) # --allocator NAME Memory allocator for libtidesdb.so: system (default), jemalloc, mimalloc, or tcmalloc. # Only affects TidesDB's internal allocations; mysqld's allocator is unchanged. # For a process-wide swap also LD_PRELOAD the allocator at mysqld startup. @@ -143,12 +146,18 @@ _fetch_url() { fi } +# The TidesDB release this TideSQL is paired with and tested against. Pinned rather than resolved +# to whatever is newest: a TideSQL release stores data in one library's on-disk format, and the +# pairing is part of what the version means -- see VERSIONING.md. --tidesdb-version overrides it +# for trying a different library, and the plugin's CMake still refuses a different major. +TIDESDB_PINNED_VERSION="v10.1.1" + get_latest_tidesdb_version() { local version version=$(_fetch_url "https://api.github.com/repos/tidesdb/tidesdb/releases/latest" \ | grep '"tag_name":' | sed -E 's/.*"tag_name": *"([^"]+)".*/\1/') if [[ -z "$version" ]]; then - echo "10.0.0" # fallback, TidesDB 10.x tags drop the leading v + echo "${TIDESDB_PINNED_VERSION}" else echo "$version" fi @@ -177,6 +186,7 @@ get_latest_mysql_version() { TIDESDB_VERSION="" MYSQL_VERSION="" +MYSQL_SRC="" TIDESDB_PREFIX="${DEFAULT_TIDESDB_PREFIX}" MYSQL_PREFIX="${DEFAULT_MYSQL_PREFIX}" BUILD_DIR="${DEFAULT_BUILD_DIR}" @@ -186,7 +196,6 @@ SKIP_TIDESDB=false REBUILD_PLUGIN=false PGO_ENABLED=false SKIP_ENGINES="" -WITH_S3=false # Memory allocator to build libtidesdb against. One of: # system glibc / platform default (no extra dep) # jemalloc routes TidesDB allocations through jemalloc (pkg: libjemalloc-dev) @@ -291,7 +300,6 @@ while [[ $# -gt 0 ]]; do --skip-engines) SKIP_ENGINES="$2"; shift 2 ;; --list-engines) list_engines ;; --pgo) PGO_ENABLED=true; shift ;; - --s3) WITH_S3=true; shift ;; --allocator) ALLOCATOR="$2" case "$ALLOCATOR" in @@ -299,15 +307,27 @@ while [[ $# -gt 0 ]]; do *) die "--allocator must be one of system|jemalloc|mimalloc|tcmalloc (got '$ALLOCATOR')" ;; esac shift 2 ;; + --mysql-src) MYSQL_SRC="$2"; shift 2 ;; --help|-h) usage ;; *) die "Unknown option: $1 (try --help)" ;; esac done -# Resolve versions (fetch from GitHub if not specified) +# The MySQL source tree: the one the caller named, or the one this script clones. +MYSQL_SRC_GIVEN=false +if [[ -n "$MYSQL_SRC" ]]; then + [[ -d "$MYSQL_SRC" ]] || die "--mysql-src: no such directory: ${MYSQL_SRC}" + [[ -f "$MYSQL_SRC/sql/mysqld.cc" ]] || die "--mysql-src: ${MYSQL_SRC} is not a MySQL source tree" + MYSQL_SRC_GIVEN=true +else + MYSQL_SRC="${BUILD_DIR}/mysql-server" +fi + +# Resolve versions. TidesDB defaults to the release this TideSQL is paired with; MySQL defaults +# to the newest, since the plugin tracks the server rather than pinning it. if [[ -z "$TIDESDB_VERSION" ]]; then - info "Fetching latest TidesDB version from GitHub..." - TIDESDB_VERSION="$(get_latest_tidesdb_version)" + TIDESDB_VERSION="${TIDESDB_PINNED_VERSION}" + info "TidesDB: using the paired release ${TIDESDB_VERSION} (override with --tidesdb-version)" fi if [[ -z "$MYSQL_VERSION" ]]; then info "Fetching latest MySQL version from GitHub..." @@ -549,11 +569,6 @@ build_tidesdb() { -DBUILD_SHARED_LIBS=ON ) - if $WITH_S3; then - cmake_args+=(-DTIDESDB_WITH_S3=ON) - info "S3 object store connector enabled" - fi - case "$ALLOCATOR" in jemalloc) cmake_args+=(-DTIDESDB_WITH_JEMALLOC=ON) @@ -615,18 +630,20 @@ build_tidesdb() { # Clone MySQL and copy TidesDB storage engine prepare_mysql() { - info "Cloning MySQL (branch/tag: ${MYSQL_VERSION})..." + local mysql_src="${MYSQL_SRC}" - local mysql_src="${BUILD_DIR}/mysql-server" - - if [[ -d "${mysql_src}" ]]; then - info "Removing previous MySQL source..." - rm -rf "${mysql_src}" + if $MYSQL_SRC_GIVEN; then + info "Using the MySQL source tree at ${mysql_src} (not cloning)" + else + info "Cloning MySQL (branch/tag: ${MYSQL_VERSION})..." + if [[ -d "${mysql_src}" ]]; then + info "Removing previous MySQL source..." + rm -rf "${mysql_src}" + fi + git clone --depth 1 --branch "${MYSQL_VERSION}" \ + https://github.com/mysql/mysql-server.git "${mysql_src}" fi - git clone --depth 1 --branch "${MYSQL_VERSION}" \ - https://github.com/mysql/mysql-server.git "${mysql_src}" - info "Copying TidesDB storage engine plugin into MySQL source..." cp -R "${SCRIPT_DIR}/tidesdb" "${mysql_src}/storage/" @@ -646,7 +663,7 @@ prepare_mysql() { build_mysql() { info "Building MySQL with InnoDB + TidesDB..." - local mysql_src="${BUILD_DIR}/mysql-server" + local mysql_src="${MYSQL_SRC}" local mysql_build="${mysql_src}/build" mkdir -p "${mysql_build}" @@ -670,11 +687,6 @@ build_mysql() { -DWITH_UNIT_TESTS=OFF ) - # S3 object store connector for the plugin - if $WITH_S3; then - cmake_args+=(-DTIDESDB_WITH_S3=ON) - fi - # Disable skipped engines if [[ -n "$SKIP_ENGINES" ]]; then IFS=',' read -ra _engines <<< "$SKIP_ENGINES" @@ -723,7 +735,7 @@ build_mysql() { install_mysql() { info "Installing MySQL to ${MYSQL_PREFIX}..." - local mysql_build="${BUILD_DIR}/mysql-server/build" + local mysql_build="${MYSQL_SRC}/build" local build_config="RelWithDebInfo" if $PGO_ENABLED; then build_config="Release" @@ -1049,7 +1061,7 @@ ${mysqld_safe_section}" local mysqld_bin="" for candidate in \ "${MYSQL_PREFIX}/bin/mysqld" \ - "${BUILD_DIR}/mysql-server/build/runtime_output_directory/mysqld"; do + "${MYSQL_SRC}/build/runtime_output_directory/mysqld"; do if [[ -x "$candidate" ]]; then mysqld_bin="$candidate" break @@ -1117,7 +1129,7 @@ ${mysqld_safe_section}" print_summary() { local cnf_name="my.cnf" local start_cmd="${MYSQL_PREFIX}/bin/mysqld-safe" - local test_dir="${BUILD_DIR}/mysql-server/build/mysql-test" + local test_dir="${MYSQL_SRC}/build/mysql-test" # Use the correct connect user, root for system installs, current user otherwise local connect_user="root" @@ -1190,7 +1202,7 @@ print_summary() { # Rebuild only the TidesDB plugin (fast dev cycle) rebuild_plugin() { - local mysql_src="${BUILD_DIR}/mysql-server" + local mysql_src="${MYSQL_SRC}" local mysql_build="${mysql_src}/build" if [[ ! -d "${mysql_build}" ]]; then @@ -1216,14 +1228,6 @@ rebuild_plugin() { # Point cmake at the TidesDB library export TIDESDB_ROOT="${TIDESDB_PREFIX}" - # Sync the S3 setting with the current --s3 flag so cached builds - # don't keep a stale TIDESDB_WITH_S3 value from a previous configure. - if $WITH_S3; then - cmake "${mysql_build}" -DTIDESDB_WITH_S3=ON - else - cmake "${mysql_build}" -DTIDESDB_WITH_S3=OFF - fi - # Build just the plugin target info "Building tidesdb plugin target (${JOBS} jobs)..." cmake --build "${mysql_build}" --target tidesdb --parallel "${JOBS}" @@ -1278,7 +1282,7 @@ rebuild_plugin() { pgo_instrument() { info "PGO Phase 1/3: Building MySQL with profiling instrumentation..." - local mysql_src="${BUILD_DIR}/mysql-server" + local mysql_src="${MYSQL_SRC}" local mysql_build="${mysql_src}/build" local profile_dir="${BUILD_DIR}/pgo-profiles" @@ -1362,7 +1366,7 @@ pgo_instrument() { pgo_train() { info "PGO Phase 2/3: Running TidesDB test suite to generate profile data..." - local mysql_src="${BUILD_DIR}/mysql-server" + local mysql_src="${MYSQL_SRC}" local mtr_dir="${mysql_src}/build/mysql-test" if [[ ! -f "${mtr_dir}/mtr" ]]; then @@ -1403,7 +1407,7 @@ pgo_train() { pgo_optimize() { info "PGO Phase 3/3: Rebuilding MySQL with profile-guided optimizations..." - local mysql_src="${BUILD_DIR}/mysql-server" + local mysql_src="${MYSQL_SRC}" local mysql_build="${mysql_src}/build" local profile_dir="${BUILD_DIR}/pgo-profiles" diff --git a/tidesdb/CMakeLists.txt b/tidesdb/CMakeLists.txt index e3b2825..41da99e 100644 --- a/tidesdb/CMakeLists.txt +++ b/tidesdb/CMakeLists.txt @@ -109,6 +109,43 @@ IF(NOT TIDESDB_INCLUDE_DIR) RETURN() ENDIF() +# The library major this plugin major is paired with. A TidesDB major opens a new on-disk format +# line, so a plugin built against a different one would read and write a format it does not claim to +# support -- see VERSIONING.md. Checked here rather than left to the linker, which would bind an +# incompatible libtidesdb happily: the soname carries the major, so the mismatch surfaces at load +# time as a missing library, or at run time as a format the engine misreads. +SET(TIDESQL_REQUIRED_TIDESDB_MAJOR 10) + +IF(NOT EXISTS "${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h") + MESSAGE(FATAL_ERROR + "TidesDB: ${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h is missing, so the library " + "version cannot be read. TideSQL requires TidesDB v${TIDESQL_REQUIRED_TIDESDB_MAJOR}.x; " + "a release that old predates it. Install a current TidesDB or point TIDESDB_ROOT at one.") +ENDIF() + +FILE(STRINGS "${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h" _tdb_version_line + REGEX "^#define[ \t]+TIDESDB_VERSION[ \t]+\"[0-9]+\\.[0-9]+\\.[0-9]+\"") +STRING(REGEX REPLACE ".*\"([0-9]+\\.[0-9]+\\.[0-9]+)\".*" "\\1" TIDESDB_VERSION "${_tdb_version_line}") + +IF(NOT TIDESDB_VERSION MATCHES "^[0-9]+\\.[0-9]+\\.[0-9]+$") + MESSAGE(FATAL_ERROR + "TidesDB: could not read TIDESDB_VERSION from " + "${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h.") +ENDIF() + +STRING(REGEX REPLACE "^([0-9]+)\\..*" "\\1" TIDESDB_VERSION_MAJOR "${TIDESDB_VERSION}") + +IF(NOT TIDESDB_VERSION_MAJOR EQUAL TIDESQL_REQUIRED_TIDESDB_MAJOR) + MESSAGE(FATAL_ERROR + "TidesDB version mismatch: found v${TIDESDB_VERSION} at ${TIDESDB_INCLUDE_DIR}, but this " + "TideSQL pairs with TidesDB v${TIDESQL_REQUIRED_TIDESDB_MAJOR}.x. A different library " + "major is a different on-disk format, so the two cannot be mixed. Install TidesDB " + "v${TIDESQL_REQUIRED_TIDESDB_MAJOR}.x, or use the TideSQL release that pairs with " + "v${TIDESDB_VERSION_MAJOR}.x -- the pairing is recorded in VERSIONING.md.") +ENDIF() + +MESSAGE(STATUS "TidesDB: v${TIDESDB_VERSION} (major ${TIDESDB_VERSION_MAJOR}, as required)") + INCLUDE_DIRECTORIES(${TIDESDB_INCLUDE_DIR}) # The compression backends are optional. The library encodes with NONE when it is