TideSQL-MySQL 2 MAJOR (v2.0.0)

VERSIONING.md

@@ -17,10 +17,17 @@ than in engine code. The capabilities MySQL does and does not offer an engine
 are listed in **What MySQL offers the engine** below, and a change to that list
 versions like any other change to the public surface.
 
-## TideSQL 2.0.0 pairs with TidesDB v10.0.1
+## TideSQL 2.0.0 pairs with TidesDB v10.1.1
+
+The pairing is enforced at build time, not left to the operator: the plugin's
+CMake reads `TIDESDB_VERSION` from the installed `tidesdb/tidesdb_version.h` and
+refuses to configure against a library whose major is not the one this TideSQL
+major pairs with. The linker would not catch it -- the soname carries the major,
+so a mismatch surfaces later as a missing library at load, or as a format the
+engine misreads.
 
 Each TideSQL release links exactly one TidesDB release and stores data in that
-library's on-disk format. TideSQL 2.0.0 links TidesDB v10.0.1 and writes the
+library's on-disk format. TideSQL 2.0.0 links TidesDB v10.1.1 and writes the
 v10 format line. The plugin version and the library version keep their own
 cadence, so the pairing is recorded here and surfaced at runtime through the
 `tidesdb_version` status variable for the plugin and `tidesdb_library_version`
@@ -89,9 +96,9 @@ format line.
 
 | TideSQL | TidesDB library | On-disk format | Rollback boundary            |
 |---------|-----------------|----------------|------------------------------|
-| 2.0.0   | 10.0.1          | 10             | any TideSQL linking format 10 |
+| 2.0.0   | 10.1.1          | 10             | any TideSQL linking format 10 |
 
-- **TideSQL 2.0.0 writes the v10 format line** by linking TidesDB v10.0.1. It
+- **TideSQL 2.0.0 writes the v10 format line** by linking TidesDB v10.1.1. It
   reads only the v10 format, so a database created by an earlier release line
   does not open in place and no in-place migration ships for that step. A
   database from an earlier release moves across by dumping with `mysqldump` and

doc/00-preface/preface.md

@@ -10,7 +10,7 @@ library. It lets a table keep its data in a TidesDB LSM B+tree instead of InnoDB
 ordinary SQL. Switching a table from InnoDB to TidesDB is a change to the `ENGINE` clause and
 nothing more.
 
-This is the TideSQL for MySQL version 2.x manual, the storage engine for TidesDB v10.1.0, and the companion to the
+This is the TideSQL for MySQL version 2.x manual, the storage engine for TidesDB v10.1.1, and the companion to the
 [TidesDB library manual](/preface). It tracks the 2.x line as a whole, so a minor or patch release
 extends this manual rather than starting a separate one. Where this book says "the library" it means TidesDB, and
 where it needs to explain a storage behavior in depth it points at the library manual rather than

doc/01-getting-started/install.md

@@ -35,12 +35,12 @@ The flags worth knowing:
 | `--mysql-version VERSION` | MySQL branch or tag; defaults to the latest on GitHub |
 | `--tidesdb-prefix DIR`, `--mysql-prefix DIR` | Where each is installed |
 | `--build-dir DIR` | Working directory for the build |
+| `--mysql-src DIR` | Build against a MySQL source tree you already have, instead of cloning one |
 | `--jobs N` | Parallel build jobs; auto-detected otherwise |
 | `--skip-deps`, `--skip-tidesdb` | Skip dependency installation, or the library build if it is already installed |
 | `--skip-engines ENGINES` | Comma-separated engines to leave out of the server; `--list-engines` shows what can be skipped |
 | `--rebuild-plugin` | Rebuild only the plugin against an existing server build, for a fast edit-build-test cycle |
 | `--allocator NAME` | Allocator for `libtidesdb`: `system` (default), `jemalloc`, `mimalloc` or `tcmalloc`. See the loading note below |
-| `--s3` | Build the library's S3 object-store connector; needs libcurl |
 | `--pgo` | Three-phase profile-guided build: instrument, train on the test suite, rebuild optimised |
 
 ```bash
@@ -50,8 +50,21 @@ The flags worth knowing:
 ```
 
 `--rebuild-plugin` needs a full run to have happened first; it reuses that build tree rather than
-configuring a new one. It does not rebuild `libtidesdb`, so a changed `--allocator` or `--s3` needs
-a full run to take effect.
+configuring a new one. It does not rebuild `libtidesdb`, so a changed `--allocator` needs a full run
+to take effect.
+
+`--mysql-src` points the build at a MySQL source tree you already have rather than cloning one, and
+combines with `--rebuild-plugin` to build only the engine against it. The tree has to be the same
+version as the server the plugin will be loaded into. A storage engine records the
+`MYSQL_VERSION_ID` it was compiled against, and the server loads only a plugin whose id matches its
+own exactly -- patch releases included, so a plugin built against 26.7.0 is refused by 26.7.1 with
+`API version for STORAGE ENGINE plugin is too different`. There is no storage-engine ABI across
+server versions to rely on.
+
+```bash
+./install.sh --rebuild-plugin --skip-deps --skip-tidesdb \
+  --mysql-src /path/to/mysql-server --mysql-prefix /usr/local/mysql
+```
 
 The rest of this page is the manual path, which is what to read if you are building against a server
 tree you already have.

doc/06-appendix/compatibility.md

@@ -16,7 +16,7 @@ that server version.
 | MySQL 26.7.0 | 2.0.0 | Yes |
 
 This is the TideSQL 2.x manual and it pairs with TidesDB v10. The current pinned release is 2.0.0,
-linking TidesDB v10.1.0, at **beta** maturity. A 2.x minor or patch links a TidesDB v10 release and
+linking TidesDB v10.1.1, at **beta** maturity. A 2.x minor or patch links a TidesDB v10 release and
 extends this same manual, so only a new major opens a new manual. See
 [Versioning](https://github.com/tidesdb/tidesql/blob/master/VERSIONING.md) for how the plugin and
 the library versions relate.

doc/manual.json

@@ -1,7 +1,7 @@
 {
   "$comment": "Table of contents for the TideSQL manual, in reading order. TideSQL is the MySQL storage engine that embeds the TidesDB library, and this manual is the companion to the TidesDB library manual. It documents the TideSQL 2.x line, which pairs with TidesDB v10. Minor and patch releases extend this same manual rather than starting a new one. The exact pinned release comes from the website's versions.js and is not repeated here.",
   "title": "TideSQL Manual",
-  "tidesdb": "10.0.1",
+  "tidesdb": "10.1.1",
   "parts": [
     {
       "id": "preface",

install.sh

@@ -40,19 +40,22 @@
 #  ./install.sh [OPTIONS]
 #
 # Options:
-#   --tidesdb-version VERSION   TidesDB release tag        (default: latest from GitHub)
+#   --tidesdb-version VERSION   TidesDB release tag        (default: the paired release)
 #   --mysql-version   VERSION   MySQL branch or tag        (default: latest from GitHub)
 #   --tidesdb-prefix  DIR       TidesDB install prefix     (default: platform-dependent)
 #   --mysql-prefix    DIR       MySQL install prefix       (default: platform-dependent)
 #   --build-dir       DIR       Working directory          (default: platform-dependent)
+#   --mysql-src       DIR       Use this MySQL source tree instead of cloning one.  The tree must be
+#                               the same version as the server the plugin will load into: a storage
+#                               engine records the server's MYSQL_VERSION_ID and the server refuses
+#                               a plugin whose id differs, patch releases included.
 #   --jobs            N         Parallel build jobs        (default: auto-detected)
 #   --skip-deps                 Skip system dependency installation
 #   --skip-tidesdb              Skip TidesDB library build (use if already installed)
 #   --skip-engines  ENGINES     Comma-separated list of storage engines to skip
 #   --list-engines              List storage engines that can be skipped and exit
 #   --rebuild-plugin             Rebuild only the TidesDB plugin (fast dev cycle)
 #   --pgo                       Enable Profile-Guided Optimization (3-phase build)
-#   --s3                        Build TidesDB with S3 object store connector (requires libcurl)
 #   --allocator  NAME           Memory allocator for libtidesdb.so: system (default), jemalloc, mimalloc, or tcmalloc.
 #                               Only affects TidesDB's internal allocations; mysqld's allocator is unchanged.
 #                               For a process-wide swap also LD_PRELOAD the allocator at mysqld startup.
@@ -143,12 +146,18 @@ _fetch_url() {
     fi
 }
 
+# The TidesDB release this TideSQL is paired with and tested against.  Pinned rather than resolved
+# to whatever is newest: a TideSQL release stores data in one library's on-disk format, and the
+# pairing is part of what the version means -- see VERSIONING.md.  --tidesdb-version overrides it
+# for trying a different library, and the plugin's CMake still refuses a different major.
+TIDESDB_PINNED_VERSION="v10.1.1"
+
 get_latest_tidesdb_version() {
     local version
     version=$(_fetch_url "https://api.github.com/repos/tidesdb/tidesdb/releases/latest" \
         | grep '"tag_name":' | sed -E 's/.*"tag_name": *"([^"]+)".*/\1/')
     if [[ -z "$version" ]]; then
-        echo "10.0.0"  # fallback, TidesDB 10.x tags drop the leading v
+        echo "${TIDESDB_PINNED_VERSION}"
     else
         echo "$version"
     fi
@@ -177,6 +186,7 @@ get_latest_mysql_version() {
 
 TIDESDB_VERSION=""
 MYSQL_VERSION=""
+MYSQL_SRC=""
 TIDESDB_PREFIX="${DEFAULT_TIDESDB_PREFIX}"
 MYSQL_PREFIX="${DEFAULT_MYSQL_PREFIX}"
 BUILD_DIR="${DEFAULT_BUILD_DIR}"
@@ -186,7 +196,6 @@ SKIP_TIDESDB=false
 REBUILD_PLUGIN=false
 PGO_ENABLED=false
 SKIP_ENGINES=""
-WITH_S3=false
 # Memory allocator to build libtidesdb against.  One of:
 #   system     glibc / platform default (no extra dep)
 #   jemalloc   routes TidesDB allocations through jemalloc (pkg: libjemalloc-dev)
@@ -291,23 +300,34 @@ while [[ $# -gt 0 ]]; do
         --skip-engines)     SKIP_ENGINES="$2";      shift 2 ;;
         --list-engines)     list_engines ;;
         --pgo)              PGO_ENABLED=true;       shift   ;;
-        --s3)               WITH_S3=true;           shift   ;;
         --allocator)
             ALLOCATOR="$2"
             case "$ALLOCATOR" in
                 system|jemalloc|mimalloc|tcmalloc) ;;
                 *) die "--allocator must be one of system|jemalloc|mimalloc|tcmalloc (got '$ALLOCATOR')" ;;
             esac
             shift 2 ;;
+        --mysql-src)        MYSQL_SRC="$2";         shift 2 ;;
         --help|-h)          usage ;;
         *) die "Unknown option: $1 (try --help)" ;;
     esac
 done
 
-# Resolve versions (fetch from GitHub if not specified) 
+# The MySQL source tree: the one the caller named, or the one this script clones.
+MYSQL_SRC_GIVEN=false
+if [[ -n "$MYSQL_SRC" ]]; then
+    [[ -d "$MYSQL_SRC" ]] || die "--mysql-src: no such directory: ${MYSQL_SRC}"
+    [[ -f "$MYSQL_SRC/sql/mysqld.cc" ]] || die "--mysql-src: ${MYSQL_SRC} is not a MySQL source tree"
+    MYSQL_SRC_GIVEN=true
+else
+    MYSQL_SRC="${BUILD_DIR}/mysql-server"
+fi
+
+# Resolve versions.  TidesDB defaults to the release this TideSQL is paired with; MySQL defaults
+# to the newest, since the plugin tracks the server rather than pinning it.
 if [[ -z "$TIDESDB_VERSION" ]]; then
-    info "Fetching latest TidesDB version from GitHub..."
-    TIDESDB_VERSION="$(get_latest_tidesdb_version)"
+    TIDESDB_VERSION="${TIDESDB_PINNED_VERSION}"
+    info "TidesDB: using the paired release ${TIDESDB_VERSION} (override with --tidesdb-version)"
 fi
 if [[ -z "$MYSQL_VERSION" ]]; then
     info "Fetching latest MySQL version from GitHub..."
@@ -549,11 +569,6 @@ build_tidesdb() {
         -DBUILD_SHARED_LIBS=ON
     )
 
-    if $WITH_S3; then
-        cmake_args+=(-DTIDESDB_WITH_S3=ON)
-        info "S3 object store connector enabled"
-    fi
-
     case "$ALLOCATOR" in
         jemalloc)
             cmake_args+=(-DTIDESDB_WITH_JEMALLOC=ON)
@@ -615,18 +630,20 @@ build_tidesdb() {
 
 # Clone MySQL and copy TidesDB storage engine 
 prepare_mysql() {
-    info "Cloning MySQL (branch/tag: ${MYSQL_VERSION})..."
+    local mysql_src="${MYSQL_SRC}"
 
-    local mysql_src="${BUILD_DIR}/mysql-server"
-
-    if [[ -d "${mysql_src}" ]]; then
-        info "Removing previous MySQL source..."
-        rm -rf "${mysql_src}"
+    if $MYSQL_SRC_GIVEN; then
+        info "Using the MySQL source tree at ${mysql_src} (not cloning)"
+    else
+        info "Cloning MySQL (branch/tag: ${MYSQL_VERSION})..."
+        if [[ -d "${mysql_src}" ]]; then
+            info "Removing previous MySQL source..."
+            rm -rf "${mysql_src}"
+        fi
+        git clone --depth 1 --branch "${MYSQL_VERSION}" \
+            https://github.com/mysql/mysql-server.git "${mysql_src}"
     fi
 
-    git clone --depth 1 --branch "${MYSQL_VERSION}" \
-        https://github.com/mysql/mysql-server.git "${mysql_src}"
-
     info "Copying TidesDB storage engine plugin into MySQL source..."
     cp -R "${SCRIPT_DIR}/tidesdb" "${mysql_src}/storage/"
 
@@ -646,7 +663,7 @@ prepare_mysql() {
 build_mysql() {
     info "Building MySQL with InnoDB + TidesDB..."
 
-    local mysql_src="${BUILD_DIR}/mysql-server"
+    local mysql_src="${MYSQL_SRC}"
     local mysql_build="${mysql_src}/build"
 
     mkdir -p "${mysql_build}"
@@ -670,11 +687,6 @@ build_mysql() {
         -DWITH_UNIT_TESTS=OFF
     )
 
-    # S3 object store connector for the plugin
-    if $WITH_S3; then
-        cmake_args+=(-DTIDESDB_WITH_S3=ON)
-    fi
-
     # Disable skipped engines
     if [[ -n "$SKIP_ENGINES" ]]; then
         IFS=',' read -ra _engines <<< "$SKIP_ENGINES"
@@ -723,7 +735,7 @@ build_mysql() {
 install_mysql() {
     info "Installing MySQL to ${MYSQL_PREFIX}..."
 
-    local mysql_build="${BUILD_DIR}/mysql-server/build"
+    local mysql_build="${MYSQL_SRC}/build"
     local build_config="RelWithDebInfo"
     if $PGO_ENABLED; then
         build_config="Release"
@@ -1049,7 +1061,7 @@ ${mysqld_safe_section}"
     local mysqld_bin=""
     for candidate in \
         "${MYSQL_PREFIX}/bin/mysqld" \
-        "${BUILD_DIR}/mysql-server/build/runtime_output_directory/mysqld"; do
+        "${MYSQL_SRC}/build/runtime_output_directory/mysqld"; do
         if [[ -x "$candidate" ]]; then
             mysqld_bin="$candidate"
             break
@@ -1117,7 +1129,7 @@ ${mysqld_safe_section}"
 print_summary() {
     local cnf_name="my.cnf"
     local start_cmd="${MYSQL_PREFIX}/bin/mysqld-safe"
-    local test_dir="${BUILD_DIR}/mysql-server/build/mysql-test"
+    local test_dir="${MYSQL_SRC}/build/mysql-test"
 
     # Use the correct connect user, root for system installs, current user otherwise
     local connect_user="root"
@@ -1190,7 +1202,7 @@ print_summary() {
 
 # Rebuild only the TidesDB plugin (fast dev cycle) 
 rebuild_plugin() {
-    local mysql_src="${BUILD_DIR}/mysql-server"
+    local mysql_src="${MYSQL_SRC}"
     local mysql_build="${mysql_src}/build"
 
     if [[ ! -d "${mysql_build}" ]]; then
@@ -1216,14 +1228,6 @@ rebuild_plugin() {
     # Point cmake at the TidesDB library
     export TIDESDB_ROOT="${TIDESDB_PREFIX}"
 
-    # Sync the S3 setting with the current --s3 flag so cached builds
-    # don't keep a stale TIDESDB_WITH_S3 value from a previous configure.
-    if $WITH_S3; then
-        cmake "${mysql_build}" -DTIDESDB_WITH_S3=ON
-    else
-        cmake "${mysql_build}" -DTIDESDB_WITH_S3=OFF
-    fi
-
     # Build just the plugin target
     info "Building tidesdb plugin target (${JOBS} jobs)..."
     cmake --build "${mysql_build}" --target tidesdb --parallel "${JOBS}"
@@ -1278,7 +1282,7 @@ rebuild_plugin() {
 pgo_instrument() {
     info "PGO Phase 1/3: Building MySQL with profiling instrumentation..."
 
-    local mysql_src="${BUILD_DIR}/mysql-server"
+    local mysql_src="${MYSQL_SRC}"
     local mysql_build="${mysql_src}/build"
     local profile_dir="${BUILD_DIR}/pgo-profiles"
 
@@ -1362,7 +1366,7 @@ pgo_instrument() {
 pgo_train() {
     info "PGO Phase 2/3: Running TidesDB test suite to generate profile data..."
 
-    local mysql_src="${BUILD_DIR}/mysql-server"
+    local mysql_src="${MYSQL_SRC}"
     local mtr_dir="${mysql_src}/build/mysql-test"
 
     if [[ ! -f "${mtr_dir}/mtr" ]]; then
@@ -1403,7 +1407,7 @@ pgo_train() {
 pgo_optimize() {
     info "PGO Phase 3/3: Rebuilding MySQL with profile-guided optimizations..."
 
-    local mysql_src="${BUILD_DIR}/mysql-server"
+    local mysql_src="${MYSQL_SRC}"
     local mysql_build="${mysql_src}/build"
     local profile_dir="${BUILD_DIR}/pgo-profiles"
 

tidesdb/CMakeLists.txt

@@ -109,6 +109,43 @@ IF(NOT TIDESDB_INCLUDE_DIR)
   RETURN()
 ENDIF()
 
+# The library major this plugin major is paired with.  A TidesDB major opens a new on-disk format
+# line, so a plugin built against a different one would read and write a format it does not claim to
+# support -- see VERSIONING.md.  Checked here rather than left to the linker, which would bind an
+# incompatible libtidesdb happily: the soname carries the major, so the mismatch surfaces at load
+# time as a missing library, or at run time as a format the engine misreads.
+SET(TIDESQL_REQUIRED_TIDESDB_MAJOR 10)
+
+IF(NOT EXISTS "${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h")
+  MESSAGE(FATAL_ERROR
+          "TidesDB: ${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h is missing, so the library "
+          "version cannot be read.  TideSQL requires TidesDB v${TIDESQL_REQUIRED_TIDESDB_MAJOR}.x; "
+          "a release that old predates it.  Install a current TidesDB or point TIDESDB_ROOT at one.")
+ENDIF()
+
+FILE(STRINGS "${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h" _tdb_version_line
+     REGEX "^#define[ \t]+TIDESDB_VERSION[ \t]+\"[0-9]+\\.[0-9]+\\.[0-9]+\"")
+STRING(REGEX REPLACE ".*\"([0-9]+\\.[0-9]+\\.[0-9]+)\".*" "\\1" TIDESDB_VERSION "${_tdb_version_line}")
+
+IF(NOT TIDESDB_VERSION MATCHES "^[0-9]+\\.[0-9]+\\.[0-9]+$")
+  MESSAGE(FATAL_ERROR
+          "TidesDB: could not read TIDESDB_VERSION from "
+          "${TIDESDB_INCLUDE_DIR}/tidesdb/tidesdb_version.h.")
+ENDIF()
+
+STRING(REGEX REPLACE "^([0-9]+)\\..*" "\\1" TIDESDB_VERSION_MAJOR "${TIDESDB_VERSION}")
+
+IF(NOT TIDESDB_VERSION_MAJOR EQUAL TIDESQL_REQUIRED_TIDESDB_MAJOR)
+  MESSAGE(FATAL_ERROR
+          "TidesDB version mismatch: found v${TIDESDB_VERSION} at ${TIDESDB_INCLUDE_DIR}, but this "
+          "TideSQL pairs with TidesDB v${TIDESQL_REQUIRED_TIDESDB_MAJOR}.x.  A different library "
+          "major is a different on-disk format, so the two cannot be mixed.  Install TidesDB "
+          "v${TIDESQL_REQUIRED_TIDESDB_MAJOR}.x, or use the TideSQL release that pairs with "
+          "v${TIDESDB_VERSION_MAJOR}.x -- the pairing is recorded in VERSIONING.md.")
+ENDIF()
+
+MESSAGE(STATUS "TidesDB: v${TIDESDB_VERSION} (major ${TIDESDB_VERSION_MAJOR}, as required)")
+
 INCLUDE_DIRECTORIES(${TIDESDB_INCLUDE_DIR})
 
 # The compression backends are optional.  The library encodes with NONE when it is