Skip to content

Adds a pluggable database configuration backend#756

Open
rbarreiros wants to merge 4 commits into
sm0svx:masterfrom
rbarreiros:config_backend
Open

Adds a pluggable database configuration backend#756
rbarreiros wants to merge 4 commits into
sm0svx:masterfrom
rbarreiros:config_backend

Conversation

@rbarreiros
Copy link
Copy Markdown
Contributor

Sorry, this is an extensive patch!

Introduces a pluggable backend system for AsyncConfig that allows SVXLink configuration to be stored in a relational database (SQLite, MySQL/MariaDB, or PostgreSQL) in addition to the traditional .conf file, with optional live change notifications and hot-reload support. This backend is extendable and allows for other backends to be implemented following the same guidelines as Async*ConfigBackend classes.

Please read src/doc/README_DBCONFIG.md for more information and examples.

New library components (src/async/core/):

  • AsyncConfigBackend: abstract base class for all backends, with a factory registry, table-prefix support, and a non-blocking polling infrastructure (background thread + self-pipe + FdWatch) so that slow DB queries never stall the event loop.

  • AsyncFileConfigBackend: wraps the existing file parser as a proper backend plugin, preserving full backward compatibility.

  • AsyncSQLiteConfigBackend: single-file embedded database backend (optional, enabled when libsqlite3 is detected at build time).

  • AsyncMySQLConfigBackend: MySQL/MariaDB backend (optional, enabled when libmysqlclient or libmariadb is detected at build time). Connection string format: key=value pairs separated by semicolons.

  • AsyncPostgreSQLConfigBackend: PostgreSQL backend (optional, enabled when libpq is detected at build time). Connection string is passed directly to PQconnectdb() as libpq keyword=value pairs.

  • AsyncConfigSource: URL parser and backend-availability registry used by the factory functions createConfigBackend() and createConfigBackendByType().

AsyncConfig rework (src/async/core/AsyncConfig.*):

  • Replaced the monolithic open(filename) call with a clean fallback chain implemented in openWithFallback(): --config file → --dbconfig db.conf → auto-discover db.conf in standard paths → auto-discover application .conf file → fatal error. This change shifts to AsyncConfig all the config (file or other) reading logic instead of the multiple code repetition in each binary.

  • Added openFromDbConfig(path) and openDirect(url) as lower-level entry points for callers that bypass the fallback chain.

  • Centralized CFG_DIR processing into Config::loadCfgDir(); removed duplicate code from every application main().

  • Added valueUpdated sigc++ signal, emitted whenever any config value changes (from setValue, backend poll, or SIGHUP reload).

  • Added subscribeValue() templates that register a typed callback invoked immediately with the current value and again on every future change. Returns a SubId handle.

  • Added subscribeOptionalValue() for keys that must not be auto-created when absent.

  • Added unsubscribeValue(section, tag, SubId) to safely remove a registered callback, preventing dangling-lambda crashes when a subscriber is destroyed before the Config object.

  • Fixed reload() to correctly handle sections and tags that were added to the database after startup (previously new keys were silently ignored).

  • Added importFromConfigFile(filename) as the public entry point for seeding an empty database from an installed .conf file.

  • Added getLastError(), getMainConfigFile(), getBackendType() for better diagnostics in application main().

Application changes (all binaries):

  • Added --dbconfig option to specify a db.conf path explicitly, overriding the automatic search.

  • Added --init-db option to import the installed example .conf file (e.g. svxlink.conf) into an empty database backend, then exit. Import is skipped with a warning if the database already has rows.

  • Removed per-application CFG_DIR processing loops and hardcoded database seeding blocks; replaced with calls to the centralized AsyncConfig methods.

Runtime hot-reload (config value changes without restart):

  • Logic, RepeaterLogic, ReflectorLogic, ReflectorV2Logic: connect to valueUpdated and implement cfgUpdated() for parameters that can be changed on-the-fly (callsign, RF thresholds, IDs, timeouts, etc.).

  • Module base class: cfgUpdated() virtual method propagates valueUpdated events to all loaded modules. ModuleDtmfRepeater, ModuleEchoLink, ModuleFrn, ModuleMetarInfo, ModuleParrot, ModuleTrx: override cfgUpdated() for their own hot-updatable parameters.

  • Squelch, LocalRxBase: cfgUpdated() for squelch thresholds and related RF parameters.

  • QsoRecorder: cfgUpdated() for recording parameters.

  • RfUplink, NetUplink, NetTrxAdapter: cfgUpdated() for remote TRX and network parameters.

  • SipLogic, QsoFrn: cfgUpdated() support for SIP and FRN parameters.

  • Voter, DtmfDecoder: subscribeValue() for voting and DTMF parameters.

  • Logic.tcl: new proc config_updated {section tag value} called by the C++ cfgUpdated() handler so TCL event scripts can react to runtime configuration changes.

Build:

  • CMakeLists.txt: optional detection of libsqlite3, libmysqlclient / libmariadb, and libpq; each backend is compiled in only when its development library is found. Defines HAS_SQLITE_SUPPORT, HAS_MYSQL_SUPPORT, HAS_POSTGRESQL_SUPPORT accordingly.

  • Added -D_REENTRANT for the background polling thread.

New example files:

  • examples/db.conf: fully-commented template for the database config file, placed in the standard search paths.

  • examples/db.conf.sqlite, db.conf.mysql, db.conf.postgresql, db.conf.file: per-backend example files.

New tests added for async config backend (requires catch2)

  • added some test cases to check async config backend, they can be enabled by passing -DBUILD_TESTS=ON to cmake.

  • make all will build tests if enabled, make test will run ctest, running all tests.

  • Single test can be compiled and run,
    make AsyncConfigBackend_test
    ./tests/async/core/AsyncConfigBackend_test

  • A specific backend can be tested
    ./tests/async/core/AsyncConfigBackend_test "[sqlite]"
    ./tests/async/core/AsyncConfigBackend_test "[mysql]"
    ./tests/async/core/AsyncConfigBackend_test "[postgres]"
    ./tests/async/core/AsyncConfigBackend_test "[file]"

Tests which require a database like mysql and postgresql require a connection string.

SVXLINK_TEST_MYSQL_CONN 
SVXLINK_TEST_POSTGRESQL_CONN

example: 
export SVXLINK_TEST_POSTGRESQL_CONN="host=localhost port=5432 dbname=svxlink user=svxlink password=svxlink" 
export SVXLINK_TEST_MYSQL_CONN="host=localhost;port=3306;user=svxlink;password=svxlink;database=svxlink"

Introduces a pluggable backend system for AsyncConfig that allows
SVXLink configuration to be stored in a relational database (SQLite,
MySQL/MariaDB, or PostgreSQL) in addition to the traditional .conf
file, with optional live change notifications and hot-reload support.
This backend is extendable and allows for other backends to be
implemented following the same guidelines as Async*ConfigBackend
classes.

New library components (src/async/core/):

  - AsyncConfigBackend: abstract base class for all backends, with a
    factory registry, table-prefix support, and a non-blocking polling
    infrastructure (background thread + self-pipe + FdWatch) so that
    slow DB queries never stall the event loop.

  - AsyncFileConfigBackend: wraps the existing file parser as a proper
    backend plugin, preserving full backward compatibility.

  - AsyncSQLiteConfigBackend: single-file embedded database backend
    (optional, enabled when libsqlite3 is detected at build time).

  - AsyncMySQLConfigBackend: MySQL/MariaDB backend (optional, enabled
    when libmysqlclient or libmariadb is detected at build time).
    Connection string format: key=value pairs separated by semicolons.

  - AsyncPostgreSQLConfigBackend: PostgreSQL backend (optional, enabled
    when libpq is detected at build time).  Connection string is passed
    directly to PQconnectdb() as libpq keyword=value pairs.

  - AsyncConfigSource: URL parser and backend-availability registry used
    by the factory functions createConfigBackend() and
    createConfigBackendByType().

AsyncConfig rework (src/async/core/AsyncConfig.*):

  - Replaced the monolithic open(filename) call with a clean fallback
    chain implemented in openWithFallback(): --config file →
    --dbconfig db.conf → auto-discover db.conf in standard paths →
    auto-discover application .conf file → fatal error.
    This change allos for the shit to AsyncConfig of all the config
    reading logic instead of the multiple repetition in each binary.

  - Added openFromDbConfig(path) and openDirect(url) as lower-level
    entry points for callers that bypass the fallback chain.

  - Centralized CFG_DIR processing into Config::loadCfgDir(); removed
    duplicate code from every application main().

  - Added valueUpdated sigc++ signal, emitted whenever any config value
    changes (from setValue, backend poll, or SIGHUP reload).

  - Added subscribeValue() templates that register a typed callback
    invoked immediately with the current value and again on every future
    change.  Returns a SubId handle.

  - Added subscribeOptionalValue() for keys that must not be
    auto-created when absent.

  - Added unsubscribeValue(section, tag, SubId) to safely remove a
    registered callback, preventing dangling-lambda crashes when a
    subscriber is destroyed before the Config object.

  - Fixed reload() to correctly handle sections and tags that were added
    to the database after startup (previously new keys were silently
    ignored).

  - Added importFromConfigFile(filename) as the public entry point for
    seeding an empty database from an installed .conf file.

  - Added getLastError(), getMainConfigFile(), getBackendType() for
    better diagnostics in application main().

  - Removed AsyncConfigManager (its logic was absorbed into AsyncConfig).

Application changes (all binaries):

  - Added --dbconfig <file> option to specify a db.conf path explicitly,
    overriding the automatic search.

  - Added --init-db option to import the installed example .conf file
    (e.g. svxlink.conf) into an empty database backend, then exit.
    Import is skipped with a warning if the database already has rows.

  - Removed per-application CFG_DIR processing loops and hardcoded
    database seeding blocks; replaced with calls to the centralized
    AsyncConfig methods.

Runtime hot-reload (config value changes without restart):

  - Logic, RepeaterLogic, ReflectorLogic, ReflectorV2Logic: connect to
    valueUpdated and implement cfgUpdated() for parameters that can be
    changed on-the-fly (callsign, RF thresholds, IDs, timeouts, etc.).

  - Module base class: cfgUpdated() virtual method propagates valueUpdated
    events to all loaded modules.  ModuleDtmfRepeater, ModuleEchoLink,
    ModuleFrn, ModuleMetarInfo, ModuleParrot, ModuleTrx: override
    cfgUpdated() for their own hot-updatable parameters.

  - Squelch, LocalRxBase: cfgUpdated() for squelch thresholds and
    related RF parameters.

  - QsoRecorder: cfgUpdated() for recording parameters.

  - RfUplink, NetUplink, NetTrxAdapter: cfgUpdated() for remote TRX
    and network parameters.

  - SipLogic, QsoFrn: cfgUpdated() support for SIP and FRN parameters.

  - Voter, DtmfDecoder: subscribeValue() for voting and DTMF parameters.

  - Logic.tcl: new proc config_updated {section tag value} called by
    the C++ cfgUpdated() handler so TCL event scripts can react to
    runtime configuration changes.

  - svxreflector / Reflector: cfgUpdated() for all hot-updatable
    reflector parameters.  Fixed RANDOM_QSY_RANGE parsing in cfgUpdated
    to use SepPair<uint32_t,uint32_t> (lo + span - 1), matching the
    initialization path; previously both lo and hi were set to atoi of
    the raw string, producing wrong values on hot-update.

Bug fixes:

  - AsyncConfig::subscribeValue(): changed Value::subs from std::vector
    to std::map<SubId, Subscriber> so individual callbacks can be
    removed without invalidating others if necessary....

Build:

  - CMakeLists.txt: optional detection of libsqlite3, libmysqlclient /
    libmariadb, and libpq; each backend is compiled in only when its
    development library is found.  Defines HAS_SQLITE_SUPPORT,
    HAS_MYSQL_SUPPORT, HAS_POSTGRESQL_SUPPORT accordingly.

  - Added -D_REENTRANT for the background polling thread.

New example files:

  - examples/db.conf: fully-commented template for the database config
    file, placed in the standard search paths.

  - examples/db.conf.sqlite, db.conf.mysql, db.conf.postgresql,
    db.conf.file: per-backend example files.

  - README_DBCONFIG.md: end-user documentation covering configuration
    format, table schema, per-backend setup commands, --init-db
    workflow, live change notifications, and table prefix rules.

New tests added for async config backend (requires catch2)

  - added some test cases to check async config backend, they can be
    enabled by passing -DBUILD_TESTS=ON to cmake.
    make all will build tests if enabled, or, make tests will build
    only the tests.
    Single test can be compiled and run
      make AsyncConfigBackend_test
      ./tests/async/core/AsyncConfigBackend_test

    Or, a specific backend
      ./tests/async/core/AsyncConfigBackend_test "[sqlite]"
      ./tests/async/core/AsyncConfigBackend_test "[mysql]"
      ./tests/async/core/AsyncConfigBackend_test "[postgres]"
      ./tests/async/core/AsyncConfigBackend_test "[file]"

    Tests which require a database like mysql and postgresql require
    a connection string.

    SVXLINK_TEST_MYSQL_CONN
    SVXLINK_TEST_POSTGRESQL_CONN

    example:
    export SVXLINK_TEST_POSTGRESQL_CONN="host=localhost port=5432 dbname=svxlink user=svxlink password=svxlink"
    export SVXLINK_TEST_MYSQL_CONN="host=localhost;port=3306;user=svxlink;password=svxlink;database=svxlink"

    All tests can be run with ctest
@rbarreiros
Copy link
Copy Markdown
Contributor Author

rbarreiros commented Apr 15, 2026

I forgot I'm using std::optional and didn't specify the c++17 in cmake. Do you wish me to remove std::optional or add c++17 requirement ?

P.S. I forgot the structured bindings as well, I think the old debian buster already supports c++17 :)

Comment thread src/async/demo/AsyncConfigBackend_demo.cpp Fixed
…as a basis to the sqlite demo db location, it's hardcoded to /tmp when no config dir is supplied and if it is, it expects the full path exists and is writeable.
- Forgot some old code while refactoring which introduced a connection issue with mysql and postgresql, which is now fixed.

- Stupid segfault which was a headache to find, when a loaded module subscribed a config change, when exiting the application
and destroying and unloading the modules it would keep dangling pointers in the cfg object from the value change subscriptions
and cause a segfault. Created a Subscription handle returned by subscribeValue and subscribeOptionalValue so that when the holder
is destroyed, this class destruction is called and the subscription removed, this way all modules only have to worry about
having a class member holding this subscription, and everything else is dont in asyncconfig.

I'm left wondering, is subscribeValue/subscribeOptionalValue really needed with the new cfgUpdated signal ?
subscribeValue and subscribeOptionalValue fire immediately and do type conversion, cfgUpated doesn't.... do they really need
to be updated immediately ?!?! Removing subscribeValue and subscribeOptionalValue would simplify things a bit :/
@rbarreiros
Copy link
Copy Markdown
Contributor Author

So sorry for these late commits Tobias... Pretty tiring few days and was in a hurry to get the PR so I missed a few important things in a refactoring I did a while ago. I made a quick test on all backends, including polling and the thread handling when the backend goes away and everything seems ok now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants