Adds a pluggable database configuration backend#756
Open
rbarreiros wants to merge 4 commits into
Open
Conversation
Introduces a pluggable backend system for AsyncConfig that allows
SVXLink configuration to be stored in a relational database (SQLite,
MySQL/MariaDB, or PostgreSQL) in addition to the traditional .conf
file, with optional live change notifications and hot-reload support.
This backend is extendable and allows for other backends to be
implemented following the same guidelines as Async*ConfigBackend
classes.
New library components (src/async/core/):
- AsyncConfigBackend: abstract base class for all backends, with a
factory registry, table-prefix support, and a non-blocking polling
infrastructure (background thread + self-pipe + FdWatch) so that
slow DB queries never stall the event loop.
- AsyncFileConfigBackend: wraps the existing file parser as a proper
backend plugin, preserving full backward compatibility.
- AsyncSQLiteConfigBackend: single-file embedded database backend
(optional, enabled when libsqlite3 is detected at build time).
- AsyncMySQLConfigBackend: MySQL/MariaDB backend (optional, enabled
when libmysqlclient or libmariadb is detected at build time).
Connection string format: key=value pairs separated by semicolons.
- AsyncPostgreSQLConfigBackend: PostgreSQL backend (optional, enabled
when libpq is detected at build time). Connection string is passed
directly to PQconnectdb() as libpq keyword=value pairs.
- AsyncConfigSource: URL parser and backend-availability registry used
by the factory functions createConfigBackend() and
createConfigBackendByType().
AsyncConfig rework (src/async/core/AsyncConfig.*):
- Replaced the monolithic open(filename) call with a clean fallback
chain implemented in openWithFallback(): --config file →
--dbconfig db.conf → auto-discover db.conf in standard paths →
auto-discover application .conf file → fatal error.
This change allos for the shit to AsyncConfig of all the config
reading logic instead of the multiple repetition in each binary.
- Added openFromDbConfig(path) and openDirect(url) as lower-level
entry points for callers that bypass the fallback chain.
- Centralized CFG_DIR processing into Config::loadCfgDir(); removed
duplicate code from every application main().
- Added valueUpdated sigc++ signal, emitted whenever any config value
changes (from setValue, backend poll, or SIGHUP reload).
- Added subscribeValue() templates that register a typed callback
invoked immediately with the current value and again on every future
change. Returns a SubId handle.
- Added subscribeOptionalValue() for keys that must not be
auto-created when absent.
- Added unsubscribeValue(section, tag, SubId) to safely remove a
registered callback, preventing dangling-lambda crashes when a
subscriber is destroyed before the Config object.
- Fixed reload() to correctly handle sections and tags that were added
to the database after startup (previously new keys were silently
ignored).
- Added importFromConfigFile(filename) as the public entry point for
seeding an empty database from an installed .conf file.
- Added getLastError(), getMainConfigFile(), getBackendType() for
better diagnostics in application main().
- Removed AsyncConfigManager (its logic was absorbed into AsyncConfig).
Application changes (all binaries):
- Added --dbconfig <file> option to specify a db.conf path explicitly,
overriding the automatic search.
- Added --init-db option to import the installed example .conf file
(e.g. svxlink.conf) into an empty database backend, then exit.
Import is skipped with a warning if the database already has rows.
- Removed per-application CFG_DIR processing loops and hardcoded
database seeding blocks; replaced with calls to the centralized
AsyncConfig methods.
Runtime hot-reload (config value changes without restart):
- Logic, RepeaterLogic, ReflectorLogic, ReflectorV2Logic: connect to
valueUpdated and implement cfgUpdated() for parameters that can be
changed on-the-fly (callsign, RF thresholds, IDs, timeouts, etc.).
- Module base class: cfgUpdated() virtual method propagates valueUpdated
events to all loaded modules. ModuleDtmfRepeater, ModuleEchoLink,
ModuleFrn, ModuleMetarInfo, ModuleParrot, ModuleTrx: override
cfgUpdated() for their own hot-updatable parameters.
- Squelch, LocalRxBase: cfgUpdated() for squelch thresholds and
related RF parameters.
- QsoRecorder: cfgUpdated() for recording parameters.
- RfUplink, NetUplink, NetTrxAdapter: cfgUpdated() for remote TRX
and network parameters.
- SipLogic, QsoFrn: cfgUpdated() support for SIP and FRN parameters.
- Voter, DtmfDecoder: subscribeValue() for voting and DTMF parameters.
- Logic.tcl: new proc config_updated {section tag value} called by
the C++ cfgUpdated() handler so TCL event scripts can react to
runtime configuration changes.
- svxreflector / Reflector: cfgUpdated() for all hot-updatable
reflector parameters. Fixed RANDOM_QSY_RANGE parsing in cfgUpdated
to use SepPair<uint32_t,uint32_t> (lo + span - 1), matching the
initialization path; previously both lo and hi were set to atoi of
the raw string, producing wrong values on hot-update.
Bug fixes:
- AsyncConfig::subscribeValue(): changed Value::subs from std::vector
to std::map<SubId, Subscriber> so individual callbacks can be
removed without invalidating others if necessary....
Build:
- CMakeLists.txt: optional detection of libsqlite3, libmysqlclient /
libmariadb, and libpq; each backend is compiled in only when its
development library is found. Defines HAS_SQLITE_SUPPORT,
HAS_MYSQL_SUPPORT, HAS_POSTGRESQL_SUPPORT accordingly.
- Added -D_REENTRANT for the background polling thread.
New example files:
- examples/db.conf: fully-commented template for the database config
file, placed in the standard search paths.
- examples/db.conf.sqlite, db.conf.mysql, db.conf.postgresql,
db.conf.file: per-backend example files.
- README_DBCONFIG.md: end-user documentation covering configuration
format, table schema, per-backend setup commands, --init-db
workflow, live change notifications, and table prefix rules.
New tests added for async config backend (requires catch2)
- added some test cases to check async config backend, they can be
enabled by passing -DBUILD_TESTS=ON to cmake.
make all will build tests if enabled, or, make tests will build
only the tests.
Single test can be compiled and run
make AsyncConfigBackend_test
./tests/async/core/AsyncConfigBackend_test
Or, a specific backend
./tests/async/core/AsyncConfigBackend_test "[sqlite]"
./tests/async/core/AsyncConfigBackend_test "[mysql]"
./tests/async/core/AsyncConfigBackend_test "[postgres]"
./tests/async/core/AsyncConfigBackend_test "[file]"
Tests which require a database like mysql and postgresql require
a connection string.
SVXLINK_TEST_MYSQL_CONN
SVXLINK_TEST_POSTGRESQL_CONN
example:
export SVXLINK_TEST_POSTGRESQL_CONN="host=localhost port=5432 dbname=svxlink user=svxlink password=svxlink"
export SVXLINK_TEST_MYSQL_CONN="host=localhost;port=3306;user=svxlink;password=svxlink;database=svxlink"
All tests can be run with ctest
Contributor
Author
|
I forgot I'm using std::optional and didn't specify the c++17 in cmake. Do you wish me to remove std::optional or add c++17 requirement ? P.S. I forgot the structured bindings as well, I think the old debian buster already supports c++17 :) |
…as a basis to the sqlite demo db location, it's hardcoded to /tmp when no config dir is supplied and if it is, it expects the full path exists and is writeable.
- Forgot some old code while refactoring which introduced a connection issue with mysql and postgresql, which is now fixed. - Stupid segfault which was a headache to find, when a loaded module subscribed a config change, when exiting the application and destroying and unloading the modules it would keep dangling pointers in the cfg object from the value change subscriptions and cause a segfault. Created a Subscription handle returned by subscribeValue and subscribeOptionalValue so that when the holder is destroyed, this class destruction is called and the subscription removed, this way all modules only have to worry about having a class member holding this subscription, and everything else is dont in asyncconfig. I'm left wondering, is subscribeValue/subscribeOptionalValue really needed with the new cfgUpdated signal ? subscribeValue and subscribeOptionalValue fire immediately and do type conversion, cfgUpated doesn't.... do they really need to be updated immediately ?!?! Removing subscribeValue and subscribeOptionalValue would simplify things a bit :/
Contributor
Author
|
So sorry for these late commits Tobias... Pretty tiring few days and was in a hurry to get the PR so I missed a few important things in a refactoring I did a while ago. I made a quick test on all backends, including polling and the thread handling when the backend goes away and everything seems ok now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sorry, this is an extensive patch!
Introduces a pluggable backend system for AsyncConfig that allows SVXLink configuration to be stored in a relational database (SQLite, MySQL/MariaDB, or PostgreSQL) in addition to the traditional .conf file, with optional live change notifications and hot-reload support. This backend is extendable and allows for other backends to be implemented following the same guidelines as Async*ConfigBackend classes.
Please read src/doc/README_DBCONFIG.md for more information and examples.
New library components (src/async/core/):
AsyncConfigBackend: abstract base class for all backends, with a factory registry, table-prefix support, and a non-blocking polling infrastructure (background thread + self-pipe + FdWatch) so that slow DB queries never stall the event loop.
AsyncFileConfigBackend: wraps the existing file parser as a proper backend plugin, preserving full backward compatibility.
AsyncSQLiteConfigBackend: single-file embedded database backend (optional, enabled when libsqlite3 is detected at build time).
AsyncMySQLConfigBackend: MySQL/MariaDB backend (optional, enabled when libmysqlclient or libmariadb is detected at build time). Connection string format: key=value pairs separated by semicolons.
AsyncPostgreSQLConfigBackend: PostgreSQL backend (optional, enabled when libpq is detected at build time). Connection string is passed directly to PQconnectdb() as libpq keyword=value pairs.
AsyncConfigSource: URL parser and backend-availability registry used by the factory functions createConfigBackend() and createConfigBackendByType().
AsyncConfig rework (src/async/core/AsyncConfig.*):
Replaced the monolithic open(filename) call with a clean fallback chain implemented in openWithFallback(): --config file → --dbconfig db.conf → auto-discover db.conf in standard paths → auto-discover application .conf file → fatal error. This change shifts to AsyncConfig all the config (file or other) reading logic instead of the multiple code repetition in each binary.
Added openFromDbConfig(path) and openDirect(url) as lower-level entry points for callers that bypass the fallback chain.
Centralized CFG_DIR processing into Config::loadCfgDir(); removed duplicate code from every application main().
Added valueUpdated sigc++ signal, emitted whenever any config value changes (from setValue, backend poll, or SIGHUP reload).
Added subscribeValue() templates that register a typed callback invoked immediately with the current value and again on every future change. Returns a SubId handle.
Added subscribeOptionalValue() for keys that must not be auto-created when absent.
Added unsubscribeValue(section, tag, SubId) to safely remove a registered callback, preventing dangling-lambda crashes when a subscriber is destroyed before the Config object.
Fixed reload() to correctly handle sections and tags that were added to the database after startup (previously new keys were silently ignored).
Added importFromConfigFile(filename) as the public entry point for seeding an empty database from an installed .conf file.
Added getLastError(), getMainConfigFile(), getBackendType() for better diagnostics in application main().
Application changes (all binaries):
Added --dbconfig option to specify a db.conf path explicitly, overriding the automatic search.
Added --init-db option to import the installed example .conf file (e.g. svxlink.conf) into an empty database backend, then exit. Import is skipped with a warning if the database already has rows.
Removed per-application CFG_DIR processing loops and hardcoded database seeding blocks; replaced with calls to the centralized AsyncConfig methods.
Runtime hot-reload (config value changes without restart):
Logic, RepeaterLogic, ReflectorLogic, ReflectorV2Logic: connect to valueUpdated and implement cfgUpdated() for parameters that can be changed on-the-fly (callsign, RF thresholds, IDs, timeouts, etc.).
Module base class: cfgUpdated() virtual method propagates valueUpdated events to all loaded modules. ModuleDtmfRepeater, ModuleEchoLink, ModuleFrn, ModuleMetarInfo, ModuleParrot, ModuleTrx: override cfgUpdated() for their own hot-updatable parameters.
Squelch, LocalRxBase: cfgUpdated() for squelch thresholds and related RF parameters.
QsoRecorder: cfgUpdated() for recording parameters.
RfUplink, NetUplink, NetTrxAdapter: cfgUpdated() for remote TRX and network parameters.
SipLogic, QsoFrn: cfgUpdated() support for SIP and FRN parameters.
Voter, DtmfDecoder: subscribeValue() for voting and DTMF parameters.
Logic.tcl: new proc config_updated {section tag value} called by the C++ cfgUpdated() handler so TCL event scripts can react to runtime configuration changes.
Build:
CMakeLists.txt: optional detection of libsqlite3, libmysqlclient / libmariadb, and libpq; each backend is compiled in only when its development library is found. Defines HAS_SQLITE_SUPPORT, HAS_MYSQL_SUPPORT, HAS_POSTGRESQL_SUPPORT accordingly.
Added -D_REENTRANT for the background polling thread.
New example files:
examples/db.conf: fully-commented template for the database config file, placed in the standard search paths.
examples/db.conf.sqlite, db.conf.mysql, db.conf.postgresql, db.conf.file: per-backend example files.
New tests added for async config backend (requires catch2)
added some test cases to check async config backend, they can be enabled by passing -DBUILD_TESTS=ON to cmake.
make all will build tests if enabled, make test will run ctest, running all tests.
Single test can be compiled and run,
make AsyncConfigBackend_test
./tests/async/core/AsyncConfigBackend_test
A specific backend can be tested
./tests/async/core/AsyncConfigBackend_test "[sqlite]"
./tests/async/core/AsyncConfigBackend_test "[mysql]"
./tests/async/core/AsyncConfigBackend_test "[postgres]"
./tests/async/core/AsyncConfigBackend_test "[file]"