Skip to content

WIP: onie-tools: add a tool for configuration backup#315

Draft
KanjiMonster wants to merge 9 commits intomainfrom
jogo_backup_restore
Draft

WIP: onie-tools: add a tool for configuration backup#315
KanjiMonster wants to merge 9 commits intomainfrom
jogo_backup_restore

Conversation

@KanjiMonster
Copy link
Copy Markdown
Contributor

@KanjiMonster KanjiMonster commented May 26, 2025

In order to make configuration handling on BISDN Linux more transparent and user friendly, let's add a tool for creating and restoring backups.

The tool can be used in three ways:

  1. Create a configuration backup:
$ onie-bisdn-backup -c backup.tgz
  1. Restore a configuration backup:
$ onie-bisdn-backup -r backup.tgz
  1. Display what will be included in backup
$ sudo onie-bisdn-backup -l
Files to backup:
/etc/ca-certificates.conf
/etc/default/baseboxd
/etc/default/ofagent
/etc/default/ofdpa
/etc/default/ofdpa_log.cfg
/etc/default/ryu-manager
/etc/default/user-backup.txt
/etc/frr/babeld.conf
/etc/frr/bfdd.conf
/etc/frr/bgpd.conf
/etc/frr/eigrpd.conf
/etc/frr/isisd.conf
/etc/frr/ldpd.conf
/etc/frr/mgmtd.conf
/etc/frr/nhrpd.conf
/etc/frr/ospf6d.conf
/etc/frr/ospfd.conf
/etc/frr/pathd.conf
/etc/frr/pbrd.conf
/etc/frr/pimd.conf
/etc/frr/ripd.conf
/etc/frr/ripngd.conf
/etc/frr/sharpd.conf
/etc/frr/staticd.conf
/etc/frr/vtysh.conf
/etc/fstab
/etc/group
/etc/gshadow
/etc/hostapd.conf
/etc/hostapd/hostapd-wired.conf.example
/etc/hostname
/etc/hosts
/etc/inputrc
/etc/modprobe.d/linux-kernel-bde.conf
/etc/modules-load.d/openvswitch.conf
/etc/motd
/etc/nsswitch.conf
/etc/ofdpa-grpc.conf
/etc/ofdpa/rc.soc
/etc/passwd
/etc/profile
/etc/raddb/README.rst
/etc/raddb/certs/01.pem
/etc/raddb/certs/01.pem-default
/etc/raddb/certs/02.pem
/etc/raddb/certs/02.pem-default
/etc/raddb/certs/Makefile
/etc/raddb/certs/README.md
/etc/raddb/certs/bootstrap
/etc/raddb/certs/ca.cnf
/etc/raddb/certs/ca.crl
/etc/raddb/certs/ca.crl-default
/etc/raddb/certs/ca.der
/etc/raddb/certs/ca.der-default
/etc/raddb/certs/ca.key
/etc/raddb/certs/ca.key-default
/etc/raddb/certs/ca.pem
/etc/raddb/certs/ca.pem-default
/etc/raddb/certs/client.cnf
/etc/raddb/certs/client.crt
/etc/raddb/certs/client.crt-default
/etc/raddb/certs/client.csr
/etc/raddb/certs/client.csr-default
/etc/raddb/certs/client.key
/etc/raddb/certs/client.key-default
/etc/raddb/certs/client.p12
/etc/raddb/certs/client.p12-default
/etc/raddb/certs/client.pem
/etc/raddb/certs/client.pem-default
/etc/raddb/certs/dh
/etc/raddb/certs/dh-default
/etc/raddb/certs/index.txt
/etc/raddb/certs/index.txt-default
/etc/raddb/certs/index.txt.attr
/etc/raddb/certs/index.txt.attr.old
/etc/raddb/certs/index.txt.old
/etc/raddb/certs/index.txt.old-default
/etc/raddb/certs/inner-server.cnf
/etc/raddb/certs/passwords.mk
/etc/raddb/certs/serial
/etc/raddb/certs/serial.old
/etc/raddb/certs/server.cnf
/etc/raddb/certs/server.crt
/etc/raddb/certs/server.crt-default
/etc/raddb/certs/server.csr
/etc/raddb/certs/server.csr-default
/etc/raddb/certs/server.key
/etc/raddb/certs/server.key-default
/etc/raddb/certs/server.p12
/etc/raddb/certs/server.p12-default
/etc/raddb/certs/server.pem
/etc/raddb/certs/server.pem-default
/etc/raddb/certs/user@example.org.p12
/etc/raddb/certs/user@example.org.p12-default
/etc/raddb/certs/user@example.org.pem
/etc/raddb/certs/user@example.org.pem-default
/etc/raddb/certs/xpextensions
/etc/raddb/clients.conf
/etc/raddb/dictionary
/etc/raddb/mods-available/README.rst
/etc/raddb/mods-available/abfab_psk_sql
/etc/raddb/mods-available/always
/etc/raddb/mods-available/attr_filter
/etc/raddb/mods-available/cache
/etc/raddb/mods-available/cache_auth
/etc/raddb/mods-available/chap
/etc/raddb/mods-available/couchbase
/etc/raddb/mods-available/counter
/etc/raddb/mods-available/cui
/etc/raddb/mods-available/date
/etc/raddb/mods-available/detail
/etc/raddb/mods-available/detail.example.com
/etc/raddb/mods-available/detail.log
/etc/raddb/mods-available/dhcp
/etc/raddb/mods-available/dhcp_files
/etc/raddb/mods-available/dhcp_passwd
/etc/raddb/mods-available/dhcp_sql
/etc/raddb/mods-available/dhcp_sqlippool
/etc/raddb/mods-available/digest
/etc/raddb/mods-available/dynamic_clients
/etc/raddb/mods-available/eap
/etc/raddb/mods-available/echo
/etc/raddb/mods-available/etc_group
/etc/raddb/mods-available/exec
/etc/raddb/mods-available/expiration
/etc/raddb/mods-available/expr
/etc/raddb/mods-available/files
/etc/raddb/mods-available/idn
/etc/raddb/mods-available/inner-eap
/etc/raddb/mods-available/ippool
/etc/raddb/mods-available/ldap_google
/etc/raddb/mods-available/linelog
/etc/raddb/mods-available/logintime
/etc/raddb/mods-available/mac2ip
/etc/raddb/mods-available/mac2vlan
/etc/raddb/mods-available/moonshot-targeted-ids
/etc/raddb/mods-available/mschap
/etc/raddb/mods-available/ntlm_auth
/etc/raddb/mods-available/opendirectory
/etc/raddb/mods-available/otp
/etc/raddb/mods-available/pam
/etc/raddb/mods-available/pap
/etc/raddb/mods-available/passwd
/etc/raddb/mods-available/preprocess
/etc/raddb/mods-available/python
/etc/raddb/mods-available/radutmp
/etc/raddb/mods-available/realm
/etc/raddb/mods-available/redis
/etc/raddb/mods-available/rediswho
/etc/raddb/mods-available/replicate
/etc/raddb/mods-available/rest
/etc/raddb/mods-available/smbpasswd
/etc/raddb/mods-available/smsotp
/etc/raddb/mods-available/soh
/etc/raddb/mods-available/sometimes
/etc/raddb/mods-available/sql_map
/etc/raddb/mods-available/sqlcounter
/etc/raddb/mods-available/sqlippool
/etc/raddb/mods-available/sradutmp
/etc/raddb/mods-available/totp
/etc/raddb/mods-available/unbound
/etc/raddb/mods-available/unix
/etc/raddb/mods-available/unpack
/etc/raddb/mods-available/utf8
/etc/raddb/mods-available/wimax
/etc/raddb/mods-available/yubikey
/etc/raddb/mods-config/README.rst
/etc/raddb/mods-config/attr_filter/access_challenge
/etc/raddb/mods-config/attr_filter/access_reject
/etc/raddb/mods-config/attr_filter/accounting_response
/etc/raddb/mods-config/attr_filter/coa
/etc/raddb/mods-config/attr_filter/post-proxy
/etc/raddb/mods-config/attr_filter/pre-proxy
/etc/raddb/mods-config/files/accounting
/etc/raddb/mods-config/files/authorize
/etc/raddb/mods-config/files/authorize-default
/etc/raddb/mods-config/files/dhcp
/etc/raddb/mods-config/files/pre-proxy
/etc/raddb/mods-config/preprocess/hints
/etc/raddb/mods-config/preprocess/huntgroups
/etc/raddb/mods-config/sql/counter/sqlite/dailycounter.conf
/etc/raddb/mods-config/sql/counter/sqlite/expire_on_login.conf
/etc/raddb/mods-config/sql/counter/sqlite/monthlycounter.conf
/etc/raddb/mods-config/sql/counter/sqlite/noresetcounter.conf
/etc/raddb/mods-config/sql/counter/sqlite/weeklycounter.conf
/etc/raddb/mods-config/sql/cui/sqlite/queries.conf
/etc/raddb/mods-config/sql/cui/sqlite/schema.sql
/etc/raddb/mods-config/sql/dhcp/mssql/queries.conf
/etc/raddb/mods-config/sql/dhcp/mssql/schema.sql
/etc/raddb/mods-config/sql/dhcp/oracle/queries.conf
/etc/raddb/mods-config/sql/dhcp/oracle/schema.sql
/etc/raddb/mods-config/sql/dhcp/sqlite/queries.conf
/etc/raddb/mods-config/sql/dhcp/sqlite/schema.sql
/etc/raddb/mods-config/sql/ippool-dhcp/mssql/procedure.sql
/etc/raddb/mods-config/sql/ippool-dhcp/mssql/queries.conf
/etc/raddb/mods-config/sql/ippool-dhcp/mssql/schema.sql
/etc/raddb/mods-config/sql/ippool-dhcp/oracle/procedure.sql
/etc/raddb/mods-config/sql/ippool-dhcp/oracle/queries.conf
/etc/raddb/mods-config/sql/ippool-dhcp/oracle/schema.sql
/etc/raddb/mods-config/sql/ippool-dhcp/sqlite/queries.conf
/etc/raddb/mods-config/sql/ippool-dhcp/sqlite/schema.sql
/etc/raddb/mods-config/sql/ippool/mongo/queries.conf
/etc/raddb/mods-config/sql/ippool/mssql/procedure.sql
/etc/raddb/mods-config/sql/ippool/mssql/queries.conf
/etc/raddb/mods-config/sql/ippool/mssql/schema.sql
/etc/raddb/mods-config/sql/ippool/oracle/procedure.sql
/etc/raddb/mods-config/sql/ippool/oracle/queries.conf
/etc/raddb/mods-config/sql/ippool/oracle/schema.sql
/etc/raddb/mods-config/sql/ippool/sqlite/queries.conf
/etc/raddb/mods-config/sql/ippool/sqlite/schema.sql
/etc/raddb/mods-config/sql/main/mongo/queries.conf
/etc/raddb/mods-config/sql/main/mssql/process-radacct.sql
/etc/raddb/mods-config/sql/main/mssql/queries.conf
/etc/raddb/mods-config/sql/main/mssql/schema.sql
/etc/raddb/mods-config/sql/main/ndb/README
/etc/raddb/mods-config/sql/main/ndb/schema.sql
/etc/raddb/mods-config/sql/main/ndb/setup.sql
/etc/raddb/mods-config/sql/main/oracle/process-radacct.sql
/etc/raddb/mods-config/sql/main/oracle/queries.conf
/etc/raddb/mods-config/sql/main/oracle/schema.sql
/etc/raddb/mods-config/sql/main/sqlite/process-radacct-refresh.sh
/etc/raddb/mods-config/sql/main/sqlite/process-radacct-schema.sql
/etc/raddb/mods-config/sql/main/sqlite/queries.conf
/etc/raddb/mods-config/sql/main/sqlite/schema.sql
/etc/raddb/mods-config/sql/moonshot-targeted-ids/sqlite/queries.conf
/etc/raddb/mods-config/sql/moonshot-targeted-ids/sqlite/schema.sql
/etc/raddb/mods-config/unbound/default.conf
/etc/raddb/panic.gdb
/etc/raddb/policy.d/abfab-tr
/etc/raddb/policy.d/accounting
/etc/raddb/policy.d/canonicalization
/etc/raddb/policy.d/control
/etc/raddb/policy.d/cui
/etc/raddb/policy.d/debug
/etc/raddb/policy.d/dhcp
/etc/raddb/policy.d/eap
/etc/raddb/policy.d/filter
/etc/raddb/policy.d/moonshot-targeted-ids
/etc/raddb/policy.d/operator-name
/etc/raddb/policy.d/rfc7542
/etc/raddb/proxy.conf
/etc/raddb/radiusd.conf
/etc/raddb/sites-available/README
/etc/raddb/sites-available/abfab-tls
/etc/raddb/sites-available/abfab-tr-idp
/etc/raddb/sites-available/buffered-sql
/etc/raddb/sites-available/challenge
/etc/raddb/sites-available/channel_bindings
/etc/raddb/sites-available/check-eap-tls
/etc/raddb/sites-available/coa
/etc/raddb/sites-available/coa-relay
/etc/raddb/sites-available/control-socket
/etc/raddb/sites-available/copy-acct-to-home-server
/etc/raddb/sites-available/decoupled-accounting
/etc/raddb/sites-available/default
/etc/raddb/sites-available/default-default
/etc/raddb/sites-available/dhcp
/etc/raddb/sites-available/dhcp.relay
/etc/raddb/sites-available/dynamic-clients
/etc/raddb/sites-available/example
/etc/raddb/sites-available/google-ldap-auth
/etc/raddb/sites-available/inner-tunnel
/etc/raddb/sites-available/originate-coa
/etc/raddb/sites-available/proxy-inner-tunnel
/etc/raddb/sites-available/resource-check
/etc/raddb/sites-available/robust-proxy-accounting
/etc/raddb/sites-available/soh
/etc/raddb/sites-available/status
/etc/raddb/sites-available/tls
/etc/raddb/sites-available/totp
/etc/raddb/sites-available/virtual.example.com
/etc/raddb/sites-available/vmps
/etc/raddb/templates.conf
/etc/raddb/trigger.conf
/etc/shadow
/etc/shells
/etc/ssh/ssh_config
/etc/ssh/ssh_host_ecdsa_key
/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub
/etc/ssh/sshd_config
/etc/ssl/openssl.cnf
/etc/sudoers
/etc/sudoers.d/basebox
/etc/systemd/coredump.conf
/etc/systemd/journald.conf
/etc/systemd/logind.conf
/etc/systemd/networkd.conf
/etc/systemd/pstore.conf
/etc/systemd/resolved.conf
/etc/systemd/sleep.conf
/etc/systemd/system.conf
/etc/systemd/timesyncd.conf
/etc/systemd/user.conf

Services to enable:
 docker

Services to disable:

$

In order to make it work the same as backing up when upgrading, the code from the installer is reused as a library. To allow this, the file is moved here.

In order to keep stdout clean, move all instanced of DEBUG output to
stderr.

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
The two paths for enabled and disabled services have nothing in common,
so let's split these to two functions to avoid one level of indentation.

No functional changes.

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
Move apply fixups to the backup of files, since they belong together.

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
Only create .SERVICES_{DIS,EN}ABLED when not empty on backup, else the
check for if we even backed up anything would always trigger since we
would always have these files.

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
@KanjiMonster KanjiMonster changed the title onie-tools: add a tool for configuration backup WIP: onie-tools: add a tool for configuration backup May 26, 2025
@KanjiMonster KanjiMonster force-pushed the jogo_backup_restore branch from f8c3954 to 2b31b74 Compare May 26, 2025 13:08
Instead of directly copying files while parsing backup configuration,
just create a list of files to copy, then copy everthing in one step.

As a side effect this allows using "-/" to prevent backup of any files
(it won't prevent backing up system service states though).

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
To be able to tell users what will be backed up, make the backup
function only output what will be backed up if the enviornment variable
DRYRUN is set.

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
In order to have the backup library available in BISDN Linux, move it
into the onie-tools package and pull it from there during image
generation.

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
Enable the comm applet as it's needed for our configuration backup
scripts.

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
Add a simple configuration backup and restore tool. It has three modes:

- backup: create a backup as a gzipped tar
- restore: restore a previous backup
- list: list everything that would be backed up

Signed-off-by: Jonas Gorski <jonas.gorski@bisdn.de>
@KanjiMonster KanjiMonster force-pushed the jogo_backup_restore branch from 2b31b74 to dd2fcc9 Compare May 26, 2025 15:40
@ideaship
Copy link
Copy Markdown
Contributor

I welcome the possibility to dry-run and test the backup process without going through a reinstallation of the OS.

@KanjiMonster
Copy link
Copy Markdown
Contributor Author

KanjiMonster commented Jun 17, 2025

I welcome the possibility to dry-run and test the backup process without going through a reinstallation of the OS.

Thanks! And what do you think of the (example) output? Do you think it is understandable that way?

if [ -d "$oldpath" ]; then
for path in $oldpath/*; do
[ -e "$path" ] || break
add_to_backup "/${path#${2%/}/}" $2 $3
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment explaining this line might be helpful.

*.service)
service_file=$(readlink $1/etc/systemd/system/multi-user.target.wants/$service)
base=$(basename $service_file)
preset=$(grep "$base" $1/lib/systemd/system-preset/*.preset | cut -d ':' -f 2 | cut -d ' ' -f 1)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment explaining the format we are processing here maybe?

for file in $1/var/lib/opkg/info/*.conffiles; do
[ -f "$file" ] || break
parse_file $file "+" $1 $2
add_files="$(echo -e "$add_files\n$(parse_file $file "+" $1 $2)")"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An explanation might be useful here, too.

Example usage:
$(basename "$0") -b bisdn-linux-backup.tgz

For additional information, see the man page for onie-bisdn-backup(1).
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure the man page is actually there.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is why the PR is still a draft, I didn't want to bother writing a man page just to have to rewrite it because the output should look totally different.

-h show this message
-l list backup contents (show only what would be backed up)
-b <FILENAME> create a tar.gz configuration backup as <FILENAME>
-r <FILENAME> restore a tar.gz configuraton backup from <FILENAME>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
-r <FILENAME> restore a tar.gz configuraton backup from <FILENAME>
-r <FILENAME> restore a tar.gz configuration backup from <FILENAME>


BACKUP_FILE=
MODE=

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not passing any arguments should give the same result as -h (instead of "Please run this program as root" or some such).

disabled_services=$(collect_disabled_services $1)

if [ -n "$DRYRUN" ]; then
echo "Services to enable:"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a user, I would wonder what this actually does (if I create a backup while, where do I find the backup for the service state?). Maybe the usage text or the man page could elaborate (list stored in .SERVICES_ENABLED/.SERVICES_DISABLED, then enabled/disabled via systemctl).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely rather something for the man page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants