Setting up the backup server (bkp_target)

Go to the contents

bkp_target accepts backups from any number of agents and keeps them in stores below <storage>/<directory>/<store>.cbs. Everything goes over TLS. Setting it up takes four steps.

4.1 Step 1: create a certificate and key#

On the server:

bkp_maintenance tls-init --cert /etc/bkp/target.crt --key /etc/bkp/target.key --cn backup.example.org

Output:

wrote certificate to /etc/bkp/target.crt and private key to /etc/bkp/target.key

The command creates a self-signed ECDSA certificate (P-256, with subjectAltName from --cn, without calling an external openssl; Windows and browsers cannot handle Ed25519). Existing files are not overwritten unless --force is given; --days <n> sets the validity.

target.crt is also needed on every client: the agent checks the server against exactly this certificate (--tls-ca). The private key target.key stays on the server and is readable only by root.

4.2 Step 2: create an API key#

bkp_maintenance acl-add --acl-file /etc/bkp/acl --dir vm1,vm2

Output:

added to acl: key=6df1922d599969e0d2c8c10ad723335a95d3cab8f4678e77ba209992aa330b1c dirs=vm1,vm2

The key is printed once (without --key a random 256-bit value). On the client, put it into a file that only the user running the backup may read:

printf '%s' 6df1922d...330b1c > /etc/bkp/vm1.key
chmod 600 /etc/bkp/vm1.key

The ACL file has one line <api_key> <dir1>[,<dir2>,...] per entry; # or ; starts a comment (at the start of a line or after a space/tab, see “Comments” below). Instead of the key, its SHA-256 may be given as sha256:<64 hex digits>; acl-add writes only this checksum, the key itself is nowhere on the server. If it is lost, create a new one.

Every directory is a pattern relative to --storage; * and ? only apply within one level: vm1 allows exactly this directory, vm* all directories whose name starts with vm, vm1/* one level below it, vm1/** any depth below it (but not vm1 itself) and vm1/*** vm1 itself and any depth below it. .. and \ are never allowed.

Contents of the ACL

cat /etc/bkp/acl

Output:

6df1922d599969e0d2c8c10ad723335a95d3cab8f4678e77ba209992aa330b1c vm1,vm2

4.3 Step 3: start the server#

bkp_target --storage /srv/backups --tls-cert /etc/bkp/target.crt --tls-key /etc/bkp/target.key \
           --acl-file /etc/bkp/acl [--bind 0.0.0.0] [--port 9990]

Output:

[bkp_target] INFO  loaded 1 ACL entry from /etc/bkp/acl
[bkp_target] INFO  bkp_target listening on 0.0.0.0:9990 (storage_root=/srv/backups, TLS enabled)

Simpler is the parameter file /etc/bkp/bkp_target.cfg (template packaging/bkp_target.cfg.example): every option has a key of the same name there, and the server starts with a plain bkp_target. An option on the command line takes precedence over the file. bkp_target --config-validate checks it; changes take effect after a restart. bkp_setup --configure-server writes it itself and leaves every line alone that does not concern it.

/etc/bkp/bkp_target.cfg

storage=/srv/backups
acl-file=/etc/bkp/acl
tls-cert=/etc/bkp/target.crt
tls-key=/etc/bkp/target.key
port=9990

Every connection runs in its own thread. A directory that does not exist yet but is allowed by the ACL is created by the server at the first backup.

Every store has an index (index.lmdb), which by default lives in the store. With --index-dir <dir> the indexes live in <dir>/<store-uuid>.lmdb instead, for example on an SSD; the directory must be writable for the account aegisgrid. If --storage is on NFS/SMB, a local --index-dir is required.

groupadd --system aegisgrid
useradd --system --no-create-home --shell /usr/sbin/nologin --gid aegisgrid aegisgrid
chown aegisgrid:aegisgrid /etc/bkp/target.crt /etc/bkp/target.key /etc/bkp/acl
chmod 640 /etc/bkp/target.key
chown -R aegisgrid:aegisgrid /srv/backups

4.3.1 Management through the Target API#

The server can also be managed from another machine, for example with the target console: watch and cancel running sessions, change settings and accesses, control the target scheduler, view, check and clean up stores. The API listens on port 9991 (HTTPS) and is off at first. bkp_setup --configure-server --enable-target-api does the following steps during the setup; by hand on the server:

bkp_target --api-init
bkp_target --api-enable
bkp_target --api-token-add konsole --role admin
systemctl reload bkp_target
bkp_target --api-info

The token is shown only once; together with the fingerprint from --api-info you use it to set up the console. Open port 9991 in the firewall only for the management machines. Deleting runs and jobs that start arbitrary programs are blocked through the API at first (allow-delete=yes or allow-exec=yes in the section [api] of /etc/bkp/bkp_target.cfg). How the target console works with it is described in Chapter 15; the full description for programmers is in docs/target-api.txt.

4.3.2 As a systemd service#

The repository contains packaging/systemd/bkp_target.service:

/etc/systemd/system/bkp_target.service

[Unit]
Description=Backup target server (bkp_target)
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/bkp_target
KillSignal=SIGTERM
TimeoutStopSec=330
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now bkp_target

The unit deliberately has no User=/Group=: the service must start as root so that the root part (see above) can carry out LVM actions; bkp_target switches to aegisgrid itself. With User=aegisgrid it runs too, but then without the root part. systemctl reload bkp_target (SIGHUP) reads certificate, key and ACL again without dropping connections. If the worker process dies unexpectedly, the root part ends with an error too, and Restart=on-failure restarts the whole service.

Gentle stop: SIGTERM (and SIGINT) accepts no new connections and lets running sessions finish for up to --drain-timeout seconds (default 300). TimeoutStopSec=330 in the unit is deliberately above that, so that systemd does not end the service early. A restart in the middle of the night thus does not cancel a running backup.

4.4 Step 4: first backup over the network#

bkp_agent --source data.img --target backup.example.org:9990 --target-dir vm1 \
          --tls-ca /etc/bkp/target.crt --api-key file:/etc/bkp/vm1.key --name data

Output:

[bkp_agent] INFO  source is not an LVM logical volume: backing it up directly (no snapshot)
[bkp_agent] INFO  created the store backup.example.org:9990/vm1/default.cbs (id hash blake3, encryption none,
                  chunks 65536/262144/1048576, segments of 1073741824 bytes)
[bkp_agent] INFO  done: full run 20261015T121825Z-6aa5d6f2-... in backup.example.org:9990/vm1/default.cbs:
                  source 16777216 bytes, data 13631488, holes 3145728 (fs-free 0); chunks 44, new 44 (0.0% deduplicated);
                  12587566 bytes sent, 12587566 newly stored

The server creates the store at the first run in /srv/backups/vm1/default.cbs (another name with --store). For every chunk the agent asks in batches whether the store already has it, and sends only the missing ones. Several machines that back up into the same store thus deduplicate against each other over the network too. A run only comes into being once the server has stored all data safely. --verify has no effect in network mode (the restore checks every chunk anyway).

Every further backup is again the same command; only what the store is missing is sent. --out and --target may be given together: the source is then read once and every store gets its own, complete run (local copy plus server). More destinations, encryption per destination and optional destinations work through sinks (below).

4.4.1 A configuration file instead of a long command line#

For cron/systemd timers and several backups on one host, bkp_agent reads /etc/bkp/bkp_agent.cfg when it starts (Windows: %ProgramData%\AegisGrid\bkp_agent.cfg, can be overridden with --config-path). Copy packaging/bkp_agent.cfg.example to /etc/bkp/bkp_agent.cfg and adjust it:

/etc/bkp/bkp_agent.cfg

[targets]
srv1:backup.example.org:9990:/etc/bkp/target.crt

[data]
source=data.img
target=srv1
target-dir=vm1
api-key=file:/etc/bkp/vm1.key

After that bkp_agent --name data is enough. Every parameter also given on the command line overrides the one of the same name from the configuration; bkp_agent --help lists all key=value names usable for this. api-key= accepts three forms: the key directly as plain text, file:<path> (as here, equivalent to --api-key file:<path> on the command line) and the reserved, not yet implemented kms:<url>. Plain text is allowed only in the configuration file -- there it never ends up on the command line or in the job description stored for crash-safe resuming (see below); on the command line --api-key always requires file:/kms:, because argv can be seen through /proc/ps.

Comments: # or ; at the start of a line or after a space/tab starts a comment that runs to the end of the line, e.g. timeout-seconds=259200 # 3 days. This applies to all .cfg files (agent, bkp_xenV, bkp_proxmox, bkp_sched, updater, setup answer file, ACL). If a value starts with " or ', it runs up to the matching closing quote; the quotes are removed, and # and ; inside belong to the value, e.g. smtp-password='secret #1' # comment. Inside "...", \" and \\ stand for " and \; '...' is taken literally. After the closing quote only a comment may follow. Quotes in the middle of a value are normal characters (args=--x "a b" passes them on to the program).

4.5 Several destinations: sinks#

A sink is a store a run writes into. --out and --target create unencrypted sinks directly on the command line. Everything else (encryption, several servers, a store name of its own per destination, optional destinations) goes into named sections [sink:<name>], which a profile selects with sinks= (on the command line --sink <name>):

/etc/bkp/bkp_agent.cfg (sink sections may also be in /etc/bkp/sinks.cfg)

[targets]
rz1:backup.example.org:9990:/etc/bkp/rz1-ca.pem

[sink:lokal]
type=out                       # out | target (required)
path=/srv/local-copy           # out: the store is kept in <path>/<store>.cbs
encryption=aes-256-gcm         # none (default) | aes-256-gcm | chacha20-poly1305
key=file:/etc/bkp/local.key    # required exactly when encrypted
retention=7d                   # overrides --retention of the job
required=no                    # yes (default) | no: a failure makes the run only “partial” (exit 3)

[sink:rz1]
type=target
target=rz1                     # entry from [targets]
target-dir=kunde-mueller
api-key=file:/etc/bkp/rz1.key
encryption=chacha20-poly1305
key=file:/etc/bkp/rz1-daten.key

[srv01]
source=/dev/vg0/root
sinks=lokal,rz1                # order = preference for a later restore

[targets] and the [sink:…] sections may also be kept in a file of their own, sinks.cfg, next to bkp_agent.cfg (Linux /etc/bkp/sinks.cfg, Windows C:\ProgramData\AegisGrid\sinks.cfg); other sections do not belong there. bkp_agent reads both together. The management console maintains this file too (“Destinations (sinks)”, Chapter 14): what it creates goes there, keys as files in keys/, uploaded CA certificates in ca/.

• The source is read and chunked once; every sink gets its own, complete run. The runs of one pass carry the same run_group (twins). Every store reports by itself which chunks it is missing; compression happens once, encryption once per key.
• At most 8 sinks. All stores of a run must chunk alike (same chunk size); a store that joins takes it over.
• Failure of a sink (connection gone longer than --net-retry, disk full, key rejected): it is dropped, the others finish their run. Exit code 0 all sinks ok, 3 partial (a sink with required=no failed; bkp_sched shows PARTIAL), 1 a sink with required=yes failed. The next run fills up the missing sink automatically.
• If the command line gives --out, --target or --sink, it replaces the sinks of the profile as a whole.
• The restore reads from the sinks of the profile (Chapter 7): from the first one that has the backup, and fetches what is missing from the twins.

4.6 Firewall and network#

• The server needs only one incoming TCP port (default 9990); the clients only need outgoing access to this port.
• Example with ufw: ufw allow from 192.0.2.0/24 to any port 9990 proto tcp.
• With --bind <address> the server listens on one interface only, for example in the management network.

4.7 Crash-safe network backup#

If the server, the network or the workstation fails during a network backup (power cut, shutdown, crash), the work done so far is not lost. This follows from the store: what is safely in the store once, no later run has to send again. A run only comes into being at the end; a cancelled run therefore never leaves half a backup. This is always active; no parameter is needed to turn it on.

4.7.1 Server crash or network failure during a run#

The agent has the server confirm its data regularly (at the latest every 60 s or after 256 MiB) and keeps everything unconfirmed in memory until then. If the connection breaks, it reconnects automatically (--net-retry <minutes>, default 30, 0 = give up at once), asks what the server still has, sends the rest again and carries on. The run stays the same, for the target too: it counts it as one backup even if it restarted in between (chapter target scheduler, “Events and chains”), and the target console shows the session as “Resumed” from the start of the backup.

4.7.2 Crash or restart of the workstation#

The agent remembers a running job in a state file (--resume-dir, default /var/lib/bkp_agent/resume or on Windows %ProgramData%\AegisGrid\resume; also through the environment variable BKP_AGENT_RESUME_DIR). If the state file is in a directory of its own, the agent leaves a pointer to it in the default directory (elsewhere/<hash>.dir); this way --resume-pending and the daily cleanup find these jobs too. The pointer disappears as soon as its directory no longer holds an interrupted job. The next run of the same command (cron job, scheduler, by hand or --resume-pending) is a new run, but sends only what the store does not have yet, so essentially the rest:

--resume-max-age <hours> (default 24) sets how long after its last activity (so from the interruption, not from the start) a started backup is still resumed. The earlier modes verify and trust no longer exist: a new run is always complete and consistent, and the store makes sure that nothing is sent twice.

4.7.3 Restarting without a daemon of its own#

bkp_agent --resume-pending [--resume-dir <dir>] [--resume-max-age <hours>]

starts all interrupted jobs again and then ends. bkp_setup sets this up at system start when installing the component agent: on Linux the systemd unit bkp_agent-resume.service (without systemd /etc/cron.d/bkp_agent-resume, both also in packaging/), on Windows the scheduled task “AegisGrid resume” (2 minutes after the start). --no-resume-service turns this off during the installation. A second run of the same job at the same time ends at once with exit code 75 (like the updater) instead of waiting or disturbing the backup.

--resume-pending also cleans up: an interrupted backup that has been idle longer than its own --resume-max-age would not be resumed anyway. Its state file and its kept snapshot (LVM snapshot or persistent shadow copy) are removed. So nothing is left behind when a job was deleted or changed and never runs again. A backup that its own limit still allows stays, even if --resume-pending no longer starts it with a shorter limit; its next regular run resumes it. Cleanup happens at system start and, where bkp_sched runs, also once a day (first one hour after the service starts) with bkp_agent --resume-cleanup. This call only removes and starts nothing; what it removes is in the log of bkp_sched. By hand, --discard-interrupted removes an interrupted backup at once.

4.7.4 Interrupting or discarding#

A crash, a time limit or SIGTERM interrupts a backup: it is taken up again as described above. If it is to be discarded instead, --discard-interrupted with otherwise the same options cleans up: no backup, the source is not read; deleted are the kept snapshot or shadow copies and the state file. What the run had already stored stays in the store until the next prune. Exit 0 (also when there is nothing to discard), 75 while the backup is still running. bkp_proxmox and bkp_xenV know the same option (it removes the snapshots left behind, with bkp_proxmox also ZFS, RBD and qcow2 snapshots). If a VM backup was ended hard (such as kill -9 or a power failure), the next run of bkp_proxmox removes the snapshots left behind by itself when no other backup of the VM is still running and no held run still needs them; the log notes it as a warning.

End a running job for good, then --discard-interrupted

bkp_sched stop <job> [--no-wait]

The stop first goes through the job's run channel (Chapter 13): bkp_agent, bkp_restore, bkp_m365 and the VM programs then end the run themselves, usually after a few seconds. If the run is still stuck after 60 seconds (for example in a snapshot or hook) or the program does not follow the channel, it gets SIGTERM, or Ctrl+Break on Windows.

bkp_sched stop reaches the job whether the daemon, bkp_sched run or bkp_sched start runs it, and then calls --discard-interrupted automatically; its output is in the job log. The run counts as FAILED, the log ends with cancelled: 1 and who cancelled. stop waits until everything is done (--no-wait: returns at once). Cancelling through the API (POST /jobs/{name}/cancel) works the same way. For the scheduler of bkp_target there is bkp_target --sched-stop <job>: the run does not go back into the queue.

4.7.5 Pausing and throttling: bkp_sched pause, resume, throttle#

A running job can be paused from the command line, whether the service, bkp_sched run or start runs it: the program stops reading and sending at once and waits; connections and snapshot are kept. The paused time does not count toward the job's time limit.

Pause, resume and throttle running jobs

bkp_sched pause <job>... | --all [--for 30m|4h|2d] [--reason "<text>"]
bkp_sched resume <job>... | --all
bkp_sched throttle <job>... | --all [--bw 20M|500K|1G|off] [--background on|off|default]

• --for pauses for the given time (at most hold-max); after that the run carries on by itself.
• Without --for the job stays paused until resume continues it. A backup (agent, proxmox, xenv, hyperv) that is paused longer than pause-max (default 4h, 10m to 7d, section [sched]) is then held (see below) and releases snapshot and target session; other jobs carry on after pause-max.
• --reason records a reason (one line, at most 200 characters). If a paused job is paused again, the new time and the new reason apply.
• resume continues paused jobs at once, no matter who paused them (command line, API, progress window or time window; then the job carries on outside its window, which is logged). Held jobs (HELD) are continued by resume at once in the background, like start.
• throttle limits the bandwidth of a running job (--bw, bytes per second, binary units; off lifts the limit) or switches the background mode, which leaves processor and disk to other programs (default: as at the start). The next run starts as configured again.
• --all acts on all running (with resume: all paused or held) jobs; jobs whose program cannot do this are named.

bkp_sched list shows this: in the column STATUS for example RUNNING 42 % (paused), below it paused since … with originator and reason and limited to …; the summary names jobs that have been paused for more than an hour. Exit code 0 when everything is done, 1 when a job is not running, cannot pause, is neither paused nor held or does not answer within 35 seconds, 2 for a wrong call.

4.7.6 Holding instead of ending: bkp_sched hold#

If a machine or the line must be free for a while (maintenance, a move, a video conference), a running backup is better held than ended: the run stops cleanly, keeps its snapshot and continues later, without an error mail and without losing the work.

Hold running backups or continue them at once

bkp_sched hold <job>... | --all [--for 30m|4h|2d|1w | --until "2026-11-03 08:00"] [--reason "<text>"] [--no-wait]
bkp_sched resume <job>... | --all

• With a time (--for or --until, local time) the scheduler continues the run at the given time, within the job's time window; appointments in between are skipped.
• Without a time the next regular run continues it.
• --all holds all running backups. agent, proxmox, xenv and hyperv jobs can be held; other running jobs (such as restore or M365) are named by --all but go on running.
• --reason records a reason (one line, at most 200 characters); it appears in bkp_sched list, in the consoles and in the job log.
• If a job is already held, hold gives it the new time and the new reason; hold without a time leaves the continuation to the next regular run.
• bkp_sched resume continues a held job at once in the background; bkp_sched run or start also continues at once.
• A hold lasts at most hold-max (default 7d, at most 30d, in the section [sched] of bkp_sched.cfg); after that the scheduler continues.
• Follow-up jobs with after= wait: held is not finished.

hold waits until the programs have ended the run (--no-wait: returns at once). A program that is still running after 120 seconds is ended hard; the run is continued later all the same. Exit code 0 when everything is held, 1 for an error, 2 for a wrong call. If a job started with bkp_sched run is held, run ends with exit code 78.

The continuation is a new run that carries on where the held one stopped. A disk or volume backup uses the same snapshot and reads only the parts that are still missing; it takes over the free space of the file systems that the held run found instead of reading the allocation bitmap again (that saves minutes with large volumes). A VM backup with proxmox or xenv keeps its snapshots and backs up only the open disks and the rest of the started one. A file backup reads from its kept snapshots, takes over every file already done without reading it, and reads a started large file on from the point where it stopped; all files thus come from the same point in time. If the snapshot is no longer there, it takes a new one, takes over only files that have not changed since the hold (size, times, inode) and reads a started file from the beginning. If paths or exclusions were changed in the meantime, it compares with the last backup as usual. The log says what was taken over. bkp_agent reports a hold with exit code 78 and the lines held_until= and hold_kind=; --resume-pending skips held backups whose time has not come yet.

A run is held automatically when its time window ends with window-end=hold, when its time limit runs out, when the service bkp_sched is stopped and when the target restarts or the target administrator holds it (Chapter 15). If a job has been held longer than remind-held-days days (section [mail], default 3), a mail reminds of it, after that weekly.

4.7.7 Snapshots across a restart#

• LVM: the snapshot stays after a cancellation or SIGTERM (tagged with the job) as long as the backup can be resumed, and is reused by the next run of the same job: the same point in time. Finished backups, --resume off and jobs without a state file clean up the snapshot at once.
• Windows: --vss-persistent auto|off (default auto) tries a persistent shadow copy (survives a restart) for a resumable network backup; if that fails, it falls back automatically to the normal, non-persistent copy (then the next run backs up from the new point in time).

Tests: scripts/resume_test.sh (server crash in the middle of a run), scripts/resume_agent_test.sh (agent crash, --resume off, --resume-pending, --discard-interrupted), scripts/lvm_resume_test.sh (LVM snapshot survives a crash), scripts/resume_agent_test_wine.sh (Windows agent under Wine), scripts/resume_test.ps1 (real Windows, persistent shadow copy).