Maintenance

Go to the contents

All maintenance functions are in bkp_maintenance. They work on stores, locally or straight on the server (<storage>/<directory>/<store>.cbs). --search-dir <dir> takes all stores of a directory, --store <dir> exactly one. All store commands know --index-dir in case the index is not in the store. M365 stores (<name>.m365) are handled by prune, scrub, mirror and export in the same way.

10.1 Overview: store-info#

bkp_maintenance store-info --store backup/default.cbs

shortened

store:        backup/default.cbs
kind:         block
encryption:   none
id hash:      blake3
objects:      56 in 2 pack(s), 14.2 MiB stored
runs:         2 (1 name(s)), oldest 2026-10-15T14:18:25+02:00, newest 2026-10-15T14:18:31+02:00
data in runs: 26.0 MiB (1.84x the stored bytes)
last cleanup: never
last scrub:   never

NAME                               RUNS   FULL  NEWEST                     DATA (NEWEST)
data                                  2      1  2026-10-15T14:18:31+02:00       13.0 MiB

data in runs is the sum of the data of all runs; divided by the stored bytes it gives the deduplication factor. --exclusive adds per name what only its runs take up, that is what deleting all its runs would free (reads all manifests, slow for large stores); --fragmentation shows how scattered the newest run is; --json prints everything in machine-readable form. For block stores the chunk size probe and recommendation follow (Chapter 25).

10.2 Retention and prune#

Every backup can bring its own retention: bkp_agent --retention <spec> (or retention= in the profile or in the sink) writes it into the run. <spec> is <n>d (days), <n>w (weeks), <n>m (calendar months), <n>y (calendar years), forever or a GFS scheme such as gfs:7d,4w,12m,5y (below). An optional class can be added (--retention-class daily|weekly|...). Without --retention a backup is never deleted automatically. Over the network bkp_target computes the expiry date with its own clock.

bkp_maintenance prune --search-dir /backup --dry-run      # only show what would be deleted
bkp_maintenance prune --search-dir /backup [--name data]  # delete; --name: only runs of this backup name
bkp_maintenance prune --store /backup/default.cbs         # only this store (with its generations)

Output of --dry-run

[bkp_maintenance] INFO  keep: block run 20261015T121825Z-6aa5d6f2-... 'data' (no retention)
[bkp_maintenance] INFO  keep: block run 20261015T121831Z-8c47f436-... 'data' (newest run of this name)
[bkp_maintenance] INFO  prune backup/default.cbs: would delete 0 run(s), kept 2; objects: would remove 0 (0.0 MiB),
                        packs deleted 0, rewritten 0
[bkp_maintenance] INFO  prune backup: 1 store(s) checked

Rules: a run is deleted when it has expired (with GFS: when no rule keeps it any more). Then prune removes the data no remaining run needs (space reclamation: unused packs are deleted, partly used ones rewritten). No key is needed.

• The newest backup of a backup name and the newest complete one always stay, even if they have expired (protection against failed jobs or a retention set too short).
• --gc-min-age <seconds> (default 24 h): younger packs are never deleted or rewritten (protects running backups).
• --pending-max-age <seconds> (default 48 h): a running VM run protects its disks backed up so far for this long.
• A store being backed up into right now is skipped; the next prune catches up.
• A generation without runs is removed completely.

10.2.1 Grandfather-father-son (GFS)#

Instead of a fixed duration the retention can be a GFS scheme: gfs: and a list of <n>h (hourly), <n>d (daily), <n>w (weekly), <n>m (monthly) and <n>y (yearly). gfs:7d,4w,12m,5y keeps the newest backup of each of the last 7 days, 4 weeks, 12 months and 5 years; one job backing up daily (or more often) is enough, no separate weekly or monthly jobs are needed.

• Counting follows the calendar: the current day, the current week (starting on Monday), the current month and the current year count. The local time of the machine prune runs on decides (for network backups the server).
• A backup stays as long as one of its rules keeps it. Within a period the newest complete backup counts; only if there is none, the newest incomplete one.
• Every backup carries its own scheme. If it is changed, the new one applies to new backups; the older ones keep theirs until set-retention (below) changes it.
• Backups with a fixed duration or without retention in the same series are not affected. The newest backup of a name stays as always.
• As an upper limit (for example for WORM locks and for an older prune) a GFS backup gets an expiry date one period after its longest rule (gfs:7d,4w → 5 weeks).

bkp_agent --source /dev/vg0/data --target backup.example:9990 ... --name data --retention gfs:7d,4w,12m,5y
bkp_maintenance prune --search-dir /srv/bkp_storage/vm1 --dry-run

Output of --dry-run with GFS

[bkp_maintenance] INFO  keep: block run 20261023T020011Z-41c0... 'data' (GFS gfs:7d,4w,12m,5y: daily)
[bkp_maintenance] INFO  keep: block run 20260930T020007Z-9be2... 'data' (GFS gfs:7d,4w,12m,5y: monthly)
[bkp_maintenance] INFO  [dry-run] would delete: block run 20261015T020009Z-c71d... 'data' (GFS gfs:7d,4w,12m,5y keeps it
                        no longer) in /srv/bkp_storage/vm1/default.cbs

10.2.2 Showing the retention#

bkp_restore --list and bkp_m365_restore --list show the column RETENTION per run in the language of the machine, locally and over the network:

• 30 days, until 2026-11-23: fixed duration, expiry date in local time; Forever: never deleted; -: without retention, also never deleted.
• GFS (7 days, 4 weeks, 12 months, 5 years), kept as daily, weekly: the GFS rules and the periods for which they keep the run at the moment.
• 30 days, expired: the retention is over, the next prune deletes the run. 30 days, kept as newest: also over, but prune keeps it as the newest (complete) backup of its name. 30 days, held until 2026-12-01: over, but a later conspicuous run keeps it until then.

The column TYPE shows Full or Incremental. For programs --list --json stays English (keep: until, expired, newest …).

10.3 Changing the retention afterwards: set-retention#

set-retention gives runs already stored a new retention, counted from the time they were created. prune and the lists follow it. The run itself stays unchanged; the change is stored next to it in the store. This also works while backups are running, and on the server straight in the storage. Exactly one selection is required: --run (name or UUID), --name (all runs of this backup name, across all generations; for M365 the store <name>.m365) or --all.

bkp_maintenance set-retention --store backup/default.cbs --run 20261015T121825Z-6aa5d6f2-... --retention 1y
bkp_maintenance set-retention --store backup/default.cbs --name data --retention gfs:7d,4w,12m --dry-run
bkp_maintenance set-retention --search-dir /srv/bkp_storage/vm1 --all --retention 90d

Output:

[bkp_maintenance] INFO  set: block run 20261015T121825Z-6aa5d6f2-... 'data': retention 30d -> 1y (kept until
                        2027-10-15T14:18:25+02:00)

• Copies made by mirror keep their own retention; use set-retention there as well if needed.
• If a retention is shortened so much that it is already over, set-retention warns: the next prune deletes the run (except the newest backup of the name).
• WORM: only extending is allowed. worm=extend first extends the locks of the packs concerned and of the run file; with worm=generation the new duration must fit into the lock of the generation.
• In bkp_sched as the job command=maint-set-retention with the same parameters in args=, usually one-off with once=. Through the API the command is blocked like maint-prune until [api] allowed-commands names it.
• The management console shows the retention per run in the column “Retention” (chapter “Management console (Windows)”); bkp_restore --list --json <file> writes the list as JSON for programs.

10.3.1 Changing it from the client, also on a target#

bkp_restore --set-retention <spec> --run <run> changes the retention of every copy of a run in the stores bkp_restore can reach (as when restoring: --store, --server or the sinks of a profile with --name and --sink). --run takes the name, the UUID or the run group, that is all sinks of a backup. Every copy is checked first; if one refuses, nothing changes. --dry-run only checks, --extend-only refuses a shorter retention in local stores too, --json <file> writes the result per copy.

bkp_restore --name web1 --run <run-group> --set-retention 1y
bkp_restore --server rz1:9990 --server-dir kunde-a --tls-ca rz1.pem --api-key file:/etc/bkp/rz1.key --run <run> --set-retention forever

On a bkp_target its operator decides with client-retention= in bkp_target.cfg: extend (default) lets clients only keep a backup longer (no GFS rules, “forever” stays “forever”), no not change it at all. Shorter is only possible there with bkp_maintenance or the target console. This way even a client whose credentials were stolen cannot make a backup disappear earlier.

10.4 Deleting a run: delete#

delete removes exactly one run, also the newest, and then the data no other run needs. Other runs of the same name are not affected: there are no dependencies between runs. With --dry-run it only shows what would be deleted.

bkp_maintenance delete --store backup/default.cbs --run 20261015T121825Z-6aa5d6f2-8419-482a-ab3f-0373e33b449d --dry-run

Output:

[bkp_maintenance] INFO  [dry-run] would delete: block run 20261015T121825Z-6aa5d6f2-... 'data' in backup/default.cbs
[bkp_maintenance] INFO  delete backup/default.cbs: would delete 1 run(s), kept 1; objects: would remove 0 (0.0 MiB),
                        packs deleted 0, rewritten 0

That no data would be freed here has two reasons: the remaining run still uses most chunks, and the packs are younger than --gc-min-age. The space of the other chunks is freed by a later prune.

10.5 Scrub: checking for bit rot#

bkp_maintenance scrub --store backup/default.cbs [--decrypt-key file:<path>] [--result-file <json>]

Output:

[bkp_maintenance] INFO  scrub backup/default.cbs: 2 run(s), 56 object(s) in 2 pack(s) (14.2 MiB): 56 checked (read back and verified),
                        0 bad, 0 missing, 0 unreadable manifest(s); 0 run(s) damaged
[bkp_maintenance] INFO  scrub backup/default.cbs: result in scrub/20261015T121843.612Z.json

Reads every object, checks its checksum and that every run is complete. For unencrypted stores and with --decrypt-key every content is also checked against its ID. Damaged objects and the runs affected are in the log and in the store (scrub/<time>.json, store-info shows the last scrub). Exit 1 on damage. Scrub repairs nothing; run it regularly, for example weekly as a maint-scrub job.

10.6 Repair: replacing damage from a second copy#

bkp_maintenance repair --store /backup/default.cbs --from /mnt/mirror/default.cbs [--dry-run]
bkp_maintenance repair --store /backup/default.cbs --from sink:rz1                  # sink from bkp_agent.cfg
bkp_maintenance repair --store /backup/default.cbs --from server:backup.example.org:9990 --from-server-dir vm1 \
                       --from-store default --tls-ca /etc/bkp/target.crt --api-key file:/etc/bkp/vm1.key

repair replaces the damaged objects found by the last scrub from a copy: another sink of the same backups or a mirror. If the copy has the same ID hash and key, its objects are taken over by ID. Otherwise by position: the twin run has the same chunks at the same places; each one is decrypted with the key of the copy (--from-key), hashed again and only taken over if it gives exactly the damaged ID. Run scrub again afterwards. Exit 1 if objects could not be replaced; what cannot be repaired is restored by a partial restore with zeros at the damaged places (Chapter 7).

10.7 Mirror: mirroring a store#

bkp_maintenance mirror --store /backup/default.cbs --to /mnt/seconddisk/default.cbs
bkp_maintenance mirror --search-dir /backup --to server:rz2.example.net:9990 --to-server-dir kunde1 \
                       --to-tls-ca /etc/bkp/rz2.crt --to-api-key file:/etc/bkp/rz2.key

mirror copies the runs of a store into a second one, in any direction (local → server, server → local, server → server as a job of the target scheduler). The destination is created with the ID hash, encryption and chunk size of the source, so both have the same IDs. Only what the destination is missing is copied; a mirrored run keeps its ID, time and retention, and a run already deleted there does not come back. Nothing is ever deleted in the destination; its own prune does that. An interrupted mirror run carries on the next time. No key is needed.

10.7.1 Moving stores#

bkp_maintenance move --store /srv/backups/hosted/G-000007/I-0000042/default.cbs --to /srv/backups/hosted/G-000007/standard/default.cbs --dry-run
bkp_maintenance move --store /backup/default.cbs --name fs01 --to server:rz2.example.net:9990 --to-server-dir kunde1 \
                     --to-tls-ca /etc/bkp/rz2.crt --to-api-key file:/etc/bkp/rz2.key

move takes a store, or only the runs of single backup names (--name), to another place, on the machine that holds the store. If the destination is free and on the same file system, the whole store is renamed with all generations; otherwise it is copied as with mirror, checked in the destination, the name register merged, and only then deleted in the source. A WORM-locked run prevents the move before anything happens. No key is needed; in the target console this is “Move...” (tab “Data”).

10.7.2 Re-encrypting#

With --reencrypt, mirror and move copy the runs into a store with another key or ID hash: every object is read with the key of the source, encrypted again with --to-key and gets its own ID there. The program looks for the source key among --decrypt-key and --previous-keys by the check value of the store. With --store sink:<name> and --to sink:<name> place, access and key come from the configuration of bkp_agent (--sink-store names the stores in it). The source is only read. This is how the backups from before joining a group get into the group folder (section “Groups”, Chapter 23).

10.8 Export and import: a run as an archive#

bkp_maintenance export --store /backup/default.cbs --name data [--at <time>] --to /mnt/usb/
bkp_maintenance export --store /backup/default.cbs --run <run> --to - | dd of=/dev/nst0 bs=1M
bkp_maintenance import --archive /mnt/usb/data-<run>.cba --to /backup/default.cbs [--dry-run]

Output:

[bkp_maintenance] INFO  export: run 20261015T121849Z-44fdc352-... 'data' of backup/default.g20261015.cbs
                        -> /mnt/usb/data-20261015T121849Z-44fdc352-....cba
[bkp_maintenance] INFO  export: run 20261015T121849Z-44fdc352-... 'data' exported: 46 object(s) (2 tree(s)), 12.0 MiB stored data,
                        source 16.0 MiB, archive 12588833 bytes

export writes a run (block, VM or M365) into a self-contained archive: one file with everything the run needs, without a store; encrypted stays encrypted. Meant for tape, USB disk, WORM file system or auditors. --to <directory> names the file <backup>-<run>.cba, --to - writes to stdout; an existing archive file is never replaced (exit 2). It is restored with bkp_restore --archive (Chapter 7). import brings the run of an archive back into a new store or one with the same ID hash, encryption and key; import --dry-run checks an archive completely without writing.

10.9 WORM: retention locks#

A store can be set up as a WORM store when it is created (bkp_agent --worm, worm= in the [sink:]). Written objects are never changed; packs and runs are locked against deletion up to a point in time:

Under WORM, prune only deletes whole unused packs after their lock has ended and never rewrites; --dry-run names day by day what becomes free when. delete refuses a locked run. On a local file system the lock is metadata (<file>.retain) that prune/delete respect; on S3 it is Object Lock and enforced by the provider (Chapter 6).

10.10 Rebuilding the index#

bkp_maintenance reindex --store /backup/default.cbs

Rebuilds index.lmdb from the packs. This also happens automatically if the index is missing or damaged.

10.11 Certificates and keys#

tls-init and acl-add are described in Chapter 4, encrypt-init in Chapter 6. To renew a certificate: tls-init --force creates a new pair; then target.crt must be distributed to all clients and bkp_target restarted, because the clients pin exactly this certificate (scenario in Chapter 17).

10.12 Scheduling maintenance#

Regular maintenance is scheduled with maint-* jobs in bkp_sched (maint-prune, maint-scrub, maint-mirror, maint-export) or straight on the server in the target scheduler (/etc/bkp/bkp_target_sched.cfg, see README). There a job can also react to an event, for example mirror every new backup at once:

/etc/bkp/bkp_target_sched.cfg (excerpt)

[prune-kunde1]
command=maint-prune
schedule=30 2 * * *
args=--search-dir {storage}/kunde1

[mirror-kunde1]
command=maint-mirror
on=backup-success
match-dir=kunde1
args=--store {store} --to server:rz2.example.net:9990 --to-server-dir kunde1 --to-tls-ca /etc/bkp/rz2.crt --to-api-key file:/etc/bkp/rz2.key

10.12.1 Schedule notation#

schedule= has the same notation in bkp_sched and in the target scheduler: minute, hour, day, month, weekday and optionally the year, in the local time of the server. Each field takes *, numbers, ranges (9-17), steps (*/15) and lists (0,30). In addition:

10.12.2 More schedule settings#

A job in bkp_sched and in the target scheduler can have several schedule lines; it runs at all their times, minus the exceptions. bkp_sched preview <job> or bkp_target --sched-preview <job> describes the schedule in one sentence (for example “On the 2nd Tuesday of the month at 22:00, except on public holidays (de-by)”) and shows the next starts as the service carries them out; the console shows the same sentence in its language:

Durations are written 90m, 1h30m, 4h or 2d. Calendars of your own are <name>.cal files in /etc/bkp/calendars (Windows %ProgramData%\AegisGrid\calendars), one line per day or range: 2026-12-24 Christmas Eve, 2026-12-28..2027-01-01 Company holidays, *-12-31 New Year's Eve (every year), include=de-by.

Working days at 22:00 except on Bavarian public holidays, plus the last Friday of the month at 23:30; only 22:00 to 06:00, up to 20 minutes later

[nightly]
command=agent
profile=srv1
schedule=0 22 * * MON-FRI
rrule=FREQ=MONTHLY;BYDAY=-1FR;BYHOUR=23;BYMINUTE=30
dtstart=2026-11-01T00:00
exclude-calendar=de-by
window=22:00-06:00
window-end=pause
jitter=20m

Daylight saving time: a date in the hour skipped at the start of daylight saving time (02:00 to 02:59) is dropped on that day. At the end of daylight saving time the clock runs from 02:00 to 02:59 twice; a date in it counts once, on the first pass, with dst-repeat=both on both.

10.13 The target scheduler#

The target scheduler runs as a process of its own in the process tree of bkp_target and has no network contact; a problem of the scheduler can therefore never hold up a backup. It is on as soon as /etc/bkp/bkp_target_sched.cfg exists. All keys are in the template packaging/bkp_target_sched.cfg.example and in the README.

10.13.1 Default jobs#

bkp_setup --configure-server creates the file if it does not exist yet (--no-default-jobs prevents this). Every job can be turned off on its own with enabled=no:

10.13.2 Events and chains#

A job starts by schedule (all settings from “More schedule settings”), on an event (on=) or after other jobs (after=). Events are <kind>-<phase> with the kind backup, mirror-in (another server mirrors to this one), restore or transfer (backup or incoming mirror) and the phase start, success, failed or finished. after-mode= decides whether the follow-up job runs after success (default), after every end (finished) or only after a failure (failed). If a job follows exactly one other, it takes over its event: {store} is then the same store. This makes a chain backup → mirror → check of the store just backed up (example above: scrub-after-mirror with after=mirror-kunde1).

A flow is such a chain with a name: jobs with the same flow=<name> are its steps. Exactly one first step has the triggers (schedule, on=, match-*, wait-for=), every further one has after= on the step before; branches and triggers on later steps are refused by the check. The target console shows and edits a flow as one list (chapter Target console, section “Actions and scheduler”) and always saves it as a whole.

After certain backups a job starts with match-backup=<directory>/<name>[,…]: directory (pattern as in the ACL) and name belong together, so match-backup=kunde1/srv1,kunde2/db01 matches exactly these two – unlike match-dir= with match-name=, which allow every combination. The scheduler remembers the 40 backups and mirrors that came in last (one entry per backup, file recent-backups next to state-file=); the target console offers them as the basis of a job.

An action is a job without schedule, on= and after=: it only starts by hand (bkp_target --sched-run <job>, target API, target console). backup=<directory>/<name> (with backup-kind=mirror-in for a mirror) gives it a backup: a start works with its newest successful run as if it had just come in – {store}, {uuid} and the jobs after it use it. This way, for example, a check restore of the last backup of kunde1/srv1 runs at the push of a button. backup= takes several backups and patterns (backup=kunde1/*,kunde2/db01, one run per backup), backup-run=<uuid>@<store> a certain run instead of the newest. If no successful run of the backup has come in yet, the action does not start and says so. note= is a note without effect. Audit log, job log, mail and evidence report name backup, run and who started it.

On the fly is the name for steps meant to work while a backup comes in: otf-encrypt (encrypt, key=file:… or kms:…), otf-restore (restore to restore-path=), otf-forward-target and otf-forward-s3 (forward to a destination dest=<name>). The first step has on=backup-start (or mirror-in-start, transfer-start) and match-*, every further one after= on the step before. Destinations are in sections [dest:<name>] (kind=target or kind=s3), their keys only as file: or kms:. This version checks and saves on-the-fly steps but does not carry them out yet; the backup itself always stays untouched by them. All keys are in the README and in the template.

If a backup or restore loses its connection, it has not failed yet: the client reconnects within its --net-retry time and carries on, even if the target itself has restarted (for example after an update). The target holds back backup-failed or restore-failed for that long (queue, history, run log: “backup connection lost”). If the run goes on, the failure is dropped, backup-start does not come a second time, and the history shows one backup from its original start; only if the client does not come back does it count as failed. Agents before 2.0.0-alpha.027 do not report this: with them the interruption counts as a failure at once and the rest as a new backup.

10.13.3 Time windows in the target scheduler#

window= holds back every run of the job, also one from on= or after=: it waits in the queue until the window opens. This way, for example, a check after every backup only runs at night. With window-end=stop a running job ends when the window closes; --sched-list then shows it as stopped, and the next run starts the work again. window-end=pause does not exist here, because a paused maintenance run would keep its store locked.

Check all stores on the last Saturday of the month, not on Bavarian public holidays, only 22:00 to 06:00

[check-month]
command=maint-scrub
schedule=0 22 * * SATL
exclude-calendar=de-by
window=22:00-06:00
window-end=stop
jitter=30m
args=--search-dir {storage} --recursive

10.13.4 Parallel runs#

Up to parallel-jobs runs (section [sched], default 2) run at the same time, but every job only once. Two runs on the same store never run at the same time; lock=<name> also forms a group of its own, for example for all exports to the same tape. Check and mirror wait as long as a backup writes into their store.

10.13.5 Waiting for backups and jobs#

after= starts a job when others have finished. wait-for=, on the other hand, holds back a run that is due anyway (schedule, event or after=) in the queue until everything it is meant to wait for is there:

What counts is what came since the run became due; wait-since=12h also accepts the twelve hours before. wait-timeout= (default 6 hours) limits the wait; then wait-timeout-action= decides: fail (default) counts the run as failed and reports by mail what was missing, skip drops it, run starts it all the same with a note in the job log. A waiting run does not lock a store yet; runs started by hand do not wait. Cycles of after= and wait-for= are refused by the check of the file.

Check at 06:00 as soon as the night's backups are there – at 09:00 at the latest all the same

[check-mornings]
command=maint-scrub
schedule=0 6 * * *
wait-for=backup:kunde1/**
wait-since=12h
wait-timeout=3h
wait-timeout-action=run
args=--search-dir {storage}/kunde1 --recursive --only-new

--sched-list shows what a run is waiting for and until when (“waits for backup:kunde1/** … until …”); every wait and its outcome are in the evidence log (next chapter).

10.13.6 Showing, starting, cancelling#

bkp_target --sched-validate                              # check the configuration
bkp_target --sched-list --storage /srv/backups           # jobs, chains, queue
bkp_target --sched-preview check-weekly --count 5       # the next five starts
bkp_target --sched-run prune-all --storage /srv/backups  # at once in the foreground
bkp_target --sched-stop prune-all                        # cancel a running job for good
bkp_target --sched-pause prune-all --for 2h --reason "Maintenance"   # pause a running job
bkp_target --sched-resume prune-all                      # continue a paused job

--sched-pause stops the program of the run at a safe point; connections stay open, and the pause does not count towards the time limit of the job. With --for (at most 7d) the run goes on by itself afterwards, otherwise only with --sched-resume. A target job is never held: if it is paused longer than pause-max (section [sched], default 4h, 10m to 7d), the log notes this once, and it stays paused. Exit code 1 if the job is not running, its program knows no pause or it (with --sched-resume) is not paused. Through the target API: POST /sched/jobs/{name}/pause or resume (docs/target-api.txt).

--sched-list shows the jobs, below them the chains and the queue. For every waiting run it says what it is waiting for:

bkp_target --sched-list --storage /srv/backups

JOB                      COMMAND              RUN-AS    STATUS     LAST RUN         EXIT  NEXT RUN         QUEUED TRIGGER
prune-all                maint-prune          aegisgrid  done       2026-10-24 02:30 0     2026-10-25 02:30 0      schedule=30 2 * * *
check-new                maint-scrub          aegisgrid  done       2026-10-24 21:05 0     -                1      on=backup-success,mirror-in-success
check-weekly             maint-scrub          aegisgrid  never-run  -                -     2026-10-25 04:00 0      schedule=0 4 * * 0
mirror-kunde1            maint-mirror         aegisgrid  running    2026-10-24 21:40 0     -                1      on=backup-success
scrub-after-mirror       maint-scrub          aegisgrid  done       2026-10-24 21:12 0     -                0      after=mirror-kunde1

Chains (after=):
  mirror-kunde1
    -> scrub-after-mirror (success)

Queue: 2 run(s), 1 of at most 2 running at once (parallel-jobs)
QUEUED           JOB                      TRIGGER                      STORE                            STATE
2026-10-24 21:40 mirror-kunde1            backup-success               kunde1/srv1.cbs                  running (pid 4711)
2026-10-24 21:40 check-new                backup-success               kunde1/srv1.cbs                  waits: job 'mirror-kunde1' works on the same store

Further reasons to wait: the same job is already running, all parallel runs are taken, a job holds the same lock=, a --sched-run is running, the time window of the job is closed, a backup writes into the store (only with --storage) or an interrupted restore is still waiting for the client to reconnect. --sched-run also triggers the follow-up jobs from after=, --no-after does not. Every run writes a log to /var/log/bkp/bkp_target-jobs/<job>/. Without mail (next section) monitoring goes by the exit code in the job log.

10.13.7 Mails to operators and customers#

With a section [mail] in bkp_target_sched.cfg the target itself sends mails through the operator's SMTP server (the same SMTP keys as for bkp_sched, plus operator-to= as a requirement). For every job the operator gets what notify= asks for (default: failures; a job that fails again and again at most once a day and once “back to normal”), with result, job log, next step and the last lines of the log. Every mail is first put in the spool and is not lost even after a restart or a disruption of the mail server.

A target serves many customers. Every customer only gets mails about their own stores, without error codes and without a job log, with an explanation (“What this means”) and the next step. Who owns which directory is in /etc/bkp/target-customers.cfg for the target's own directories; for shares with other accounts (hosted/G-…) company name, contacts, the supporting partner and the mail recipients come from the portal through the hourly sync. If a customer is looked after by a partner, their mails refer questions to “your AegisGrid partner”, and replies go to the partner.

Only the operator gets the mail server health (betreiber-serverzustand): as soon as a message of the system check newly becomes a warning or an error – for example “could start VMs, now no longer”, the certificate at 30, 7 and 0 days, the store at 90 and 97 % – and once “back to normal” when everything is fixed. An improvement only counts when it lasts for an hour; this way no mail goes back and forth at a limit, and never the same one every day. Mails stuck in the spool do not trigger a mail.

Per customer and mail kind the recipient can be redirected: kunde (default, the customer), to the partner or, for the operator's own customers, to the operator, to beide (both) or aus (off). If a customer has nobody with the matching role, the mails go to all their contacts, without contacts to the operator. The operator also gets an overview of all stores, customers and whom the target has notified.

Checking and adapting the target's mails

bkp_target --mail-test                                    # test mail to operator-to=
bkp_target --mail-recipients --dir mueller/srv1           # who owns the directory, who gets which mail
bkp_target --mail-preview schaden --customer mueller      # the mail with sample values
bkp_target --mail-templates-export /etc/bkp/mail-templates # put the built-in templates there for adapting
bkp_target --mail-templates-check                         # check your own templates

The wording comes from templates (<mail kind>.tmpl, for the operator betreiber-<mail kind>.tmpl, if wanted with an HTML part <mail kind>.html.tmpl). Your own are in /etc/bkp/mail-templates, for a single customer in kunden/<id>/, for all customers of a partner in partner/<A-…>/; before sending, the target fills in the placeholders (--mail-preview names them all). A faulty template is passed over, the mail then comes from the next level, and the operator learns about it once by mail.

Every customer gets their mails in their language (language= in target-customers.cfg, per contact language=<code> at the end of the contact line), otherwise in the language of the server; mails to the operator are always in the language of the server. A template of your own for only one language is called <mail kind>.<code>.tmpl (for example fehler.fr.tmpl); --mail-templates-export <dir> --lang fr puts the built-in templates there in this language.

10.14 Consumption of the jobs#

The jobs of the target scheduler – check restores with VMs, virus check, scrub, mirror, export, prune – need very different amounts of computing time, memory and disk access on the server. The target therefore measures every run, converts it into resource units (RE), predicts what the next run will need and reports the daily totals per account to the license server. The incoming backups themselves (the sessions of the agents) do not count; they are covered by points or TB days.

10.14.1 Requirement: cgroup v2#

Measuring works through cgroup v2: every run gets a cgroup of its own, all its child processes (clamscan, qemu-nbd, fsck …) count too; check VMs that outlive their job run in a cgroup of their chain. For this the service needs Delegate=yes; the bkp_target.service supplied has it. Without cgroup v2 or without delegation (container, started by hand) the target only measures what the kernel reports when a process ends; such runs are called “estimated” in console, report and message, and the system check names the next step as a note (systemctl edit bkp_target, add Delegate=yes).

The CPU factor evens out fast and slow servers: the system check measures it once a day with a fixed computing load; every run stores the factor of its time, so that old runs do not change afterwards.

10.14.2 Resource units and classes#

By default one RE is 1 normalized CPU core hour = 8 GiB hours of memory = 50 GB of disk access = 500 GB hours of workspace = 1 vCPU hour of a VM = 100 GB of network; the parts are added up. Weights, time rates and price are set by the license server; the target gets them with the renewal and uses them for display and prediction – what counts is the bill at the license server. Every run gets a class: the kind by the largest share (CPU-heavy, memory-heavy, IO-heavy, space-heavy, VM-heavy, network-heavy) and the size S (under 0.1 RE), M (under 1), L (under 10) or XL. If a run spent more than a fifth of its time waiting for processor or disk, “slowed down by the server” is added.

A run of a check chain belongs to the owner of the backup checked (share under hosted/G-…, otherwise customer according to target-customers.cfg, otherwise the account of the target). Jobs across several stores (scrub, prune, mirror, export) are split by the target by the bytes per store; a run without a store counts for the target itself (in the console “the target itself (no single account)”). The target account's own jobs cost nothing.

10.14.3 Limits and fair sharing#

Limits can be set per job; without cgroup they are passed over with a note in the log:

If runs of several accounts compete, they share processor and disks in equal parts (fair-share=yes, default, in [sched] of bkp_target_sched.cfg); this way an account with many jobs does not push out another.

10.14.4 Planning by prediction#

From the last 20 runs of a job the target knows median and 90 % value of every quantity; without runs of its own it estimates from the size of the backup and the kind of job. With capacity-planning=yes (default) a job only starts when its expected memory and cores fit beside the running jobs (memory up to 85 %, cores up to the number of cores); until then it shows “waits for free capacity”. After capacity-wait-max= (default 4h, 10 minutes to 7 days) it starts all the same, with a note in the log; a job with nothing running beside it always starts.

A run that uses more than three times or less than a third of the median (from 5 runs) gets the note outlier. The result of the job stays as it is; the note is in console, job mail and report.

10.14.5 Budget#

There is a monthly budget per job and per account:

At 80 % and at 100 % the mail verbrauch-budget goes to the operator and to the customer, once a month per level. With hold a scheduled run no longer starts and the job shows yellow “Not started: budget reached” – not red, because nothing is broken. Started by hand it runs all the same, with a note in the log. In the next month it carries on by itself.

10.14.6 Report, evidence, energy#

On the first of the month the target creates the consumption report of the previous month as a PDF (RE per job and kind, classes, history over 12 months, prediction, outliers, energy) and sends it with the mail verbrauchsbericht: every customer only their runs, the operator all of them. The reports stay for 13 months under usage-reports/ next to the state of the scheduler. Into the evidence log go job-usage per run, usage-day per day and account and every budget level as job-budget.

The target estimates energy and CO₂ from the measured values; the factors are in bkp_target.cfg and are marked everywhere as “estimate”:

10.14.7 In the target console#

The job list has the column Usage (RE of the last run, class, outlier), the run list the measured values per run. The context menu of a job opens its runs with breakdown, history, prediction and energy through “Usage...”; “Usage of all jobs...” sums up by account, day or job, with budget status, and “Export...” saves CSV or JSON. In the job dialog “Usage...” opens the window Usage and limits: the limits and the budget of the job, what a run is expected to use and cost, and – if the license server has time rates – how much cheaper the job would be in a cheaper time window; “Take over window” enters it as the time window of the job. In the graph the tooltip of a box shows the consumption of the last 31 days.

Customers see the consumption of their checks on the target with the next backup there: bkp_sched list and the scheduler console name the RE of the current month and the budget status.