Stores and runs

Go to the contents

6.1 Layout of a store#

A store is a directory <store>.cbs below the --out directory or, on the server, below <storage>/<directory>/. It contains:

A run consists of a manifest: the source in segments of 1 GiB, every segment a list of chunks and holes. Two runs that contain the same data refer to the same chunks. That is why every run is complete and still small.

• Several backup names (for example several machines) may write into the same store at the same time; every writer creates packs of its own.
• Maintenance that rewrites data (prune, delete, repair, reindex) needs the store to itself and otherwise skips it or waits.
• A cancelled backup leaves no half run behind, at most data without a run. The next prune clears it away (at the earliest after --gc-min-age, default 24 h), unless a new run uses it by then.

6.2 Backup names per store#

Several machines may back up into the same store; shared data is kept there only once. So that their backups never get mixed up, a backup name (the profile name, e.g. system) belongs to exactly one source in a store: for machine and M365 backups to the installation (installation ID I-…, which stays when the same machine is activated again), for VM backups to the VM (its UUID). Who owns a name is recorded in the name register <store>.names next to the store and in every run (owner). Restore “newest run”, retention and GFS count per name.

• If a machine backs up under a name that already belongs to another one in the store, the backup is refused before the first block (agent and target check). The message names the other machine and suggests a free name, usually with the host name (system-fs01).
• If the machine really replaces the other one (new hardware, reinstalled), it takes over the name on purpose: bkp_agent … --adopt-name or, in the management console, “Take over backups...” in the job menu. The previous runs then count for it; the target records the takeover in the audit log.
• The job scheduler checks the names of its profiles daily, after every change and after a refused backup. A conflict appears as the state “name taken” (API NAME_TAKEN) with a mail; the console shows it as soon as the profile is created and offers to rename it or to take it over.
• Older runs without an owner belong to the machine with their host name. Builds without a license do not check.

Viewing the names

bkp_maintenance store-info --store /srv/backups/default.cbs --names     # who owns each name, which machines have runs
bkp_restore --sink rz1 --name system --check-name /tmp/name.json        # check without writing (free, own, taken)

In the target console the tab “Data” shows in the column “Computer” which machine the runs come from (“several” if a name has runs of several machines; store-info --names then says what to do).

6.3 Generations#

Because runs share chunks, damage to a frequently used chunk hits many runs. Whoever wants a physically independent base (like a new FULL in the past) starts a new store generation: a store of its own, <store>.g<YYYYMMDD>.cbs, next to <store>.cbs, which shares no data with the older ones.

• --type full starts a new generation (in all sinks of the run; with Proxmox/Xen once for the whole VM run). The first run in it is stored in full again.
• --store-generation <n>d|<n>m (e.g. 1m) starts it automatically as soon as the current one is older.
• Restore, --list, prune and mirror see all generations of a store as one timeline; prune removes an old generation as soon as it has no runs left.
• A new generation may get a different ID hash (--hash) and a different chunk size (--chunk-size). That is how to change both without a break.

bkp_agent --source data.img --out backup --name data --type full --hash blake3

Output:

[bkp_agent] INFO  started the new store generation default.g20261015 (--type full; the previous one is default)
[bkp_agent] INFO  created the store backup/default.g20261015.cbs (id hash blake3, encryption none, ...)
[bkp_agent] INFO  done: full run 20261015T121849Z-44fdc352-... in backup/default.g20261015.cbs: ... chunks 44, new 44 (0.0% deduplicated); ...

bkp_restore --store backup/default.cbs --list --name data

Output:

RUN                                     CREATED              NAME  TYPE         ...   NEW       STATUS  STORE
20261015T121825Z-6aa5d6f2-...           2026-10-15 14:18:25  data  Full         ...   12.0 MiB  ok      default
20261015T121831Z-8c47f436-...           2026-10-15 14:18:31  data  Incremental  ...    2.2 MiB  ok      default
20261015T121849Z-44fdc352-...           2026-10-15 14:18:49  data  Full         ...   12.0 MiB  ok      default.g20261015

6.4 Encryption#

The agent can encrypt chunks before they are written locally or sent to the server, independently for every sink (encryption=/key= in the [sink:<name>] section, Chapter 4). AES-256-GCM and ChaCha20-Poly1305 are available; both reliably detect a wrong key or tampered data.

Create a key and back up encrypted

bkp_maintenance encrypt-init --key-file /etc/bkp/out.key       # create the key once
bkp_agent --source /dev/vg0/data --sink lokal --name data       # [sink:lokal] with encryption= and key=

• The store gets the method and key check value when it is created; a run with a different method or key is refused before anything is written.
• The IDs are formed with a key derived from the key. Without the key the store reveals only structure and sizes, never content.
• The key applies for the lifetime of the store or generation; a new key is only possible with a new store or --type full.
• Given like the API key: file:<path>, the key itself (64 hex characters, only in configuration files) or the reserved kms:<url>.
• prune, delete, mirror, export/import and a scrub of the checksums work without the key. The server never needs the key.

6.5 S3 object storage#

A store can live in S3-compatible object storage (AWS S3, Wasabi, MinIO and others). Backups get there in three ways: the agent writes directly (sink type=s3, Linux and Windows), the target copies every backup that comes in (destination [dest:<name>] kind=s3 and a maint-mirror job with dest=<name>, in the target console the action “Copy to S3”), and restore, maintenance and bkp_dump take a store as s3:<name>/<path> (<name> = the section with the S3 settings, --s3-config <file>).

An S3 sink in sinks.cfg

[sink:wasabi1]
type=s3
s3-endpoint=https://s3.eu-central-2.wasabisys.com
s3-bucket=aegisgrid-backup
s3-region=eu-central-2
s3-prefix=customer1
s3-access-key=file:/etc/bkp/wasabi1.access
s3-secret-key=file:/etc/bkp/wasabi1.secret
encryption=aes-256-gcm
key=file:/etc/bkp/wasabi1.key

• What goes to S3 is always encrypted; a type=s3 sink without encryption=/key= is refused when it is loaded. The target encrypts stores that came in unencrypted with the destination's key= while copying; without it, it does not copy them.
• Access keys only as file: or kms:. The consoles store them as files (readable by administrators only) and never show them again.
• bkp_agent --s3-check <sink> or bkp_maintenance s3-check s3:<name> checks connection, certificate, keys, bucket, clock, conditional writes, versioning, Object Lock and a test object; every step names the next step. The consoles show the same under “Check connection”.
• The index is kept locally (/var/cache/aegisgrid/s3/<store-uuid>, Windows %ProgramData%\AegisGrid\cache\s3\<store-uuid>); if it is missing, it is rebuilt from the pack footers.
• Only one writer: a lease in the bucket (locks/, 10 minutes, renewed all the time); an expired one is taken over by the next program with a warning.
• Temporary errors (5xx, 429, a dropped connection) are repeated up to 6 times; final ones come with the reason and the next step.
• WORM: worm=extend locks packs and runs with Object Lock (worm-mode=governance or compliance). The bucket needs Object Lock without a default retention; attempts to delete locked objects are INFO lines.

Details and all keys: README, section “S3-Objektspeicher”; how a store looks in the bucket: docs/store-format.txt 2.4.

6.6 Viewing stores and runs: bkp_dump#

bkp_dump backup/default.cbs

Output:

=== store backup/default.cbs ===
format              : 1
uuid                : e694d0f7-9e67-4409-a3c1-6c44a3a842a2
kind                : block
encryption          : none
key_check           : (none)
id_hash             : blake3
segment_size        : 1073741824
chunker             : fastcdc 65536 / 262144 / 1048576 (min / avg / max)
objects             : 56
stored_bytes        : 14849939
packs               : 2
index               : backup/default.cbs/index.lmdb

=== runs (2, oldest first) ===
run                                 created                   kind   backup_name  type        status  source_size  logical_bytes  sent_bytes  new_bytes  expires
20261015T121825Z-6aa5d6f2-...       2026-10-15T14:18:25+02:00 block  data         Full        ok         16777216       13631488    12587566   12587566  never
20261015T121831Z-8c47f436-...       2026-10-15T14:18:31+02:00 block  data         Incremental ok         16777216       13631488     2262373    2262373  never

With --run <run>, bkp_dump also shows the run (point in time, machine, sizes, retention, run_group), the source and the segments; --extents lists every chunk and every hole (--limit <n>, default 50, 0 = all). With encrypted stores the manifest needs --decrypt-key. An overview with deduplication, last cleanup and last scrub is given by bkp_maintenance store-info (Chapter 10).

bkp_dump backup/default.cbs --run 20261015T121831Z-8c47f436-cdc7-4638-a3f8-d347e0c5ba7c --extents --limit 3

Excerpt

=== segments (1) ===
segment  offset           length       chunks   holes    data_bytes     id
0        0                16777216     44       1        13631488       edcc0ef0d0bd68e2...
    0                328147       data  79596f0301091b2334ddb6ead02f207789986b4a2003f5bc4d3a263c926614aa
    328147           281294       data  1e5cc22f23588a0318012478db8c7e83b6b05dfa8d98c1541c5f50d5415d177c
    609441           328454       data  5a40c945daf7bea90ee554230ac3a7d81cb5083dee156ceb7840305e8b34f9e1
... (more extents not shown; use --limit 0 for all)

total: 44 chunks, 13631488 data bytes, 1 holes, 3145728 hole bytes

6.7 Managing stores safely#

• Never delete or copy single files in a store. Runs are deleted by bkp_maintenance prune or delete, copies are made by mirror, single runs are written as an archive by export (Chapter 10).
• Run scrub regularly and keep a second copy (second sink or mirror) ready, from which repair can replace damage.