Consistent backups: snapshots and hooks

Go to the contents

A volume that is in use, for example the disk of a running virtual machine, changes while it is being read. The image then matches no point in time. That is why, for an LVM volume, the agent first takes a snapshot, backs it up and then removes it again. This applies to normal (thick) and to thin volumes.

5.1 LVM snapshots#

# default (--snapshot auto): if the source is an LVM volume, a snapshot is used
bkp_agent --source /dev/vg0/icinga.disk --out /backup --name icinga

If --snapshot on points to a source without LVM, the agent stops:

Output:

[bkp_agent] ERROR --snapshot on: data.img is not an LVM logical volume
[bkp_agent] ERROR backup failed (see the message above)

5.1.1 What to keep in mind#

• If the space for changes (--snapshot-size) fills up during the backup, the snapshot becomes invalid. The agent then stops and publishes nothing. For volumes with heavy writes, raise the value.
• Only one agent runs per volume at a time; a second one reports this and stops.
• Snapshots left over after a crash (<volume>_bkpsnap, with the LVM tag bkp_agent) are cleaned up by the next run. A volume of the same name without this tag is never touched.
• Run and monitor go on showing the original volume, not the snapshot. Deduplication against earlier runs works as usual.
• With a resumable network backup the snapshot deliberately stays after a crash instead of being removed at once — see “Crash-safe network backup” in Chapter 4.

5.2 Hooks: freezing the guest#

A snapshot is *crash-consistent*, as if the power had failed. Journaling file systems and databases with a transaction log (SQL Server, PostgreSQL, MySQL/InnoDB) usually cope with that. What an application still holds in its own memory is missing, though. For *application-consistent* backups the guest must write back its buffers before the snapshot and stand still for a moment. The hooks do that. They are deliberately not tied to a hypervisor.

Cron job that freezes the guest (libvirt/QEMU guest agent)

bkp_agent --source /dev/vg0/icinga.disk --target backup.example.org:9990 --target-dir vm1 \
          --tls-ca /etc/bkp/target.crt --api-key file:/etc/bkp/vm1.key --name icinga \
          --pre-snapshot  'virsh domfsfreeze icinga' \
          --post-snapshot 'virsh domfsthaw icinga'

The hooks get BKP_SOURCE (original volume), BKP_NAME, BKP_SNAPSHOT (snapshot device) and BKP_PHASE (pre or post) in their environment. The agent reports how long the guest stood still between pre and post hook (consistency window), usually a few seconds. The guest stays frozen only as long as passes between the two hooks; the post hook also runs on errors, timeouts and SIGTERM.

Freezing over SSH: always thaw the guest through the connection that was opened for freezing (SSH multiplexing). A *new* login on a frozen guest can hang, on a Debian guest after only about 20 seconds, because the guest wants to write to its frozen file system during login. The post hook then does not get through, and the guest stays frozen. Through the open connection it thaws at once, even after a long freeze. Thaw several file systems in reverse order.

Xen domain with a Linux guest, frozen over SSH and thawed through the same connection

O="-o ControlMaster=auto -o ControlPath=/run/bkp-freeze-%h -o ControlPersist=300"
bkp_xenV --name web01 \
    --pre-snapshot  "ssh $O root@web01 'fsfreeze -f /data; fsfreeze -f /'" \
    --post-snapshot "ssh $O root@web01 'fsfreeze -u /; fsfreeze -u /data'; ssh $O -O exit root@web01" \
    -- --sink backup

With Windows guests the QEMU guest agent does the freezing through VSS. VSS-aware applications such as SQL Server are thus paused cleanly; applications without a VSS writer are backed up only crash-consistently.

5.3 Windows: VSS shadow copy#

bkp_agent.exe backs up a volume (\\.\C:) or a whole disk (\\.\PhysicalDrive0) from a VSS shadow copy and needs administrator rights for that. The options --snapshot, --pre-snapshot, --post-snapshot and --hook-timeout apply as on Linux (hooks run through cmd.exe).

Windows: back up drive C: through VSS (command prompt as administrator)

bkp_agent.exe --source \\.\C: --target backup.example.org:9990 --target-dir srv-win01 ^
    --tls-ca C:\ProgramData\AegisGrid\target.crt --api-key file:C:\ProgramData\AegisGrid\srv-win01.key ^
    --name srv-win01-c

• Whole disk: VSS only knows volumes. The agent therefore takes a shadow copy of all volumes of the disk in one step (the volumes are thus consistent with each other) and puts the image together from several sources: partition table, gaps and everything VSS does not support (for example a FAT EFI partition, dynamic disks) are read directly from the disk, the volumes from the shadow copies. The log names for every partition where it was read from. With --snapshot on the backup fails if a volume cannot be backed up through VSS.
• Disk by its ID: the number in \\.\PhysicalDrive<n> can change after a restart or when another disk is connected; then a different disk would be backed up. Better is the ID of the disk: --source disk-id:{<GPT disk GUID>} or disk-id:mbr-<signature, 8 hex digits> (this is also how the scheduler's source list, GET /sources, names it). The agent looks for the disk on every run and gives its current number in the log; chain, resuming and index carry on by the ID, even if the number changes. If the disk is missing or two disks carry the same ID (a copied disk), the run stops with a message instead of backing up another one. On Linux the link in /dev/disk/by-id serves the same purpose (for a ZFS volume the link in /dev/zvol).
• --vss-mode copy|full (default copy): copy corresponds to VSS_BT_COPY; the application logs (for example of SQL Server) stay unchanged and other backup software is not disturbed. full corresponds to VSS_BT_FULL: after a successful backup the VSS writers are told that a full backup has been completed, and they truncate their logs.
• Limits: VSS makes only applications with a VSS writer application-consistent (SQL Server, Exchange, Active Directory, Hyper-V and others). For software without a writer the snapshot is crash-consistent; the hooks are there for that (stop the service and start it again). The VSS freeze window is limited to a few seconds.
• Persistent shadow copy: for a resumable network backup (--target) the agent by default (--vss-persistent auto) tries a shadow copy that survives a restart; if the backup is interrupted, it stays and the next run continues from the same point in time without reading the source again. See “Crash-safe network backup” in Chapter 4.