Monitoring

Go to the contents

13.1 Live view: bkp_monitor#

Agent and restore write their progress into the run channel, a shared memory segment: for a scheduler job into the channel bkp_sched creates for this run (/dev/shm/bkp_runctl.<pid>.<n>), for a program started by hand into one of its own (/dev/shm/bkp_runctl.<pid>). bkp_monitor shows it: throughput, compression ratio, data and holes (of them free in the file system), chunks and how many of them were new (deduplication rate), fill level of the read, processing and write queues, load of the workers, the current zstd level and the CPU features used (AVX2, SHA-NI). The segment is created right at program start: during the preparation (snapshot including hooks, connection, opening the store, file system bitmap, pre-allocating the restore destination) the job is starting and the status line names the current phase. The ETA only counts from the start of the data transfer. bkp_proxmox, bkp_xenV and their restore programs also create a segment of their own (phase: freeze, snapshots, disk n/m, ...); bkp_monitor shows the orchestrator as one entry and automatically follows the bkp_agent/bkp_restore running at the moment within it (also with --dump).

bkp_monitor                  # first running job, on the left the list of all jobs (arrow up/down switches)
bkp_monitor --pid <pid>      # show a certain process

The monitor helps to find the bottleneck: if the read queue is mostly empty, the source is too slow; if the write queue is full, the destination slows things down; if all workers are busy, CPU is lacking (more --threads or a lower compression level).

13.2 Scheduler jobs at a glance: bkp_sched list#

bkp_sched list shows all scheduler jobs on this machine in a table, read-only. Times are short and in local time (“01:30”, “yesterday 02:40”, “Sun 03:00”, further ones with a date); the line above the table names the offset to UTC; --full shows the full time stamps instead. “DURATION” is the duration of the last run, for a running job the time so far with the estimated time left. A running job shows its progress in percent, a failed or partly successful one its exit code. Below a waiting job it says which after= jobs it is still waiting for, below a blocked one the reason, below a held one (HELD, bkp_sched hold) since when, by whom, when it will continue and the reason, for example “held since 22:00 · by bkp_sched hold (root) · continues tomorrow 06:00 · reason: relocation”. A summary ends the list.

bkp_sched list

Times in local time (UTC+01:00)
NAME          COMMAND       STATUS      NEXT-RUN       LAST-RUN          DURATION              AFTER
srv-db        agent         RUNNING 43% tomorrow 01:30 01:30             18 min (~24 min left) -
m365-mail     m365          FAILED (1)  tomorrow 02:00 02:00             3 min                 -
mirror        maint-mirror  PENDING     -              yesterday 02:40   1 h 12 min            srv-db,m365-mail
  waiting for: srv-db (RUNNING), m365-mail (FAILED)
prune         maint-prune   DONE        Sun 03:00      Sun 03:00         45 s                  -
restore-test  restore       DISABLED    -              2026-10-24 10:15  6 min                 -

5 jobs: 1 done, 1 running, 1 failed, 1 waiting, 1 disabled

In a terminal only the column “STATUS” is coloured: done green, running cyan, failed red, partial and blocked yellow (brown on a light background), waiting, held and disabled without colour. The colours adapt to a dark or light background if the terminal tells it (COLORFGBG or the query of the background colour); NO_COLOR=1 turns the colours off. If the terminal is too narrow, “AFTER” is shortened first, then “NAME”. Into a file or pipe the table always goes complete and without colours.

13.3 A job in detail: bkp_sched show#

bkp_sched show <job> shows a job line by line: command, profile (the name of the backup), the sinks of the profile, the schedule as in the configuration and in words, the next start, the last run with status and duration, after= and the state (on, off, held, paused, with reason). For a backup job the last line is the ready-made command that gets you to the backups: for a file backup the search, otherwise the list of runs. The command only reads and creates nothing.

bkp_sched show data-night

Output:

Times in local time (UTC+01:00)
Job:         data-night
Command:     agent
Profile:     daten
Sinks:       usb1, usb2
Schedule:    schedule=0 22 * * *
             (Daily at 22:00; missed runs are caught up once)
Next start:  22:00
Last run:    never
After:       -
State:       switched on
Restore:     bkp_restore --name daten --files find "<pattern>"

• With command=agent the sinks come from bkp_agent.cfg, with Proxmox, Xen and Hyper-V jobs from sinks= of the profile in their configuration; if the profile is missing there, the line says so. Restore, M365 and maintenance jobs have no line “Sinks” and no line “Restore”.
• An unknown job name names the existing jobs (at most 20) and bkp_sched list.
• --json gives the same information for programs (name, command, profile, sinks, schedule, next_run, last_run, status, after, restore_hint; times in UTC, never translated).

13.4 For scripts: --dump#

bkp_monitor --dump --pid <pid> --interval 200 | cut -f1-5,21-27

Excerpt (columns 1–5 and 21–27)

#elapsed_ms	blocks_done	blocks_total	bytes_raw	bytes_compressed	state	logical_bytes	hole_bytes	fs_free_bytes	chunks	chunks_new	new_bytes
5934	33	48	35487305	35412856	running	35487305	0	0	119	118	35412856
6167	48	48	50331648	50341056	DONE	50331648	0	0	168	168	50341056

The complete line has 27 columns, among them fill levels of the queues, workers, zstd level and the values of the self-throttling (bg_*, with --background). blocks_* counts units of the source read here, chunks/chunks_new the chunks created and those of them newly stored.

Per interval (default 500 ms) --dump prints a tab-separated line and ends when the process is finished. --wait <s> (default 10) sets how long to wait for the segment of the process. This way, for example, the progress can be taken over into a log or monitoring software.

bkp_agent ... --name data &
bkp_monitor --dump --pid $! --interval 5000 >> /var/log/bkp/progress.tsv

13.5 Controlling running jobs: pause, background mode, bandwidth#

Through the same run channel a scheduler job takes instructions as long as it runs. Control goes through the API of bkp_sched (GET /jobs/{name}/progress shows the progress of every program of the run, PUT /jobs/{name}/control changes what is requested; details in docs/sched-api.txt, sections 7.17 and 7.18):

bkp_agent and bkp_restore follow all instructions, bkp_m365 all except the background mode. With VM jobs (bkp_proxmox, bkp_xenV and their restores) the bkp_agent or bkp_restore of the disk being transferred takes over the instruction, also the programs of the following disks. An instruction only applies to the current run; the next one starts again as the job is set up.

On the server itself the same works with bkp_sched pause, resume and throttle (Chapter 4), also for runs started with bkp_sched run/start. Programs with --no-progress and the cleanup phase after a cancellation cannot be controlled.

13.6 Progress window on the Windows desktop#

When the scheduler starts a backup or restore on Windows, the users logged on see it in a small window at the top right of the screen: per run the job, a bar and a line such as “42 % · 1.3 GB of 3.0 GB · 85.0 MB/s · about 5 min left”. The window lies (in the default display) above the other windows, is slightly transparent and lets clicks through: whoever works below it is not disturbed. If the mouse rests on it briefly, it becomes opaque and shows “Pause” (or “Resume”) and “Hide”; this way it can also be dragged to another place, which it remembers for the user. More than three runs are summed up as “and 2 more runs”; the end of a run (“Completed”, “Ended with errors – details in the log”, for a held run “Held – it continues later”) stays visible for ten more seconds.

Display: where the window lies can be chosen. “In the foreground” (default) is the behaviour described above. “On the desktop” puts it on the desktop: above wallpaper and icons, but behind all program windows; with “Show desktop” (Windows key+D) it can be seen, and there too it can be used and moved with the mouse. “In the desktop background” makes it part of the wallpaper, behind the desktop icons; there it takes no mouse input. The default for the machine is [sched] overlay-mode (or the console, “Scheduler settings”).

As long as the window runs, it has an icon in the notification area of the taskbar (bottom right; Windows 11 and Server 2025 show a new icon behind the arrow ^ at first). Its tooltip shows the progress, a click opens a menu: “Display” with “As set on this computer” and the three positions, “Pause backup “…”” or “Resume”, “Hide” and “Show again”. The choice applies at once and only for this user (remembered in HKCU\Software\AegisGrid\overlay, value mode); “As set on this computer” follows the default again. If bkp_overlay.exe is started by hand with --mode top|desktop|wallpaper, this position applies, and the menu does not let it be changed. If Explorer is restarted, the window appears again after about a second; hidden runs can then be seen again.

The service starts the program bkp_overlay.exe (component “Job scheduler”) itself in every session logged on, also in Remote Desktop sessions and at a logon during a run, under the account of the user concerned. It asks the service through a named pipe that can only be reached on the machine itself (docs/sched-api.txt, appendix A), and ends when there is nothing more to show. While a program takes up the whole screen (presentation, game, video), it stays hidden. Runs started by hand with bkp_sched run/start are not shown.

Pausing is allowed for every user logged on. The run stops at a safe point (as through the console) and goes on by itself after 60 minutes; the window shows when. The job log notes who paused and resumed. “Resume” only exists for a pause from the window: a run paused through the console or bkp_sched pause is shown as “Paused through the management”. If the console pauses or resumes a run paused in the window, its decision applies, and the pause no longer ends by itself. A run cannot be held (bkp_sched hold, Chapter 4) in the window.

The settings in [sched] take effect at once, without restarting the service; in the console they are under “Scheduler settings” and in the job dialog.

13.7 Monitoring the success of backups#

• The exit code of bkp_agent is 0 only if all sinks have completed their run, 3 for a failed optional sink, 75 if the same job is already running, 77 if the license allows no new backups (chapter “Automatic updates”, section “Trial version and license”), 78 if the run was held and will continue later (bkp_sched hold, not an error). Other values are errors, in a cron job therefore a reason for an alarm mail. bkp_sched sends the mail itself and shows the figures of the run in it.
• The line done: ... chunks N, new M (x% deduplicated); ... newly stored at the end of the log shows how much was new. A suddenly very large value for newly stored (or 100 % deduplicated for days with an active source) is suspicious. An unusually large increase can also point to encryption by malware.
• bkp_restore --list against the server shows whether the expected backup of the day is there (columns NEW and STATUS); bkp_maintenance store-info shows deduplication, last cleanup and last scrub.
• Run scrub regularly (Chapter 10).

13.8 Windows event log#

On Windows the scheduler service bkp_sched and bkp_updater also write the most important events into the event log Application, source AegisGrid. Monitoring tools (for example an RMM system or a scheduled task on an event) can thus filter by fixed event IDs. The text is a whole sentence in the language of the system and names the job log for errors. The setup registers the source and removes it on uninstall; the events written stay in the log. If the source is missing (for example after an update of an older installation), the service registers it itself when it starts.

Runs that were held and continue by themselves (bkp_sched hold, restart, update), and failed attempts followed by another attempt with retry=, do not appear. On Linux there is no counterpart; there the log and the mails apply.