Storage needs and chunk size – Manual

Go to the contents

The need grows with the number of chunks, not with a block size. With the default chunk size (average 256 KiB, in practice about 300 KiB), 1 TB of used source has around 3.5 to 4 million chunks:

Where	Need (order of magnitude per TB of stored data)
Index of the store (`index.lmdb`)	a few hundred MB on disk; fast if it lies on SSD (`--index-dir`)
Agent	buffers of the pipeline (limited), plus a map of the free areas (at most 16 Mi entries)
`mirror` (first run of a source)	reads all manifests, about 128 MB per TB
`export`, restore from an archive	index in memory (file: about 170 MB per TB; stream from stdin: about 112 MB per TB)

bkp_maintenance store-info shows the estimated index size. If the store lies on NFS/SMB, the index must lie locally (--index-dir).

25.1 How the chunk size works#

The agent cuts the source with FastCDC at content-dependent places. The average size (--chunk-size, power of two, default 256K; the minimum is a quarter, the maximum four times that) is a trade-off:

	smaller chunks (e.g. 64K)	larger chunks (e.g. 1M)
Deduplication	finds more duplicates, small changes cost little	a small change makes a whole large chunk new
Index	larger (more entries)	smaller
Restore	more single reads, slower on HDD	fewer, larger reads

The size is fixed per store (or generation); --chunk-size only takes effect when creating it. A new generation takes over the size of the previous one if none is given.
All sinks of a run must chunk the same way; a store newly added takes over the size of the others.

25.2 Measuring instead of guessing#

With --chunk-probe yes (default) the agent measures on the side on a sample (1/32 of the chunks) how a quarter and four times the current size would have deduplicated, and reports it to the store, also over the network. Nothing of it is stored; it costs about one more chunker pass. store-info shows the estimate per size and, as soon as there is enough data (7 backups over 5 days), a recommendation:

bkp_maintenance store-info --store /backup/default.cbs

End of the output (test data, hence no recommendation yet)

chunk size probe: 2 backup(s) over 0.0 day(s), 2 sample(s) of 256K chunks
SIZE     RUNS    EST. WRITTEN   VS. NOW  COMPR.    EST. INDEX
64K         2        17.4 MiB      +86%    100%       0.6 MiB
256K        2         9.4 MiB     (now)    100%       0.1 MiB
1M          2        33.6 MiB     +258%    100%       0.1 MiB
memory:       1.4 GiB; index on an unknown medium, store on an unknown medium
restores:     1, 188.4 MiB/s, 0.15 ms per read
recommended:  not enough data yet (needs 7 backups over 5 days and 256 samples)

EST. WRITTEN estimates how much would have been written at this size over the measured backups, EST. INDEX the index size. Memory and storage medium also go into the recommendation (if the index no longer fits into memory and lies on a spinning disk, that speaks for larger chunks); restores shows the speed measured during restores.

25.3 Changing the chunk size#

bkp_agent ... --type full --chunk-size 1M

starts a new generation with the new size in all sinks of the run together. Older runs stay readable in their generation until prune removes them. Re-cutting existing runs (rechunk) is not provided. The first run of the new generation costs space and time like a complete backup.

Tip: For very large, rarely changed volumes on HDD a larger chunk size pays off (index and restore), for many small changes and similar machines in one store rather the default size or smaller. When in doubt, wait for the recommendation of store-info.