Re: Anyone tried out btrbk yet?

[Sander <sander@humilis.net>]

Marc MERLIN wrote (ao):
> On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote:
> > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup
> > as well), local snapshots continue to happen as normal (Eg. daily or
> > hourly) and so when your backup media or backup server is finally
> > available again, the size of each individual incremental is still the
> > same as usual, it just has to perform more of them.
>  
> Good point. My system is not as smart. Every night, it'll make a new
> backup and only send one incremental and hope it gets there. It doesn't
> make a bunch of incrementals and send multiple.
> 
> The other options do a better job here.

FWIW, I've written a bunch of scripts for making backups. The lot has
grown over the past years to what is is now. Not very pretty to see, but
reliable.

The subvolumes backupadmin home root rootvolume and var are snapshotted
every hour.

Each subvolume has their own entry in crontab for the actual backup.
For example rootvolume once a day, home and backupadmin every hour.

The scripts uses tar to make a full backup every first backup of a
subvolume that month, an incremental daily backup, and an incremental
hourly backup if applicable.

For a full backup the oldest available snapshot for that month is used,
regardless of when the backup is started. This way the backup of each
subvolume can be spread not to overload a system.

Backups are running in the idle queue to not hinder other processes, are
compressed with lbzip2 to utilize all cores, and are encrypted with gpg
for obvious reasons. In my tests lbzip2 gives the best size/speed ratio
compared to lzop, xz, bzip2, gzip, pxz and lz4(hc).

The script outputs what files and directories are in the backup to the
backupadmin subvolume. This data is compressed with lz4hc as lz4hc is
the fastest to decompress (useful to determine which archive contains
what you want restored).

Archives get transfered to a remote server by ftp, as ftp is the leanest
way of file transfer and supports resume. The initial connection is
encrypted to hide username/password, but as the archive is already
encrypted, the data channel is not. The ftp transfer is throttled to
only use part of the available bandwith.

A daily running script checks for archives which are not transfered yet
due to remote server not available or failed connection or the like, and
retransmits those archives.

Snapshots and archives are pruned based on disk usage (yet another
script).

Restore can be done by hand from snapshots (obviously), or by a script
from the locale archive if still available, or the remote archive.

The restore script can search a specific date-time range, and checks
both local and remote for the availability of an archive that contains
the wanted.

A bare metal restore can be done by fetching the archives from the
remote host and pipe them directly into gpg/tar. No need for additional
local storage and no delay. First the monthly full backup is restored,
then every daily incremental since, and then every hourly since the
youngest daily, if applicable. tar incremental restore is smart, and
removes the files and directories that were removed between backups.

	Sander