backing up a collection of snapshot subvolumes

backing up a collection of snapshot subvolumes

[J. Hart]

I have a remote machine with a filesystem for which I periodically take 
incremental snapshots for historical reasons.  These snapshots are 
stored in an archival filesystem tree on a file server.  Older snapshots 
are removed and newer ones added on a rotational basis.  I need to be 
able to backup this archive by syncing it with a set of backup drives. 
Due to the size, I need to back it up incrementally rather than sending 
the entire content each time.  Due to the snapshot rotation, I need to 
be able to update the state of the archive backup filesystem as a whole, 
in much the same manner that rsync handles file trees.

It seems that I cannot use "btrfs send", as the archive directory 
contains the snapshots as subvolumes.

I cannot use rsync as it treats subvolumes as simple directories, and 
does not preserve subvolume attributes.  Rsync also does not support 
reflinks, so the snapshot directory content will no longer be reflinked 
to other snapshots on the archive backup.  I cannot use hard links in 
the incrementals as hard links do not cross subvolume boundaries.

Thoughts anyone ?

Re: backing up a collection of snapshot subvolumes

[Graham Cobb]

On 25/04/17 05:02, J. Hart wrote:
> I have a remote machine with a filesystem for which I periodically take
> incremental snapshots for historical reasons.  These snapshots are
> stored in an archival filesystem tree on a file server.  Older snapshots
> are removed and newer ones added on a rotational basis.  I need to be
> able to backup this archive by syncing it with a set of backup drives.
> Due to the size, I need to back it up incrementally rather than sending
> the entire content each time.  Due to the snapshot rotation, I need to
> be able to update the state of the archive backup filesystem as a whole,
> in much the same manner that rsync handles file trees.

If I have understood your requirement correctly, this seems to be
exactly matched to the capabilities of btrbk. I use btrbk to maintain a
similar backup disk which contains a full copy of my main data disk
along with various snapshots.

> It seems that I cannot use "btrfs send", as the archive directory
> contains the snapshots as subvolumes.

I'm not sure that you mean. If your problem is that btrfs send does not
cross subvolume boundaries then that is true: you would need to
configure btrbk to back up each subvolume. I have a cron job that checks
that all subvolumes (except the snapshots btrbk creates) are listed in
my btrbk configuration file.

Graham

Re: backing up a collection of snapshot subvolumes

[Kai Krakow]

Am Tue, 25 Apr 2017 00:02:13 -0400
schrieb "J. Hart" <jfhart085@gmail.com>:

> I have a remote machine with a filesystem for which I periodically
> take incremental snapshots for historical reasons.  These snapshots
> are stored in an archival filesystem tree on a file server.  Older
> snapshots are removed and newer ones added on a rotational basis.  I
> need to be able to backup this archive by syncing it with a set of
> backup drives. Due to the size, I need to back it up incrementally
> rather than sending the entire content each time.  Due to the
> snapshot rotation, I need to be able to update the state of the
> archive backup filesystem as a whole, in much the same manner that
> rsync handles file trees.
> 
> It seems that I cannot use "btrfs send", as the archive directory 
> contains the snapshots as subvolumes.
> 
> I cannot use rsync as it treats subvolumes as simple directories, and 
> does not preserve subvolume attributes.  Rsync also does not support 
> reflinks, so the snapshot directory content will no longer be
> reflinked to other snapshots on the archive backup.  I cannot use
> hard links in the incrementals as hard links do not cross subvolume
> boundaries.
> 
> Thoughts anyone ?

If this is for archival purpose only and storage efficiency and speed is
your primary concern, try borgbackup.

Borgbackup deploys its own deduplication and doesn't rely on btrfs
snapshot capabilities. You can store your archives wherever you want
(even on non-btrfs), and it won't store any data blocks twice.

Upon restore, you could simply recreate the snapshot/subvolume from a
similar tree and then rsync a tree restored from borgbackup back to
this snapshot to get back a state with the benefits of btrfs snapshots.

Borg also has an adapter for fuse to get a mounted view into the
archives, tho, it is slow and probably takes a lot of RAM. But it is
good enough for single file lookups and navigation.

-- 
Regards,
Kai

Replies to list-only preferred.