Re: btrfs based backup?

[Oliver Freyermuth <freyermuth@physik.uni-bonn.de>]

[-- Attachment #1: Type: text/plain, Size: 4379 bytes --]

Hi,

I'm not sure if the btrfs list is the correct place for a generic answer - but I'll try to give one
mentioning all the backup solutions I have collected experience with (all open source, of course). 

1) btrbk ( https://github.com/digint/btrbk )
   I use it on all my personal machines, both for local snapshotting (to unroll my own mistakes easily...) and for sending the incrementals
   to an external storage. It's basically a well-working btrfs send / receive automation, so it needs btrfs at both ends (or becomes less efficient), which may not match your use case. 

2) Borg Backup ( https://borgbackup.readthedocs.io/en/stable/ )
   I use this whenever I do not have btrfs at one / both ends. It can also do encryption, compression and deduplication, purge old incrementals without ever doing a full backup,
   you can even mount your backups. 
   I use this for some smaller machines (e.g. on a Raspberry Pi) and we use it on our infrastructure for some configuration backups. 

3) Restic ( https://restic.readthedocs.io/en/latest/ )
   Restic is (feature-wise) like borg (but no compression yet). The main difference is that it can (but does not have to) back up to cloud-like storages such as S3. 
   We intend to use this heavily to a local Ceph storage system with 3x replication offering S3/Swift via Rados Gateway nodes. 
   If you want something less heavy than a Ceph cluster (we love it, it does not bite!) you can try minio ( https://min.io/ ). I never used minio myself,
   but only heard good things about it. 

4) Duplicati ( https://www.duplicati.com/ )
   Like Borg / Restic (can also talk S3 if wanted, or store to a file system, also does compression). 
   The main advantage here is that it has a GUI. Probably not interesting for your use case, but we intend to recommend that to our users
   with Windows / MacOS X who may prefer some buttons to click. 

5) Since you mention VMs in your signature, I'll also mention:
   https://benji-backup.me/
   http://backy2.com/
   https://bitbucket.org/flyingcircus/backy
   I'll personally recommend benji here, due to a large featureset, very active development and high efficiency. 
   It does differential backups of RBD volumes, so it will only be really useful to you if you use Ceph RBD
   (you can also get it to work with LVM and raw block devices, I think). 
   You can find some of our experiences with it here:
   https://indico.cern.ch/event/765214/contributions/3517132/

I think all of these are not too complex (of course, they only work well if your infrastructure matches them)
since you can essentially arrive at a working backup and restore in a few minutes. 
I'll also add that for almost all of our servers, we do not do any backups at all - file servers and services with data have their storage replicated to their HA partner node(s),
and all configuration is "backed up" by having it completely in Foreman / Puppet, so a machine can be reinstalled at the push of a button. 

The main functionalities you give up with your original idea (rsync to one replicated node) is that you do not have deduplication built-in and would need to do encryption at the source yourself if needed. 
Also, you would have to do regular "full" backups and think about how you can keep incrementals - rsnapshot (which nowadays seems rather dead) could do something similar via rsync with hardlinks,
but that meant you had tons of files which no FS really likes, and would always have to store a full file if a single byte changed. 

Cheers and hope that helps,
	Oliver

Am 12.11.19 um 19:34 schrieb Ulli Horlacher:
> 
> I need a new backup system for some servers. Destination is a RAID, not
> tapes.
> 
> So far I have used a self written shell script. 25 years old, over 1000
> lines of (HORRIBLE) code, no longer maintenable :-}
> 
> All backup software I know is either too primitive (e.g. no versioning) or
> very complex and needs a long time to master it.
> 
> My new idea is:
> 
> Set up a backup server with btrfs storage (with compress mount option),
> the clients do their backup with rsync over nfs.
> 
> For versioning I make btrfs snapshots.
> 
> 
> To have a secondary backup I will use btrfs send / receive,
> 
> 
> Any comments on this? Or better suggestions?
> 
> The backup software must be open source!
> 



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5432 bytes --]