btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

[Dave T]

Are btrbk-specific questions OK here?

I have a small LAN with a fileserver that should store backups from
each attached host on the LAN. What is the most efficient (performant)
way to do this with btrbk?

Each host (laptops, desktops and a few other devices) does hourly
local snapshots with btrbk. Once per day, I would like to send backups
of each volume on each device to the local fileserver. This has to be
done via SSH (as NFS isn't supported by btrfs send|receive, afaik).

The options I'm aware of from the btrbk readme
(https://digint.ch/btrbk/doc/readme.html) are:

1. host-initiated backup to the fileserver from each host

2. fileserver-initiated backups from all hosts

My guess is that the second option is preferred. Is that correct?

Assuming I use the second option, do I need to be concerned about it
initiating a backup on a host while that host is also performing a
local hourly snapshot?

What are the disadvantages of the  fileserver-initiated approach?

If one host is offline, will the backup procedure continue on with the
other hosts it can reach at that time?

Since deleting snapshots can potentially be a costly operation (in
terms of performance), should I split the process into two steps,
where one step would pull the backups from each host without any
deletions, and a second step would then prune the backups according to
configured retention policies?

How many backups (snapshots) can I safely retain for each host volume?
I would like to keep as many as possible, but I know there is a
threshold at which performance can become a problem.

I mount btrfs volumes on the **hosts** with these mount options:

    autodefrag,noatime,nodiratime,compress=lzo,space_cache=v2

And I have the systemd fstrim.service enabled.

The fileserver is a dedicated backup server, not a general-purpose
fileserver. I plan to use most of those same mount options. Do I need
the autodefrag option? Will autodefrag help or hurt performance in
this use-case? The following message from this list caused me some
confusion as I would have expected the opposite:

[freezes during snapshot creation/deletion -- to be expected? November
2019, 00:21:18 CET]

> So just to follow up on this, reducing the total number of snapshots and  increasing the time between their creation from hourly to once every six hours  did help a *little* bit.  However, about a week ago I decided to try an  experiment and added the "autodefrag" mount option (which I don't usually do  on SSDs), and that helped *massively*.  Ever since, snapper-cleanup.service  runs without me noticing at all!.

Are there any other recommendations?

Re: btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

[Joshua]

September 12, 2021 10:42 AM, "Dave T" <davestechshop@gmail.com> wrote:

> Are btrbk-specific questions OK here?
> 
> I have a small LAN with a fileserver that should store backups from
> each attached host on the LAN. What is the most efficient (performant)
> way to do this with btrbk?
> 
> Each host (laptops, desktops and a few other devices) does hourly
> local snapshots with btrbk. Once per day, I would like to send backups
> of each volume on each device to the local fileserver. This has to be
> done via SSH (as NFS isn't supported by btrfs send|receive, afaik).
> 
> The options I'm aware of from the btrbk readme
> (https://digint.ch/btrbk/doc/readme.html) are:
> 
> 1. host-initiated backup to the fileserver from each host
> 
> 2. fileserver-initiated backups from all hosts
> 
> My guess is that the second option is preferred. Is that correct?

I personally prefer it, yes.

I can manage all my retention in one place, and my backups are isolated. If a client is
compromised, the backups on the server cannot be deleted by an attacker, since my clients have no
access to the server, rather the server has access to the clients.

> Assuming I use the second option, do I need to be concerned about it
> initiating a backup on a host while that host is also performing a
> local hourly snapshot?

I don't think so.  Hopefully someone will correct me if so.

> What are the disadvantages of the fileserver-initiated approach?

If a client is offline, it will not be backed up at that time.

There's probably other disadvantages, but that's the main one I can think of.

> If one host is offline, will the backup procedure continue on with the
> other hosts it can reach at that time?

It should, but I don't know 100%

> Since deleting snapshots can potentially be a costly operation (in
> terms of performance), should I split the process into two steps,
> where one step would pull the backups from each host without any
> deletions, and a second step would then prune the backups according to
> configured retention policies?

If it's important that the backup process complete as soon as possible, perhaps this would be a
good idea.

If that's not important, I don't see why it would matter.

> How many backups (snapshots) can I safely retain for each host volume?
> I would like to keep as many as possible, but I know there is a
> threshold at which performance can become a problem.

I would think the limits would be relatively high, but I personally only run dailies for a week,
then weeklies for a month, then monthlies for a year.

> I mount btrfs volumes on the **hosts** with these mount options:
> 
> autodefrag,noatime,nodiratime,compress=lzo,space_cache=v2

Just FYI, noatime implies nodiratime. Source: https://lwn.net/Articles/245002

> And I have the systemd fstrim.service enabled.
> 
> The fileserver is a dedicated backup server, not a general-purpose
> fileserver. I plan to use most of those same mount options. Do I need
> the autodefrag option? Will autodefrag help or hurt performance in
> this use-case? The following message from this list caused me some
> confusion as I would have expected the opposite:

Sorry, I honestly don't know what impact this might have.
I personally run autodefrag on my clients, and not on my backup server.

> [freezes during snapshot creation/deletion -- to be expected? November
> 2019, 00:21:18 CET]
> 
>> So just to follow up on this, reducing the total number of snapshots and increasing the time
>> between their creation from hourly to once every six hours did help a *little* bit. However, about
>> a week ago I decided to try an experiment and added the "autodefrag" mount option (which I don't
>> usually do on SSDs), and that helped *massively*. Ever since, snapper-cleanup.service runs without
>> me noticing at all!.
> 
> Are there any other recommendations?

Re: btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

[Forza]

On 2021-09-14 03:49, Joshua wrote:
> September 12, 2021 10:42 AM, "Dave T" <davestechshop@gmail.com> wrote:
> 
>> Are btrbk-specific questions OK here?

There is also #btrbk on libera.chat

>>
>> I have a small LAN with a fileserver that should store backups from
>> each attached host on the LAN. What is the most efficient (performant)
>> way to do this with btrbk?
>>
>> Each host (laptops, desktops and a few other devices) does hourly
>> local snapshots with btrbk. Once per day, I would like to send backups
>> of each volume on each device to the local fileserver. This has to be
>> done via SSH (as NFS isn't supported by btrfs send|receive, afaik).
>>
>> The options I'm aware of from the btrbk readme
>> (https://digint.ch/btrbk/doc/readme.html) are:
>>
>> 1. host-initiated backup to the fileserver from each host
>>
>> 2. fileserver-initiated backups from all hosts
>>
>> My guess is that the second option is preferred. Is that correct?
> 
> I personally prefer it, yes.

It might also be easier to manage load on the server if you can backup 
each client serially instead of in parallel.

> 
> I can manage all my retention in one place, and my backups are isolated. If a client is
> compromised, the backups on the server cannot be deleted by an attacker, since my clients have no
> access to the server, rather the server has access to the clients.
> 
>> Assuming I use the second option, do I need to be concerned about it
>> initiating a backup on a host while that host is also performing a
>> local hourly snapshot?
> 
> I don't think so.  Hopefully someone will correct me if so.

No. I believe the only caveats are if you were running old versions of 
"bees" on the clients. Bees is an advanced deduplication tool.

> 
>> What are the disadvantages of the fileserver-initiated approach?
> 
> If a client is offline, it will not be backed up at that time.
> 
> There's probably other disadvantages, but that's the main one I can think of.
> 
>> If one host is offline, will the backup procedure continue on with the
>> other hosts it can reach at that time?
> 
> It should, but I don't know 100%
> 
>> Since deleting snapshots can potentially be a costly operation (in
>> terms of performance), should I split the process into two steps,
>> where one step would pull the backups from each host without any
>> deletions, and a second step would then prune the backups according to
>> configured retention policies?

As far as I know, Btrbk transfers all backups before attempting pruning. 
All sources would have to be in the same btrbk.conf though.

> 
> If it's important that the backup process complete as soon as possible, perhaps this would be a
> good idea.
> 
> If that's not important, I don't see why it would matter.
> 
>> How many backups (snapshots) can I safely retain for each host volume?
>> I would like to keep as many as possible, but I know there is a
>> threshold at which performance can become a problem.
> 
> I would think the limits would be relatively high, but I personally only run dailies for a week,
> then weeklies for a month, then monthlies for a year.
> 
>> I mount btrfs volumes on the **hosts** with these mount options:
>>
>> autodefrag,noatime,nodiratime,compress=lzo,space_cache=v2

autodefrag can break reflinks between snapshots which can lead to 
somewhat higher disk usage. In my experience it has been a little hit or 
miss if autodefrag helps. If it does help, it is great, otherwise just 
disable it.

These days I prefer zstd:1 over lzo for compression. Zstd allows for 
different compression efforts to be set. It generally has better 
compression ratios than LZO and is only a little slower, unless you use 
a high compression level. Default zstd level is zstd:3.

https://btrfs.wiki.kernel.org/index.php/Compression
https://wiki.tnonline.net/w/Btrfs/Compression

> 
> Just FYI, noatime implies nodiratime. Source: https://lwn.net/Articles/245002
> 
>> And I have the systemd fstrim.service enabled.
>>
>> The fileserver is a dedicated backup server, not a general-purpose
>> fileserver. I plan to use most of those same mount options. Do I need
>> the autodefrag option? Will autodefrag help or hurt performance in
>> this use-case? The following message from this list caused me some
>> confusion as I would have expected the opposite:
> 
> Sorry, I honestly don't know what impact this might have.
> I personally run autodefrag on my clients, and not on my backup server.
> 
>> [freezes during snapshot creation/deletion -- to be expected? November
>> 2019, 00:21:18 CET]
>>
>>> So just to follow up on this, reducing the total number of snapshots and increasing the time
>>> between their creation from hourly to once every six hours did help a *little* bit. However, about
>>> a week ago I decided to try an experiment and added the "autodefrag" mount option (which I don't
>>> usually do on SSDs), and that helped *massively*. Ever since, snapper-cleanup.service runs without
>>> me noticing at all!.
>>
>> Are there any other recommendations?

Is your backup server SSD only? If you add HDD's, avoid SMR drives. It 
also a good idea to mix different ages or models of drives to reduce the 
chance that multiple drives fails at the same time.

Don't use raid56 profiles at the moment.

Monitor your disk usage with "btrfs filesystem usage -T /mnt" to make 
sure you don't run out of space for metadata allocation.

Run manual fstrim on off-hours.

Do a full scrub every now and then.

Re: btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

[Graham Cobb]

On 12/09/2021 18:40, Dave T wrote:
> Are btrbk-specific questions OK here?
> 
> I have a small LAN with a fileserver that should store backups from
> each attached host on the LAN. What is the most efficient (performant)
> way to do this with btrbk?

My main goal is not performance but safety - but I realise there is
always a tradeoff to be made! And security and data protection also feed
into the analysis (ransomware, personal data, etc etc).

> Each host (laptops, desktops and a few other devices) does hourly
> local snapshots with btrbk. Once per day, I would like to send backups
> of each volume on each device to the local fileserver. This has to be
> done via SSH (as NFS isn't supported by btrfs send|receive, afaik).

That is similar to my setup. But in my case the server is always in control.

> The options I'm aware of from the btrbk readme
> (https://digint.ch/btrbk/doc/readme.html) are:
> 
> 1. host-initiated backup to the fileserver from each host
> 
> 2. fileserver-initiated backups from all hosts
> 
> My guess is that the second option is preferred. Is that correct?
> 
> Assuming I use the second option, do I need to be concerned about it
> initiating a backup on a host while that host is also performing a
> local hourly snapshot?

I use the second option, but I rely on btrbk on the server to take the
local snapshots on the hosts as well. In other words, btrbk software is
installed on the host but I never run it there explicitly. btrbk on the
server controls making both host and server snapshots.

> What are the disadvantages of the  fileserver-initiated approach?

Laptops, and other intermittently connected hosts, don't even get local
backups done unless they are connected at the time the server tries to
do them. Not a big problem for me.

> If one host is offline, will the backup procedure continue on with the
> other hosts it can reach at that time?

I run separate cron jobs (with separate btrbk conf files) for each host.

> Since deleting snapshots can potentially be a costly operation (in
> terms of performance), should I split the process into two steps,
> where one step would pull the backups from each host without any
> deletions, and a second step would then prune the backups according to
> configured retention policies?

I don't. I just let btrbk run through the process.

> How many backups (snapshots) can I safely retain for each host volume?
> I would like to keep as many as possible, but I know there is a
> threshold at which performance can become a problem.

On the server I use a separate btrfs filesystem for snapshots (a mixture
of btrbk snapshots and rsnapshot snapshots). It is currently 18TB (data
single, metadata raid1 on two spinning disks, with LUKS and LVM), of
which about 15TB is in use. It has about 1300 btrbk subvolumes on it
(and about 50 rsnapshot subvolumes). The btrbk jobs run (mostly at
night) using cron so I don't pay any attention to how long they take but
it isn't excessive. It doesn't seem to slow the system down or cause any
problems.

The only problem is that check (run monthly) takes a few days! I just
let it run in the background.

I don't keep many snapshots on the hosts - they take up disk space and
can cause unnecessary issues. Keep the main snapshots on the server,
with just a small number of recent ones on the host for easy access when
someone deletes the wrong file by mistake. For laptops you need to trade
off keeping more so older data can be accessed when on the move or fewer
so that deleted files don't hang around if the laptop is lost.

> 
> I mount btrfs volumes on the **hosts** with these mount options:
> 
>     autodefrag,noatime,nodiratime,compress=lzo,space_cache=v2

On the hosts I use nothing special. For example:

    noatime,nodiratime,nossd

On the server, I use:


noatime,nodiratime,compress-force=zstd:15,skip_balance,commit=15,space_cache=v2,x-systemd.mount-timeout=180s,nofail

> 
> And I have the systemd fstrim.service enabled.
> 
> The fileserver is a dedicated backup server, not a general-purpose
> fileserver. I plan to use most of those same mount options. Do I need
> the autodefrag option? Will autodefrag help or hurt performance in
> this use-case? The following message from this list caused me some
> confusion as I would have expected the opposite:
> 
> [freezes during snapshot creation/deletion -- to be expected? November
> 2019, 00:21:18 CET]

I don't use autodefrag or any other defrag. As these are backups I don't
see any need to improve read access, and I prefer to avoid the concern
over defrag breaking something.

>> So just to follow up on this, reducing the total number of snapshots and  increasing the time between their creation from hourly to once every six hours  did help a *little* bit.  However, about a week ago I decided to try an  experiment and added the "autodefrag" mount option (which I don't usually do  on SSDs), and that helped *massively*.  Ever since, snapper-cleanup.service  runs without me noticing at all!.
> 
> Are there any other recommendations?

Don't use snapshots as your only backup. They are great for easy access
and for being bang up to date but I maintain more traditional backups as
well (using DAR, daily in my case, and encrypted and sent to a cloud
service). This is mainly in case some bug or disk problem caused me to
lose the btrfs filesystem structure of the snapshots filesystem, but it
also provides protection against a fire or similar.

Graham

P.S. Just fyi, here is an example of one of my btrbk conf files (for an
intermittently connected laptop in this particular case, others are more
complex with multiple subvolumes but they are all similar):

volume <REDACTED>:/mnt/rootdisk
 ssh_identity /root/.ssh/<REDACTED>
 snapshot_dir btrbk_snapshots
 snapshot_create onchange
 preserve_day_of_week monday

# On the disk itself only keep recent snapshots
 snapshot_preserve_min  5d
 snapshot_preserve 5d 2w
 timestamp_format long-iso

# On the backup disk keep historic monthlies
 target_preserve_min no
 target_preserve 30d 8w *m

subvolume rootfs
 target send-receive    /snapshots/<REDACTED>_snapshots

Re: btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

[Dave T]

On Tue, Sep 14, 2021 at 1:11 AM Forza <forza@tnonline.net> wrote:
>
>
>
> On 2021-09-14 03:49, Joshua wrote:
> > September 12, 2021 10:42 AM, "Dave T" <davestechshop@gmail.com> wrote:
> >
> >> Are btrbk-specific questions OK here?
>
> There is also #btrbk on libera.chat
>
> >>
> >> I have a small LAN with a fileserver that should store backups from
> >> each attached host on the LAN. What is the most efficient (performant)
> >> way to do this with btrbk?
> >>
> >> Each host (laptops, desktops and a few other devices) does hourly
> >> local snapshots with btrbk. Once per day, I would like to send backups
> >> of each volume on each device to the local fileserver. This has to be
> >> done via SSH (as NFS isn't supported by btrfs send|receive, afaik).
> >>
> >> The options I'm aware of from the btrbk readme
> >> (https://digint.ch/btrbk/doc/readme.html) are:
> >>
> >> 1. host-initiated backup to the fileserver from each host
> >>
> >> 2. fileserver-initiated backups from all hosts
> >>
> >> My guess is that the second option is preferred. Is that correct?
> >
> > I personally prefer it, yes.
>
> It might also be easier to manage load on the server if you can backup
> each client serially instead of in parallel.
>
> >
> > I can manage all my retention in one place, and my backups are isolated. If a client is
> > compromised, the backups on the server cannot be deleted by an attacker, since my clients have no
> > access to the server, rather the server has access to the clients.
> >
> >> Assuming I use the second option, do I need to be concerned about it
> >> initiating a backup on a host while that host is also performing a
> >> local hourly snapshot?
> >
> > I don't think so.  Hopefully someone will correct me if so.
>
> No. I believe the only caveats are if you were running old versions of
> "bees" on the clients. Bees is an advanced deduplication tool.
>
> >
> >> What are the disadvantages of the fileserver-initiated approach?
> >
> > If a client is offline, it will not be backed up at that time.
> >
> > There's probably other disadvantages, but that's the main one I can think of.
> >
> >> If one host is offline, will the backup procedure continue on with the
> >> other hosts it can reach at that time?
> >
> > It should, but I don't know 100%
> >
> >> Since deleting snapshots can potentially be a costly operation (in
> >> terms of performance), should I split the process into two steps,
> >> where one step would pull the backups from each host without any
> >> deletions, and a second step would then prune the backups according to
> >> configured retention policies?
>
> As far as I know, Btrbk transfers all backups before attempting pruning.
> All sources would have to be in the same btrbk.conf though.
>
> >
> > If it's important that the backup process complete as soon as possible, perhaps this would be a
> > good idea.
> >
> > If that's not important, I don't see why it would matter.
> >
> >> How many backups (snapshots) can I safely retain for each host volume?
> >> I would like to keep as many as possible, but I know there is a
> >> threshold at which performance can become a problem.
> >
> > I would think the limits would be relatively high, but I personally only run dailies for a week,
> > then weeklies for a month, then monthlies for a year.
> >
> >> I mount btrfs volumes on the **hosts** with these mount options:
> >>
> >> autodefrag,noatime,nodiratime,compress=lzo,space_cache=v2
>
> autodefrag can break reflinks between snapshots which can lead to
> somewhat higher disk usage. In my experience it has been a little hit or
> miss if autodefrag helps. If it does help, it is great, otherwise just
> disable it.

I will remove autodefrag.

>
> These days I prefer zstd:1 over lzo for compression. Zstd allows for
> different compression efforts to be set. It generally has better
> compression ratios than LZO and is only a little slower, unless you use
> a high compression level. Default zstd level is zstd:3.
>
> https://btrfs.wiki.kernel.org/index.php/Compression
> https://wiki.tnonline.net/w/Btrfs/Compression

I will probably switch to zstd too, but not until I have the entire
backup implementation working the way I expect it to work.

>
> >
> > Just FYI, noatime implies nodiratime. Source: https://lwn.net/Articles/245002
> >
> >> And I have the systemd fstrim.service enabled.
> >>
> >> The fileserver is a dedicated backup server, not a general-purpose
> >> fileserver. I plan to use most of those same mount options. Do I need
> >> the autodefrag option? Will autodefrag help or hurt performance in
> >> this use-case? The following message from this list caused me some
> >> confusion as I would have expected the opposite:
> >
> > Sorry, I honestly don't know what impact this might have.
> > I personally run autodefrag on my clients, and not on my backup server.
> >
> >> [freezes during snapshot creation/deletion -- to be expected? November
> >> 2019, 00:21:18 CET]
> >>
> >>> So just to follow up on this, reducing the total number of snapshots and increasing the time
> >>> between their creation from hourly to once every six hours did help a *little* bit. However, about
> >>> a week ago I decided to try an experiment and added the "autodefrag" mount option (which I don't
> >>> usually do on SSDs), and that helped *massively*. Ever since, snapper-cleanup.service runs without
> >>> me noticing at all!.
> >>
> >> Are there any other recommendations?
>
> Is your backup server SSD only?

No, my backup server uses only hard disks.

> If you add HDD's, avoid SMR drives.

Thanks for the tip. I had to look up "SMR". I do have a couple Seagate
"archive" hard drives in some other systems, but fortunately I do not
have any of that type in this server.


> It also a good idea to mix different ages or models of drives to reduce the
> chance that multiple drives fails at the same time.
>
> Don't use raid56 profiles at the moment.
>
> Monitor your disk usage with "btrfs filesystem usage -T /mnt" to make
> sure you don't run out of space for metadata allocation.
>
> Run manual fstrim on off-hours.
>
> Do a full scrub every now and then.
>

Re: btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

[Dave T]

On Tue, Sep 14, 2021 at 6:00 AM Graham Cobb <g.btrfs@cobb.uk.net> wrote:
>
> On 12/09/2021 18:40, Dave T wrote:
> > Are btrbk-specific questions OK here?
> >
> > I have a small LAN with a fileserver that should store backups from
> > each attached host on the LAN. What is the most efficient (performant)
> > way to do this with btrbk?
>
> My main goal is not performance but safety - but I realise there is
> always a tradeoff to be made! And security and data protection also feed
> into the analysis (ransomware, personal data, etc etc).
>
> > Each host (laptops, desktops and a few other devices) does hourly
> > local snapshots with btrbk. Once per day, I would like to send backups
> > of each volume on each device to the local fileserver. This has to be
> > done via SSH (as NFS isn't supported by btrfs send|receive, afaik).
>
> That is similar to my setup. But in my case the server is always in control.
>
> > The options I'm aware of from the btrbk readme
> > (https://digint.ch/btrbk/doc/readme.html) are:
> >
> > 1. host-initiated backup to the fileserver from each host
> >
> > 2. fileserver-initiated backups from all hosts
> >
> > My guess is that the second option is preferred. Is that correct?
> >
> > Assuming I use the second option, do I need to be concerned about it
> > initiating a backup on a host while that host is also performing a
> > local hourly snapshot?
>
> I use the second option, but I rely on btrbk on the server to take the
> local snapshots on the hosts as well. In other words, btrbk software is
> installed on the host but I never run it there explicitly. btrbk on the
> server controls making both host and server snapshots.
>
> > What are the disadvantages of the  fileserver-initiated approach?
>
> Laptops, and other intermittently connected hosts, don't even get local
> backups done unless they are connected at the time the server tries to
> do them. Not a big problem for me.
>
> > If one host is offline, will the backup procedure continue on with the
> > other hosts it can reach at that time?
>
> I run separate cron jobs (with separate btrbk conf files) for each host.

That's a very interesting approach. How many hosts do you have?

>
> > Since deleting snapshots can potentially be a costly operation (in
> > terms of performance), should I split the process into two steps,
> > where one step would pull the backups from each host without any
> > deletions, and a second step would then prune the backups according to
> > configured retention policies?
>
> I don't. I just let btrbk run through the process.

I will try it that way. I think I will try to keep my configuration as
simple as possible, while still accomplishing my goals.

>
> > How many backups (snapshots) can I safely retain for each host volume?
> > I would like to keep as many as possible, but I know there is a
> > threshold at which performance can become a problem.
>
> On the server I use a separate btrfs filesystem for snapshots (a mixture
> of btrbk snapshots and rsnapshot snapshots). It is currently 18TB (data
> single, metadata raid1 on two spinning disks, with LUKS and LVM), of
> which about 15TB is in use. It has about 1300 btrbk subvolumes on it
> (and about 50 rsnapshot subvolumes). The btrbk jobs run (mostly at
> night) using cron so I don't pay any attention to how long they take but
> it isn't excessive. It doesn't seem to slow the system down or cause any
> problems.
>
> The only problem is that check (run monthly) takes a few days! I just
> let it run in the background.

Do you run btrfs-check on the mounted or unmounted filesystems? What
check options do you use?

>
> I don't keep many snapshots on the hosts - they take up disk space and
> can cause unnecessary issues. Keep the main snapshots on the server,
> with just a small number of recent ones on the host for easy access when
> someone deletes the wrong file by mistake. For laptops you need to trade
> off keeping more so older data can be accessed when on the move or fewer
> so that deleted files don't hang around if the laptop is lost.
>
> >
> > I mount btrfs volumes on the **hosts** with these mount options:
> >
> >     autodefrag,noatime,nodiratime,compress=lzo,space_cache=v2
>
> On the hosts I use nothing special. For example:
>
>     noatime,nodiratime,nossd
>
> On the server, I use:
>
>
> noatime,nodiratime,compress-force=zstd:15,skip_balance,commit=15,space_cache=v2,x-systemd.mount-timeout=180s,nofail


Why do you use the skip_balance mount option? I have never had any
problem related to what this option seems intended to do. I'm curious
if you use it due to encountering some problem that it solves for you.

Also, I can't find the documentation for the commit=15 mount option.
I'm curious to know about it. Why do you use it?

>
> >
> > And I have the systemd fstrim.service enabled.
> >
> > The fileserver is a dedicated backup server, not a general-purpose
> > fileserver. I plan to use most of those same mount options. Do I need
> > the autodefrag option? Will autodefrag help or hurt performance in
> > this use-case? The following message from this list caused me some
> > confusion as I would have expected the opposite:
> >
> > [freezes during snapshot creation/deletion -- to be expected? November
> > 2019, 00:21:18 CET]
>
> I don't use autodefrag or any other defrag. As these are backups I don't
> see any need to improve read access, and I prefer to avoid the concern
> over defrag breaking something.

That makes sense.

>
> >> So just to follow up on this, reducing the total number of snapshots and  increasing the time between their creation from hourly to once every six hours  did help a *little* bit.  However, about a week ago I decided to try an  experiment and added the "autodefrag" mount option (which I don't usually do  on SSDs), and that helped *massively*.  Ever since, snapper-cleanup.service  runs without me noticing at all!.
> >
> > Are there any other recommendations?
>
> Don't use snapshots as your only backup. They are great for easy access
> and for being bang up to date but I maintain more traditional backups as
> well (using DAR, daily in my case, and encrypted and sent to a cloud
> service). This is mainly in case some bug or disk problem caused me to
> lose the btrfs filesystem structure of the snapshots filesystem, but it
> also provides protection against a fire or similar.
>
> Graham
>
> P.S. Just fyi, here is an example of one of my btrbk conf files

Thank you for sharing this.


> (for an intermittently connected laptop in this particular case, others are more
> complex with multiple subvolumes but they are all similar):
>
> volume <REDACTED>:/mnt/rootdisk
>  ssh_identity /root/.ssh/<REDACTED>
>  snapshot_dir btrbk_snapshots
>  snapshot_create onchange
>  preserve_day_of_week monday
>
> # On the disk itself only keep recent snapshots
>  snapshot_preserve_min  5d
>  snapshot_preserve 5d 2w
>  timestamp_format long-iso
>
> # On the backup disk keep historic monthlies
>  target_preserve_min no
>  target_preserve 30d 8w *m
>
> subvolume rootfs
>  target send-receive    /snapshots/<REDACTED>_snapshots
>
>
>

Re: btrbk question: Should I Prefer Fileserver-initiated Backups from Several Hosts (Instead of Each Host Sending to the Server)?

[Graham Cobb]

On 14/09/2021 17:17, Dave T wrote:
> On Tue, Sep 14, 2021 at 6:00 AM Graham Cobb <g.btrfs@cobb.uk.net> wrote:
>>
>> On 12/09/2021 18:40, Dave T wrote:
>>> Are btrbk-specific questions OK here?
>>>
>>> I have a small LAN with a fileserver that should store backups from
>>> each attached host on the LAN. What is the most efficient (performant)
>>> way to do this with btrbk?
>>
>> My main goal is not performance but safety - but I realise there is
>> always a tradeoff to be made! And security and data protection also feed
>> into the analysis (ransomware, personal data, etc etc).
>>
>>> Each host (laptops, desktops and a few other devices) does hourly
>>> local snapshots with btrbk. Once per day, I would like to send backups
>>> of each volume on each device to the local fileserver. This has to be
>>> done via SSH (as NFS isn't supported by btrfs send|receive, afaik).
>>
>> That is similar to my setup. But in my case the server is always in control.
>>
>>> The options I'm aware of from the btrbk readme
>>> (https://digint.ch/btrbk/doc/readme.html) are:
>>>
>>> 1. host-initiated backup to the fileserver from each host
>>>
>>> 2. fileserver-initiated backups from all hosts
>>>
>>> My guess is that the second option is preferred. Is that correct?
>>>
>>> Assuming I use the second option, do I need to be concerned about it
>>> initiating a backup on a host while that host is also performing a
>>> local hourly snapshot?
>>
>> I use the second option, but I rely on btrbk on the server to take the
>> local snapshots on the hosts as well. In other words, btrbk software is
>> installed on the host but I never run it there explicitly. btrbk on the
>> server controls making both host and server snapshots.
>>
>>> What are the disadvantages of the  fileserver-initiated approach?
>>
>> Laptops, and other intermittently connected hosts, don't even get local
>> backups done unless they are connected at the time the server tries to
>> do them. Not a big problem for me.
>>
>>> If one host is offline, will the backup procedure continue on with the
>>> other hosts it can reach at that time?
>>
>> I run separate cron jobs (with separate btrbk conf files) for each host.
> 
> That's a very interesting approach. How many hosts do you have?

It varies a bit as some are intermittently connected but there are
currently about 5.

>>> Since deleting snapshots can potentially be a costly operation (in
>>> terms of performance), should I split the process into two steps,
>>> where one step would pull the backups from each host without any
>>> deletions, and a second step would then prune the backups according to
>>> configured retention policies?
>>
>> I don't. I just let btrbk run through the process.
> 
> I will try it that way. I think I will try to keep my configuration as
> simple as possible, while still accomplishing my goals.
> 
>>
>>> How many backups (snapshots) can I safely retain for each host volume?
>>> I would like to keep as many as possible, but I know there is a
>>> threshold at which performance can become a problem.
>>
>> On the server I use a separate btrfs filesystem for snapshots (a mixture
>> of btrbk snapshots and rsnapshot snapshots). It is currently 18TB (data
>> single, metadata raid1 on two spinning disks, with LUKS and LVM), of
>> which about 15TB is in use. It has about 1300 btrbk subvolumes on it
>> (and about 50 rsnapshot subvolumes). The btrbk jobs run (mostly at
>> night) using cron so I don't pay any attention to how long they take but
>> it isn't excessive. It doesn't seem to slow the system down or cause any
>> problems.
>>
>> The only problem is that check (run monthly) takes a few days! I just
>> let it run in the background.
> 
> Do you run btrfs-check on the mounted or unmounted filesystems? What
> check options do you use?

DOH! I don't know why I wrote "check"! I meant "scrub". Must be losing it...

> 
>>
>> I don't keep many snapshots on the hosts - they take up disk space and
>> can cause unnecessary issues. Keep the main snapshots on the server,
>> with just a small number of recent ones on the host for easy access when
>> someone deletes the wrong file by mistake. For laptops you need to trade
>> off keeping more so older data can be accessed when on the move or fewer
>> so that deleted files don't hang around if the laptop is lost.
>>
>>>
>>> I mount btrfs volumes on the **hosts** with these mount options:
>>>
>>>     autodefrag,noatime,nodiratime,compress=lzo,space_cache=v2
>>
>> On the hosts I use nothing special. For example:
>>
>>     noatime,nodiratime,nossd
>>
>> On the server, I use:
>>
>>
>> noatime,nodiratime,compress-force=zstd:15,skip_balance,commit=15,space_cache=v2,x-systemd.mount-timeout=180s,nofail
> 
> 
> Why do you use the skip_balance mount option? I have never had any
> problem related to what this option seems intended to do. I'm curious
> if you use it due to encountering some problem that it solves for you.

I have been using btrfs for a long time (I first used it in the Meego
days IIRC, then with Mer & Sailfish). Back then balance sometimes got
very confused and did some nasty things so it was important to make sure
you could kill a balance by rebooting! I have never turned skip_balance
off but I don't think I have needed it for a long time.

> 
> Also, I can't find the documentation for the commit=15 mount option.
> I'm curious to know about it. Why do you use it?

It forces btrfs to commit the log every 15 seconds. I use it because
power is not massively reliable here so I wanted to reduce lost
transactions in a power failure. Note: btrfs is good at preserving
filesystem integrity on a power failure but the filesystem will revert
to the state at the last completed transaction.

Graham

> 
>>
>>>
>>> And I have the systemd fstrim.service enabled.
>>>
>>> The fileserver is a dedicated backup server, not a general-purpose
>>> fileserver. I plan to use most of those same mount options. Do I need
>>> the autodefrag option? Will autodefrag help or hurt performance in
>>> this use-case? The following message from this list caused me some
>>> confusion as I would have expected the opposite:
>>>
>>> [freezes during snapshot creation/deletion -- to be expected? November
>>> 2019, 00:21:18 CET]
>>
>> I don't use autodefrag or any other defrag. As these are backups I don't
>> see any need to improve read access, and I prefer to avoid the concern
>> over defrag breaking something.
> 
> That makes sense.
> 
>>
>>>> So just to follow up on this, reducing the total number of snapshots and  increasing the time between their creation from hourly to once every six hours  did help a *little* bit.  However, about a week ago I decided to try an  experiment and added the "autodefrag" mount option (which I don't usually do  on SSDs), and that helped *massively*.  Ever since, snapper-cleanup.service  runs without me noticing at all!.
>>>
>>> Are there any other recommendations?
>>
>> Don't use snapshots as your only backup. They are great for easy access
>> and for being bang up to date but I maintain more traditional backups as
>> well (using DAR, daily in my case, and encrypted and sent to a cloud
>> service). This is mainly in case some bug or disk problem caused me to
>> lose the btrfs filesystem structure of the snapshots filesystem, but it
>> also provides protection against a fire or similar.
>>
>> Graham
>>
>> P.S. Just fyi, here is an example of one of my btrbk conf files
> 
> Thank you for sharing this.
> 
> 
>> (for an intermittently connected laptop in this particular case, others are more
>> complex with multiple subvolumes but they are all similar):
>>
>> volume <REDACTED>:/mnt/rootdisk
>>  ssh_identity /root/.ssh/<REDACTED>
>>  snapshot_dir btrbk_snapshots
>>  snapshot_create onchange
>>  preserve_day_of_week monday
>>
>> # On the disk itself only keep recent snapshots
>>  snapshot_preserve_min  5d
>>  snapshot_preserve 5d 2w
>>  timestamp_format long-iso
>>
>> # On the backup disk keep historic monthlies
>>  target_preserve_min no
>>  target_preserve 30d 8w *m
>>
>> subvolume rootfs
>>  target send-receive    /snapshots/<REDACTED>_snapshots
>>
>>
>>