Anyone tried out btrbk yet?

Anyone tried out btrbk yet?

[Martin Steigerwald]

Hi!

I see Alex, the developer of btrbk posted here once about btrfs send and 
receive, but well any other users of btrbk¹? What are your experiences?

I consider switching to it from my home grown rsync based backup script to 
it.

Well I may try it for one of my BTRFS volumes in addition to the rsync 
backup for now. I would like to give all options on command line, but well, 
maybe it can completely replace my current script if I put everything in its 
configuration.

Any other handy BTRFS backup solutions?

[1] http://www.digint.ch/btrbk/

Thanks,
-- 
Martin

Re: Anyone tried out btrbk yet?

[Henri Valta]

On Thursday 09 July 2015 14:26:55 you wrote:

> Well I may try it for one of my BTRFS volumes in addition to the rsync
> backup for now. I would like to give all options on command line, but well,
> maybe it can completely replace my current script if I put everything in its
> configuration.
> 
> Any other handy BTRFS backup solutions?

Hi,

I've been using btrfs-sxbackup for a couple of weeks, and it has been working 
great. Everything is configured on command line, so that's a plus.

https://pypi.python.org/pypi/btrfs-sxbackup
https://github.com/masc3d/btrfs-sxbackup

-Henri

Re: Anyone tried out btrbk yet?

[Marc MERLIN]

On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote:
> Hi!
> 
> I see Alex, the developer of btrbk posted here once about btrfs send and 
> receive, but well any other users of btrbk¹? What are your experiences?
> 
> I consider switching to it from my home grown rsync based backup script to 
> it.
> 
> Well I may try it for one of my BTRFS volumes in addition to the rsync 
> backup for now. I would like to give all options on command line, but well, 
> maybe it can completely replace my current script if I put everything in its 
> configuration.
> 
> Any other handy BTRFS backup solutions?

I use my own which I wrote :)
http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive
http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Anyone tried out btrbk yet?

[Donald Pearson]

Marc,

I thought I'd yours a try, and I'm probably embarassing myself here
but I'm running in to this issue.  Centos 7.

[root@san01 tank]# ./btrfs-subvolume-backup store /mnt2/backups
./btrfs-subvolume-backup: line 177: shlock: command not found
/var/run/btrfs-subvolume-backup held for btrfs-subvolume-backup, quitting
[root@san01 tank]# yum whatprovides shlock
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base: dist1.800hosting.com
 * elrepo: repos.dfw.lax-noc.com
 * epel: mirror.umd.edu
 * extras: mirrors.usc.edu
 * updates: mirror.keystealth.orgNo matches found
[root@san01 tank]# shlock
-bash: shlock: command not found
[root@san01 tank]# yum search all shlock
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base: dist1.800hosting.com
 * elrepo: repos.dfw.lax-noc.com
 * epel: mirror.utexas.edu
 * extras: mirror.thelinuxfix.com
 * updates: dallas.tx.mirror.xygenhosting.com
Warning: No matches found for: shlock
No matches found

On Thu, Jul 9, 2015 at 12:17 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote:
>> Hi!
>>
>> I see Alex, the developer of btrbk posted here once about btrfs send and
>> receive, but well any other users of btrbk¹? What are your experiences?
>>
>> I consider switching to it from my home grown rsync based backup script to
>> it.
>>
>> Well I may try it for one of my BTRFS volumes in addition to the rsync
>> backup for now. I would like to give all options on command line, but well,
>> maybe it can completely replace my current script if I put everything in its
>> configuration.
>>
>> Any other handy BTRFS backup solutions?
>
> I use my own which I wrote :)
> http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive
> http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Donald Pearson]

... and I just found your other block about stealing shlock out of inn.

Officially embarassed!

On Thu, Jul 9, 2015 at 8:35 PM, Donald Pearson
<donaldwhpearson@gmail.com> wrote:
> Marc,
>
> I thought I'd yours a try, and I'm probably embarassing myself here
> but I'm running in to this issue.  Centos 7.
>
> [root@san01 tank]# ./btrfs-subvolume-backup store /mnt2/backups
> ./btrfs-subvolume-backup: line 177: shlock: command not found
> /var/run/btrfs-subvolume-backup held for btrfs-subvolume-backup, quitting
> [root@san01 tank]# yum whatprovides shlock
> Loaded plugins: changelog, fastestmirror
> Loading mirror speeds from cached hostfile
>  * base: dist1.800hosting.com
>  * elrepo: repos.dfw.lax-noc.com
>  * epel: mirror.umd.edu
>  * extras: mirrors.usc.edu
>  * updates: mirror.keystealth.orgNo matches found
> [root@san01 tank]# shlock
> -bash: shlock: command not found
> [root@san01 tank]# yum search all shlock
> Loaded plugins: changelog, fastestmirror
> Loading mirror speeds from cached hostfile
>  * base: dist1.800hosting.com
>  * elrepo: repos.dfw.lax-noc.com
>  * epel: mirror.utexas.edu
>  * extras: mirror.thelinuxfix.com
>  * updates: dallas.tx.mirror.xygenhosting.com
> Warning: No matches found for: shlock
> No matches found
>
> On Thu, Jul 9, 2015 at 12:17 PM, Marc MERLIN <marc@merlins.org> wrote:
>> On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote:
>>> Hi!
>>>
>>> I see Alex, the developer of btrbk posted here once about btrfs send and
>>> receive, but well any other users of btrbk¹? What are your experiences?
>>>
>>> I consider switching to it from my home grown rsync based backup script to
>>> it.
>>>
>>> Well I may try it for one of my BTRFS volumes in addition to the rsync
>>> backup for now. I would like to give all options on command line, but well,
>>> maybe it can completely replace my current script if I put everything in its
>>> configuration.
>>>
>>> Any other handy BTRFS backup solutions?
>>
>> I use my own which I wrote :)
>> http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive
>> http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup
>>
>> Marc
>> --
>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> Microsoft is to operating systems ....
>>                                       .... what McDonalds is to gourmet cooking
>> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Paul Harvey]

In my research, I've found btrbk and btrfs-sxbackup certainly to be
the leading contenders in terms of feature completeness. sanoid [1]
will be another interesting possibility once btrfs compatibility is
added (currently zfs only).

I just wish I'd discovered all these before I went to all the effort
of creating snazzer [1] :)

I've been meaning to split some stuff out of snazzer that might be
generically useful to other folks, such as filesystem cloning of all
subvols/snapshots via send/receive, and it seems as if it should be
possible to automatically prune any idiosyncratic snapshot naming
convention - I just haven't found the time to write unit tests.

[1] https://github.com/jimsalterjrs/sanoid
[2] https://github.com/csirac2/snazzer

On 10 July 2015 at 11:38, Donald Pearson <donaldwhpearson@gmail.com> wrote:
> ... and I just found your other block about stealing shlock out of inn.
>
> Officially embarassed!
>
> On Thu, Jul 9, 2015 at 8:35 PM, Donald Pearson
> <donaldwhpearson@gmail.com> wrote:
>> Marc,
>>
>> I thought I'd yours a try, and I'm probably embarassing myself here
>> but I'm running in to this issue.  Centos 7.
>>
>> [root@san01 tank]# ./btrfs-subvolume-backup store /mnt2/backups
>> ./btrfs-subvolume-backup: line 177: shlock: command not found
>> /var/run/btrfs-subvolume-backup held for btrfs-subvolume-backup, quitting
>> [root@san01 tank]# yum whatprovides shlock
>> Loaded plugins: changelog, fastestmirror
>> Loading mirror speeds from cached hostfile
>>  * base: dist1.800hosting.com
>>  * elrepo: repos.dfw.lax-noc.com
>>  * epel: mirror.umd.edu
>>  * extras: mirrors.usc.edu
>>  * updates: mirror.keystealth.orgNo matches found
>> [root@san01 tank]# shlock
>> -bash: shlock: command not found
>> [root@san01 tank]# yum search all shlock
>> Loaded plugins: changelog, fastestmirror
>> Loading mirror speeds from cached hostfile
>>  * base: dist1.800hosting.com
>>  * elrepo: repos.dfw.lax-noc.com
>>  * epel: mirror.utexas.edu
>>  * extras: mirror.thelinuxfix.com
>>  * updates: dallas.tx.mirror.xygenhosting.com
>> Warning: No matches found for: shlock
>> No matches found
>>
>> On Thu, Jul 9, 2015 at 12:17 PM, Marc MERLIN <marc@merlins.org> wrote:
>>> On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote:
>>>> Hi!
>>>>
>>>> I see Alex, the developer of btrbk posted here once about btrfs send and
>>>> receive, but well any other users of btrbk¹? What are your experiences?
>>>>
>>>> I consider switching to it from my home grown rsync based backup script to
>>>> it.
>>>>
>>>> Well I may try it for one of my BTRFS volumes in addition to the rsync
>>>> backup for now. I would like to give all options on command line, but well,
>>>> maybe it can completely replace my current script if I put everything in its
>>>> configuration.
>>>>
>>>> Any other handy BTRFS backup solutions?
>>>
>>> I use my own which I wrote :)
>>> http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive
>>> http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup
>>>
>>> Marc
>>> --
>>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>>> Microsoft is to operating systems ....
>>>                                       .... what McDonalds is to gourmet cooking
>>> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Axel Burri]

On 2015-07-09 14:26, Martin Steigerwald wrote:

> Well I may try it for one of my BTRFS volumes in addition to the rsync 
> backup for now. I would like to give all options on command line, but well, 
> maybe it can completely replace my current script if I put everything in its 
> configuration.

One reason why btrbk exists is that I wanted to have all my backups
configured in a config file. I use btrbk to backup several hosts to
several backup locations (around 20 subvolumes), which made setting
everything up with command-line options very cumbersome.

For simpler setups you might be better off with command-line based tools
(e.g. Marcs "btrfs-subvolume-backup", which I like for it's smallness).

Give it a try, there is a "dryrun" command in btrbk :)
Ask on github if you need help.


- Axel

Re: Anyone tried out btrbk yet?

[Marc MERLIN]

On Fri, Jul 10, 2015 at 12:46:24PM +0200, Axel Burri wrote:
> On 2015-07-09 14:26, Martin Steigerwald wrote:
> 
> > Well I may try it for one of my BTRFS volumes in addition to the rsync 
> > backup for now. I would like to give all options on command line, but well, 
> > maybe it can completely replace my current script if I put everything in its 
> > configuration.
> 
> One reason why btrbk exists is that I wanted to have all my backups
> configured in a config file. I use btrbk to backup several hosts to
> several backup locations (around 20 subvolumes), which made setting
> everything up with command-line options very cumbersome.
> 
> For simpler setups you might be better off with command-line based tools
> (e.g. Marcs "btrfs-subvolume-backup", which I like for it's smallness).

I just had another look at btrbk, and it's obviously a lot more
featureful. It's almost 10 times bigger in lines of code than
btrfs-subvolume-backup :)

Anyway, to others, if you're happy using a tool that just does that you
need without having to dig into it and without you caring how btrfs
send/receive, works, btrbk looks like the better choice.

If you'd like something short-ish to look at and see how it works and/or
want something simple in shell you can modify quickly, btrfs-subvolume-backup
might be better for you.

Hope that helps others.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Anyone tried out btrbk yet?

[Marc MERLIN]

On Sat, Jul 11, 2015 at 08:55:23PM -0700, Marc MERLIN wrote:
> On Fri, Jul 10, 2015 at 12:46:24PM +0200, Axel Burri wrote:
> > On 2015-07-09 14:26, Martin Steigerwald wrote:
> > 
> > > Well I may try it for one of my BTRFS volumes in addition to the rsync 
> > > backup for now. I would like to give all options on command line, but well, 
> > > maybe it can completely replace my current script if I put everything in its 
> > > configuration.
> > 
> > One reason why btrbk exists is that I wanted to have all my backups
> > configured in a config file. I use btrbk to backup several hosts to
> > several backup locations (around 20 subvolumes), which made setting
> > everything up with command-line options very cumbersome.
> > 
> > For simpler setups you might be better off with command-line based tools
> > (e.g. Marcs "btrfs-subvolume-backup", which I like for it's smallness).
> 
> I just had another look at btrbk, and it's obviously a lot more
> featureful. It's almost 10 times bigger in lines of code than
> btrfs-subvolume-backup :)
> 
> Anyway, to others, if you're happy using a tool that just does that you
> need without having to dig into it and without you caring how btrfs
> send/receive, works, btrbk looks like the better choice.
> 
> If you'd like something short-ish to look at and see how it works and/or
> want something simple in shell you can modify quickly, btrfs-subvolume-backup
> might be better for you.

Actually there is one thing my backup script doesn't do and it looks
like btrbk doesn't do either.
When I travel I'm often on crappy hotel wireless and my incremental
backups never succeed, and they just get bigger every day, so succeed
even less next time.
At the same time, I have an issue with btrfs where my backup server
cause a backup over ssh on a local network to take 12H or more when the
data to transfer isn't that much.
>From what I can tell it's a problem where btrfs receive just pauses for
one second or more, and kills the TCP flow.
TCP restarts slowly and ramps up just to be stopped again. As a result,
it takes forever to finish.

Since I don't know if this btrfs stall problem will get looked into or
fixed anytime soon, it would be great for the incremental backup to go
to a local spool directory, and then for the backup script to just copy
it with rsync until it makes it to the other side. Only then is btrfs
receive run with that file, and then the next incremental backup is
tried.

But this is a fair amount of logic to get this right, and I just haven't
taken the time to write it.

Would it be helpful to others, or am I the only one with this problem?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Anyone tried out btrbk yet?

[Paul Harvey]

The way it works in snazzer (and btrbk and I think also btrfs-sxbackup
as well), local snapshots continue to happen as normal (Eg. daily or
hourly) and so when your backup media or backup server is finally
available again, the size of each individual incremental is still the
same as usual, it just has to perform more of them.

Separating snapshotting from transport lends to more flexibility IMHO,
Eg. with snazzer I can keep multiple physical backup media in sync
with each other even if I only rotate/attach those disks once a
week/month (maintain backup filesystems in parallel), the
snazzer-receive script is very dumb - it just receives all the missing
snapshots from the source. However it does filter them cf. "btrfs
subvolume list /subvolume | snazzer-prune-candidates --invert" first
in case some would just be deleted again shortly after according to
retention policy.

For the ssh transport, you can do the same things but in series: push
the snapshots up to a local server and then on to remote storage
elsewhere (maintain backup filesystems in series).

Because the snapshotting, transport and pruning operations are
asynchronous the logic for all this is relatively simple.

It's thanks to seeing send/receive struggles such as yours on this
list (which has also happened to me, but only very rarely: it seems I
tend to have reliable connectivity), among other issues, that I wrote
snazzer-measure. It ends up appending reproducible sha512sum and pgp
signatures to a measurements file for each snapshot, measurements
happen more than just once so they're timestamped with hostname - the
hope is I should spot any corruption that happens after the first
measurements are taken.

This is also a separate/async operation (it's the most I/O and CPU
intense operation of all).

Re: Anyone tried out btrbk yet?

[Marc MERLIN]

On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote:
> The way it works in snazzer (and btrbk and I think also btrfs-sxbackup
> as well), local snapshots continue to happen as normal (Eg. daily or
> hourly) and so when your backup media or backup server is finally
> available again, the size of each individual incremental is still the
> same as usual, it just has to perform more of them.
 
Good point. My system is not as smart. Every night, it'll make a new
backup and only send one incremental and hope it gets there. It doesn't
make a bunch of incrementals and send multiple.

The other options do a better job here.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Anyone tried out btrbk yet?

[Sander]

Marc MERLIN wrote (ao):
> On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote:
> > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup
> > as well), local snapshots continue to happen as normal (Eg. daily or
> > hourly) and so when your backup media or backup server is finally
> > available again, the size of each individual incremental is still the
> > same as usual, it just has to perform more of them.
>  
> Good point. My system is not as smart. Every night, it'll make a new
> backup and only send one incremental and hope it gets there. It doesn't
> make a bunch of incrementals and send multiple.
> 
> The other options do a better job here.

FWIW, I've written a bunch of scripts for making backups. The lot has
grown over the past years to what is is now. Not very pretty to see, but
reliable.

The subvolumes backupadmin home root rootvolume and var are snapshotted
every hour.

Each subvolume has their own entry in crontab for the actual backup.
For example rootvolume once a day, home and backupadmin every hour.

The scripts uses tar to make a full backup every first backup of a
subvolume that month, an incremental daily backup, and an incremental
hourly backup if applicable.

For a full backup the oldest available snapshot for that month is used,
regardless of when the backup is started. This way the backup of each
subvolume can be spread not to overload a system.

Backups are running in the idle queue to not hinder other processes, are
compressed with lbzip2 to utilize all cores, and are encrypted with gpg
for obvious reasons. In my tests lbzip2 gives the best size/speed ratio
compared to lzop, xz, bzip2, gzip, pxz and lz4(hc).

The script outputs what files and directories are in the backup to the
backupadmin subvolume. This data is compressed with lz4hc as lz4hc is
the fastest to decompress (useful to determine which archive contains
what you want restored).

Archives get transfered to a remote server by ftp, as ftp is the leanest
way of file transfer and supports resume. The initial connection is
encrypted to hide username/password, but as the archive is already
encrypted, the data channel is not. The ftp transfer is throttled to
only use part of the available bandwith.

A daily running script checks for archives which are not transfered yet
due to remote server not available or failed connection or the like, and
retransmits those archives.

Snapshots and archives are pruned based on disk usage (yet another
script).

Restore can be done by hand from snapshots (obviously), or by a script
from the locale archive if still available, or the remote archive.

The restore script can search a specific date-time range, and checks
both local and remote for the availability of an archive that contains
the wanted.

A bare metal restore can be done by fetching the archives from the
remote host and pipe them directly into gpg/tar. No need for additional
local storage and no delay. First the monthly full backup is restored,
then every daily incremental since, and then every hourly since the
youngest daily, if applicable. tar incremental restore is smart, and
removes the files and directories that were removed between backups.

	Sander

Re: Anyone tried out btrbk yet?

[Donald Pearson]

Implementation question about your scripts Marc..

I've set up some routines for different backup and retention intervals
and periods in cron but quickly ran in to stepping on my own toes by
the locking mechanism.  I could just disable the locking but I'm not
sure if that's the best approach and I don't know what it was
implemented to prevent in the first place.

Thoughts?

Thanks,
Donald

On Wed, Jul 15, 2015 at 3:00 AM, Sander <sander@humilis.net> wrote:
> Marc MERLIN wrote (ao):
>> On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote:
>> > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup
>> > as well), local snapshots continue to happen as normal (Eg. daily or
>> > hourly) and so when your backup media or backup server is finally
>> > available again, the size of each individual incremental is still the
>> > same as usual, it just has to perform more of them.
>>
>> Good point. My system is not as smart. Every night, it'll make a new
>> backup and only send one incremental and hope it gets there. It doesn't
>> make a bunch of incrementals and send multiple.
>>
>> The other options do a better job here.
>
> FWIW, I've written a bunch of scripts for making backups. The lot has
> grown over the past years to what is is now. Not very pretty to see, but
> reliable.
>
> The subvolumes backupadmin home root rootvolume and var are snapshotted
> every hour.
>
> Each subvolume has their own entry in crontab for the actual backup.
> For example rootvolume once a day, home and backupadmin every hour.
>
> The scripts uses tar to make a full backup every first backup of a
> subvolume that month, an incremental daily backup, and an incremental
> hourly backup if applicable.
>
> For a full backup the oldest available snapshot for that month is used,
> regardless of when the backup is started. This way the backup of each
> subvolume can be spread not to overload a system.
>
> Backups are running in the idle queue to not hinder other processes, are
> compressed with lbzip2 to utilize all cores, and are encrypted with gpg
> for obvious reasons. In my tests lbzip2 gives the best size/speed ratio
> compared to lzop, xz, bzip2, gzip, pxz and lz4(hc).
>
> The script outputs what files and directories are in the backup to the
> backupadmin subvolume. This data is compressed with lz4hc as lz4hc is
> the fastest to decompress (useful to determine which archive contains
> what you want restored).
>
> Archives get transfered to a remote server by ftp, as ftp is the leanest
> way of file transfer and supports resume. The initial connection is
> encrypted to hide username/password, but as the archive is already
> encrypted, the data channel is not. The ftp transfer is throttled to
> only use part of the available bandwith.
>
> A daily running script checks for archives which are not transfered yet
> due to remote server not available or failed connection or the like, and
> retransmits those archives.
>
> Snapshots and archives are pruned based on disk usage (yet another
> script).
>
> Restore can be done by hand from snapshots (obviously), or by a script
> from the locale archive if still available, or the remote archive.
>
> The restore script can search a specific date-time range, and checks
> both local and remote for the availability of an archive that contains
> the wanted.
>
> A bare metal restore can be done by fetching the archives from the
> remote host and pipe them directly into gpg/tar. No need for additional
> local storage and no delay. First the monthly full backup is restored,
> then every daily incremental since, and then every hourly since the
> youngest daily, if applicable. tar incremental restore is smart, and
> removes the files and directories that were removed between backups.
>
>         Sander
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Donald Pearson]

BTW, is anybody else experiencing btrfs-cleaner consuming heavy
resources for a very long time when snapshots are removed?

Note the TIME on one of these btrfs-cleaner processes.

top - 13:01:15 up 21:09,  2 users,  load average: 5.30, 4.80, 3.83
Tasks: 315 total,   3 running, 312 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us, 50.2 sy,  0.0 ni, 47.8 id,  1.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16431800 total,   177448 free,  1411876 used, 14842476 buff/cache
KiB Swap:  8257532 total,  8257316 free,      216 used. 14420732 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4134 root      20   0       0      0      0 R 100.0  0.0   2:41.40
btrfs-cleaner
 4183 root      20   0       0      0      0 R  99.7  0.0 191:11.33
btrfs-cleaner

On Wed, Jul 15, 2015 at 9:42 AM, Donald Pearson
<donaldwhpearson@gmail.com> wrote:
> Implementation question about your scripts Marc..
>
> I've set up some routines for different backup and retention intervals
> and periods in cron but quickly ran in to stepping on my own toes by
> the locking mechanism.  I could just disable the locking but I'm not
> sure if that's the best approach and I don't know what it was
> implemented to prevent in the first place.
>
> Thoughts?
>
> Thanks,
> Donald
>
> On Wed, Jul 15, 2015 at 3:00 AM, Sander <sander@humilis.net> wrote:
>> Marc MERLIN wrote (ao):
>>> On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote:
>>> > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup
>>> > as well), local snapshots continue to happen as normal (Eg. daily or
>>> > hourly) and so when your backup media or backup server is finally
>>> > available again, the size of each individual incremental is still the
>>> > same as usual, it just has to perform more of them.
>>>
>>> Good point. My system is not as smart. Every night, it'll make a new
>>> backup and only send one incremental and hope it gets there. It doesn't
>>> make a bunch of incrementals and send multiple.
>>>
>>> The other options do a better job here.
>>
>> FWIW, I've written a bunch of scripts for making backups. The lot has
>> grown over the past years to what is is now. Not very pretty to see, but
>> reliable.
>>
>> The subvolumes backupadmin home root rootvolume and var are snapshotted
>> every hour.
>>
>> Each subvolume has their own entry in crontab for the actual backup.
>> For example rootvolume once a day, home and backupadmin every hour.
>>
>> The scripts uses tar to make a full backup every first backup of a
>> subvolume that month, an incremental daily backup, and an incremental
>> hourly backup if applicable.
>>
>> For a full backup the oldest available snapshot for that month is used,
>> regardless of when the backup is started. This way the backup of each
>> subvolume can be spread not to overload a system.
>>
>> Backups are running in the idle queue to not hinder other processes, are
>> compressed with lbzip2 to utilize all cores, and are encrypted with gpg
>> for obvious reasons. In my tests lbzip2 gives the best size/speed ratio
>> compared to lzop, xz, bzip2, gzip, pxz and lz4(hc).
>>
>> The script outputs what files and directories are in the backup to the
>> backupadmin subvolume. This data is compressed with lz4hc as lz4hc is
>> the fastest to decompress (useful to determine which archive contains
>> what you want restored).
>>
>> Archives get transfered to a remote server by ftp, as ftp is the leanest
>> way of file transfer and supports resume. The initial connection is
>> encrypted to hide username/password, but as the archive is already
>> encrypted, the data channel is not. The ftp transfer is throttled to
>> only use part of the available bandwith.
>>
>> A daily running script checks for archives which are not transfered yet
>> due to remote server not available or failed connection or the like, and
>> retransmits those archives.
>>
>> Snapshots and archives are pruned based on disk usage (yet another
>> script).
>>
>> Restore can be done by hand from snapshots (obviously), or by a script
>> from the locale archive if still available, or the remote archive.
>>
>> The restore script can search a specific date-time range, and checks
>> both local and remote for the availability of an archive that contains
>> the wanted.
>>
>> A bare metal restore can be done by fetching the archives from the
>> remote host and pipe them directly into gpg/tar. No need for additional
>> local storage and no delay. First the monthly full backup is restored,
>> then every daily incremental since, and then every hourly since the
>> youngest daily, if applicable. tar incremental restore is smart, and
>> removes the files and directories that were removed between backups.
>>
>>         Sander
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Marc MERLIN]

On Wed, Jul 15, 2015 at 01:02:29PM -0500, Donald Pearson wrote:
> BTW, is anybody else experiencing btrfs-cleaner consuming heavy
> resources for a very long time when snapshots are removed?
 
Yes, that's normal. It spends a long time to reclaim blocks and free
them, especially if they are on a hard drive and not SSD.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Anyone tried out btrbk yet?

[Donald Pearson]

I'm starting to think there's something wrong with creating and
removing snapshots that leaves btrfs-cleaner either locked up or
nearly so.  If the btrfs-cleaner process was hard-disk limited I
should be seeing some HDD I/O to coincide but I don't.

So far btrfs-cleaner is has been using lots of CPU for 1900+ hours and
my disk I/O is basically idle.  My hourly snaps via cronjob stalled 11
hours ago.

Otherwise attempts to read/write to the filesystem appear to be
perfectly normal.

[root@san01 virtual_machines]# ps aux | grep btrfs-cleaner
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      1292     0.0      0.0      0     0     ?        S    Jul16
     0:01   [btrfs-cleaner]
root      5796     0.0      0.0      0     0     ?        S    Jul16
     0:00   [btrfs-cleaner]
root      6005     21.3    0.0      0     0     ?        S    Jul16
 943:34   [btrfs-cleaner]
root     14040    43.3    0.0      0     0     ?        R    Jul16
1916:05   [btrfs-cleaner]

[root@san01 virtual_machines]# ls -lah /run | grep backup
-rw-r--r--   1 root     root       11 Jul 20 00:00 backup.home.daily
-rw-r--r--   1 root     root       11 Jul 20 00:00 backup.root.daily
-rw-r--r--   1 root     root       11 Jul 20 00:01 backup.store.daily
-rw-r--r--   1 root     root       11 Jul 19 13:00 backup.store.hourly
-rw-r--r--   1 root     root       11 Jul 20 00:01 backup.virtual_machines.daily
-rw-r--r--   1 root     root       11 Jul 19 13:00
backup.virtual_machines.hourly
[root@san01 virtual_machines]# date
Mon Jul 20 00:14:05 CDT 2015

On Wed, Jul 15, 2015 at 4:49 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Wed, Jul 15, 2015 at 01:02:29PM -0500, Donald Pearson wrote:
>> BTW, is anybody else experiencing btrfs-cleaner consuming heavy
>> resources for a very long time when snapshots are removed?
>
> Yes, that's normal. It spends a long time to reclaim blocks and free
> them, especially if they are on a hard drive and not SSD.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Anyone tried out btrbk yet?

[Duncan]

Donald Pearson posted on Mon, 20 Jul 2015 00:15:26 -0500 as excerpted:

> I'm starting to think there's something wrong with creating and removing
> snapshots that leaves btrfs-cleaner either locked up or nearly so.  If
> the btrfs-cleaner process was hard-disk limited I should be seeing some
> HDD I/O to coincide but I don't.
> 
> So far btrfs-cleaner is has been using lots of CPU for 1900+ hours and
> my disk I/O is basically idle.  My hourly snaps via cronjob stalled 11
> hours ago.
> 
> Otherwise attempts to read/write to the filesystem appear to be
> perfectly normal.

Hourly snaps.  How many snapshots/subvolumes on the filesystem?  I assume 
the snap removal you mention is scheduled thinning?

A general rule of thumb is under 2000 snapshots per filesystem total, 
under 1000 if at all reasonable.  At 250 snapshots per subvolume, 2000 
snapshots is eight subvolumes worth, and 250-ish snapshots per subvolume 
is well within reason with reasonable thinning.

If you have lots of subvolumes, 3000 snapshots per filesystem isn't /too/ 
bad, but filesystem maintenance including snapshot deletion simply does 
/not/ scale well, and you'll likely run into scalability issues if you 
let it reach 10K snapshots on the filesystem.

If you're at 10K snapshots on the filesystem, it's quite likely the usual 
scalability issue's you're seeing, almost certain at 100K+ snapshots.  
OTOH, if you're under 1K or even 2K snapshots, it's very likely something 
abnormal.  2K-10K, should be usable in most cases, but there are likely 
corner-cases where it's bad.

Also, FWIW, the btrfs quota subsystem increases snapshot management 
complexity dramatically, so if you're using that, aim for the low ends of 
the above recommendation if at all possible, and/or consider either 
turning off the quota stuff or using a filesystem other than btrfs, as in 
addition to the scaling issues, the quota management code has been a 
source of repeated bugs and isn't a feature I'd recommend relying on 
until it has at least several kernel cycles worth of trouble-free history 
behind it.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Anyone tried out btrbk yet?

[Donald Pearson]

On Mon, Jul 20, 2015 at 3:28 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Donald Pearson posted on Mon, 20 Jul 2015 00:15:26 -0500 as excerpted:
>
>> I'm starting to think there's something wrong with creating and removing
>> snapshots that leaves btrfs-cleaner either locked up or nearly so.  If
>> the btrfs-cleaner process was hard-disk limited I should be seeing some
>> HDD I/O to coincide but I don't.
>>
>> So far btrfs-cleaner is has been using lots of CPU for 1900+ hours and
>> my disk I/O is basically idle.  My hourly snaps via cronjob stalled 11
>> hours ago.
>>
>> Otherwise attempts to read/write to the filesystem appear to be
>> perfectly normal.
>
> Hourly snaps.  How many snapshots/subvolumes on the filesystem?  I assume
> the snap removal you mention is scheduled thinning?

I'm using Marc's scripts currently which handles the creation and
retention.  Including snapshots there are currently 94 subvolumes.
The maximum possible with my settings is 118 after 3 months.

>
> A general rule of thumb is under 2000 snapshots per filesystem total,
> under 1000 if at all reasonable.  At 250 snapshots per subvolume, 2000
> snapshots is eight subvolumes worth, and 250-ish snapshots per subvolume
> is well within reason with reasonable thinning.
>
> If you have lots of subvolumes, 3000 snapshots per filesystem isn't /too/
> bad, but filesystem maintenance including snapshot deletion simply does
> /not/ scale well, and you'll likely run into scalability issues if you
> let it reach 10K snapshots on the filesystem.
>
> If you're at 10K snapshots on the filesystem, it's quite likely the usual
> scalability issue's you're seeing, almost certain at 100K+ snapshots.
> OTOH, if you're under 1K or even 2K snapshots, it's very likely something
> abnormal.  2K-10K, should be usable in most cases, but there are likely
> corner-cases where it's bad.
>
> Also, FWIW, the btrfs quota subsystem increases snapshot management
> complexity dramatically, so if you're using that, aim for the low ends of
> the above recommendation if at all possible, and/or consider either
> turning off the quota stuff or using a filesystem other than btrfs, as in
> addition to the scaling issues, the quota management code has been a
> source of repeated bugs and isn't a feature I'd recommend relying on
> until it has at least several kernel cycles worth of trouble-free history
> behind it.

Thanks for the insight.  I just took a look at dmesg and found this.
Is this coincidental or is this maybe the reason things appear to be
stuck?  I'm not sure how to read this.

[195400.023165] ------------[ cut here ]------------
[195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028
__qgroup_excl_accounting+0x1dc/0x270 [btrfs]()
[195400.023201] Modules linked in: ext4(E) mbcache(E) jbd2(E) ppdev(E)
snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_intel(E)
snd_hda_controller(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E)
snd_seq(E) snd_seq_device(E) kvm_amd(E) kvm(E) snd_pcm(E) pcspkr(E)
serio_raw(E) k10temp(E) edac_mce_amd(E) edac_core(E) snd_timer(E)
snd(E) soundcore(E) sp5100_tco(E) i2c_piix4(E) ses(E) enclosure(E)
8250_fintek(E) parport_pc(E) parport(E) tpm_infineon(E) shpchp(E)
acpi_cpufreq(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E)
sunrpc(E) btrfs(E) xor(E) raid6_pq(E) ata_generic(E) pata_acpi(E)
sd_mod(E) nouveau(E) video(E) mxm_wmi(E) i2c_algo_bit(E)
drm_kms_helper(E) ttm(E) drm(E) pata_atiixp(E) ahci(E) pata_jmicron(E)
libahci(E) lpfc(E) scsi_transport_fc(E) firewire_ohci(E) libata(E)
firewire_core(E)
[195400.023225]  crc_itu_t(E) r8169(E) mii(E) mpt2sas(E) raid_class(E)
scsi_transport_sas(E) wmi(E)
[195400.023231] CPU: 2 PID: 16987 Comm: kworker/u12:5 Tainted: G
     E   4.1.0-1.el7.elrepo.x86_64 #1
[195400.023232] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD
MS-7577/790FX-GD70(MS-7577), BIOS V1.16 12/01/2010
[195400.023244] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs]
[195400.023246]  0000000000000000 0000000031fd5f58 ffff880368a03b68
ffffffff816d4638
[195400.023248]  0000000000000000 0000000000000000 ffff880368a03ba8
ffffffff8107c51a
[195400.023250]  c000000000000000 ffff880409018d60 ffff88040a28ec88
ffffffffffff9000
[195400.023252] Call Trace:
[195400.023257]  [<ffffffff816d4638>] dump_stack+0x45/0x57
[195400.023261]  [<ffffffff8107c51a>] warn_slowpath_common+0x8a/0xc0
[195400.023263]  [<ffffffff8107c64a>] warn_slowpath_null+0x1a/0x20
[195400.023274]  [<ffffffffa05b5bdc>]
__qgroup_excl_accounting+0x1dc/0x270 [btrfs]
[195400.023277]  [<ffffffff811dd8b9>] ? kmem_cache_alloc_trace+0x199/0x220
[195400.023289]  [<ffffffffa05b8e37>]
btrfs_delayed_qgroup_accounting+0x317/0xc60 [btrfs]
[195400.023291]  [<ffffffff810afd68>] ? __enqueue_entity+0x78/0x80
[195400.023293]  [<ffffffff811dd699>] ? kmem_cache_alloc+0x1a9/0x230
[195400.023302]  [<ffffffffa053cb0a>]
btrfs_run_delayed_refs.part.66+0x20a/0x270 [btrfs]
[195400.023311]  [<ffffffffa053cc18>] delayed_ref_async_start+0x88/0xa0 [btrfs]
[195400.023322]  [<ffffffffa0580562>] normal_work_helper+0xc2/0x280 [btrfs]
[195400.023332]  [<ffffffffa0580952>] btrfs_extent_refs_helper+0x12/0x20 [btrfs]
[195400.023335]  [<ffffffff810959cd>] process_one_work+0x14d/0x420
[195400.023337]  [<ffffffff81096192>] worker_thread+0x112/0x520
[195400.023339]  [<ffffffff81096080>] ? rescuer_thread+0x3e0/0x3e0
[195400.023341]  [<ffffffff8109bee8>] kthread+0xd8/0xf0
[195400.023342]  [<ffffffff8109be10>] ? kthread_create_on_node+0x1b0/0x1b0
[195400.023344]  [<ffffffff816dc3a2>] ret_from_fork+0x42/0x70
[195400.023346]  [<ffffffff8109be10>] ? kthread_create_on_node+0x1b0/0x1b0
[195400.023347] ---[ end trace 1abc27647fe906a5 ]---


>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Duncan]

Donald Pearson posted on Mon, 20 Jul 2015 08:33:47 -0500 as excerpted:

>> Also, FWIW, the btrfs quota subsystem increases snapshot management
>> complexity dramatically, so if you're using that, aim for the low ends
>> of the above recommendation if at all possible, and/or consider either
>> turning off the quota stuff or using a filesystem other than btrfs, as
>> in addition to the scaling issues, the quota management code has been a
>> source of repeated bugs and isn't a feature I'd recommend relying on
>> until it has at least several kernel cycles worth of trouble-free
>> history behind it.
> 
> Thanks for the insight.  I just took a look at dmesg and found this.
> Is this coincidental or is this maybe the reason things appear to be
> stuck?  I'm not sure how to read this.
> 
> [195400.023165] ------------[ cut here ]------------
> [195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028
> __qgroup_excl_accounting+0x1dc/0x270 [btrfs]()

I don't know.  I'm not a btrfs quota feature user.

What I do know is that there has been... again... more quota code patches 
recently, fixing what sound like serious problems.

You already have my recommendation above; unless you are actively testing 
the btrfs quota code, if you don't need it, don't use it; if you do need 
it, use something where it's actually working well enough to /rely/ on.

But in fairness that's potentially a negatively biased view, since as I'm 
on the list but not actually using that feature I'm seeing the bug 
reports without much in the way of knowing how common they actually are.  
If it's a big deal to turn quotas off, I'd say wait and see if the dev 
actually working on it cares to comment with his opinion of quota feature 
stability in general, and perhaps of that specific trace, before deciding 
for sure.

But if you have the inclination and don't really need quotas, and 
assuming disabling them isn't /too/ big a hassle, it might be worthwhile 
disabling the feature and seeing how things go.  I can't see how not 
having to deal with quotas could /hurt/ scaling, and with luck, it'll 
improve things noticeably.

FWIW, the developer post where the effect of quotas on scaling was 
explained was actually in the context of snapshot-aware-defrag, which has 
been disabled for awhile now, /because/ of the scaling issues (it was so 
bad the defrag would basically hang, taking hours to defrag a single 
moderately sized file as it tracked it thru the various snapshots, so 
processing time for a full filesystem could easily be weeks and could in 
some cases be months, obviously well outside any practically workable 
range!), and while quota processing was explained to be horrible in that 
context at that time (I read it as at least doubling the necessary work), 
there has been at least one quota-code rewrite since then, as well as 
additional work, and it may actually not be nearly as bad any more, 
barring a direct bug, of course.  But I don't know as I've not seen any 
direct statements or numbers either way, only the bug reports and patches 
going by.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Anyone tried out btrbk yet?

[Donald Pearson]

Thanks for the feedback Duncan.

It doesn't appear to be a big deal to disable quotas so that's what
I'll do for now.

On Tue, Jul 21, 2015 at 4:29 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Donald Pearson posted on Mon, 20 Jul 2015 08:33:47 -0500 as excerpted:
>
>>> Also, FWIW, the btrfs quota subsystem increases snapshot management
>>> complexity dramatically, so if you're using that, aim for the low ends
>>> of the above recommendation if at all possible, and/or consider either
>>> turning off the quota stuff or using a filesystem other than btrfs, as
>>> in addition to the scaling issues, the quota management code has been a
>>> source of repeated bugs and isn't a feature I'd recommend relying on
>>> until it has at least several kernel cycles worth of trouble-free
>>> history behind it.
>>
>> Thanks for the insight.  I just took a look at dmesg and found this.
>> Is this coincidental or is this maybe the reason things appear to be
>> stuck?  I'm not sure how to read this.
>>
>> [195400.023165] ------------[ cut here ]------------
>> [195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028
>> __qgroup_excl_accounting+0x1dc/0x270 [btrfs]()
>
> I don't know.  I'm not a btrfs quota feature user.
>
> What I do know is that there has been... again... more quota code patches
> recently, fixing what sound like serious problems.
>
> You already have my recommendation above; unless you are actively testing
> the btrfs quota code, if you don't need it, don't use it; if you do need
> it, use something where it's actually working well enough to /rely/ on.
>
> But in fairness that's potentially a negatively biased view, since as I'm
> on the list but not actually using that feature I'm seeing the bug
> reports without much in the way of knowing how common they actually are.
> If it's a big deal to turn quotas off, I'd say wait and see if the dev
> actually working on it cares to comment with his opinion of quota feature
> stability in general, and perhaps of that specific trace, before deciding
> for sure.
>
> But if you have the inclination and don't really need quotas, and
> assuming disabling them isn't /too/ big a hassle, it might be worthwhile
> disabling the feature and seeing how things go.  I can't see how not
> having to deal with quotas could /hurt/ scaling, and with luck, it'll
> improve things noticeably.
>
> FWIW, the developer post where the effect of quotas on scaling was
> explained was actually in the context of snapshot-aware-defrag, which has
> been disabled for awhile now, /because/ of the scaling issues (it was so
> bad the defrag would basically hang, taking hours to defrag a single
> moderately sized file as it tracked it thru the various snapshots, so
> processing time for a full filesystem could easily be weeks and could in
> some cases be months, obviously well outside any practically workable
> range!), and while quota processing was explained to be horrible in that
> context at that time (I read it as at least doubling the necessary work),
> there has been at least one quota-code rewrite since then, as well as
> additional work, and it may actually not be nearly as bad any more,
> barring a direct bug, of course.  But I don't know as I've not seen any
> direct statements or numbers either way, only the bug reports and patches
> going by.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Donald Pearson]

I can confirm that getting rid of the quotas fixed the issue for me.
Just disabling quotas wasn't enough, I had to enable, delete all
qgroups, reboot because disable was hung on one of the filesystems,
then disable quotas.  Now when btrfs-cleaner runs it doesn't
completely consume a core, I can see corresponding disk i/o, and the
process goes away after a reasonable amount of time.

On Tue, Jul 21, 2015 at 8:29 PM, Donald Pearson
<donaldwhpearson@gmail.com> wrote:
> Thanks for the feedback Duncan.
>
> It doesn't appear to be a big deal to disable quotas so that's what
> I'll do for now.
>
> On Tue, Jul 21, 2015 at 4:29 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> Donald Pearson posted on Mon, 20 Jul 2015 08:33:47 -0500 as excerpted:
>>
>>>> Also, FWIW, the btrfs quota subsystem increases snapshot management
>>>> complexity dramatically, so if you're using that, aim for the low ends
>>>> of the above recommendation if at all possible, and/or consider either
>>>> turning off the quota stuff or using a filesystem other than btrfs, as
>>>> in addition to the scaling issues, the quota management code has been a
>>>> source of repeated bugs and isn't a feature I'd recommend relying on
>>>> until it has at least several kernel cycles worth of trouble-free
>>>> history behind it.
>>>
>>> Thanks for the insight.  I just took a look at dmesg and found this.
>>> Is this coincidental or is this maybe the reason things appear to be
>>> stuck?  I'm not sure how to read this.
>>>
>>> [195400.023165] ------------[ cut here ]------------
>>> [195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028
>>> __qgroup_excl_accounting+0x1dc/0x270 [btrfs]()
>>
>> I don't know.  I'm not a btrfs quota feature user.
>>
>> What I do know is that there has been... again... more quota code patches
>> recently, fixing what sound like serious problems.
>>
>> You already have my recommendation above; unless you are actively testing
>> the btrfs quota code, if you don't need it, don't use it; if you do need
>> it, use something where it's actually working well enough to /rely/ on.
>>
>> But in fairness that's potentially a negatively biased view, since as I'm
>> on the list but not actually using that feature I'm seeing the bug
>> reports without much in the way of knowing how common they actually are.
>> If it's a big deal to turn quotas off, I'd say wait and see if the dev
>> actually working on it cares to comment with his opinion of quota feature
>> stability in general, and perhaps of that specific trace, before deciding
>> for sure.
>>
>> But if you have the inclination and don't really need quotas, and
>> assuming disabling them isn't /too/ big a hassle, it might be worthwhile
>> disabling the feature and seeing how things go.  I can't see how not
>> having to deal with quotas could /hurt/ scaling, and with luck, it'll
>> improve things noticeably.
>>
>> FWIW, the developer post where the effect of quotas on scaling was
>> explained was actually in the context of snapshot-aware-defrag, which has
>> been disabled for awhile now, /because/ of the scaling issues (it was so
>> bad the defrag would basically hang, taking hours to defrag a single
>> moderately sized file as it tracked it thru the various snapshots, so
>> processing time for a full filesystem could easily be weeks and could in
>> some cases be months, obviously well outside any practically workable
>> range!), and while quota processing was explained to be horrible in that
>> context at that time (I read it as at least doubling the necessary work),
>> there has been at least one quota-code rewrite since then, as well as
>> additional work, and it may actually not be nearly as bad any more,
>> barring a direct bug, of course.  But I don't know as I've not seen any
>> direct statements or numbers either way, only the bug reports and patches
>> going by.
>>
>> --
>> Duncan - List replies preferred.   No HTML msgs.
>> "Every nonfree program has a lord, a master --
>> and if you use the program, he is your master."  Richard Stallman
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Anyone tried out btrbk yet?

[Duncan]

Donald Pearson posted on Sat, 25 Jul 2015 08:27:03 -0500 as excerpted:

[Duncan wrote...]

>>>>> Also, FWIW, the btrfs quota subsystem increases snapshot management
>>>>> complexity dramatically, so if you're using that, aim for the low
>>>>> ends of the above recommendation if at all possible, and/or consider
>>>>> either turning off the quota stuff or using a filesystem other than
>>>>> btrfs, as in addition to the scaling issues, the quota management
>>>>> code has been a source of repeated bugs and isn't a feature I'd
>>>>> recommend relying on until it has at least several kernel cycles
>>>>> worth of trouble-free history behind it.

[Big intervening thread snip...]

> I can confirm that getting rid of the quotas fixed the issue for me.
> Just disabling quotas wasn't enough, I had to enable, delete all
> qgroups, reboot because disable was hung on one of the filesystems,
> then disable quotas.  Now when btrfs-cleaner runs it doesn't completely
> consume a core, I can see corresponding disk i/o, and the process goes
> away after a reasonable amount of time.

Thanks for the confirmation!

Good to have a recent and concrete report to point to, on the subject of 
btrfs and quotas, now.  Now I can at least say something like "we do have 
at least one report (4.1 timeframe) of snapshot deletion scaling or 
lockup issues where quotas was implicated -- after deleting qgroups and 
disabling quotas, the problem disappeared."

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Anyone tried out btrbk yet?

[Marc MERLIN]

On Wed, Jul 15, 2015 at 09:42:28AM -0500, Donald Pearson wrote:
> Implementation question about your scripts Marc..

make sure you Cc me then, I could have missed that Email :)

> I've set up some routines for different backup and retention intervals
> and periods in cron but quickly ran in to stepping on my own toes by
> the locking mechanism.  I could just disable the locking but I'm not
> sure if that's the best approach and I don't know what it was
> implemented to prevent in the first place.

Try --postfix servername
it'll add the destination server in the snapshot rotation and the
lockfile.

Otherwise, you can just trivially modify the script to take --lock as an
argument, or you can even 
ln -s btrfs-subvolume-backup btrfs-subvolume-backupserver2
and the script will automatically make a /var/run/btrfs-subvolume-backupserver2
as a lockfile.

Hope this helps.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901