* Anyone tried out btrbk yet? @ 2015-07-09 12:26 Martin Steigerwald 2015-07-09 17:12 ` Henri Valta ` (2 more replies) 0 siblings, 3 replies; 23+ messages in thread From: Martin Steigerwald @ 2015-07-09 12:26 UTC (permalink / raw) To: linux-btrfs Hi! I see Alex, the developer of btrbk posted here once about btrfs send and receive, but well any other users of btrbk¹? What are your experiences? I consider switching to it from my home grown rsync based backup script to it. Well I may try it for one of my BTRFS volumes in addition to the rsync backup for now. I would like to give all options on command line, but well, maybe it can completely replace my current script if I put everything in its configuration. Any other handy BTRFS backup solutions? [1] http://www.digint.ch/btrbk/ Thanks, -- Martin ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-09 12:26 Anyone tried out btrbk yet? Martin Steigerwald @ 2015-07-09 17:12 ` Henri Valta 2015-07-09 17:17 ` Marc MERLIN 2015-07-10 10:46 ` Axel Burri 2 siblings, 0 replies; 23+ messages in thread From: Henri Valta @ 2015-07-09 17:12 UTC (permalink / raw) To: Martin Steigerwald; +Cc: linux-btrfs On Thursday 09 July 2015 14:26:55 you wrote: > Well I may try it for one of my BTRFS volumes in addition to the rsync > backup for now. I would like to give all options on command line, but well, > maybe it can completely replace my current script if I put everything in its > configuration. > > Any other handy BTRFS backup solutions? Hi, I've been using btrfs-sxbackup for a couple of weeks, and it has been working great. Everything is configured on command line, so that's a plus. https://pypi.python.org/pypi/btrfs-sxbackup https://github.com/masc3d/btrfs-sxbackup -Henri ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-09 12:26 Anyone tried out btrbk yet? Martin Steigerwald 2015-07-09 17:12 ` Henri Valta @ 2015-07-09 17:17 ` Marc MERLIN 2015-07-10 1:35 ` Donald Pearson 2015-07-10 10:46 ` Axel Burri 2 siblings, 1 reply; 23+ messages in thread From: Marc MERLIN @ 2015-07-09 17:17 UTC (permalink / raw) To: Martin Steigerwald; +Cc: linux-btrfs On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote: > Hi! > > I see Alex, the developer of btrbk posted here once about btrfs send and > receive, but well any other users of btrbk¹? What are your experiences? > > I consider switching to it from my home grown rsync based backup script to > it. > > Well I may try it for one of my BTRFS volumes in addition to the rsync > backup for now. I would like to give all options on command line, but well, > maybe it can completely replace my current script if I put everything in its > configuration. > > Any other handy BTRFS backup solutions? I use my own which I wrote :) http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-09 17:17 ` Marc MERLIN @ 2015-07-10 1:35 ` Donald Pearson 2015-07-10 1:38 ` Donald Pearson 0 siblings, 1 reply; 23+ messages in thread From: Donald Pearson @ 2015-07-10 1:35 UTC (permalink / raw) To: Marc MERLIN; +Cc: Martin Steigerwald, Btrfs BTRFS Marc, I thought I'd yours a try, and I'm probably embarassing myself here but I'm running in to this issue. Centos 7. [root@san01 tank]# ./btrfs-subvolume-backup store /mnt2/backups ./btrfs-subvolume-backup: line 177: shlock: command not found /var/run/btrfs-subvolume-backup held for btrfs-subvolume-backup, quitting [root@san01 tank]# yum whatprovides shlock Loaded plugins: changelog, fastestmirror Loading mirror speeds from cached hostfile * base: dist1.800hosting.com * elrepo: repos.dfw.lax-noc.com * epel: mirror.umd.edu * extras: mirrors.usc.edu * updates: mirror.keystealth.orgNo matches found [root@san01 tank]# shlock -bash: shlock: command not found [root@san01 tank]# yum search all shlock Loaded plugins: changelog, fastestmirror Loading mirror speeds from cached hostfile * base: dist1.800hosting.com * elrepo: repos.dfw.lax-noc.com * epel: mirror.utexas.edu * extras: mirror.thelinuxfix.com * updates: dallas.tx.mirror.xygenhosting.com Warning: No matches found for: shlock No matches found On Thu, Jul 9, 2015 at 12:17 PM, Marc MERLIN <marc@merlins.org> wrote: > On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote: >> Hi! >> >> I see Alex, the developer of btrbk posted here once about btrfs send and >> receive, but well any other users of btrbk¹? What are your experiences? >> >> I consider switching to it from my home grown rsync based backup script to >> it. >> >> Well I may try it for one of my BTRFS volumes in addition to the rsync >> backup for now. I would like to give all options on command line, but well, >> maybe it can completely replace my current script if I put everything in its >> configuration. >> >> Any other handy BTRFS backup solutions? > > I use my own which I wrote :) > http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive > http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup > > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking > Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-10 1:35 ` Donald Pearson @ 2015-07-10 1:38 ` Donald Pearson 2015-07-10 4:02 ` Paul Harvey 0 siblings, 1 reply; 23+ messages in thread From: Donald Pearson @ 2015-07-10 1:38 UTC (permalink / raw) To: Marc MERLIN; +Cc: Martin Steigerwald, Btrfs BTRFS ... and I just found your other block about stealing shlock out of inn. Officially embarassed! On Thu, Jul 9, 2015 at 8:35 PM, Donald Pearson <donaldwhpearson@gmail.com> wrote: > Marc, > > I thought I'd yours a try, and I'm probably embarassing myself here > but I'm running in to this issue. Centos 7. > > [root@san01 tank]# ./btrfs-subvolume-backup store /mnt2/backups > ./btrfs-subvolume-backup: line 177: shlock: command not found > /var/run/btrfs-subvolume-backup held for btrfs-subvolume-backup, quitting > [root@san01 tank]# yum whatprovides shlock > Loaded plugins: changelog, fastestmirror > Loading mirror speeds from cached hostfile > * base: dist1.800hosting.com > * elrepo: repos.dfw.lax-noc.com > * epel: mirror.umd.edu > * extras: mirrors.usc.edu > * updates: mirror.keystealth.orgNo matches found > [root@san01 tank]# shlock > -bash: shlock: command not found > [root@san01 tank]# yum search all shlock > Loaded plugins: changelog, fastestmirror > Loading mirror speeds from cached hostfile > * base: dist1.800hosting.com > * elrepo: repos.dfw.lax-noc.com > * epel: mirror.utexas.edu > * extras: mirror.thelinuxfix.com > * updates: dallas.tx.mirror.xygenhosting.com > Warning: No matches found for: shlock > No matches found > > On Thu, Jul 9, 2015 at 12:17 PM, Marc MERLIN <marc@merlins.org> wrote: >> On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote: >>> Hi! >>> >>> I see Alex, the developer of btrbk posted here once about btrfs send and >>> receive, but well any other users of btrbk¹? What are your experiences? >>> >>> I consider switching to it from my home grown rsync based backup script to >>> it. >>> >>> Well I may try it for one of my BTRFS volumes in addition to the rsync >>> backup for now. I would like to give all options on command line, but well, >>> maybe it can completely replace my current script if I put everything in its >>> configuration. >>> >>> Any other handy BTRFS backup solutions? >> >> I use my own which I wrote :) >> http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive >> http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup >> >> Marc >> -- >> "A mouse is a device used to point at the xterm you want to type in" - A.S.R. >> Microsoft is to operating systems .... >> .... what McDonalds is to gourmet cooking >> Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-10 1:38 ` Donald Pearson @ 2015-07-10 4:02 ` Paul Harvey 0 siblings, 0 replies; 23+ messages in thread From: Paul Harvey @ 2015-07-10 4:02 UTC (permalink / raw) To: Donald Pearson; +Cc: Marc MERLIN, Martin Steigerwald, Btrfs BTRFS In my research, I've found btrbk and btrfs-sxbackup certainly to be the leading contenders in terms of feature completeness. sanoid [1] will be another interesting possibility once btrfs compatibility is added (currently zfs only). I just wish I'd discovered all these before I went to all the effort of creating snazzer [1] :) I've been meaning to split some stuff out of snazzer that might be generically useful to other folks, such as filesystem cloning of all subvols/snapshots via send/receive, and it seems as if it should be possible to automatically prune any idiosyncratic snapshot naming convention - I just haven't found the time to write unit tests. [1] https://github.com/jimsalterjrs/sanoid [2] https://github.com/csirac2/snazzer On 10 July 2015 at 11:38, Donald Pearson <donaldwhpearson@gmail.com> wrote: > ... and I just found your other block about stealing shlock out of inn. > > Officially embarassed! > > On Thu, Jul 9, 2015 at 8:35 PM, Donald Pearson > <donaldwhpearson@gmail.com> wrote: >> Marc, >> >> I thought I'd yours a try, and I'm probably embarassing myself here >> but I'm running in to this issue. Centos 7. >> >> [root@san01 tank]# ./btrfs-subvolume-backup store /mnt2/backups >> ./btrfs-subvolume-backup: line 177: shlock: command not found >> /var/run/btrfs-subvolume-backup held for btrfs-subvolume-backup, quitting >> [root@san01 tank]# yum whatprovides shlock >> Loaded plugins: changelog, fastestmirror >> Loading mirror speeds from cached hostfile >> * base: dist1.800hosting.com >> * elrepo: repos.dfw.lax-noc.com >> * epel: mirror.umd.edu >> * extras: mirrors.usc.edu >> * updates: mirror.keystealth.orgNo matches found >> [root@san01 tank]# shlock >> -bash: shlock: command not found >> [root@san01 tank]# yum search all shlock >> Loaded plugins: changelog, fastestmirror >> Loading mirror speeds from cached hostfile >> * base: dist1.800hosting.com >> * elrepo: repos.dfw.lax-noc.com >> * epel: mirror.utexas.edu >> * extras: mirror.thelinuxfix.com >> * updates: dallas.tx.mirror.xygenhosting.com >> Warning: No matches found for: shlock >> No matches found >> >> On Thu, Jul 9, 2015 at 12:17 PM, Marc MERLIN <marc@merlins.org> wrote: >>> On Thu, Jul 09, 2015 at 02:26:55PM +0200, Martin Steigerwald wrote: >>>> Hi! >>>> >>>> I see Alex, the developer of btrbk posted here once about btrfs send and >>>> receive, but well any other users of btrbk¹? What are your experiences? >>>> >>>> I consider switching to it from my home grown rsync based backup script to >>>> it. >>>> >>>> Well I may try it for one of my BTRFS volumes in addition to the rsync >>>> backup for now. I would like to give all options on command line, but well, >>>> maybe it can completely replace my current script if I put everything in its >>>> configuration. >>>> >>>> Any other handy BTRFS backup solutions? >>> >>> I use my own which I wrote :) >>> http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive >>> http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup >>> >>> Marc >>> -- >>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R. >>> Microsoft is to operating systems .... >>> .... what McDonalds is to gourmet cooking >>> Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-09 12:26 Anyone tried out btrbk yet? Martin Steigerwald 2015-07-09 17:12 ` Henri Valta 2015-07-09 17:17 ` Marc MERLIN @ 2015-07-10 10:46 ` Axel Burri 2015-07-12 3:55 ` Marc MERLIN 2 siblings, 1 reply; 23+ messages in thread From: Axel Burri @ 2015-07-10 10:46 UTC (permalink / raw) To: Martin Steigerwald; +Cc: linux-btrfs On 2015-07-09 14:26, Martin Steigerwald wrote: > Well I may try it for one of my BTRFS volumes in addition to the rsync > backup for now. I would like to give all options on command line, but well, > maybe it can completely replace my current script if I put everything in its > configuration. One reason why btrbk exists is that I wanted to have all my backups configured in a config file. I use btrbk to backup several hosts to several backup locations (around 20 subvolumes), which made setting everything up with command-line options very cumbersome. For simpler setups you might be better off with command-line based tools (e.g. Marcs "btrfs-subvolume-backup", which I like for it's smallness). Give it a try, there is a "dryrun" command in btrbk :) Ask on github if you need help. - Axel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-10 10:46 ` Axel Burri @ 2015-07-12 3:55 ` Marc MERLIN 2015-07-13 0:42 ` Marc MERLIN 0 siblings, 1 reply; 23+ messages in thread From: Marc MERLIN @ 2015-07-12 3:55 UTC (permalink / raw) To: Axel Burri; +Cc: Martin Steigerwald, linux-btrfs On Fri, Jul 10, 2015 at 12:46:24PM +0200, Axel Burri wrote: > On 2015-07-09 14:26, Martin Steigerwald wrote: > > > Well I may try it for one of my BTRFS volumes in addition to the rsync > > backup for now. I would like to give all options on command line, but well, > > maybe it can completely replace my current script if I put everything in its > > configuration. > > One reason why btrbk exists is that I wanted to have all my backups > configured in a config file. I use btrbk to backup several hosts to > several backup locations (around 20 subvolumes), which made setting > everything up with command-line options very cumbersome. > > For simpler setups you might be better off with command-line based tools > (e.g. Marcs "btrfs-subvolume-backup", which I like for it's smallness). I just had another look at btrbk, and it's obviously a lot more featureful. It's almost 10 times bigger in lines of code than btrfs-subvolume-backup :) Anyway, to others, if you're happy using a tool that just does that you need without having to dig into it and without you caring how btrfs send/receive, works, btrbk looks like the better choice. If you'd like something short-ish to look at and see how it works and/or want something simple in shell you can modify quickly, btrfs-subvolume-backup might be better for you. Hope that helps others. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-12 3:55 ` Marc MERLIN @ 2015-07-13 0:42 ` Marc MERLIN 2015-07-15 0:03 ` Paul Harvey 0 siblings, 1 reply; 23+ messages in thread From: Marc MERLIN @ 2015-07-13 0:42 UTC (permalink / raw) To: Axel Burri; +Cc: Martin Steigerwald, linux-btrfs On Sat, Jul 11, 2015 at 08:55:23PM -0700, Marc MERLIN wrote: > On Fri, Jul 10, 2015 at 12:46:24PM +0200, Axel Burri wrote: > > On 2015-07-09 14:26, Martin Steigerwald wrote: > > > > > Well I may try it for one of my BTRFS volumes in addition to the rsync > > > backup for now. I would like to give all options on command line, but well, > > > maybe it can completely replace my current script if I put everything in its > > > configuration. > > > > One reason why btrbk exists is that I wanted to have all my backups > > configured in a config file. I use btrbk to backup several hosts to > > several backup locations (around 20 subvolumes), which made setting > > everything up with command-line options very cumbersome. > > > > For simpler setups you might be better off with command-line based tools > > (e.g. Marcs "btrfs-subvolume-backup", which I like for it's smallness). > > I just had another look at btrbk, and it's obviously a lot more > featureful. It's almost 10 times bigger in lines of code than > btrfs-subvolume-backup :) > > Anyway, to others, if you're happy using a tool that just does that you > need without having to dig into it and without you caring how btrfs > send/receive, works, btrbk looks like the better choice. > > If you'd like something short-ish to look at and see how it works and/or > want something simple in shell you can modify quickly, btrfs-subvolume-backup > might be better for you. Actually there is one thing my backup script doesn't do and it looks like btrbk doesn't do either. When I travel I'm often on crappy hotel wireless and my incremental backups never succeed, and they just get bigger every day, so succeed even less next time. At the same time, I have an issue with btrfs where my backup server cause a backup over ssh on a local network to take 12H or more when the data to transfer isn't that much. >From what I can tell it's a problem where btrfs receive just pauses for one second or more, and kills the TCP flow. TCP restarts slowly and ramps up just to be stopped again. As a result, it takes forever to finish. Since I don't know if this btrfs stall problem will get looked into or fixed anytime soon, it would be great for the incremental backup to go to a local spool directory, and then for the backup script to just copy it with rsync until it makes it to the other side. Only then is btrfs receive run with that file, and then the next incremental backup is tried. But this is a fair amount of logic to get this right, and I just haven't taken the time to write it. Would it be helpful to others, or am I the only one with this problem? Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-13 0:42 ` Marc MERLIN @ 2015-07-15 0:03 ` Paul Harvey 2015-07-15 3:14 ` Marc MERLIN 0 siblings, 1 reply; 23+ messages in thread From: Paul Harvey @ 2015-07-15 0:03 UTC (permalink / raw) To: Marc MERLIN; +Cc: Axel Burri, Martin Steigerwald, linux-btrfs The way it works in snazzer (and btrbk and I think also btrfs-sxbackup as well), local snapshots continue to happen as normal (Eg. daily or hourly) and so when your backup media or backup server is finally available again, the size of each individual incremental is still the same as usual, it just has to perform more of them. Separating snapshotting from transport lends to more flexibility IMHO, Eg. with snazzer I can keep multiple physical backup media in sync with each other even if I only rotate/attach those disks once a week/month (maintain backup filesystems in parallel), the snazzer-receive script is very dumb - it just receives all the missing snapshots from the source. However it does filter them cf. "btrfs subvolume list /subvolume | snazzer-prune-candidates --invert" first in case some would just be deleted again shortly after according to retention policy. For the ssh transport, you can do the same things but in series: push the snapshots up to a local server and then on to remote storage elsewhere (maintain backup filesystems in series). Because the snapshotting, transport and pruning operations are asynchronous the logic for all this is relatively simple. It's thanks to seeing send/receive struggles such as yours on this list (which has also happened to me, but only very rarely: it seems I tend to have reliable connectivity), among other issues, that I wrote snazzer-measure. It ends up appending reproducible sha512sum and pgp signatures to a measurements file for each snapshot, measurements happen more than just once so they're timestamped with hostname - the hope is I should spot any corruption that happens after the first measurements are taken. This is also a separate/async operation (it's the most I/O and CPU intense operation of all). ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-15 0:03 ` Paul Harvey @ 2015-07-15 3:14 ` Marc MERLIN 2015-07-15 8:00 ` Sander 0 siblings, 1 reply; 23+ messages in thread From: Marc MERLIN @ 2015-07-15 3:14 UTC (permalink / raw) To: Paul Harvey; +Cc: Axel Burri, Martin Steigerwald, linux-btrfs On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote: > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup > as well), local snapshots continue to happen as normal (Eg. daily or > hourly) and so when your backup media or backup server is finally > available again, the size of each individual incremental is still the > same as usual, it just has to perform more of them. Good point. My system is not as smart. Every night, it'll make a new backup and only send one incremental and hope it gets there. It doesn't make a bunch of incrementals and send multiple. The other options do a better job here. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-15 3:14 ` Marc MERLIN @ 2015-07-15 8:00 ` Sander 2015-07-15 14:42 ` Donald Pearson 0 siblings, 1 reply; 23+ messages in thread From: Sander @ 2015-07-15 8:00 UTC (permalink / raw) To: linux-btrfs Marc MERLIN wrote (ao): > On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote: > > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup > > as well), local snapshots continue to happen as normal (Eg. daily or > > hourly) and so when your backup media or backup server is finally > > available again, the size of each individual incremental is still the > > same as usual, it just has to perform more of them. > > Good point. My system is not as smart. Every night, it'll make a new > backup and only send one incremental and hope it gets there. It doesn't > make a bunch of incrementals and send multiple. > > The other options do a better job here. FWIW, I've written a bunch of scripts for making backups. The lot has grown over the past years to what is is now. Not very pretty to see, but reliable. The subvolumes backupadmin home root rootvolume and var are snapshotted every hour. Each subvolume has their own entry in crontab for the actual backup. For example rootvolume once a day, home and backupadmin every hour. The scripts uses tar to make a full backup every first backup of a subvolume that month, an incremental daily backup, and an incremental hourly backup if applicable. For a full backup the oldest available snapshot for that month is used, regardless of when the backup is started. This way the backup of each subvolume can be spread not to overload a system. Backups are running in the idle queue to not hinder other processes, are compressed with lbzip2 to utilize all cores, and are encrypted with gpg for obvious reasons. In my tests lbzip2 gives the best size/speed ratio compared to lzop, xz, bzip2, gzip, pxz and lz4(hc). The script outputs what files and directories are in the backup to the backupadmin subvolume. This data is compressed with lz4hc as lz4hc is the fastest to decompress (useful to determine which archive contains what you want restored). Archives get transfered to a remote server by ftp, as ftp is the leanest way of file transfer and supports resume. The initial connection is encrypted to hide username/password, but as the archive is already encrypted, the data channel is not. The ftp transfer is throttled to only use part of the available bandwith. A daily running script checks for archives which are not transfered yet due to remote server not available or failed connection or the like, and retransmits those archives. Snapshots and archives are pruned based on disk usage (yet another script). Restore can be done by hand from snapshots (obviously), or by a script from the locale archive if still available, or the remote archive. The restore script can search a specific date-time range, and checks both local and remote for the availability of an archive that contains the wanted. A bare metal restore can be done by fetching the archives from the remote host and pipe them directly into gpg/tar. No need for additional local storage and no delay. First the monthly full backup is restored, then every daily incremental since, and then every hourly since the youngest daily, if applicable. tar incremental restore is smart, and removes the files and directories that were removed between backups. Sander ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-15 8:00 ` Sander @ 2015-07-15 14:42 ` Donald Pearson 2015-07-15 18:02 ` Donald Pearson 2015-07-15 21:48 ` Marc MERLIN 0 siblings, 2 replies; 23+ messages in thread From: Donald Pearson @ 2015-07-15 14:42 UTC (permalink / raw) To: sander; +Cc: Btrfs BTRFS Implementation question about your scripts Marc.. I've set up some routines for different backup and retention intervals and periods in cron but quickly ran in to stepping on my own toes by the locking mechanism. I could just disable the locking but I'm not sure if that's the best approach and I don't know what it was implemented to prevent in the first place. Thoughts? Thanks, Donald On Wed, Jul 15, 2015 at 3:00 AM, Sander <sander@humilis.net> wrote: > Marc MERLIN wrote (ao): >> On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote: >> > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup >> > as well), local snapshots continue to happen as normal (Eg. daily or >> > hourly) and so when your backup media or backup server is finally >> > available again, the size of each individual incremental is still the >> > same as usual, it just has to perform more of them. >> >> Good point. My system is not as smart. Every night, it'll make a new >> backup and only send one incremental and hope it gets there. It doesn't >> make a bunch of incrementals and send multiple. >> >> The other options do a better job here. > > FWIW, I've written a bunch of scripts for making backups. The lot has > grown over the past years to what is is now. Not very pretty to see, but > reliable. > > The subvolumes backupadmin home root rootvolume and var are snapshotted > every hour. > > Each subvolume has their own entry in crontab for the actual backup. > For example rootvolume once a day, home and backupadmin every hour. > > The scripts uses tar to make a full backup every first backup of a > subvolume that month, an incremental daily backup, and an incremental > hourly backup if applicable. > > For a full backup the oldest available snapshot for that month is used, > regardless of when the backup is started. This way the backup of each > subvolume can be spread not to overload a system. > > Backups are running in the idle queue to not hinder other processes, are > compressed with lbzip2 to utilize all cores, and are encrypted with gpg > for obvious reasons. In my tests lbzip2 gives the best size/speed ratio > compared to lzop, xz, bzip2, gzip, pxz and lz4(hc). > > The script outputs what files and directories are in the backup to the > backupadmin subvolume. This data is compressed with lz4hc as lz4hc is > the fastest to decompress (useful to determine which archive contains > what you want restored). > > Archives get transfered to a remote server by ftp, as ftp is the leanest > way of file transfer and supports resume. The initial connection is > encrypted to hide username/password, but as the archive is already > encrypted, the data channel is not. The ftp transfer is throttled to > only use part of the available bandwith. > > A daily running script checks for archives which are not transfered yet > due to remote server not available or failed connection or the like, and > retransmits those archives. > > Snapshots and archives are pruned based on disk usage (yet another > script). > > Restore can be done by hand from snapshots (obviously), or by a script > from the locale archive if still available, or the remote archive. > > The restore script can search a specific date-time range, and checks > both local and remote for the availability of an archive that contains > the wanted. > > A bare metal restore can be done by fetching the archives from the > remote host and pipe them directly into gpg/tar. No need for additional > local storage and no delay. First the monthly full backup is restored, > then every daily incremental since, and then every hourly since the > youngest daily, if applicable. tar incremental restore is smart, and > removes the files and directories that were removed between backups. > > Sander > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-15 14:42 ` Donald Pearson @ 2015-07-15 18:02 ` Donald Pearson 2015-07-15 21:49 ` Marc MERLIN 2015-07-15 21:48 ` Marc MERLIN 1 sibling, 1 reply; 23+ messages in thread From: Donald Pearson @ 2015-07-15 18:02 UTC (permalink / raw) To: sander; +Cc: Btrfs BTRFS BTW, is anybody else experiencing btrfs-cleaner consuming heavy resources for a very long time when snapshots are removed? Note the TIME on one of these btrfs-cleaner processes. top - 13:01:15 up 21:09, 2 users, load average: 5.30, 4.80, 3.83 Tasks: 315 total, 3 running, 312 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.7 us, 50.2 sy, 0.0 ni, 47.8 id, 1.2 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16431800 total, 177448 free, 1411876 used, 14842476 buff/cache KiB Swap: 8257532 total, 8257316 free, 216 used. 14420732 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4134 root 20 0 0 0 0 R 100.0 0.0 2:41.40 btrfs-cleaner 4183 root 20 0 0 0 0 R 99.7 0.0 191:11.33 btrfs-cleaner On Wed, Jul 15, 2015 at 9:42 AM, Donald Pearson <donaldwhpearson@gmail.com> wrote: > Implementation question about your scripts Marc.. > > I've set up some routines for different backup and retention intervals > and periods in cron but quickly ran in to stepping on my own toes by > the locking mechanism. I could just disable the locking but I'm not > sure if that's the best approach and I don't know what it was > implemented to prevent in the first place. > > Thoughts? > > Thanks, > Donald > > On Wed, Jul 15, 2015 at 3:00 AM, Sander <sander@humilis.net> wrote: >> Marc MERLIN wrote (ao): >>> On Wed, Jul 15, 2015 at 10:03:16AM +1000, Paul Harvey wrote: >>> > The way it works in snazzer (and btrbk and I think also btrfs-sxbackup >>> > as well), local snapshots continue to happen as normal (Eg. daily or >>> > hourly) and so when your backup media or backup server is finally >>> > available again, the size of each individual incremental is still the >>> > same as usual, it just has to perform more of them. >>> >>> Good point. My system is not as smart. Every night, it'll make a new >>> backup and only send one incremental and hope it gets there. It doesn't >>> make a bunch of incrementals and send multiple. >>> >>> The other options do a better job here. >> >> FWIW, I've written a bunch of scripts for making backups. The lot has >> grown over the past years to what is is now. Not very pretty to see, but >> reliable. >> >> The subvolumes backupadmin home root rootvolume and var are snapshotted >> every hour. >> >> Each subvolume has their own entry in crontab for the actual backup. >> For example rootvolume once a day, home and backupadmin every hour. >> >> The scripts uses tar to make a full backup every first backup of a >> subvolume that month, an incremental daily backup, and an incremental >> hourly backup if applicable. >> >> For a full backup the oldest available snapshot for that month is used, >> regardless of when the backup is started. This way the backup of each >> subvolume can be spread not to overload a system. >> >> Backups are running in the idle queue to not hinder other processes, are >> compressed with lbzip2 to utilize all cores, and are encrypted with gpg >> for obvious reasons. In my tests lbzip2 gives the best size/speed ratio >> compared to lzop, xz, bzip2, gzip, pxz and lz4(hc). >> >> The script outputs what files and directories are in the backup to the >> backupadmin subvolume. This data is compressed with lz4hc as lz4hc is >> the fastest to decompress (useful to determine which archive contains >> what you want restored). >> >> Archives get transfered to a remote server by ftp, as ftp is the leanest >> way of file transfer and supports resume. The initial connection is >> encrypted to hide username/password, but as the archive is already >> encrypted, the data channel is not. The ftp transfer is throttled to >> only use part of the available bandwith. >> >> A daily running script checks for archives which are not transfered yet >> due to remote server not available or failed connection or the like, and >> retransmits those archives. >> >> Snapshots and archives are pruned based on disk usage (yet another >> script). >> >> Restore can be done by hand from snapshots (obviously), or by a script >> from the locale archive if still available, or the remote archive. >> >> The restore script can search a specific date-time range, and checks >> both local and remote for the availability of an archive that contains >> the wanted. >> >> A bare metal restore can be done by fetching the archives from the >> remote host and pipe them directly into gpg/tar. No need for additional >> local storage and no delay. First the monthly full backup is restored, >> then every daily incremental since, and then every hourly since the >> youngest daily, if applicable. tar incremental restore is smart, and >> removes the files and directories that were removed between backups. >> >> Sander >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-15 18:02 ` Donald Pearson @ 2015-07-15 21:49 ` Marc MERLIN 2015-07-20 5:15 ` Donald Pearson 0 siblings, 1 reply; 23+ messages in thread From: Marc MERLIN @ 2015-07-15 21:49 UTC (permalink / raw) To: Donald Pearson; +Cc: sander, Btrfs BTRFS On Wed, Jul 15, 2015 at 01:02:29PM -0500, Donald Pearson wrote: > BTW, is anybody else experiencing btrfs-cleaner consuming heavy > resources for a very long time when snapshots are removed? Yes, that's normal. It spends a long time to reclaim blocks and free them, especially if they are on a hard drive and not SSD. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-15 21:49 ` Marc MERLIN @ 2015-07-20 5:15 ` Donald Pearson 2015-07-20 8:28 ` Duncan 0 siblings, 1 reply; 23+ messages in thread From: Donald Pearson @ 2015-07-20 5:15 UTC (permalink / raw) To: Marc MERLIN; +Cc: sander, Btrfs BTRFS I'm starting to think there's something wrong with creating and removing snapshots that leaves btrfs-cleaner either locked up or nearly so. If the btrfs-cleaner process was hard-disk limited I should be seeing some HDD I/O to coincide but I don't. So far btrfs-cleaner is has been using lots of CPU for 1900+ hours and my disk I/O is basically idle. My hourly snaps via cronjob stalled 11 hours ago. Otherwise attempts to read/write to the filesystem appear to be perfectly normal. [root@san01 virtual_machines]# ps aux | grep btrfs-cleaner USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1292 0.0 0.0 0 0 ? S Jul16 0:01 [btrfs-cleaner] root 5796 0.0 0.0 0 0 ? S Jul16 0:00 [btrfs-cleaner] root 6005 21.3 0.0 0 0 ? S Jul16 943:34 [btrfs-cleaner] root 14040 43.3 0.0 0 0 ? R Jul16 1916:05 [btrfs-cleaner] [root@san01 virtual_machines]# ls -lah /run | grep backup -rw-r--r-- 1 root root 11 Jul 20 00:00 backup.home.daily -rw-r--r-- 1 root root 11 Jul 20 00:00 backup.root.daily -rw-r--r-- 1 root root 11 Jul 20 00:01 backup.store.daily -rw-r--r-- 1 root root 11 Jul 19 13:00 backup.store.hourly -rw-r--r-- 1 root root 11 Jul 20 00:01 backup.virtual_machines.daily -rw-r--r-- 1 root root 11 Jul 19 13:00 backup.virtual_machines.hourly [root@san01 virtual_machines]# date Mon Jul 20 00:14:05 CDT 2015 On Wed, Jul 15, 2015 at 4:49 PM, Marc MERLIN <marc@merlins.org> wrote: > On Wed, Jul 15, 2015 at 01:02:29PM -0500, Donald Pearson wrote: >> BTW, is anybody else experiencing btrfs-cleaner consuming heavy >> resources for a very long time when snapshots are removed? > > Yes, that's normal. It spends a long time to reclaim blocks and free > them, especially if they are on a hard drive and not SSD. > > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking > Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-20 5:15 ` Donald Pearson @ 2015-07-20 8:28 ` Duncan 2015-07-20 13:33 ` Donald Pearson 0 siblings, 1 reply; 23+ messages in thread From: Duncan @ 2015-07-20 8:28 UTC (permalink / raw) To: linux-btrfs Donald Pearson posted on Mon, 20 Jul 2015 00:15:26 -0500 as excerpted: > I'm starting to think there's something wrong with creating and removing > snapshots that leaves btrfs-cleaner either locked up or nearly so. If > the btrfs-cleaner process was hard-disk limited I should be seeing some > HDD I/O to coincide but I don't. > > So far btrfs-cleaner is has been using lots of CPU for 1900+ hours and > my disk I/O is basically idle. My hourly snaps via cronjob stalled 11 > hours ago. > > Otherwise attempts to read/write to the filesystem appear to be > perfectly normal. Hourly snaps. How many snapshots/subvolumes on the filesystem? I assume the snap removal you mention is scheduled thinning? A general rule of thumb is under 2000 snapshots per filesystem total, under 1000 if at all reasonable. At 250 snapshots per subvolume, 2000 snapshots is eight subvolumes worth, and 250-ish snapshots per subvolume is well within reason with reasonable thinning. If you have lots of subvolumes, 3000 snapshots per filesystem isn't /too/ bad, but filesystem maintenance including snapshot deletion simply does /not/ scale well, and you'll likely run into scalability issues if you let it reach 10K snapshots on the filesystem. If you're at 10K snapshots on the filesystem, it's quite likely the usual scalability issue's you're seeing, almost certain at 100K+ snapshots. OTOH, if you're under 1K or even 2K snapshots, it's very likely something abnormal. 2K-10K, should be usable in most cases, but there are likely corner-cases where it's bad. Also, FWIW, the btrfs quota subsystem increases snapshot management complexity dramatically, so if you're using that, aim for the low ends of the above recommendation if at all possible, and/or consider either turning off the quota stuff or using a filesystem other than btrfs, as in addition to the scaling issues, the quota management code has been a source of repeated bugs and isn't a feature I'd recommend relying on until it has at least several kernel cycles worth of trouble-free history behind it. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-20 8:28 ` Duncan @ 2015-07-20 13:33 ` Donald Pearson 2015-07-21 9:29 ` Duncan 0 siblings, 1 reply; 23+ messages in thread From: Donald Pearson @ 2015-07-20 13:33 UTC (permalink / raw) To: Duncan; +Cc: Btrfs BTRFS On Mon, Jul 20, 2015 at 3:28 AM, Duncan <1i5t5.duncan@cox.net> wrote: > Donald Pearson posted on Mon, 20 Jul 2015 00:15:26 -0500 as excerpted: > >> I'm starting to think there's something wrong with creating and removing >> snapshots that leaves btrfs-cleaner either locked up or nearly so. If >> the btrfs-cleaner process was hard-disk limited I should be seeing some >> HDD I/O to coincide but I don't. >> >> So far btrfs-cleaner is has been using lots of CPU for 1900+ hours and >> my disk I/O is basically idle. My hourly snaps via cronjob stalled 11 >> hours ago. >> >> Otherwise attempts to read/write to the filesystem appear to be >> perfectly normal. > > Hourly snaps. How many snapshots/subvolumes on the filesystem? I assume > the snap removal you mention is scheduled thinning? I'm using Marc's scripts currently which handles the creation and retention. Including snapshots there are currently 94 subvolumes. The maximum possible with my settings is 118 after 3 months. > > A general rule of thumb is under 2000 snapshots per filesystem total, > under 1000 if at all reasonable. At 250 snapshots per subvolume, 2000 > snapshots is eight subvolumes worth, and 250-ish snapshots per subvolume > is well within reason with reasonable thinning. > > If you have lots of subvolumes, 3000 snapshots per filesystem isn't /too/ > bad, but filesystem maintenance including snapshot deletion simply does > /not/ scale well, and you'll likely run into scalability issues if you > let it reach 10K snapshots on the filesystem. > > If you're at 10K snapshots on the filesystem, it's quite likely the usual > scalability issue's you're seeing, almost certain at 100K+ snapshots. > OTOH, if you're under 1K or even 2K snapshots, it's very likely something > abnormal. 2K-10K, should be usable in most cases, but there are likely > corner-cases where it's bad. > > Also, FWIW, the btrfs quota subsystem increases snapshot management > complexity dramatically, so if you're using that, aim for the low ends of > the above recommendation if at all possible, and/or consider either > turning off the quota stuff or using a filesystem other than btrfs, as in > addition to the scaling issues, the quota management code has been a > source of repeated bugs and isn't a feature I'd recommend relying on > until it has at least several kernel cycles worth of trouble-free history > behind it. Thanks for the insight. I just took a look at dmesg and found this. Is this coincidental or is this maybe the reason things appear to be stuck? I'm not sure how to read this. [195400.023165] ------------[ cut here ]------------ [195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028 __qgroup_excl_accounting+0x1dc/0x270 [btrfs]() [195400.023201] Modules linked in: ext4(E) mbcache(E) jbd2(E) ppdev(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_intel(E) snd_hda_controller(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) snd_seq(E) snd_seq_device(E) kvm_amd(E) kvm(E) snd_pcm(E) pcspkr(E) serio_raw(E) k10temp(E) edac_mce_amd(E) edac_core(E) snd_timer(E) snd(E) soundcore(E) sp5100_tco(E) i2c_piix4(E) ses(E) enclosure(E) 8250_fintek(E) parport_pc(E) parport(E) tpm_infineon(E) shpchp(E) acpi_cpufreq(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) sunrpc(E) btrfs(E) xor(E) raid6_pq(E) ata_generic(E) pata_acpi(E) sd_mod(E) nouveau(E) video(E) mxm_wmi(E) i2c_algo_bit(E) drm_kms_helper(E) ttm(E) drm(E) pata_atiixp(E) ahci(E) pata_jmicron(E) libahci(E) lpfc(E) scsi_transport_fc(E) firewire_ohci(E) libata(E) firewire_core(E) [195400.023225] crc_itu_t(E) r8169(E) mii(E) mpt2sas(E) raid_class(E) scsi_transport_sas(E) wmi(E) [195400.023231] CPU: 2 PID: 16987 Comm: kworker/u12:5 Tainted: G E 4.1.0-1.el7.elrepo.x86_64 #1 [195400.023232] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD MS-7577/790FX-GD70(MS-7577), BIOS V1.16 12/01/2010 [195400.023244] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [195400.023246] 0000000000000000 0000000031fd5f58 ffff880368a03b68 ffffffff816d4638 [195400.023248] 0000000000000000 0000000000000000 ffff880368a03ba8 ffffffff8107c51a [195400.023250] c000000000000000 ffff880409018d60 ffff88040a28ec88 ffffffffffff9000 [195400.023252] Call Trace: [195400.023257] [<ffffffff816d4638>] dump_stack+0x45/0x57 [195400.023261] [<ffffffff8107c51a>] warn_slowpath_common+0x8a/0xc0 [195400.023263] [<ffffffff8107c64a>] warn_slowpath_null+0x1a/0x20 [195400.023274] [<ffffffffa05b5bdc>] __qgroup_excl_accounting+0x1dc/0x270 [btrfs] [195400.023277] [<ffffffff811dd8b9>] ? kmem_cache_alloc_trace+0x199/0x220 [195400.023289] [<ffffffffa05b8e37>] btrfs_delayed_qgroup_accounting+0x317/0xc60 [btrfs] [195400.023291] [<ffffffff810afd68>] ? __enqueue_entity+0x78/0x80 [195400.023293] [<ffffffff811dd699>] ? kmem_cache_alloc+0x1a9/0x230 [195400.023302] [<ffffffffa053cb0a>] btrfs_run_delayed_refs.part.66+0x20a/0x270 [btrfs] [195400.023311] [<ffffffffa053cc18>] delayed_ref_async_start+0x88/0xa0 [btrfs] [195400.023322] [<ffffffffa0580562>] normal_work_helper+0xc2/0x280 [btrfs] [195400.023332] [<ffffffffa0580952>] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [195400.023335] [<ffffffff810959cd>] process_one_work+0x14d/0x420 [195400.023337] [<ffffffff81096192>] worker_thread+0x112/0x520 [195400.023339] [<ffffffff81096080>] ? rescuer_thread+0x3e0/0x3e0 [195400.023341] [<ffffffff8109bee8>] kthread+0xd8/0xf0 [195400.023342] [<ffffffff8109be10>] ? kthread_create_on_node+0x1b0/0x1b0 [195400.023344] [<ffffffff816dc3a2>] ret_from_fork+0x42/0x70 [195400.023346] [<ffffffff8109be10>] ? kthread_create_on_node+0x1b0/0x1b0 [195400.023347] ---[ end trace 1abc27647fe906a5 ]--- > > -- > Duncan - List replies preferred. No HTML msgs. > "Every nonfree program has a lord, a master -- > and if you use the program, he is your master." Richard Stallman > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-20 13:33 ` Donald Pearson @ 2015-07-21 9:29 ` Duncan 2015-07-22 1:29 ` Donald Pearson 0 siblings, 1 reply; 23+ messages in thread From: Duncan @ 2015-07-21 9:29 UTC (permalink / raw) To: linux-btrfs Donald Pearson posted on Mon, 20 Jul 2015 08:33:47 -0500 as excerpted: >> Also, FWIW, the btrfs quota subsystem increases snapshot management >> complexity dramatically, so if you're using that, aim for the low ends >> of the above recommendation if at all possible, and/or consider either >> turning off the quota stuff or using a filesystem other than btrfs, as >> in addition to the scaling issues, the quota management code has been a >> source of repeated bugs and isn't a feature I'd recommend relying on >> until it has at least several kernel cycles worth of trouble-free >> history behind it. > > Thanks for the insight. I just took a look at dmesg and found this. > Is this coincidental or is this maybe the reason things appear to be > stuck? I'm not sure how to read this. > > [195400.023165] ------------[ cut here ]------------ > [195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028 > __qgroup_excl_accounting+0x1dc/0x270 [btrfs]() I don't know. I'm not a btrfs quota feature user. What I do know is that there has been... again... more quota code patches recently, fixing what sound like serious problems. You already have my recommendation above; unless you are actively testing the btrfs quota code, if you don't need it, don't use it; if you do need it, use something where it's actually working well enough to /rely/ on. But in fairness that's potentially a negatively biased view, since as I'm on the list but not actually using that feature I'm seeing the bug reports without much in the way of knowing how common they actually are. If it's a big deal to turn quotas off, I'd say wait and see if the dev actually working on it cares to comment with his opinion of quota feature stability in general, and perhaps of that specific trace, before deciding for sure. But if you have the inclination and don't really need quotas, and assuming disabling them isn't /too/ big a hassle, it might be worthwhile disabling the feature and seeing how things go. I can't see how not having to deal with quotas could /hurt/ scaling, and with luck, it'll improve things noticeably. FWIW, the developer post where the effect of quotas on scaling was explained was actually in the context of snapshot-aware-defrag, which has been disabled for awhile now, /because/ of the scaling issues (it was so bad the defrag would basically hang, taking hours to defrag a single moderately sized file as it tracked it thru the various snapshots, so processing time for a full filesystem could easily be weeks and could in some cases be months, obviously well outside any practically workable range!), and while quota processing was explained to be horrible in that context at that time (I read it as at least doubling the necessary work), there has been at least one quota-code rewrite since then, as well as additional work, and it may actually not be nearly as bad any more, barring a direct bug, of course. But I don't know as I've not seen any direct statements or numbers either way, only the bug reports and patches going by. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-21 9:29 ` Duncan @ 2015-07-22 1:29 ` Donald Pearson 2015-07-25 13:27 ` Donald Pearson 0 siblings, 1 reply; 23+ messages in thread From: Donald Pearson @ 2015-07-22 1:29 UTC (permalink / raw) To: Duncan; +Cc: Btrfs BTRFS Thanks for the feedback Duncan. It doesn't appear to be a big deal to disable quotas so that's what I'll do for now. On Tue, Jul 21, 2015 at 4:29 AM, Duncan <1i5t5.duncan@cox.net> wrote: > Donald Pearson posted on Mon, 20 Jul 2015 08:33:47 -0500 as excerpted: > >>> Also, FWIW, the btrfs quota subsystem increases snapshot management >>> complexity dramatically, so if you're using that, aim for the low ends >>> of the above recommendation if at all possible, and/or consider either >>> turning off the quota stuff or using a filesystem other than btrfs, as >>> in addition to the scaling issues, the quota management code has been a >>> source of repeated bugs and isn't a feature I'd recommend relying on >>> until it has at least several kernel cycles worth of trouble-free >>> history behind it. >> >> Thanks for the insight. I just took a look at dmesg and found this. >> Is this coincidental or is this maybe the reason things appear to be >> stuck? I'm not sure how to read this. >> >> [195400.023165] ------------[ cut here ]------------ >> [195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028 >> __qgroup_excl_accounting+0x1dc/0x270 [btrfs]() > > I don't know. I'm not a btrfs quota feature user. > > What I do know is that there has been... again... more quota code patches > recently, fixing what sound like serious problems. > > You already have my recommendation above; unless you are actively testing > the btrfs quota code, if you don't need it, don't use it; if you do need > it, use something where it's actually working well enough to /rely/ on. > > But in fairness that's potentially a negatively biased view, since as I'm > on the list but not actually using that feature I'm seeing the bug > reports without much in the way of knowing how common they actually are. > If it's a big deal to turn quotas off, I'd say wait and see if the dev > actually working on it cares to comment with his opinion of quota feature > stability in general, and perhaps of that specific trace, before deciding > for sure. > > But if you have the inclination and don't really need quotas, and > assuming disabling them isn't /too/ big a hassle, it might be worthwhile > disabling the feature and seeing how things go. I can't see how not > having to deal with quotas could /hurt/ scaling, and with luck, it'll > improve things noticeably. > > FWIW, the developer post where the effect of quotas on scaling was > explained was actually in the context of snapshot-aware-defrag, which has > been disabled for awhile now, /because/ of the scaling issues (it was so > bad the defrag would basically hang, taking hours to defrag a single > moderately sized file as it tracked it thru the various snapshots, so > processing time for a full filesystem could easily be weeks and could in > some cases be months, obviously well outside any practically workable > range!), and while quota processing was explained to be horrible in that > context at that time (I read it as at least doubling the necessary work), > there has been at least one quota-code rewrite since then, as well as > additional work, and it may actually not be nearly as bad any more, > barring a direct bug, of course. But I don't know as I've not seen any > direct statements or numbers either way, only the bug reports and patches > going by. > > -- > Duncan - List replies preferred. No HTML msgs. > "Every nonfree program has a lord, a master -- > and if you use the program, he is your master." Richard Stallman > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-22 1:29 ` Donald Pearson @ 2015-07-25 13:27 ` Donald Pearson 2015-07-26 5:39 ` Duncan 0 siblings, 1 reply; 23+ messages in thread From: Donald Pearson @ 2015-07-25 13:27 UTC (permalink / raw) To: Duncan; +Cc: Btrfs BTRFS I can confirm that getting rid of the quotas fixed the issue for me. Just disabling quotas wasn't enough, I had to enable, delete all qgroups, reboot because disable was hung on one of the filesystems, then disable quotas. Now when btrfs-cleaner runs it doesn't completely consume a core, I can see corresponding disk i/o, and the process goes away after a reasonable amount of time. On Tue, Jul 21, 2015 at 8:29 PM, Donald Pearson <donaldwhpearson@gmail.com> wrote: > Thanks for the feedback Duncan. > > It doesn't appear to be a big deal to disable quotas so that's what > I'll do for now. > > On Tue, Jul 21, 2015 at 4:29 AM, Duncan <1i5t5.duncan@cox.net> wrote: >> Donald Pearson posted on Mon, 20 Jul 2015 08:33:47 -0500 as excerpted: >> >>>> Also, FWIW, the btrfs quota subsystem increases snapshot management >>>> complexity dramatically, so if you're using that, aim for the low ends >>>> of the above recommendation if at all possible, and/or consider either >>>> turning off the quota stuff or using a filesystem other than btrfs, as >>>> in addition to the scaling issues, the quota management code has been a >>>> source of repeated bugs and isn't a feature I'd recommend relying on >>>> until it has at least several kernel cycles worth of trouble-free >>>> history behind it. >>> >>> Thanks for the insight. I just took a look at dmesg and found this. >>> Is this coincidental or is this maybe the reason things appear to be >>> stuck? I'm not sure how to read this. >>> >>> [195400.023165] ------------[ cut here ]------------ >>> [195400.023199] WARNING: CPU: 2 PID: 16987 at fs/btrfs/qgroup.c:1028 >>> __qgroup_excl_accounting+0x1dc/0x270 [btrfs]() >> >> I don't know. I'm not a btrfs quota feature user. >> >> What I do know is that there has been... again... more quota code patches >> recently, fixing what sound like serious problems. >> >> You already have my recommendation above; unless you are actively testing >> the btrfs quota code, if you don't need it, don't use it; if you do need >> it, use something where it's actually working well enough to /rely/ on. >> >> But in fairness that's potentially a negatively biased view, since as I'm >> on the list but not actually using that feature I'm seeing the bug >> reports without much in the way of knowing how common they actually are. >> If it's a big deal to turn quotas off, I'd say wait and see if the dev >> actually working on it cares to comment with his opinion of quota feature >> stability in general, and perhaps of that specific trace, before deciding >> for sure. >> >> But if you have the inclination and don't really need quotas, and >> assuming disabling them isn't /too/ big a hassle, it might be worthwhile >> disabling the feature and seeing how things go. I can't see how not >> having to deal with quotas could /hurt/ scaling, and with luck, it'll >> improve things noticeably. >> >> FWIW, the developer post where the effect of quotas on scaling was >> explained was actually in the context of snapshot-aware-defrag, which has >> been disabled for awhile now, /because/ of the scaling issues (it was so >> bad the defrag would basically hang, taking hours to defrag a single >> moderately sized file as it tracked it thru the various snapshots, so >> processing time for a full filesystem could easily be weeks and could in >> some cases be months, obviously well outside any practically workable >> range!), and while quota processing was explained to be horrible in that >> context at that time (I read it as at least doubling the necessary work), >> there has been at least one quota-code rewrite since then, as well as >> additional work, and it may actually not be nearly as bad any more, >> barring a direct bug, of course. But I don't know as I've not seen any >> direct statements or numbers either way, only the bug reports and patches >> going by. >> >> -- >> Duncan - List replies preferred. No HTML msgs. >> "Every nonfree program has a lord, a master -- >> and if you use the program, he is your master." Richard Stallman >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-25 13:27 ` Donald Pearson @ 2015-07-26 5:39 ` Duncan 0 siblings, 0 replies; 23+ messages in thread From: Duncan @ 2015-07-26 5:39 UTC (permalink / raw) To: linux-btrfs Donald Pearson posted on Sat, 25 Jul 2015 08:27:03 -0500 as excerpted: [Duncan wrote...] >>>>> Also, FWIW, the btrfs quota subsystem increases snapshot management >>>>> complexity dramatically, so if you're using that, aim for the low >>>>> ends of the above recommendation if at all possible, and/or consider >>>>> either turning off the quota stuff or using a filesystem other than >>>>> btrfs, as in addition to the scaling issues, the quota management >>>>> code has been a source of repeated bugs and isn't a feature I'd >>>>> recommend relying on until it has at least several kernel cycles >>>>> worth of trouble-free history behind it. [Big intervening thread snip...] > I can confirm that getting rid of the quotas fixed the issue for me. > Just disabling quotas wasn't enough, I had to enable, delete all > qgroups, reboot because disable was hung on one of the filesystems, > then disable quotas. Now when btrfs-cleaner runs it doesn't completely > consume a core, I can see corresponding disk i/o, and the process goes > away after a reasonable amount of time. Thanks for the confirmation! Good to have a recent and concrete report to point to, on the subject of btrfs and quotas, now. Now I can at least say something like "we do have at least one report (4.1 timeframe) of snapshot deletion scaling or lockup issues where quotas was implicated -- after deleting qgroups and disabling quotas, the problem disappeared." -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Anyone tried out btrbk yet? 2015-07-15 14:42 ` Donald Pearson 2015-07-15 18:02 ` Donald Pearson @ 2015-07-15 21:48 ` Marc MERLIN 1 sibling, 0 replies; 23+ messages in thread From: Marc MERLIN @ 2015-07-15 21:48 UTC (permalink / raw) To: Donald Pearson; +Cc: sander, Btrfs BTRFS On Wed, Jul 15, 2015 at 09:42:28AM -0500, Donald Pearson wrote: > Implementation question about your scripts Marc.. make sure you Cc me then, I could have missed that Email :) > I've set up some routines for different backup and retention intervals > and periods in cron but quickly ran in to stepping on my own toes by > the locking mechanism. I could just disable the locking but I'm not > sure if that's the best approach and I don't know what it was > implemented to prevent in the first place. Try --postfix servername it'll add the destination server in the snapshot rotation and the lockfile. Otherwise, you can just trivially modify the script to take --lock as an argument, or you can even ln -s btrfs-subvolume-backup btrfs-subvolume-backupserver2 and the script will automatically make a /var/run/btrfs-subvolume-backupserver2 as a lockfile. Hope this helps. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2015-07-26 5:39 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-07-09 12:26 Anyone tried out btrbk yet? Martin Steigerwald 2015-07-09 17:12 ` Henri Valta 2015-07-09 17:17 ` Marc MERLIN 2015-07-10 1:35 ` Donald Pearson 2015-07-10 1:38 ` Donald Pearson 2015-07-10 4:02 ` Paul Harvey 2015-07-10 10:46 ` Axel Burri 2015-07-12 3:55 ` Marc MERLIN 2015-07-13 0:42 ` Marc MERLIN 2015-07-15 0:03 ` Paul Harvey 2015-07-15 3:14 ` Marc MERLIN 2015-07-15 8:00 ` Sander 2015-07-15 14:42 ` Donald Pearson 2015-07-15 18:02 ` Donald Pearson 2015-07-15 21:49 ` Marc MERLIN 2015-07-20 5:15 ` Donald Pearson 2015-07-20 8:28 ` Duncan 2015-07-20 13:33 ` Donald Pearson 2015-07-21 9:29 ` Duncan 2015-07-22 1:29 ` Donald Pearson 2015-07-25 13:27 ` Donald Pearson 2015-07-26 5:39 ` Duncan 2015-07-15 21:48 ` Marc MERLIN
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.