btrfs-subv-backup v0.1b

btrfs-subv-backup v0.1b

[Austin S. Hemmelgarn]

As previously mentioned on the list, I've written up a script to back up 
BTRFS subvolume structures in regular file-based backups.  As of right 
now, it's still a bit rough around the edges, but it's cleaned up enough 
that I consider it of at least beta quality, and therefore fit for more 
general testing (and possibly usage).   Aside from the issues and 
limitations listed below in the README, error checking is still somewhat 
lacking, and I plan to remedy that in the near future.

The script itself can be found here:
https://github.com/Ferroin/btrfs-subv-backup

Here's the official README, as that already covers pretty much 
everything else I would put in this e-mail:

# btrfs-subv-backup v0.1b
btrfs-subv-backup is a tool for recording the layout of subvolumes on
a mounted BTRFS filesystem in a way that can be stored in a regular
file-based backup (for example, using tar).  It originated out of a lack
of such existing tools, and is intended to be as portable as reasonably
possible.  As a result, it depends on nothing beyond a working
installation of Python version 3.4 or higher.

btrfs-subv-backup is licensed under a BSD-style license, check the
LICENSE file or the docstring for more information.

### Usage
Usage is extremely simple.  To generate a backup of a given mount
point, run:

`btrfs-subv-backup.py /path`

This will create a file called `.btrfs-subv-backup.json` in the root of
the mount point, make sure that gets included in any backups you run of
the mount point.

To restore the subvolumes in a filesystem after you've extracted a backup
of the mount point, run:

`btrfs-subv-backup.py --restore /path`

This will recreate the subvolume structure.

If you need to manually recreate the subvolumes, you can find a list
of them in the aforementioned JSON file under the 'subvolumes' key (the
other keys store info about the filesystem itself to make it easier to
figure out what it was).

### Limitations and Known Issues
* We don't store information about reflinks.  This means in particular
that snapshot relationships __ARE NOT__ saved.  There is currently no
way to store this data reliably short of a block-level backup, which
has it's own special issues.
* Subvolumes with spaces in their name are not supported.
* There is currently no indication of progress.
* The restoration process may take a long time, and may use a very large
amount of disk space.  Ideally this should be fixed, but I'm not sure
of any way to do so while keeping things on-line.
* The current restoration process is all-or-nothing, things are not
correctly handled if some of the subvolumes have already been restored
(although btrfs-subv-backup will clean up the temporary subvolumes it
creates during the restore if the restore fails).  This should be pretty
easy to fix, I just haven't gotten around to it yet.
* btrfs-subv-backup won't cross actual mount points, which means it
won't recurse into explicitly mounted subvolumes.  This makes usage a
bit more complicated on some distributions (such as SLES and OpenSUSE),
but greatly simplifies the code.

Re: btrfs-subv-backup v0.1b

[Marat Khalili]

Hello Austin,

Looks very useful. Two questions:

1. Can you release it under some standard license recognized by github, 
in case someone wants to include it in other projects? AGPL-3.0 would be 
nice.

2. I don't understand mentioned restore performance issues. It shouldn't 
apply if data is restored _after_ subvolume structure is re-created, but 
even if (1) data is already there, and (2) copyless move doesn't work 
between subvolumes (really a limitation of some older systems, not 
Python), there's a known workaround of creating a reflink and then 
removing the original.

--

With Best Regards,
Marat Khalili

Re: btrfs-subv-backup v0.1b

[Austin S. Hemmelgarn]

On 2017-10-26 11:25, Marat Khalili wrote:
> Hello Austin,
> 
> Looks very useful. Two questions:
> 
> 1. Can you release it under some standard license recognized by github, 
> in case someone wants to include it in other projects? AGPL-3.0 would be 
> nice.
The intent is for it to be under what Github calls the BSD 'new' 
3-clause license, I'll attempt to get it updated so that Github 
recognizes it properly shortly.  It's most likely not matching correctly 
due to the differences between the 'official' version of the license 
text, and the version I pulled out of the Gentoo portage tree on my 
local system (in particular, the portage copy uses numbers instead of 
bullet points, and assumes the project creators will be referenced 
directly in the third clause like in the original BSD licenses, instead 
of in an abstract manner like the text that Github uses).
> 
> 2. I don't understand mentioned restore performance issues. It shouldn't 
> apply if data is restored _after_ subvolume structure is re-created, but 
> even if (1) data is already there, and (2) copyless move doesn't work 
> between subvolumes (really a limitation of some older systems, not 
> Python), there's a known workaround of creating a reflink and then 
> removing the original.
As of right now, if the data is there, it will use the shutil.copytree() 
function to copy it.  This is roughly equivalent to calling `cp -a` on 
the directory in a shell, so it's potentially very slow compared to what 
it could be, and will temporarily duplicate data on-disk.  I hope to 
have it using reflinks eventually, but for the time being, I wanted to 
get something working out there so that people can use it, and then 
worry about improving performance, and I'm still not 100% confident 
about mucking around with ioctls from Python.  I'll get the README 
updated to clarify that the performance issues are only present when 
recreating subvolumes after the data has been restored.

As far as restoring subvolume structure first, that will work too, and I 
should probably mention that in the README file, I just didn't see that 
as being the most likely case (in the backup software I've dealt with, 
it's simpler to just extract everything and run a script afterwards than 
to extract one file, run a script, and then extract the rest).

Re: btrfs-subv-backup v0.1b

[Lentes, Bernd]

----- Am 26. Okt 2017 um 15:32 schrieb Austin S. Hemmelgarn ahferroin7@gmail.com:

> As previously mentioned on the list, I've written up a script to back up
> BTRFS subvolume structures in regular file-based backups.  As of right
> now, it's still a bit rough around the edges, but it's cleaned up enough
> that I consider it of at least beta quality, and therefore fit for more
> general testing (and possibly usage).   Aside from the issues and
> limitations listed below in the README, error checking is still somewhat
> lacking, and I plan to remedy that in the near future.
> 
> The script itself can be found here:
> https://github.com/Ferroin/btrfs-subv-backup
> 
> Here's the official README, as that already covers pretty much
> everything else I would put in this e-mail:
> 
> # btrfs-subv-backup v0.1b
> btrfs-subv-backup is a tool for recording the layout of subvolumes on
> a mounted BTRFS filesystem in a way that can be stored in a regular
> file-based backup (for example, using tar).  It originated out of a lack
> of such existing tools, and is intended to be as portable as reasonably
> possible.  As a result, it depends on nothing beyond a working
> installation of Python version 3.4 or higher.
> 
> btrfs-subv-backup is licensed under a BSD-style license, check the
> LICENSE file or the docstring for more information.
> 
> ### Usage
> Usage is extremely simple.  To generate a backup of a given mount
> point, run:
> 
> `btrfs-subv-backup.py /path`
> 
> This will create a file called `.btrfs-subv-backup.json` in the root of
> the mount point, make sure that gets included in any backups you run of
> the mount point.
> 
> To restore the subvolumes in a filesystem after you've extracted a backup
> of the mount point, run:
> 
> `btrfs-subv-backup.py --restore /path`
> 
> This will recreate the subvolume structure.
> 
> If you need to manually recreate the subvolumes, you can find a list
> of them in the aforementioned JSON file under the 'subvolumes' key (the
> other keys store info about the filesystem itself to make it easier to
> figure out what it was).
> 
> ### Limitations and Known Issues
> * We don't store information about reflinks.  This means in particular
> that snapshot relationships __ARE NOT__ saved.  There is currently no
> way to store this data reliably short of a block-level backup, which
> has it's own special issues.
> * Subvolumes with spaces in their name are not supported.
> * There is currently no indication of progress.
> * The restoration process may take a long time, and may use a very large
> amount of disk space.  Ideally this should be fixed, but I'm not sure
> of any way to do so while keeping things on-line.
> * The current restoration process is all-or-nothing, things are not
> correctly handled if some of the subvolumes have already been restored
> (although btrfs-subv-backup will clean up the temporary subvolumes it
> creates during the restore if the restore fails).  This should be pretty
> easy to fix, I just haven't gotten around to it yet.
> * btrfs-subv-backup won't cross actual mount points, which means it
> won't recurse into explicitly mounted subvolumes.  This makes usage a
> bit more complicated on some distributions (such as SLES and OpenSUSE),
> but greatly simplifies the code.

Hi Austin,

thanks for your effort. What are the minimum prerequesties for kernel and btrfsprogs for that script ?
Do you think it will run on my old SLES 11 SP4 ?

ha-idg-1:~ # rpm -qa|grep -i btrfs
btrfsprogs-3.18.2-0.40.48
btrfsmaintenance-0.1-3.1
libbtrfs0-3.18.2-0.40.48

ha-idg-1:~ # uname -a
Linux ha-idg-1 3.0.101-100-default #1 SMP Tue Apr 18 22:46:01 UTC 2017 (ce95309) x86_64 x86_64 x86_64 GNU/Linux

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

Re: btrfs-subv-backup v0.1b

[Austin S. Hemmelgarn]

On 2017-10-31 08:40, Lentes, Bernd wrote:
> 
> 
> ----- Am 26. Okt 2017 um 15:32 schrieb Austin S. Hemmelgarn ahferroin7@gmail.com:
> 
>> As previously mentioned on the list, I've written up a script to back up
>> BTRFS subvolume structures in regular file-based backups.  As of right
>> now, it's still a bit rough around the edges, but it's cleaned up enough
>> that I consider it of at least beta quality, and therefore fit for more
>> general testing (and possibly usage).   Aside from the issues and
>> limitations listed below in the README, error checking is still somewhat
>> lacking, and I plan to remedy that in the near future.
>>
>> The script itself can be found here:
>> https://github.com/Ferroin/btrfs-subv-backup
>>
>> Here's the official README, as that already covers pretty much
>> everything else I would put in this e-mail:
>>
>> # btrfs-subv-backup v0.1b
>> btrfs-subv-backup is a tool for recording the layout of subvolumes on
>> a mounted BTRFS filesystem in a way that can be stored in a regular
>> file-based backup (for example, using tar).  It originated out of a lack
>> of such existing tools, and is intended to be as portable as reasonably
>> possible.  As a result, it depends on nothing beyond a working
>> installation of Python version 3.4 or higher.
>>
>> btrfs-subv-backup is licensed under a BSD-style license, check the
>> LICENSE file or the docstring for more information.
>>
>> ### Usage
>> Usage is extremely simple.  To generate a backup of a given mount
>> point, run:
>>
>> `btrfs-subv-backup.py /path`
>>
>> This will create a file called `.btrfs-subv-backup.json` in the root of
>> the mount point, make sure that gets included in any backups you run of
>> the mount point.
>>
>> To restore the subvolumes in a filesystem after you've extracted a backup
>> of the mount point, run:
>>
>> `btrfs-subv-backup.py --restore /path`
>>
>> This will recreate the subvolume structure.
>>
>> If you need to manually recreate the subvolumes, you can find a list
>> of them in the aforementioned JSON file under the 'subvolumes' key (the
>> other keys store info about the filesystem itself to make it easier to
>> figure out what it was).
>>
>> ### Limitations and Known Issues
>> * We don't store information about reflinks.  This means in particular
>> that snapshot relationships __ARE NOT__ saved.  There is currently no
>> way to store this data reliably short of a block-level backup, which
>> has it's own special issues.
>> * Subvolumes with spaces in their name are not supported.
>> * There is currently no indication of progress.
>> * The restoration process may take a long time, and may use a very large
>> amount of disk space.  Ideally this should be fixed, but I'm not sure
>> of any way to do so while keeping things on-line.
>> * The current restoration process is all-or-nothing, things are not
>> correctly handled if some of the subvolumes have already been restored
>> (although btrfs-subv-backup will clean up the temporary subvolumes it
>> creates during the restore if the restore fails).  This should be pretty
>> easy to fix, I just haven't gotten around to it yet.
>> * btrfs-subv-backup won't cross actual mount points, which means it
>> won't recurse into explicitly mounted subvolumes.  This makes usage a
>> bit more complicated on some distributions (such as SLES and OpenSUSE),
>> but greatly simplifies the code.
> 
> Hi Austin,
> 
> thanks for your effort. What are the minimum prerequesties for kernel and btrfsprogs for that script ?
> Do you think it will run on my old SLES 11 SP4 ?
Dependency-wise, it needs:

kernel: Should work with any kernel version that has BTRFS support, 
untested on kernels before 4.10, but I'm fairly confident that it should 
work on just about anything.  If there's going to be a failure due to 
kernel version, it should happen when saving the subvolume information, 
so you should be safe checking this (especially since the script doesn't 
write anything when saving the info until the very end, and doesn't use 
any write capable ioctls when fetching the info).

btrfs-progs: Only uses the subvolume list, create, and delete commands, 
and the only one which may have changed significantly in the past few 
years is the list command.  I've only tested it with btrfs-progs 4.10.2 
and newer, but I expect it should work with any of them at least since 
the change to matching kernel versions.  You can quickly check this by 
making sure the output of `btrfs subvolume list` looks like this:

     ID 257 gen 769304 top level 5 path root

What matters there is the number of words between each number, as the 
parsing code is pretty naive.

util-linux: We use blkid to get the filesystem label and UUID so that 
it's more evident in the file what filesystem it's for (in case you 
decide to stroe them separately), but as far as I know the relevant 
command-line options haven't changed in at least half a decade, so any 
reasonably recent version should work fine (tested with util-linux 
2.31).  As the script is currently, this will cause it to throw an 
exception if it doesn't work, but that will happen before it does almost 
anything else, so it should still fail safe.

Python: btrfs-subv-backup requires Python 3.  I've only tested it on 3.4 
and 3.6, but I'm pretty sure it should work on most versions back to at 
least 3.2.  Of the dependencies, this is the one I'd expect to be the 
most likely possible issue on older distros, but you likely have a new 
enough version already given the kernel build date shown below.  If this 
doesn't work, you will get syntax errors (specifically around the error 
handlers).

I've been slowly updating things to improve the script too, I'll make 
sure to get this info into the README before I post the most recent version.
> 
> ha-idg-1:~ # rpm -qa|grep -i btrfs
> btrfsprogs-3.18.2-0.40.48
> btrfsmaintenance-0.1-3.1
> libbtrfs0-3.18.2-0.40.48
> 
> ha-idg-1:~ # uname -a
> Linux ha-idg-1 3.0.101-100-default #1 SMP Tue Apr 18 22:46:01 UTC 2017 (ce95309) x86_64 x86_64 x86_64 GNU/Linux
>

Re: btrfs-subv-backup v0.1b

[Lentes, Bernd]

----- On Oct 31, 2017, at 2:59 PM, Austin S. Hemmelgarn ahferroin7@gmail.com wrote:

>> Hi Austin,
>> 
>> thanks for your effort. What are the minimum prerequesties for kernel and
>> btrfsprogs for that script ?
>> Do you think it will run on my old SLES 11 SP4 ?
> Dependency-wise, it needs:
> 
> kernel: Should work with any kernel version that has BTRFS support,
> untested on kernels before 4.10, but I'm fairly confident that it should
> work on just about anything.  If there's going to be a failure due to
> kernel version, it should happen when saving the subvolume information,
> so you should be safe checking this (especially since the script doesn't
> write anything when saving the info until the very end, and doesn't use
> any write capable ioctls when fetching the info).
> 
> btrfs-progs: Only uses the subvolume list, create, and delete commands,
> and the only one which may have changed significantly in the past few
> years is the list command.  I've only tested it with btrfs-progs 4.10.2
> and newer, but I expect it should work with any of them at least since
> the change to matching kernel versions.  You can quickly check this by
> making sure the output of `btrfs subvolume list` looks like this:
> 
>     ID 257 gen 769304 top level 5 path root
> 
> What matters there is the number of words between each number, as the
> parsing code is pretty naive.
> 
> util-linux: We use blkid to get the filesystem label and UUID so that
> it's more evident in the file what filesystem it's for (in case you
> decide to stroe them separately), but as far as I know the relevant
> command-line options haven't changed in at least half a decade, so any
> reasonably recent version should work fine (tested with util-linux
> 2.31).  As the script is currently, this will cause it to throw an
> exception if it doesn't work, but that will happen before it does almost
> anything else, so it should still fail safe.
> 
> Python: btrfs-subv-backup requires Python 3.  I've only tested it on 3.4
> and 3.6, but I'm pretty sure it should work on most versions back to at
> least 3.2.  Of the dependencies, this is the one I'd expect to be the
> most likely possible issue on older distros, but you likely have a new
> enough version already given the kernel build date shown below.  If this
> doesn't work, you will get syntax errors (specifically around the error
> handlers).
> 

damned, i have 2.6.9 and the most recent i fnd in repositories for SLES 11 SP4 is 2.7.6

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

Re: btrfs-subv-backup v0.1b

[Austin S. Hemmelgarn]

On 2017-10-31 12:54, Lentes, Bernd wrote:
> ----- On Oct 31, 2017, at 2:59 PM, Austin S. Hemmelgarn ahferroin7@gmail.com wrote:
> 
>>> Hi Austin,
>>>
>>> thanks for your effort. What are the minimum prerequesties for kernel and
>>> btrfsprogs for that script ?
>>> Do you think it will run on my old SLES 11 SP4 ?
>> Dependency-wise, it needs:
>>
>> kernel: Should work with any kernel version that has BTRFS support,
>> untested on kernels before 4.10, but I'm fairly confident that it should
>> work on just about anything.  If there's going to be a failure due to
>> kernel version, it should happen when saving the subvolume information,
>> so you should be safe checking this (especially since the script doesn't
>> write anything when saving the info until the very end, and doesn't use
>> any write capable ioctls when fetching the info).
>>
>> btrfs-progs: Only uses the subvolume list, create, and delete commands,
>> and the only one which may have changed significantly in the past few
>> years is the list command.  I've only tested it with btrfs-progs 4.10.2
>> and newer, but I expect it should work with any of them at least since
>> the change to matching kernel versions.  You can quickly check this by
>> making sure the output of `btrfs subvolume list` looks like this:
>>
>>      ID 257 gen 769304 top level 5 path root
>>
>> What matters there is the number of words between each number, as the
>> parsing code is pretty naive.
>>
>> util-linux: We use blkid to get the filesystem label and UUID so that
>> it's more evident in the file what filesystem it's for (in case you
>> decide to stroe them separately), but as far as I know the relevant
>> command-line options haven't changed in at least half a decade, so any
>> reasonably recent version should work fine (tested with util-linux
>> 2.31).  As the script is currently, this will cause it to throw an
>> exception if it doesn't work, but that will happen before it does almost
>> anything else, so it should still fail safe.
>>
>> Python: btrfs-subv-backup requires Python 3.  I've only tested it on 3.4
>> and 3.6, but I'm pretty sure it should work on most versions back to at
>> least 3.2.  Of the dependencies, this is the one I'd expect to be the
>> most likely possible issue on older distros, but you likely have a new
>> enough version already given the kernel build date shown below.  If this
>> doesn't work, you will get syntax errors (specifically around the error
>> handlers).
>>
> 
> damned, i have 2.6.9 and the most recent i fnd in repositories for SLES 11 SP4 is 2.7.6
Assuming you're careful about how you install it (that is, put it in a 
custom prefix that isn't in $PATH), you could always build a local 
version of Python.  Once you've got that, it's pretty trivial to change 
the #! line at the beginning of the script to point to the appropriate 
location.

For what it's worth, it shouldn't be _too_ hard to get the script 
working with Python 2.7.  The only big syntax difference that's liable 
to be an issue is the except clauses.  I'll take a look at this and see 
if I can't get it working for both 3.X and 2.7.

Re: btrfs-subv-backup v0.1b

[Lentes, Bernd]

----- On Oct 31, 2017, at 6:00 PM, Austin S. Hemmelgarn ahferroin7@gmail.com wrote:

> Assuming you're careful about how you install it (that is, put it in a
> custom prefix that isn't in $PATH), you could always build a local
> version of Python.  Once you've got that, it's pretty trivial to change
> the #! line at the beginning of the script to point to the appropriate
> location.
> 
> For what it's worth, it shouldn't be _too_ hard to get the script
> working with Python 2.7.  The only big syntax difference that's liable
> to be an issue is the except clauses.  I'll take a look at this and see
> if I can't get it working for both 3.X and 2.7.

Austin,

do i understand correctly what i've read on github, that your script is worthless for SLES because they mount the subvolumes ?
"this of course does not work if you explicitly mount subvolumes, but the only distro I know of that does that is SUSE, and they do their own thing anyway."

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

Re: btrfs-subv-backup v0.1b

[Austin S. Hemmelgarn]

On 2017-10-31 15:54, Lentes, Bernd wrote:
> ----- On Oct 31, 2017, at 6:00 PM, Austin S. Hemmelgarn ahferroin7@gmail.com wrote:
> 
>> Assuming you're careful about how you install it (that is, put it in a
>> custom prefix that isn't in $PATH), you could always build a local
>> version of Python.  Once you've got that, it's pretty trivial to change
>> the #! line at the beginning of the script to point to the appropriate
>> location.
>>
>> For what it's worth, it shouldn't be _too_ hard to get the script
>> working with Python 2.7.  The only big syntax difference that's liable
>> to be an issue is the except clauses.  I'll take a look at this and see
>> if I can't get it working for both 3.X and 2.7.
> 
> Austin,
> 
> do i understand correctly what i've read on github, that your script is worthless for SLES because they mount the subvolumes ?
> "this of course does not work if you explicitly mount subvolumes, but the only distro I know of that does that is SUSE, and they do their own thing anyway."
It's not quite worthless, it just needs a bit more effort, although 
that's not _just_ because of the subvolumes being explicitly mounted. 
There's also the fact that SUSE has an odd mix of subvolumes outside of 
the subvolume that gets mounted as root, and ones inside, which makes 
handling a SUSE root filesystem extremely complicated (though this is 
only the case if you installed with snapshot support enabled).

In short, to use this reliably for the root filesystem for OpenSUSE and 
SLES, you need to mount the top level of the root filesystem somewhere, 
and then run the script on that (and that tree is what you should be 
backing up in an ideal situation as well).

I'll look at getting the README updated to be more clear about that and 
give more specific directions tomorrow.