Cloning / getting a full backup of a BTRFS filesystem

Cloning / getting a full backup of a BTRFS filesystem

[Swâmi Petaramesh]

Hi list,

Is there an advised way to completely “clone” a complete BTRFS
filesystem, I mean to get an exact copy of a BTRFS filesystem including
subvolumes (even readonly snapshots) and complete file attributes
including extended attributes, ACLs and so, to another storage pool,
possibly defined with a different RAID geometry or compression ?

The question boils down to getting an exact backup replica of a given
BTRFS filesystem that could be restored to something logically
absolutely identical.

The usual backup tools have no clue about share extents, snapshots and
the like, and using btrfs send/receive for individual subvols is a real
pain in a BTRFS filesystem that may contain hundreds of snapshots of
different BTRFS subvols plus deduplication etc.

So on a practical standpoint, how can one backup and restore a full
BTRFS structure ?

(I know of tools like partclone that may or may not do the job as they
usually lack behind recent BTRFS features, and may not be able to clone
BTRFS RAID setups for example...)

TIA.

ॐ

-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E

Re: Cloning / getting a full backup of a BTRFS filesystem

[Andrei Borzenkov]

On Wed, Sep 4, 2019 at 9:16 AM Swâmi Petaramesh <swami@petaramesh.org> wrote:
>
> Hi list,
>
> Is there an advised way to completely “clone” a complete BTRFS
> filesystem, I mean to get an exact copy of a BTRFS filesystem including
> subvolumes (even readonly snapshots) and complete file attributes
> including extended attributes, ACLs and so, to another storage pool,
> possibly defined with a different RAID geometry or compression ?
>

As long as you do not use top level subvolume directly (all data is
located in subolumes), send/receive should work.

> The question boils down to getting an exact backup replica of a given
> BTRFS filesystem that could be restored to something logically
> absolutely identical.
>
> The usual backup tools have no clue about share extents, snapshots and
> the like, and using btrfs send/receive for individual subvols is a real
> pain in a BTRFS filesystem that may contain hundreds of snapshots of
> different BTRFS subvols plus deduplication etc.
>

Shared extents could be challenging. You can provide this information
to "btrfs send", but for one, there is no direct visibility into which
subvolumes share extents with given subvolume, so no way to build
corresponding list for "btrfs send". I do not even know if this
information can be obtained without exhaustive search over all
extents. Second, btrfs send/receive only allows sharing of full
extents which means there is no guarantee of identical structure on
receiving side.

> So on a practical standpoint, how can one backup and restore a full
> BTRFS structure ?
>
> (I know of tools like partclone that may or may not do the job as they
> usually lack behind recent BTRFS features, and may not be able to clone
> BTRFS RAID setups for example...)
>
> TIA.
>
> ॐ
>
> --
> Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E

Re: Cloning / getting a full backup of a BTRFS filesystem

[Piotr Szymaniak]

On Wed, Sep 04, 2019 at 12:03:10PM +0300, Andrei Borzenkov wrote:
> On Wed, Sep 4, 2019 at 9:16 AM Swâmi Petaramesh <swami@petaramesh.org> wrote:
> >
> > Hi list,
> >
> > Is there an advised way to completely “clone” a complete BTRFS
> > filesystem, I mean to get an exact copy of a BTRFS filesystem including
> > subvolumes (even readonly snapshots) and complete file attributes
> > including extended attributes, ACLs and so, to another storage pool,
> > possibly defined with a different RAID geometry or compression ?
> >
> 
> As long as you do not use top level subvolume directly (all data is
> located in subolumes), send/receive should work.
> 
> > The question boils down to getting an exact backup replica of a given
> > BTRFS filesystem that could be restored to something logically
> > absolutely identical.
> >
> > The usual backup tools have no clue about share extents, snapshots and
> > the like, and using btrfs send/receive for individual subvols is a real
> > pain in a BTRFS filesystem that may contain hundreds of snapshots of
> > different BTRFS subvols plus deduplication etc.
> >
> 
> Shared extents could be challenging. You can provide this information
> to "btrfs send", but for one, there is no direct visibility into which
> subvolumes share extents with given subvolume, so no way to build
> corresponding list for "btrfs send". I do not even know if this
> information can be obtained without exhaustive search over all
> extents. Second, btrfs send/receive only allows sharing of full
> extents which means there is no guarantee of identical structure on
> receiving side.

So right now the only answer is: use good old dd?


Piotr Szymaniak.
-- 
Jedyne  napisy,  które rozumie każdy  Amerykanin,  to  "Wyprzedaż", "Za
darmo" i "Seks".  Kiedyś  widziałem w Arizonie tablicę  "Seks za darmo.
Ograniczenie prędkości do 60 km/h".
  -- Nelson DeMille, "The Lion's Game"

Re: Cloning / getting a full backup of a BTRFS filesystem

[Hugo Mills]

On Wed, Sep 04, 2019 at 04:04:44PM +0200, Piotr Szymaniak wrote:
> On Wed, Sep 04, 2019 at 12:03:10PM +0300, Andrei Borzenkov wrote:
> > On Wed, Sep 4, 2019 at 9:16 AM Swâmi Petaramesh <swami@petaramesh.org> wrote:
> > >
> > > Hi list,
> > >
> > > Is there an advised way to completely “clone” a complete BTRFS
> > > filesystem, I mean to get an exact copy of a BTRFS filesystem including
> > > subvolumes (even readonly snapshots) and complete file attributes
> > > including extended attributes, ACLs and so, to another storage pool,
> > > possibly defined with a different RAID geometry or compression ?
> > >
> > 
> > As long as you do not use top level subvolume directly (all data is
> > located in subolumes), send/receive should work.
> > 
> > > The question boils down to getting an exact backup replica of a given
> > > BTRFS filesystem that could be restored to something logically
> > > absolutely identical.
> > >
> > > The usual backup tools have no clue about share extents, snapshots and
> > > the like, and using btrfs send/receive for individual subvols is a real
> > > pain in a BTRFS filesystem that may contain hundreds of snapshots of
> > > different BTRFS subvols plus deduplication etc.
> > >
> > 
> > Shared extents could be challenging. You can provide this information
> > to "btrfs send", but for one, there is no direct visibility into which
> > subvolumes share extents with given subvolume, so no way to build
> > corresponding list for "btrfs send". I do not even know if this
> > information can be obtained without exhaustive search over all
> > extents. Second, btrfs send/receive only allows sharing of full
> > extents which means there is no guarantee of identical structure on
> > receiving side.
> 
> So right now the only answer is: use good old dd?

   If you want an exact copy, including all of the exact UUIDs, yes.

   Be aware of the problems of making block-level copies of btrfs
filesystems, though:
https://btrfs.wiki.kernel.org/index.php/Gotchas#Block-level_copies_of_devices

   Hugo.

-- 
Hugo Mills             | I have a step-ladder. My real ladder left when I was
hugo@... carfax.org.uk | a child.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Re: Cloning / getting a full backup of a BTRFS filesystem

[Chris Murphy]

On Wed, Sep 4, 2019 at 2:20 PM Hugo Mills <hugo@carfax.org.uk> wrote:
>
> On Wed, Sep 04, 2019 at 04:04:44PM +0200, Piotr Szymaniak wrote:
> > On Wed, Sep 04, 2019 at 12:03:10PM +0300, Andrei Borzenkov wrote:
> > > On Wed, Sep 4, 2019 at 9:16 AM Swâmi Petaramesh <swami@petaramesh.org> wrote:
> > > >
> > > > Hi list,
> > > >
> > > > Is there an advised way to completely “clone” a complete BTRFS
> > > > filesystem, I mean to get an exact copy of a BTRFS filesystem including
> > > > subvolumes (even readonly snapshots) and complete file attributes
> > > > including extended attributes, ACLs and so, to another storage pool,
> > > > possibly defined with a different RAID geometry or compression ?
> > > >
> > >
> > > As long as you do not use top level subvolume directly (all data is
> > > located in subolumes), send/receive should work.
> > >
> > > > The question boils down to getting an exact backup replica of a given
> > > > BTRFS filesystem that could be restored to something logically
> > > > absolutely identical.
> > > >
> > > > The usual backup tools have no clue about share extents, snapshots and
> > > > the like, and using btrfs send/receive for individual subvols is a real
> > > > pain in a BTRFS filesystem that may contain hundreds of snapshots of
> > > > different BTRFS subvols plus deduplication etc.
> > > >
> > >
> > > Shared extents could be challenging. You can provide this information
> > > to "btrfs send", but for one, there is no direct visibility into which
> > > subvolumes share extents with given subvolume, so no way to build
> > > corresponding list for "btrfs send". I do not even know if this
> > > information can be obtained without exhaustive search over all
> > > extents. Second, btrfs send/receive only allows sharing of full
> > > extents which means there is no guarantee of identical structure on
> > > receiving side.
> >
> > So right now the only answer is: use good old dd?
>
>    If you want an exact copy, including all of the exact UUIDs, yes.
>
>    Be aware of the problems of making block-level copies of btrfs
> filesystems, though:
> https://btrfs.wiki.kernel.org/index.php/Gotchas#Block-level_copies_of_devices


This might be fixed now. Anyone want to test it? With a recent kernel of course.

*runs away fast in case it isn't*


-- 
Chris Murphy

Re: Cloning / getting a full backup of a BTRFS filesystem

[Swâmi Petaramesh]

Hello and thanks for the replies all,

Le 04/09/2019 à 22:48, Chris Murphy a écrit :
>    If you want an exact copy, including all of the exact UUIDs, yes.

No I don"t really care about the UUIDs, and it's the only thing I don't
care about.

My use case is rather taking a full and complete copy of my NAS BTRFS
RAID-5 storage space onto an external, eventually big, disk, or some
other RAID combination in case something would go wrong - in a way that
I could get everything backed up and restored easily.

So far and for years I found BTRFS to be extremely reliable and
resilient in all configurations, but I've had several issues with recent
5.1 and 5.2 kernels on several machines recently .

My NAS holds a number of subvols for many different purposes, most of
them contains a hosts of RO snaphots made either manually or
automatically using snapper, and everything is offline-deduped using
bees (as there is a *lot* of redundant copies of the same files in
different subvols and structures).

Of course I cannot just “dd” as I don't start from a single device but a
BTRFS RAID, and I'm afraid that any manual scheme that I could think of
would break deduplication and thus result in a huge output that wouldn't
fit any device I could think of.

So the question reslly is : How could I backup a complex BTRFS volume to
another but differently (physically) organized volume keeping the
complete structure and being able to restore it preferably in a single
operation.

If the answer is « There's no way it can be done » then it is really
badly annoying...

ॐ

-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E

Re: Cloning / getting a full backup of a BTRFS filesystem

[Chris Murphy]

On Wed, Sep 4, 2019 at 3:04 PM Swâmi Petaramesh <swami@petaramesh.org> wrote:
>
> So the question reslly is : How could I backup a complex BTRFS volume to
> another but differently (physically) organized volume keeping the
> complete structure and being able to restore it preferably in a single
> operation.
>
> If the answer is « There's no way it can be done » then it is really
> badly annoying...

It is, but it's also a direct consequence of its features. Those
features actually being used makes the resulting file system
complicated enough that it defies being easily replicated - other than
a block copy of every drive.

ZFS has the same problem, but with some constraints on its features
that end up softening the problem. Snapshots are only read only, they
are direct children of a dataset, not independent. And therefore it's
straightforward to determine their relationship, and recursive send is
possible (-R replication option). Also it has a big advantage of
online dedup, so while there can be unnecessary data in the send
stream, it gets deduplicated on the receive side if dedup is on (which
comes with a really heavy cost on speed or resources).

On Btrfs, if you avoid reflinks and deduplication between subvolumes,
and if you have a rigorous naming scheme, to enforce restrictions on
Btrfs - you can get most of the way where you want, without having to
do deduplication again on the new destination.


-- 
Chris Murphy

Re: Cloning / getting a full backup of a BTRFS filesystem

[Alberto Bursi]

On 04/09/19 23:04, Swâmi Petaramesh wrote:
>
> So the question reslly is : How could I backup a complex BTRFS volume to
> another but differently (physically) organized volume keeping the
> complete structure and being able to restore it preferably in a single
> operation.
>
> If the answer is « There's no way it can be done » then it is really
> badly annoying...
>
> ॐ
>
There is a python-based tool that can clone a btrfs volume by sending 
subvolumes to the new filesystem,
one at a time. I never tried it, but it has a bunch of options, a decent 
readme and it's still maintained so
you may ask its developer too about your case.

https://github.com/mwilck/btrfs-clone

Personally I've never used large arrays nor RAID5/6 on Btrfs so I've 
just been cloning
partitions of UNMOUNTED Btrfs raids with pv (which is basically a "cat" 
that shows progress bars and such,
dd isn't doing anything special, and if you set the wrong block size it 
will take ages).

One disk after another, as long as the partition fits it will work fine. 
Can clone all disks at the same time, as long as they are unmounted.

If you are copying GPT disks and the target disk isn't 100% the same 
size, the backup GPT partition table at the end of the drive will be in 
the wrong place
so you will need to fix that with fdisk (should be just "sudo fdisk 
/dev/sdX" and then you aknowledge the issue
about the missing GPT table at the end of the device, and then "write 
changes to disk") or gdisk (that has a specific command to rebuild the 
backup GPT table in the advanced menu).

-Alberto

Re: Cloning / getting a full backup of a BTRFS filesystem

[Chris Murphy]

On Wed, Sep 4, 2019 at 4:14 PM Alberto Bursi <alberto.bursi@outlook.it> wrote:
>
> There is a python-based tool that can clone a btrfs volume by sending
> subvolumes to the new filesystem,
> one at a time. I never tried it, but it has a bunch of options, a decent
> readme and it's still maintained so
> you may ask its developer too about your case.
>
> https://github.com/mwilck/btrfs-clone

Wow this is really cool! Just reading the readme is enlightening.


-- 
Chris Murphy

Re: Cloning / getting a full backup of a BTRFS filesystem

[Swâmi Petaramesh]

Le 05/09/2019 à 04:47, Chris Murphy a écrit :
> On Wed, Sep 4, 2019 at 4:14 PM Alberto Bursi <alberto.bursi@outlook.it> wrote:
>>
>> There is a python-based tool that can clone a btrfs volume by sending
>> subvolumes to the new filesystem,
>> one at a time. I never tried it, but it has a bunch of options, a decent
>> readme and it's still maintained so
>> you may ask its developer too about your case.
>>
>> https://github.com/mwilck/btrfs-clone
> 
> Wow this is really cool! Just reading the readme is enlightening.

Looks real great ! This could be the tool I was looking for.

Thannk you all for your input.

ॐ

-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E

Re: Cloning / getting a full backup of a BTRFS filesystem

[Chris Murphy]

On Wed, Sep 4, 2019 at 12:16 AM Swâmi Petaramesh <swami@petaramesh.org> wrote:
>
> Hi list,
>
> Is there an advised way to completely “clone” a complete BTRFS
> filesystem, I mean to get an exact copy of a BTRFS filesystem including
> subvolumes (even readonly snapshots) and complete file attributes
> including extended attributes, ACLs and so, to another storage pool,
> possibly defined with a different RAID geometry or compression ?

The bottom line answer is no. There are only compromises.

Btrfs seed sprout will do what you want, except you can't change
geometry or compression. Last time I tested multiple devices as either
a source or destination, I ran into problems - but it's possible some
of this has been fixed, which is a question for Anand Jain.

Btrfs send receive can use a destination with different geometry,
compression, and multiple devices - but it doesn't handle relinks /
shared extents between otherwise unrelated subvolumes, although I
think that's what the -c flag is used for. I've never used it. There
also isn't a built-in way to recursively and intelligently send all
subvolumes, taking advantage of incremental send.

But in both cases, all file metadata is preserved. It's not really an
exact copy. The extent layout will be different, the subvolume UUIDs
will change in the 2nd case, and the volume UUID will change in both
cases. Etc. So not exact. But in terms of user visible aspects, yes it
would be exact with either method.

>
> The question boils down to getting an exact backup replica of a given
> BTRFS filesystem that could be restored to something logically
> absolutely identical.

Using words like "exact" and "absolutely identical" you've reduced it
down to a block copy. Only a sector copy does that. So I think you
need a better term that qualifies what you want or what your use case
really is. And also, changing compression, number of devices and
profiles, that's not exact either, and it definitely does not qualify
as identical let alone absolutely identical.


> The usual backup tools have no clue about share extents, snapshots and
> the like, and using btrfs send/receive for individual subvols is a real
> pain in a BTRFS filesystem that may contain hundreds of snapshots of
> different BTRFS subvols plus deduplication etc.

If you only ever deduplicate within a subvolume, and you aren't
deduplicating between subvolumes, then send/receive will do what you
want. Otherwise I think it's clone option. Maybe.

> So on a practical standpoint, how can one backup and restore a full
> BTRFS structure ?

Seed sprout comes the closest. But Btrfs does allow the user too
create a volume that will exceed the present feature set of seed
sprout. Therefore it's possible to create a Btrfs volume that cannot
really be replicated other than a block copy, if your goal is
exactness.


-- 
Chris Murphy

Re: Cloning / getting a full backup of a BTRFS filesystem

[Chris Murphy]

On Wed, Sep 4, 2019 at 11:55 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> Btrfs send receive can use a destination with different geometry,
> compression, and multiple devices - but it doesn't handle relinks /
> shared extents between otherwise unrelated subvolumes, although I
> think that's what the -c flag is used for. I've never used it. There
> also isn't a built-in way to recursively and intelligently send all
> subvolumes, taking advantage of incremental send.

^ reflinks

-- 
Chris Murphy

Re: Cloning / getting a full backup of a BTRFS filesystem

[Anand Jain]

> On 5 Sep 2019, at 1:55 AM, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Wed, Sep 4, 2019 at 12:16 AM Swâmi Petaramesh <swami@petaramesh.org> wrote:
>> 
>> Hi list,
>> 
>> Is there an advised way to completely “clone” a complete BTRFS
>> filesystem, I mean to get an exact copy of a BTRFS filesystem including
>> subvolumes (even readonly snapshots) and complete file attributes
>> including extended attributes, ACLs and so, to another storage pool,
>> possibly defined with a different RAID geometry or compression ?
> 

 Remote replication is a planned feature, when ready, it can be
 used with a local target to meet this requisite.

> The bottom line answer is no. There are only compromises.
> 
> Btrfs seed sprout will do what you want, except you can't change
> geometry or compression. Last time I tested multiple devices as either
> a source or destination, I ran into problems - but it's possible some
> of this has been fixed, which is a question for Anand Jain.

 Thanks for the report. I just tested the below test case[1],
 it does not fail. Any idea?

[1]
 Create and mount a two devices seed fs, and create a sprout.

 umount /btrfs; mkfs.btrfs -fq -dsingle -msingle /dev/sdb /dev/sdc && mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs && btrfstune -S1 /dev/sdb && mount /dev/sdb /btrfs && btrfs dev add /dev/sdd /btrfs && umount /btrfs

 Mount the sprout device and delete the seed devices to make
 it an independent but identical fs.

 mount /dev/sdd /btrfs && btrfs dev del /dev/sdb /btrfs && btrfs dev del /dev/sdc /btrfs

Thanks, Anand

Re: Cloning / getting a full backup of a BTRFS filesystem

[Qu Wenruo]

On 2019/9/5 下午9:06, Anand Jain wrote:
>
>
>> On 5 Sep 2019, at 1:55 AM, Chris Murphy <lists@colorremedies.com> wrote:
>>
>> On Wed, Sep 4, 2019 at 12:16 AM Swâmi Petaramesh <swami@petaramesh.org> wrote:
>>>
>>> Hi list,
>>>
>>> Is there an advised way to completely “clone” a complete BTRFS
>>> filesystem, I mean to get an exact copy of a BTRFS filesystem including
>>> subvolumes (even readonly snapshots) and complete file attributes
>>> including extended attributes, ACLs and so, to another storage pool,
>>> possibly defined with a different RAID geometry or compression ?
>>
>
>  Remote replication is a planned feature, when ready, it can be
>  used with a local target to meet this requisite.

In fact, I have submitted patches to implement btrfs-image data dump
feature, which can be used here.
https://patchwork.kernel.org/project/linux-btrfs/list/?series=141937

Further more, since it's btrfs-image, it can even support cloning a
RAID5 fs into single device!
But I doubt btrfs-image -r would restore into another profile other the
original profile or single device.

Thanks,
Qu

>
>> The bottom line answer is no. There are only compromises.
>>
>> Btrfs seed sprout will do what you want, except you can't change
>> geometry or compression. Last time I tested multiple devices as either
>> a source or destination, I ran into problems - but it's possible some
>> of this has been fixed, which is a question for Anand Jain.
>
>  Thanks for the report. I just tested the below test case[1],
>  it does not fail. Any idea?
>
> [1]
>  Create and mount a two devices seed fs, and create a sprout.
>
>  umount /btrfs; mkfs.btrfs -fq -dsingle -msingle /dev/sdb /dev/sdc && mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs && btrfstune -S1 /dev/sdb && mount /dev/sdb /btrfs && btrfs dev add /dev/sdd /btrfs && umount /btrfs
>
>  Mount the sprout device and delete the seed devices to make
>  it an independent but identical fs.
>
>  mount /dev/sdd /btrfs && btrfs dev del /dev/sdb /btrfs && btrfs dev del /dev/sdc /btrfs
>
> Thanks, Anand
>
>
>
>
>