hierarchical, tree-like structure of snapshots

hierarchical, tree-like structure of snapshots

[john terragon]

Hi.
I would like to maintain a tree-like hierarchical structure of
snapshots. Let me try to explain what I mean by that.

Let's say I have a btrfs fs with just one subvolume X, and let's say
that a make a readonly snapshot Y of X. As far as I understand there
is a parent-child relation between Y (the parent) and X the child.

Now let's say that after some time and modifications of X I do another
snapshot Z of X. Now the "temporal" stucture would be Y-Z-X. So X is
now the "child" of Z and Z is now the "child" of Y. The structure is a
path which is a special case of a tree.

Now let's suppose that I want to start modify Y but I still want to be
able to have a parent of Z which I might use as a point of reference
for Z in a
send to somewhere. That is I want to be able to still do a send -p Y Z
to another btrfs filesystem where there is previously sent copy of Y
(which, remember, as of this point has been readonly and I'm just now
wanting to start to modify it).
The only thing I think I can do would be to make a readonly snapshot
Y1 of Y and make Y writeable (so that I can start modify it). At that
point the structure would be

Y1-Y
    \
      Z-X

(yes my ascii art is atrocious...) which is a "proper" tree where Y1
is the root with two children (Y and Z), Z has one child (X) and Y and
X are leaves.
Now, my question is, would Y1 still be usable in send -p Y1 Z, just
like Y was before becoming writeable and being modified? I would say
that Y1 would be just as good as the readonly original Y was as a
parent for Z in a send. But maybe there is some implementation detail
that escapes me and that prevents Y1 to be used as a perfect
replacement for the original Y.
I hope I was clear enough.
Thanks
John

Re: hierarchical, tree-like structure of snapshots

[john terragon]

Sorry, that ascii tree came out awful and it looks like Z is the child
of Y instead of Y1. I hope this one below looks better.

Y1-Y
 \
  Z-X

On Wed, Dec 30, 2020 at 5:56 PM john terragon <jterragon@gmail.com> wrote:
>
> Hi.
> I would like to maintain a tree-like hierarchical structure of
> snapshots. Let me try to explain what I mean by that.
>
> Let's say I have a btrfs fs with just one subvolume X, and let's say
> that a make a readonly snapshot Y of X. As far as I understand there
> is a parent-child relation between Y (the parent) and X the child.
>
> Now let's say that after some time and modifications of X I do another
> snapshot Z of X. Now the "temporal" stucture would be Y-Z-X. So X is
> now the "child" of Z and Z is now the "child" of Y. The structure is a
> path which is a special case of a tree.
>
> Now let's suppose that I want to start modify Y but I still want to be
> able to have a parent of Z which I might use as a point of reference
> for Z in a
> send to somewhere. That is I want to be able to still do a send -p Y Z
> to another btrfs filesystem where there is previously sent copy of Y
> (which, remember, as of this point has been readonly and I'm just now
> wanting to start to modify it).
> The only thing I think I can do would be to make a readonly snapshot
> Y1 of Y and make Y writeable (so that I can start modify it). At that
> point the structure would be
>
> Y1-Y
>     \
>       Z-X
>
> (yes my ascii art is atrocious...) which is a "proper" tree where Y1
> is the root with two children (Y and Z), Z has one child (X) and Y and
> X are leaves.
> Now, my question is, would Y1 still be usable in send -p Y1 Z, just
> like Y was before becoming writeable and being modified? I would say
> that Y1 would be just as good as the readonly original Y was as a
> parent for Z in a send. But maybe there is some implementation detail
> that escapes me and that prevents Y1 to be used as a perfect
> replacement for the original Y.
> I hope I was clear enough.
> Thanks
> John

Re: hierarchical, tree-like structure of snapshots

[sys]

On 2020-12-30 17:56, john terragon wrote:
> Hi.
> I would like to maintain a tree-like hierarchical structure of
> snapshots. Let me try to explain what I mean by that.
> 
> Let's say I have a btrfs fs with just one subvolume X, and let's say
> that a make a readonly snapshot Y of X. As far as I understand there
> is a parent-child relation between Y (the parent) and X the child.
> 
> Now let's say that after some time and modifications of X I do another
> snapshot Z of X. Now the "temporal" stucture would be Y-Z-X. So X is
> now the "child" of Z and Z is now the "child" of Y. The structure is a
> path which is a special case of a tree.
> 
> Now let's suppose that I want to start modify Y but I still want to be
> able to have a parent of Z which I might use as a point of reference
> for Z in a
> send to somewhere. That is I want to be able to still do a send -p Y Z
> to another btrfs filesystem where there is previously sent copy of Y
> (which, remember, as of this point has been readonly and I'm just now
> wanting to start to modify it).
> The only thing I think I can do would be to make a readonly snapshot
> Y1 of Y and make Y writeable (so that I can start modify it).

You should simply make a 'read-write' snapshot (Y-rw) of the 'read-only' 
snapshot (Y) that is part of your backup/send scheme. Do not modify 
read-only snapshots to be rw.


  At that
> point the structure would be
> 
> Y1-Y
>      \
>        Z-X
> 
> (yes my ascii art is atrocious...) which is a "proper" tree where Y1
> is the root with two children (Y and Z), Z has one child (X) and Y and
> X are leaves.
> Now, my question is, would Y1 still be usable in send -p Y1 Z, just
> like Y was before becoming writeable and being modified? I would say
> that Y1 would be just as good as the readonly original Y was as a
> parent for Z in a send. But maybe there is some implementation detail
> that escapes me and that prevents Y1 to be used as a perfect
> replacement for the original Y.
> I hope I was clear enough.
> Thanks
> John
> 

Re: hierarchical, tree-like structure of snapshots

[john terragon]

On Wed, Dec 30, 2020 at 6:24 PM sys <system@lechevalier.se> wrote:
>
>
>
[...]
> You should simply make a 'read-write' snapshot (Y-rw) of the 'read-only'
> snapshot (Y) that is part of your backup/send scheme. Do not modify
> read-only snapshots to be rw.
>

OK, but then could I use Y as parent of the rw snapshot, let's call it
W, in a send?
So I would have this tree where Y is still the root.

Y-W
 \
  Z-X

Can I do a send -p Y W ?
Because I thought it was other way around, that is I do a readonly
snapshot W of Y and that will be the base for incrementally sending
the future modified Y to another  FS (provided of course W is already
there).

Re: hierarchical, tree-like structure of snapshots

[Andrei Borzenkov]

30.12.2020 20:39, john terragon пишет:
> On Wed, Dec 30, 2020 at 6:24 PM sys <system@lechevalier.se> wrote:
>>
>>
>>
> [...]
>> You should simply make a 'read-write' snapshot (Y-rw) of the 'read-only'
>> snapshot (Y) that is part of your backup/send scheme. Do not modify
>> read-only snapshots to be rw.
>>
> 
> OK, but then could I use Y as parent of the rw snapshot, let's call it
> W, in a send?

No

> So I would have this tree where Y is still the root.
> 
> Y-W
>  \
>   Z-X
> 
> Can I do a send -p Y W ?

No. All subvolumes used in send/receive must be read-only. And they must
remain read-only from the moment they are created - we have seen quite a
lot of reports when users removed read-only property from subvolume used
in the past as send source, modified it, set as read-only again and
tried to continue replication. This resulted in complete mess on receive
side. Also if you try to modify destination snapshots it will break at
some point.

The general rule - everything used for replication must remain
read-only. If you want to use any snapshot that is part of replication
you clone it and use its clone.

> Because I thought it was other way around, that is I do a readonly
> snapshot W of Y and that will be the base for incrementally sending
> the future modified Y to another  FS (provided of course W is already
> there).
> 

If you want to capture changes in W since it was cloned from Y you
create another read-only snapshot of W and use it.

btrfs subvolume snapshot -r W V
btrfs send -p Y V

It is possible that btrfs implementation is optimized for sequential
snapshots from the same subvolume so the send stream size will be
larger. I am not familiar with these low level details. From the naïve
end-user point of view there should be no difference between

btrfs subvolume snapshot -r W R1
btrfs send R1
modify W
btrfs subvolume snapshot -r W R2
btrfs send -p R1 R2

and

btrfs send R1
btrfs subvolume snapshot R1 W
modify W
btrfs subvolume snapshot -r W R2
btrfs send -p R1 R2

Re: hierarchical, tree-like structure of snapshots

[Forza]

On 2020-12-31 08:05, Andrei Borzenkov wrote:
> 30.12.2020 20:39, john terragon пишет:
>> On Wed, Dec 30, 2020 at 6:24 PM sys <system@lechevalier.se> wrote:
>>>
>>>
>>>
>> [...]
>>> You should simply make a 'read-write' snapshot (Y-rw) of the 'read-only'
>>> snapshot (Y) that is part of your backup/send scheme. Do not modify
>>> read-only snapshots to be rw.
>>>
>>
>> OK, but then could I use Y as parent of the rw snapshot, let's call it
>> W, in a send?
> 
> No
> 
>> So I would have this tree where Y is still the root.
>>
>> Y-W
>>   \
>>    Z-X
>>
>> Can I do a send -p Y W ?
> 
> No. All subvolumes used in send/receive must be read-only. And they must
> remain read-only from the moment they are created - we have seen quite a
> lot of reports when users removed read-only property from subvolume used
> in the past as send source, modified it, set as read-only again and
> tried to continue replication. This resulted in complete mess on receive
> side. Also if you try to modify destination snapshots it will break at
> some point.
> 
> The general rule - everything used for replication must remain
> read-only. If you want to use any snapshot that is part of replication
> you clone it and use its clone.
> 
>> Because I thought it was other way around, that is I do a readonly
>> snapshot W of Y and that will be the base for incrementally sending
>> the future modified Y to another  FS (provided of course W is already
>> there).
>>
> 
> If you want to capture changes in W since it was cloned from Y you
> create another read-only snapshot of W and use it.
> 
> btrfs subvolume snapshot -r W V
> btrfs send -p Y V
> 
> It is possible that btrfs implementation is optimized for sequential
> snapshots from the same subvolume so the send stream size will be
> larger. I am not familiar with these low level details. From the naïve
> end-user point of view there should be no difference between
> 
> btrfs subvolume snapshot -r W R1
> btrfs send R1
> modify W
> btrfs subvolume snapshot -r W R2
> btrfs send -p R1 R2
> 
> and
> 
> btrfs send R1
> btrfs subvolume snapshot R1 W
> modify W
> btrfs subvolume snapshot -r W R2
> btrfs send -p R1 R2
> 

I think you are correct. The man page specifies that all snapshots must 
be read-only, but it is rather unclear if you modify some snaps in between.

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-send

Re: hierarchical, tree-like structure of snapshots

[john terragon]

On Thu, Dec 31, 2020 at 8:05 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>

> >
> > OK, but then could I use Y as parent of the rw snapshot, let's call it
> > W, in a send?
>
> No
>

Of course I didn't mean to use Y as a parent of W itself but to a
readonly snapshot of W whenever I want to send it to the second FS.

And I just tried the following steps and they worked:

1) created subvol X
2) created readonly snap Y of X
3) sent Y to second FS
4) modified X
5) created readonly snap X1 of X
6) sent -p Y X1 to second FS
7) created readwrite snap Y1 of Y
8) modified Y1
9) created readonly snap Y1_RO of Y1
10) sent -p Y Y1_RO to second FS

So, as you can see,

-in 6) I've used the RO snap Y of X as the parent of X1 (and X) to
send X1 to the second FS

-in 10) I did the opposite, Y is still used as the parent but this
time I've sent the RO snap of a subvol that is a snap of Y.

So it seems to work both ways

Re: hierarchical, tree-like structure of snapshots

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

On Thu, Dec 31, 2020 at 05:08:57PM +0100, john terragon wrote:
> On Thu, Dec 31, 2020 at 8:05 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> >
> 
> > >
> > > OK, but then could I use Y as parent of the rw snapshot, let's call it
> > > W, in a send?
> >
> > No
> >
> 
> Of course I didn't mean to use Y as a parent of W itself but to a
> readonly snapshot of W whenever I want to send it to the second FS.
> 
> And I just tried the following steps and they worked:
> 
> 1) created subvol X
> 2) created readonly snap Y of X
> 3) sent Y to second FS
> 4) modified X
> 5) created readonly snap X1 of X
> 6) sent -p Y X1 to second FS
> 7) created readwrite snap Y1 of Y
> 8) modified Y1
> 9) created readonly snap Y1_RO of Y1
> 10) sent -p Y Y1_RO to second FS
> 
> So, as you can see,
> 
> -in 6) I've used the RO snap Y of X as the parent of X1 (and X) to
> send X1 to the second FS
> 
> -in 10) I did the opposite, Y is still used as the parent but this
> time I've sent the RO snap of a subvol that is a snap of Y.
> 
> So it seems to work both ways

I think your confusion is that you are thinking of these as a tree.
There is no tree, each subvol is an equal peer in the filesystem.

"send -p A B" just walks over subvol A and B and sends a diff of the
parts of B not in A.  You can pick any subvol with -p as long as it's
read-only and present on the receiving side.  Obviously it's much more
efficient if the two subvols have a lot of shared extents (e.g. because
B and A were both snapshots made at different times of some other subvol
C), but this is not required.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: hierarchical, tree-like structure of snapshots

[john terragon]

On Thu, Dec 31, 2020 at 6:28 PM Zygo Blaxell
<ce3g8jdj@umail.furryterror.org> wrote:

>
> I think your confusion is that you are thinking of these as a tree.
> There is no tree, each subvol is an equal peer in the filesystem.
>
> "send -p A B" just walks over subvol A and B and sends a diff of the
> parts of B not in A.  You can pick any subvol with -p as long as it's
> read-only and present on the receiving side.  Obviously it's much more
> efficient if the two subvols have a lot of shared extents (e.g. because
> B and A were both snapshots made at different times of some other subvol
> C), but this is not required.

Can you really use ANY subvol to use with -p. Because if I

1) create a subvol X
2) create a subvol W with the exact same content of X (but created
independently)
3) do a RO snap X_RO of X
4) do a RO snap W_RO of W
5) send W_RO to the other FS
6) send -p W_RO X_RO to the other FS

I get this:

At subvol X_RO
At snapshot X_RO
ERROR: chown o257-1648413-0 failed: No such file or directory

any idea?

Re: hierarchical, tree-like structure of snapshots

[Andrei Borzenkov]

31.12.2020 21:19, john terragon пишет:
> On Thu, Dec 31, 2020 at 6:28 PM Zygo Blaxell
> <ce3g8jdj@umail.furryterror.org> wrote:
> 
>>
>> I think your confusion is that you are thinking of these as a tree.
>> There is no tree, each subvol is an equal peer in the filesystem.
>>
>> "send -p A B" just walks over subvol A and B and sends a diff of the
>> parts of B not in A.  You can pick any subvol with -p as long as it's
>> read-only and present on the receiving side.  Obviously it's much more
>> efficient if the two subvols have a lot of shared extents (e.g. because
>> B and A were both snapshots made at different times of some other subvol
>> C), but this is not required.
> 
> Can you really use ANY subvol to use with -p. Because if I
> 
> 1) create a subvol X
> 2) create a subvol W with the exact same content of X (but created
> independently)

How exactly you create subvolume with the same content? There are many
possible interpretations.

> 3) do a RO snap X_RO of X
> 4) do a RO snap W_RO of W
> 5) send W_RO to the other FS

Show actual command please.

> 6) send -p W_RO X_RO to the other FS
> 

Again show full command please. Which include also receive command.

> I get this:
> 
> At subvol X_RO
> At snapshot X_RO
> ERROR: chown o257-1648413-0 failed: No such file or directory
> 

You get where? On source, on destination?

> any idea?
> 

Re: hierarchical, tree-like structure of snapshots

[john terragon]

On Thu, Dec 31, 2020 at 8:42 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>

>
> How exactly you create subvolume with the same content? There are many
> possible interpretations.
>

Zygo wrote that any subvol could be used with -p. So, out of
curiosity, I did the following

1) btrfs sub create X
2) I unpacked some source (linux kernel) in X
3) btrfs sub create W
4) I unpacked the same source in W (so X and W have the same content
but they are independent)
5) btrfs sub snap -r X X_RO
6) btrfs sub snap -r W W_RO
7) btrfs send W_RO | btrfs receive /mnt/btrfs2
8) btrfs send -p W_RO X_RO | btrfs receive /mnt/btrfs2

And this is the exact output of 8)

At subvol X_RO
At snapshot X_RO
ERROR: chown o257-1648413-0 failed: No such file or directory

Re: hierarchical, tree-like structure of snapshots

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 4922 bytes --]

On Thu, Dec 31, 2020 at 09:48:54PM +0100, john terragon wrote:
> On Thu, Dec 31, 2020 at 8:42 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> >
> 
> >
> > How exactly you create subvolume with the same content? There are many
> > possible interpretations.
> >
> 
> Zygo wrote that any subvol could be used with -p. So, out of
> curiosity, I did the following
> 
> 1) btrfs sub create X
> 2) I unpacked some source (linux kernel) in X
> 3) btrfs sub create W
> 4) I unpacked the same source in W (so X and W have the same content
> but they are independent)
> 5) btrfs sub snap -r X X_RO
> 6) btrfs sub snap -r W W_RO
> 7) btrfs send W_RO | btrfs receive /mnt/btrfs2
> 8) btrfs send -p W_RO X_RO | btrfs receive /mnt/btrfs2
> 
> And this is the exact output of 8)
> 
> At subvol X_RO
> At snapshot X_RO
> ERROR: chown o257-1648413-0 failed: No such file or directory

Yeah, I only checked that send completed without error and produced a
smaller stream.

I just dumped the send metadata stream from the incremental snapshot now,
and it's more or less garbage at the start:

	# btrfs sub create A
	# btrfs sub create B
	# date > A/date
	# date > B/date
	# mkdir A/t B/u
	# btrfs sub snap -r A A_RO
	# btrfs sub snap -r B B_RO
	# btrfs send A_RO | btrfs receive --dump
	At subvol A_RO
	subvol          ./A_RO                          uuid=995adde4-00ac-5e49-8c6f-f01743def072 transid=7329268
	chown           ./A_RO/                         gid=0 uid=0
	chmod           ./A_RO/                         mode=755
	utimes          ./A_RO/                         atime=2020-12-31T15:51:31-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
	mkfile          ./A_RO/o257-7329268-0
	rename          ./A_RO/o257-7329268-0           dest=./A_RO/date
	utimes          ./A_RO/                         atime=2020-12-31T15:51:31-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
	write           ./A_RO/date                     offset=0 len=29
	chown           ./A_RO/date                     gid=0 uid=0
	chmod           ./A_RO/date                     mode=644
	utimes          ./A_RO/date                     atime=2020-12-31T15:51:38-0500 mtime=2020-12-31T15:51:38-0500 ctime=2020-12-31T15:51:38-0500
	mkdir           ./A_RO/o258-7329268-0
	rename          ./A_RO/o258-7329268-0           dest=./A_RO/t
	utimes          ./A_RO/                         atime=2020-12-31T15:51:31-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
	chown           ./A_RO/t                        gid=0 uid=0
	chmod           ./A_RO/t                        mode=755
	utimes          ./A_RO/t                        atime=2020-12-31T15:51:48-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
	# btrfs send B_RO -p A_RO | btrfs receive --dump
	At subvol B_RO
	snapshot        ./B_RO                          uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb transid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072 parent_transid=7329268
	utimes          ./B_RO/                         atime=2020-12-31T15:51:33-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
	link            ./B_RO/date                     dest=date
	unlink          ./B_RO/date
	utimes          ./B_RO/                         atime=2020-12-31T15:51:33-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
	write           ./B_RO/date                     offset=0 len=29
	utimes          ./B_RO/date                     atime=2020-12-31T15:51:41-0500 mtime=2020-12-31T15:51:41-0500 ctime=2020-12-31T15:51:41-0500
	rename          ./B_RO/t                        dest=./B_RO/u
	utimes          ./B_RO/                         atime=2020-12-31T15:51:33-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
	utimes          ./B_RO/u                        atime=2020-12-31T15:51:52-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
	# btrfs send A_RO | btrfs receive -v /tmp/test
	At subvol A_RO
	At subvol A_RO
	receiving subvol A_RO uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
	write date - offset=0 length=29
	BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
	# btrfs send B_RO -p A_RO | btrfs receive -v /tmp/test
	At subvol B_RO
	At snapshot B_RO
	receiving snapshot B_RO uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb, ctransid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072, parent_ctransid=7329268
	ERROR: link date -> date failed: File exists

The btrfs_compare_trees function can handle arbitrary tree differences,
but something happens in one of the support functions and we get a
bogus link command.  The rest of the stream is OK though:  we fill
in the contents of B_RO/date, rename A_RO/t to B_RO/u, and update all
the timestamps.

Oh well, I didn't say send didn't have any bugs.  ;)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: hierarchical, tree-like structure of snapshots

[john terragon]

Although I'm glad that a bug has been uncovered, maybe it's best if I
stick with good old rsync for backups.
It would be kind of ironic if the first data loss that I experienced
in many years of btrfs use would be caused by an ancillary backup
tool.

On Thu, Dec 31, 2020 at 10:36 PM Zygo Blaxell
<ce3g8jdj@umail.furryterror.org> wrote:
>
> On Thu, Dec 31, 2020 at 09:48:54PM +0100, john terragon wrote:
> > On Thu, Dec 31, 2020 at 8:42 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> > >
> >
> > >
> > > How exactly you create subvolume with the same content? There are many
> > > possible interpretations.
> > >
> >
> > Zygo wrote that any subvol could be used with -p. So, out of
> > curiosity, I did the following
> >
> > 1) btrfs sub create X
> > 2) I unpacked some source (linux kernel) in X
> > 3) btrfs sub create W
> > 4) I unpacked the same source in W (so X and W have the same content
> > but they are independent)
> > 5) btrfs sub snap -r X X_RO
> > 6) btrfs sub snap -r W W_RO
> > 7) btrfs send W_RO | btrfs receive /mnt/btrfs2
> > 8) btrfs send -p W_RO X_RO | btrfs receive /mnt/btrfs2
> >
> > And this is the exact output of 8)
> >
> > At subvol X_RO
> > At snapshot X_RO
> > ERROR: chown o257-1648413-0 failed: No such file or directory
>
> Yeah, I only checked that send completed without error and produced a
> smaller stream.
>
> I just dumped the send metadata stream from the incremental snapshot now,
> and it's more or less garbage at the start:
>
>         # btrfs sub create A
>         # btrfs sub create B
>         # date > A/date
>         # date > B/date
>         # mkdir A/t B/u
>         # btrfs sub snap -r A A_RO
>         # btrfs sub snap -r B B_RO
>         # btrfs send A_RO | btrfs receive --dump
>         At subvol A_RO
>         subvol          ./A_RO                          uuid=995adde4-00ac-5e49-8c6f-f01743def072 transid=7329268
>         chown           ./A_RO/                         gid=0 uid=0
>         chmod           ./A_RO/                         mode=755
>         utimes          ./A_RO/                         atime=2020-12-31T15:51:31-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
>         mkfile          ./A_RO/o257-7329268-0
>         rename          ./A_RO/o257-7329268-0           dest=./A_RO/date
>         utimes          ./A_RO/                         atime=2020-12-31T15:51:31-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
>         write           ./A_RO/date                     offset=0 len=29
>         chown           ./A_RO/date                     gid=0 uid=0
>         chmod           ./A_RO/date                     mode=644
>         utimes          ./A_RO/date                     atime=2020-12-31T15:51:38-0500 mtime=2020-12-31T15:51:38-0500 ctime=2020-12-31T15:51:38-0500
>         mkdir           ./A_RO/o258-7329268-0
>         rename          ./A_RO/o258-7329268-0           dest=./A_RO/t
>         utimes          ./A_RO/                         atime=2020-12-31T15:51:31-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
>         chown           ./A_RO/t                        gid=0 uid=0
>         chmod           ./A_RO/t                        mode=755
>         utimes          ./A_RO/t                        atime=2020-12-31T15:51:48-0500 mtime=2020-12-31T15:51:48-0500 ctime=2020-12-31T15:51:48-0500
>         # btrfs send B_RO -p A_RO | btrfs receive --dump
>         At subvol B_RO
>         snapshot        ./B_RO                          uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb transid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072 parent_transid=7329268
>         utimes          ./B_RO/                         atime=2020-12-31T15:51:33-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
>         link            ./B_RO/date                     dest=date
>         unlink          ./B_RO/date
>         utimes          ./B_RO/                         atime=2020-12-31T15:51:33-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
>         write           ./B_RO/date                     offset=0 len=29
>         utimes          ./B_RO/date                     atime=2020-12-31T15:51:41-0500 mtime=2020-12-31T15:51:41-0500 ctime=2020-12-31T15:51:41-0500
>         rename          ./B_RO/t                        dest=./B_RO/u
>         utimes          ./B_RO/                         atime=2020-12-31T15:51:33-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
>         utimes          ./B_RO/u                        atime=2020-12-31T15:51:52-0500 mtime=2020-12-31T15:51:52-0500 ctime=2020-12-31T15:51:52-0500
>         # btrfs send A_RO | btrfs receive -v /tmp/test
>         At subvol A_RO
>         At subvol A_RO
>         receiving subvol A_RO uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
>         write date - offset=0 length=29
>         BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
>         # btrfs send B_RO -p A_RO | btrfs receive -v /tmp/test
>         At subvol B_RO
>         At snapshot B_RO
>         receiving snapshot B_RO uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb, ctransid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072, parent_ctransid=7329268
>         ERROR: link date -> date failed: File exists
>
> The btrfs_compare_trees function can handle arbitrary tree differences,
> but something happens in one of the support functions and we get a
> bogus link command.  The rest of the stream is OK though:  we fill
> in the contents of B_RO/date, rename A_RO/t to B_RO/u, and update all
> the timestamps.
>
> Oh well, I didn't say send didn't have any bugs.  ;)

Re: hierarchical, tree-like structure of snapshots

[Andrei Borzenkov]

[-- Attachment #1.1: Type: text/plain, Size: 1841 bytes --]

01.01.2021 00:36, Zygo Blaxell пишет:
...
> 
> Yeah, I only checked that send completed without error and produced a
> smaller stream.
> 
> I just dumped the send metadata stream from the incremental snapshot now,
> and it's more or less garbage at the start:
> 
> 	# btrfs sub create A
> 	# btrfs sub create B
> 	# date > A/date
> 	# date > B/date
> 	# mkdir A/t B/u
> 	# btrfs sub snap -r A A_RO
> 	# btrfs sub snap -r B B_RO
...
> 	# btrfs send A_RO | btrfs receive -v /tmp/test
> 	At subvol A_RO
> 	At subvol A_RO
> 	receiving subvol A_RO uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
> 	write date - offset=0 length=29
> 	BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
> 	# btrfs send B_RO -p A_RO | btrfs receive -v /tmp/test
> 	At subvol B_RO
> 	At snapshot B_RO
> 	receiving snapshot B_RO uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb, ctransid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072, parent_ctransid=7329268
> 	ERROR: link date -> date failed: File exists
> 
> The btrfs_compare_trees function can handle arbitrary tree differences,

I am not sure. It apparently relies on the fact that inodes are ever
monotonically increasing. This is probably true for clones of the same
subvolume (I assume clone inherits highest_objectid) but two subvolumes
created independently have the same range of inode numbers.

Also I am not sure if using later clone as base for difference to
earlier clone will work for the same reason.

> but something happens in one of the support functions and we get a
> bogus link command.  The rest of the stream is OK though:  we fill
> in the contents of B_RO/date, rename A_RO/t to B_RO/u, and update all
> the timestamps.
> 
> Oh well, I didn't say send didn't have any bugs.  ;)
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: hierarchical, tree-like structure of snapshots

[Andrei Borzenkov]

[-- Attachment #1.1: Type: text/plain, Size: 3091 bytes --]

01.01.2021 14:42, Andrei Borzenkov пишет:
> 01.01.2021 00:36, Zygo Blaxell пишет:
> ...
>>
>> Yeah, I only checked that send completed without error and produced a
>> smaller stream.
>>
>> I just dumped the send metadata stream from the incremental snapshot now,
>> and it's more or less garbage at the start:
>>
>> 	# btrfs sub create A
>> 	# btrfs sub create B
>> 	# date > A/date
>> 	# date > B/date
>> 	# mkdir A/t B/u
>> 	# btrfs sub snap -r A A_RO
>> 	# btrfs sub snap -r B B_RO
> ...
>> 	# btrfs send A_RO | btrfs receive -v /tmp/test
>> 	At subvol A_RO
>> 	At subvol A_RO
>> 	receiving subvol A_RO uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
>> 	write date - offset=0 length=29
>> 	BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
>> 	# btrfs send B_RO -p A_RO | btrfs receive -v /tmp/test
>> 	At subvol B_RO
>> 	At snapshot B_RO
>> 	receiving snapshot B_RO uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb, ctransid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072, parent_ctransid=7329268
>> 	ERROR: link date -> date failed: File exists
>>
>> The btrfs_compare_trees function can handle arbitrary tree differences,
> 
> I am not sure. It apparently relies on the fact that inodes are ever
> monotonically increasing. This is probably true for clones of the same
> subvolume (I assume clone inherits highest_objectid) but two subvolumes
> created independently have the same range of inode numbers.
> 

In particular in your example both A/date and B/date have identical
inode numbers and in general INODE_ITEMs are identical (including
generation numbers) up to times so two inodes are compared as changed.
At the same time INODE_REFs for them are considered different because
INODE_ITEMs for root have different generations. This leads to code path
that attempts to create additional alias to existing inode, as it is
regular file it tries to link it. It does not really compares ref names
at this point at all.

This would not really be possible if A and B were clones of the same
subvolume (not necessary consecutive) as A/date and B/date would always
have different inode numbers.

If I force different generation numbers for A/date and B/date (by
syncing in between) send stream contains correct sequence of removing
old B/date (from A clone) and re-creating it again.

Which shows that unfortunately generation numbers are not reliable to
differentiate between different object generations (pun unintended). As
I understand generation is tied to transaction and multiple changes can
be packed into one transaction.

> Also I am not sure if using later clone as base for difference to
> earlier clone will work for the same reason.
> 
>> but something happens in one of the support functions and we get a
>> bogus link command.  The rest of the stream is OK though:  we fill
>> in the contents of B_RO/date, rename A_RO/t to B_RO/u, and update all
>> the timestamps.
>>
>> Oh well, I didn't say send didn't have any bugs.  ;)
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: hierarchical, tree-like structure of snapshots

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 5466 bytes --]

On Fri, Jan 01, 2021 at 11:40:28PM +0300, Andrei Borzenkov wrote:
> 01.01.2021 14:42, Andrei Borzenkov пишет:
> > 01.01.2021 00:36, Zygo Blaxell пишет:
> > ...
> >>
> >> Yeah, I only checked that send completed without error and produced a
> >> smaller stream.
> >>
> >> I just dumped the send metadata stream from the incremental snapshot now,
> >> and it's more or less garbage at the start:
> >>
> >> 	# btrfs sub create A
> >> 	# btrfs sub create B
> >> 	# date > A/date
> >> 	# date > B/date
> >> 	# mkdir A/t B/u
> >> 	# btrfs sub snap -r A A_RO
> >> 	# btrfs sub snap -r B B_RO
> > ...
> >> 	# btrfs send A_RO | btrfs receive -v /tmp/test
> >> 	At subvol A_RO
> >> 	At subvol A_RO
> >> 	receiving subvol A_RO uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
> >> 	write date - offset=0 length=29
> >> 	BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
> >> 	# btrfs send B_RO -p A_RO | btrfs receive -v /tmp/test
> >> 	At subvol B_RO
> >> 	At snapshot B_RO
> >> 	receiving snapshot B_RO uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb, ctransid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072, parent_ctransid=7329268
> >> 	ERROR: link date -> date failed: File exists
> >>
> >> The btrfs_compare_trees function can handle arbitrary tree differences,
> > 
> > I am not sure. It apparently relies on the fact that inodes are ever
> > monotonically increasing. This is probably true for clones of the same
> > subvolume (I assume clone inherits highest_objectid) but two subvolumes
> > created independently have the same range of inode numbers.
> > 
> 
> In particular in your example both A/date and B/date have identical
> inode numbers and in general INODE_ITEMs are identical (including
> generation numbers) up to times so two inodes are compared as changed.
> At the same time INODE_REFs for them are considered different because
> INODE_ITEMs for root have different generations. This leads to code path
> that attempts to create additional alias to existing inode, as it is
> regular file it tries to link it. It does not really compares ref names
> at this point at all.
> 
> This would not really be possible if A and B were clones of the same
> subvolume (not necessary consecutive) as A/date and B/date would always
> have different inode numbers.

After v5.11-rc1 inode_cache can no longer be used, but any filesystem that
has inode_cache in its history might have cases like this hiding in
metadata even with a linear series of snapshots.

The send code is mostly used to transmit linear sequences of snapshots
(a series of snapshots which capture the state of a single subvol at
different times, ordered from oldest to newest) between machines that
are not using the inode_cache mount option.  Any other case isn't getting
very well tested in the field, even if it happens to work sometimes.

> If I force different generation numbers for A/date and B/date (by
> syncing in between) send stream contains correct sequence of removing
> old B/date (from A clone) and re-creating it again.
>
> Which shows that unfortunately generation numbers are not reliable to
> differentiate between different object generations (pun unintended). As
> I understand generation is tied to transaction and multiple changes can
> be packed into one transaction.

I'm pretty sure that the 6000+ lines of special-case code in send.c still
don't cover every possible case, or even all of the likely ones, even
with linear snapshot sequences.  We still get people on IRC reporting
strange receive issues, and usually the best solution we can find is
to start over with a new full send.  That's OK for small filesystems,
but when you have to unexpectedly do a full send of dozens of terabytes
over a medium-speed link, it's probably time to switch to rsync.

Subversion used to have problems like this (maybe it still does, I
switched to git years ago) where a complicated commit that combined
multiple operations on objects of the same name would break the tool.
I'm surprised btrfs is trying to do similar things in the kernel
(though with the current send implementation there's nowhere else we
could do them).  At least for fsync we get to say "nope, too hard,
do a full commit instead" when complications arise.

> > Also I am not sure if using later clone as base for difference to
> > earlier clone will work for the same reason.

That use case can come up e.g. if you have snapshots of / and you roll
back to an earlier snapshot after a bad upgrade, but your backups are
using incremental snapshots made from '/'.  Then the last-sent-snapshot
(from the bad upgrade) is newer than the origin subvol (from an earlier
good upgrade, with new modifications on top).

Cases like these really need to work, or at least reliably throw
errors when they have failed, as the application that rolls back to
earlier snapshots might have no knowledge of the application that does
incremental send backups on a user's system if they integrated tools
from different vendors.

> >> but something happens in one of the support functions and we get a
> >> bogus link command.  The rest of the stream is OK though:  we fill
> >> in the contents of B_RO/date, rename A_RO/t to B_RO/u, and update all
> >> the timestamps.
> >>
> >> Oh well, I didn't say send didn't have any bugs.  ;)
> >>
> > 
> 
> 




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: hierarchical, tree-like structure of snapshots

[Andrei Borzenkov]

[-- Attachment #1.1: Type: text/plain, Size: 6566 bytes --]

02.01.2021 02:11, Zygo Blaxell пишет:
> On Fri, Jan 01, 2021 at 11:40:28PM +0300, Andrei Borzenkov wrote:
>> 01.01.2021 14:42, Andrei Borzenkov пишет:
>>> 01.01.2021 00:36, Zygo Blaxell пишет:
>>> ...
>>>>
>>>> Yeah, I only checked that send completed without error and produced a
>>>> smaller stream.
>>>>
>>>> I just dumped the send metadata stream from the incremental snapshot now,
>>>> and it's more or less garbage at the start:
>>>>
>>>> 	# btrfs sub create A
>>>> 	# btrfs sub create B
>>>> 	# date > A/date
>>>> 	# date > B/date
>>>> 	# mkdir A/t B/u
>>>> 	# btrfs sub snap -r A A_RO
>>>> 	# btrfs sub snap -r B B_RO
>>> ...
>>>> 	# btrfs send A_RO | btrfs receive -v /tmp/test
>>>> 	At subvol A_RO
>>>> 	At subvol A_RO
>>>> 	receiving subvol A_RO uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
>>>> 	write date - offset=0 length=29
>>>> 	BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=995adde4-00ac-5e49-8c6f-f01743def072, stransid=7329268
>>>> 	# btrfs send B_RO -p A_RO | btrfs receive -v /tmp/test
>>>> 	At subvol B_RO
>>>> 	At snapshot B_RO
>>>> 	receiving snapshot B_RO uuid=4aa7db26-b219-694e-9b3c-f8f737a46bdb, ctransid=7329268 parent_uuid=995adde4-00ac-5e49-8c6f-f01743def072, parent_ctransid=7329268
>>>> 	ERROR: link date -> date failed: File exists
>>>>
>>>> The btrfs_compare_trees function can handle arbitrary tree differences,
>>>
>>> I am not sure. It apparently relies on the fact that inodes are ever
>>> monotonically increasing. This is probably true for clones of the same
>>> subvolume (I assume clone inherits highest_objectid) but two subvolumes
>>> created independently have the same range of inode numbers.
>>>
>>
>> In particular in your example both A/date and B/date have identical
>> inode numbers and in general INODE_ITEMs are identical (including
>> generation numbers) up to times so two inodes are compared as changed.
>> At the same time INODE_REFs for them are considered different because
>> INODE_ITEMs for root have different generations. This leads to code path
>> that attempts to create additional alias to existing inode, as it is
>> regular file it tries to link it. It does not really compares ref names
>> at this point at all.
>>
>> This would not really be possible if A and B were clones of the same
>> subvolume (not necessary consecutive) as A/date and B/date would always
>> have different inode numbers.
> 
> After v5.11-rc1 inode_cache can no longer be used, but any filesystem that
> has inode_cache in its history might have cases like this hiding in
> metadata even with a linear series of snapshots.
> 
> The send code is mostly used to transmit linear sequences of snapshots
> (a series of snapshots which capture the state of a single subvol at
> different times, ordered from oldest to newest) between machines that
> are not using the inode_cache mount option.  Any other case isn't getting
> very well tested in the field, even if it happens to work sometimes.
> 

This is the only possible way to do it in NetApp and ZFS. But NetApp is
really much more usable than just that (I do not say ZFS is not, I just
have less experience with it). It retains unique identification of every
snapshot that is transferred so you can reverse replication and then
reverse it back, clone both source and destination volumes (which clones
their snapshots) and continue incremental replication of clones starting
from arbitrary snapshot pair, you can cascade replications (including
fan-pout them) and resume incremental replication between arbitrary pair
of systems in replication cascade. Nothing that is even remotely
possible in btrfs. Replication in btrfs is not really suitable for
anything more than offsite backup.

>> If I force different generation numbers for A/date and B/date (by
>> syncing in between) send stream contains correct sequence of removing
>> old B/date (from A clone) and re-creating it again.
>>
>> Which shows that unfortunately generation numbers are not reliable to
>> differentiate between different object generations (pun unintended). As
>> I understand generation is tied to transaction and multiple changes can
>> be packed into one transaction.
> 
> I'm pretty sure that the 6000+ lines of special-case code in send.c still
> don't cover every possible case, or even all of the likely ones, even
> with linear snapshot sequences.  We still get people on IRC reporting
> strange receive issues, and usually the best solution we can find is
> to start over with a new full send.  That's OK for small filesystems,
> but when you have to unexpectedly do a full send of dozens of terabytes
> over a medium-speed link, it's probably time to switch to rsync.
> 
> Subversion used to have problems like this (maybe it still does, I
> switched to git years ago) where a complicated commit that combined
> multiple operations on objects of the same name would break the tool.
> I'm surprised btrfs is trying to do similar things in the kernel
> (though with the current send implementation there's nowhere else we
> could do them).  At least for fsync we get to say "nope, too hard,
> do a full commit instead" when complications arise.
> 
>>> Also I am not sure if using later clone as base for difference to
>>> earlier clone will work for the same reason.
> 
> That use case can come up e.g. if you have snapshots of / and you roll
> back to an earlier snapshot after a bad upgrade, but your backups are
> using incremental snapshots made from '/'.  Then the last-sent-snapshot
> (from the bad upgrade) is newer than the origin subvol (from an earlier
> good upgrade, with new modifications on top).
> 

btrfs does not even support rollback. What is called "rollback" today
works only for root subvolume and is not really rollback but switch to a
different copy.

> Cases like these really need to work, or at least reliably throw
> errors when they have failed, 

Of course :)

> as the application that rolls back to
> earlier snapshots might have no knowledge of the application that does
> incremental send backups on a user's system if they integrated tools
> from different vendors.
> 
>>>> but something happens in one of the support functions and we get a
>>>> bogus link command.  The rest of the stream is OK though:  we fill
>>>> in the contents of B_RO/date, rename A_RO/t to B_RO/u, and update all
>>>> the timestamps.
>>>>
>>>> Oh well, I didn't say send didn't have any bugs.  ;)
>>>>
>>>
>>
>>
> 
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]