is back and forth incremental send/receive supported/stable?

is back and forth incremental send/receive supported/stable?

[Christoph Anton Mitterer]

Hey.

I regularly do the following with btrfs, which seems to work pretty
stable since years:
- having n+1 filesystems MASTER and COPY_n
- creating snapshots on MASTER, e.g. one each month
- incremental send/receive the new snapshot from MASTER to each of
  COPY_n (which already have the previous snapshot)


so for example:
- MASTER has
  - snapshot-2020-11/
  - snapshot-2020-12/
  and newly get's
  - snapshot-2021-01/
- each of COPY_n has only
  - snapshot-2020-11/
  - snapshot-2020-12(
- with:
  # btrfs send -p MASTER/snapshot-2020-12 MASTER/snapshot-2021-01  |  btrfs receive COPY_n/
  I incrementally send the new snapshot from MASTER to each of COPY_n
  using the already available previous snapshot as parent.

Works(TM)



Now I basically want to swap a MASTER with a COPY_n (e.g. because
MASTER's HDD has started to age).

So the plan is e.g.:
- COPY_1 becomes NEW_MASTER
- MASTER becomes OLD_MASTER later known NEW_COPY_1

a) Can I then start e.g. in February to incrementally send/receive from
NEW_MASTER back(!!) to OLD_MASTER?
Like:
# btrfs send -p NEW_MASTER/snapshot-2021-01 NEW_MASTER/snapshot-2021-02  |  btrfs receive OLD_MASTER/

b) And the same from NEW_MSTER to all the other COPY_n?
Like:
# btrfs send -p NEW_MASTER/snapshot-2021-01 NEW_MASTER/snapshot-2021-02  |  btrfs receive COPY_n


So in other words, does btrfs get, that the new parent (which is no
longer on the OLD_MASTER but the previous COPY_1, now NEW_MASTER) is
already present (and identical and usable) on the OLD_MASTER, now
NEW_COPY_1, and also on the other COPY_n ?


By the way, I'm talking about *precious* data, so I'd like to be really
sure that this works... and whether it's intended to work and ideally
have been tested.


Thanks,
Chris.

Re: is back and forth incremental send/receive supported/stable?

By the way, I'm talking about *precious* data, so I'd like to be really
sure that this works... and whether it's intended to work and ideally
have been tested.


Thanks,
Chris.


Hey Chris,
In case everything goes up in flames, just restore from your off-site, cold backup.
Never leave your data in a single place that can burn down.

Cheers,
Cedric
(I'll leave now, don't shoot me)

---

Take your mailboxes with you. Free, fast and secure Mail & Cloud: https://www.eclipso.eu - Time to change!

Re: is back and forth incremental send/receive supported/stable?

[Hugo Mills]

On Fri, Jan 29, 2021 at 08:09:49PM +0100, Christoph Anton Mitterer wrote:
> I regularly do the following with btrfs, which seems to work pretty
> stable since years:
> - having n+1 filesystems MASTER and COPY_n
> - creating snapshots on MASTER, e.g. one each month
> - incremental send/receive the new snapshot from MASTER to each of
>   COPY_n (which already have the previous snapshot)
> 
> 
> so for example:
> - MASTER has
>   - snapshot-2020-11/
>   - snapshot-2020-12/
>   and newly get's
>   - snapshot-2021-01/
> - each of COPY_n has only
>   - snapshot-2020-11/
>   - snapshot-2020-12(
> - with:
>   # btrfs send -p MASTER/snapshot-2020-12 MASTER/snapshot-2021-01  |  btrfs receive COPY_n/
>   I incrementally send the new snapshot from MASTER to each of COPY_n
>   using the already available previous snapshot as parent.
> 
> Works(TM)
> 
> 
> 
> Now I basically want to swap a MASTER with a COPY_n (e.g. because
> MASTER's HDD has started to age).
> 
> So the plan is e.g.:
> - COPY_1 becomes NEW_MASTER
> - MASTER becomes OLD_MASTER later known NEW_COPY_1
> 
> a) Can I then start e.g. in February to incrementally send/receive from
> NEW_MASTER back(!!) to OLD_MASTER?

   No.

   When you make an incremental send, you give it a reference
subvolume with -p. This subvol's UUID is sent in the send stream to
the remote side for receive.

   When receive gets told about a reference subvolume in this way, it
looks for the reference and snapshots it (RW) to use as the base to
apply the incremental on top of.

   The way it finds the reference subvol is to look for a subvol with
the "received_uuid" field matching. This field is set by the receiving
process that made it in the first place (as the result of an earlier
send).

   In your scenario with MASTER and COPY-1 swapped, you'd have to
match the received_uuid from the sending side (on old COPY-1) to the
actual UUID on old MASTER. The code doesn't do this, so you'd have to
patch send/receive to do this.

   Your best bet here is to do a new full send and then continue a new
incremental sequence based on that.

   There's a detailed and fairly formal description of this stuff that
I wrote a few years ago here:

https://www.spinics.net/lists/linux-btrfs/msg44089.html

   Hugo.

> Like:
> # btrfs send -p NEW_MASTER/snapshot-2021-01 NEW_MASTER/snapshot-2021-02  |  btrfs receive OLD_MASTER/
> 
> b) And the same from NEW_MSTER to all the other COPY_n?
> Like:
> # btrfs send -p NEW_MASTER/snapshot-2021-01 NEW_MASTER/snapshot-2021-02  |  btrfs receive COPY_n
> 
> 
> So in other words, does btrfs get, that the new parent (which is no
> longer on the OLD_MASTER but the previous COPY_1, now NEW_MASTER) is
> already present (and identical and usable) on the OLD_MASTER, now
> NEW_COPY_1, and also on the other COPY_n ?
> 
> 
> By the way, I'm talking about *precious* data, so I'd like to be really
> sure that this works... and whether it's intended to work and ideally
> have been tested.
> 
> 
> Thanks,
> Chris.
> 

-- 
Hugo Mills             | You shouldn't anthropomorphise computers. They
hugo@... carfax.org.uk | really don't like that.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Re: is back and forth incremental send/receive supported/stable?

[Christoph Anton Mitterer]

Hey Hugo.


Thanks for your explanation.
I assume such a swapped send/receive would fail at least gracefully?


On Fri, 2021-01-29 at 19:20 +0000, Hugo Mills wrote:
>    In your scenario with MASTER and COPY-1 swapped, you'd have to
> match the received_uuid from the sending side (on old COPY-1) to the
> actual UUID on old MASTER. The code doesn't do this, so you'd have to
> patch send/receive to do this.

Well from the mailing list thread you've referenced it seems that the
whole thing is rather quite non-trivial... so I guess it's nothing for
someone who has basically no insight into btrfs code ^^


It's a pity though, that this doesn't work. Especially the use case of
sending back (backup)snapshots would seem pretty useful.

Given that this thread is nearly 6 years, I'd guess the whole idea has
been abandoned upstream?!


Cheers,
Chris.

Re: is back and forth incremental send/receive supported/stable?

[Hugo Mills]

On Sun, Jan 31, 2021 at 11:50:22PM +0100, Christoph Anton Mitterer wrote:
> Hey Hugo.
> 
> 
> Thanks for your explanation.
> I assume such a swapped send/receive would fail at least gracefully?

   It'll fail *obviously*. I'm not sure how graceful it is. :)

> On Fri, 2021-01-29 at 19:20 +0000, Hugo Mills wrote:
> >    In your scenario with MASTER and COPY-1 swapped, you'd have to
> > match the received_uuid from the sending side (on old COPY-1) to the
> > actual UUID on old MASTER. The code doesn't do this, so you'd have to
> > patch send/receive to do this.
> 
> Well from the mailing list thread you've referenced it seems that the
> whole thing is rather quite non-trivial... so I guess it's nothing for
> someone who has basically no insight into btrfs code ^^
> 
> It's a pity though, that this doesn't work. Especially the use case of
> sending back (backup)snapshots would seem pretty useful.
> 
> Given that this thread is nearly 6 years, I'd guess the whole idea has
> been abandoned upstream?!

   It can be made to work, in a number of different ways -- the option
above is one way; another would be to add extra history of subvolume
identities -- but I guess it's not a priority for the devs, and at
least the latter approach would require extending the on-disk FS
format. Both approaches would need changes to the send stream format.

   Hugo.

-- 
Hugo Mills             | Great oxymorons of the world, no. 7:
hugo@... carfax.org.uk | The Simple Truth
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Re: is back and forth incremental send/receive supported/stable?

[Chris Murphy]

It needs testing but I think -c option can work for this case, because
the parent on both source and destination are identical, even if the
new destination (the old source) has an unexpected received subvolume
uuid.

At least for me, it worked once and I didn't explore it further. I
also don't know if it'll set received uuid, such that subsequent send
can use -p instead of -c.

-c generally still confuses me... in particular multiple instances of -c

--
Chris Murphy

Re: is back and forth incremental send/receive supported/stable?

[Andrei Borzenkov]

02.02.2021 00:53, Chris Murphy пишет:
> It needs testing but I think -c option can work for this case, because
> the parent on both source and destination are identical, even if the
> new destination (the old source) has an unexpected received subvolume
> uuid.
> 

Incremental send requires base snapshot. It is given either as -p option
or "btrfs send" tries to find "best" base snapshot among those given by
-c. This works only if both target snapshot and (some) snapshots given
by -c are created from the same subvolume. So "btrfs send -c foo bar" is
entirely equivalent to "btrfs send -p foo bar" as long as both foo and
bar have the same parent subvolume.

> At least for me, it worked once and I didn't explore it further. I
> also don't know if it'll set received uuid, such that subsequent send
> can use -p instead of -c.
> 

On receive side "btrfs receive" will always set received_uuid to
snapshot given as "btrfs send" argument. The question is whether receive
side will be able to find base (parent) snapshot. As far as I can tell,
"btrfs receive" will first search for subvolume with matching
received_uuid and if nothing is found it will repeat searching for
subvolume with matching uuid. "btrfs send" will send received_uuid
instead of uuid if it is set.

Which means that in case

btrfs subvolume snapshot -r source base
btrfs send base
btrfs subvolume snapshot -r source diff
btrfs send -p base diff

you should be able to reverse replication by doing on destination

btrfs subvolume snapshot diff new-source

this creates writable clone of replication snapshot that will be used to
actually store data. You cannot use replication snapshot itself because
you need its uuid and it must remain read-only

btrfs subvolume snapshot -r new-source diff2
btrfs send -p diff diff2

This should find "diff" on source that has uuid matching received_uuid
of "diff" on destination.

The key here is to not attempt to directly use snapshots involved in
replication. They must remain immutable.

> -c generally still confuses me... in particular multiple instances of -c
> 


Well, it allows you to optimize transfer. if you know that large amount
of data in subvolume foo is reflinked to data in subvolume bar, you can use

btrfs send -c bar foo

and instead of sending (duplicate copy) of data in foo "btrfs receive"
will reflink it from bar on destination. Consider

tw:/mnt/src # btrfs subvolume create source
Create subvolume './source'
tw:/mnt/src # btrfs subvolume create clone
Create subvolume './clone'
tw:/mnt/src # dd if=/dev/urandom of=clone/cln bs=8k count=2
2+0 records in
2+0 records out
16384 bytes (16 kB, 16 KiB) copied, 0.000495937 s, 33.0 MB/s
tw:/mnt/src # btrfs sub snap -r source source-base
Create a readonly snapshot of 'source' in './source-base'
tw:/mnt/src # btrfs sub snap -r clone clone-base
Create a readonly snapshot of 'clone' in './clone-base'
tw:/mnt/src # btrfs send source-base/ | btrfs receive ../dst
At subvol source-base/
At subvol source-base
tw:/mnt/src # btrfs send clone-base/ | btrfs receive ../dst
At subvol clone-base/
At subvol clone-base
tw:/mnt/src # cp --reflink=always clone/cln source
tw:/mnt/src # btrfs sub snap -r source source-diff
Create a readonly snapshot of 'source' in './source-diff'
tw:/mnt/src # btrfs send -p source-base source-diff | btrfs receive --dump
At subvol source-diff
...
write           ./source-diff/cln               offset=0 len=16384
...
tw:/mnt/src # btrfs send -p source-base -c clone-base source-diff |
btrfs receive --dump
At subvol source-diff
...
clone           ./source-diff/cln               offset=0 len=16384
from=./source-diff/cln clone_offset=0
...
tw:/mnt/src #

Ignore weird clone from= in "btrfs receive --dump". It should have
printed clone uuid instead (it does not really have name at this point).

Re: is back and forth incremental send/receive supported/stable?

[Christoph Anton Mitterer]

On Mon, 2021-02-01 at 10:46 +0000, Hugo Mills wrote:
>    It'll fail *obviously*. I'm not sure how graceful it is. :)

Okay that doesn't sound like it was very trustworthy... :-/

Especially this from the manpage:
       You must not specify clone sources unless you guarantee that these
       snapshots are exactly in the same state on both sides—both for the
       sender and the receiver.

I mean what should the user ever be able to guarantee... respectively
what's meant with above?

If the tools or any option combination thereof would allow one to
create corrupted send/received shapthots, then there's not much a user
can do.
If this sentence just means that the user mustn't have manually hacked
some UUIDs or so... well then I guess that's anyway clear and the
sentence is just confusing.


> but I guess it's not a priority for the devs

Since it seems to be a valuable feature with probably little chances to
get it working in the foreseeable future, I've added it as a feature
request to the long term records ;-)
https://bugzilla.kernel.org/show_bug.cgi?id=211521



Cheers,
Chris.

Re: is back and forth incremental send/receive supported/stable?

[Hugo Mills]

On Mon, Feb 01, 2021 at 11:51:06PM +0100, Christoph Anton Mitterer wrote:
> On Mon, 2021-02-01 at 10:46 +0000, Hugo Mills wrote:
> >    It'll fail *obviously*. I'm not sure how graceful it is. :)
> 
> Okay that doesn't sound like it was very trustworthy... :-/
> 
> Especially this from the manpage:
>        You must not specify clone sources unless you guarantee that these
>        snapshots are exactly in the same state on both sides—both for the
>        sender and the receiver.
> 
> I mean what should the user ever be able to guarantee... respectively
> what's meant with above?
> 
> If the tools or any option combination thereof would allow one to
> create corrupted send/received shapthots, then there's not much a user
> can do.
> If this sentence just means that the user mustn't have manually hacked
> some UUIDs or so... well then I guess that's anyway clear and the
> sentence is just confusing.

   It means that (a) the snapshots should exist, and (b) you shouldn't
use the tools to make any of them read-write, make modifications, and
make them read-only again. (and (c), as you say, don't modify the
UUIDs).

   Hugo.

> > but I guess it's not a priority for the devs
> 
> Since it seems to be a valuable feature with probably little chances to
> get it working in the foreseeable future, I've added it as a feature
> request to the long term records ;-)
> https://bugzilla.kernel.org/show_bug.cgi?id=211521
> 
> 
> 
> Cheers,
> Chris.
> 

-- 
Hugo Mills             |
hugo@... carfax.org.uk | __(_'>
http://carfax.org.uk/  | Squeak!
PGP: E2AB1DE4          |

Re: is back and forth incremental send/receive supported/stable?

[Andrei Borzenkov]

02.02.2021 10:53, Hugo Mills пишет:
> On Mon, Feb 01, 2021 at 11:51:06PM +0100, Christoph Anton Mitterer wrote:
>> On Mon, 2021-02-01 at 10:46 +0000, Hugo Mills wrote:
>>>    It'll fail *obviously*. I'm not sure how graceful it is. :)
>>
>> Okay that doesn't sound like it was very trustworthy... :-/
>>
>> Especially this from the manpage:
>>        You must not specify clone sources unless you guarantee that these
>>        snapshots are exactly in the same state on both sides—both for the
>>        sender and the receiver.
>>
>> I mean what should the user ever be able to guarantee... respectively
>> what's meant with above?
>>
>> If the tools or any option combination thereof would allow one to
>> create corrupted send/received shapthots, then there's not much a user
>> can do.
>> If this sentence just means that the user mustn't have manually hacked
>> some UUIDs or so... well then I guess that's anyway clear and the
>> sentence is just confusing.
> 
>    It means that (a) the snapshots should exist, and (b) you shouldn't
> use the tools to make any of them read-write, make modifications, and
> make them read-only again. (and (c), as you say, don't modify the
> UUIDs).
> 

There was patch that cleared received_uuid if snapshot was made
read-write. Not sure what happened to it.