Incremental receive completes succesfully despite missing files

Incremental receive completes succesfully despite missing files

[Dennis K]

Apologies in advance, if the issue I put forward is actually the
intended behavior of BTRFS.

I have noted while playing with sub-volumes, and trying to determine
what exactly are the requirements for a subvolume to act as a legitimate
parent during a receive operation, that modification of one subvolume,
can affect children subvolumes that are received.

It's possible I have noted this before when directories which I though
should have existed in the destination volume, where not present,
despite being present in the snapshot at the sending end.  (ie, a
subvolume is sent incrementally, but the received subvolume is missing
files that exist on the sent side).

I can replicate this as follows

Create the subvolumes and put some files in them.
# btrfs sub create 1
# btrfs sub create 2
# cd 1
# dd if=/dev/urandom bs=1M count=10 of=test
# cd ..
# btrfs sub snap 1 2
# dd if=/dev/urandom bs=1M count=1 of=test2
# cd ..

Now set as read-only to send.  Subvolume 1 has the file "test, and
subvolume 2 has the files "test" and "test2".
# btrfs prop set 1 ro true
# btrfs prop set 2 ro true

Send, snapshot 2 is an incremental send.  The files created are the
expected sizes.
# btrfs send 1 -f /tmp/1
# btrfs send -p 1 2 -f /tmp 2

Now we make subvolume one read-write
# btrfs prop set 1 ro false
# rm 1/test

Delete subvolume 2 and then recreate it be receiving it.
# btrfs sub del 2
# btrfs receive -f /tmp/2 .

What happens, is that subvolume 2 is created, but it is missing the file
'test' which was present in subvolume 1 at the time it was created as a
snapshot and sent.  It now only contains the file "test2", which is NOT
the state that it was sent.


Note the same results are obtained, if you also delete subvolume 1 and
then recreate it with btrfs-receive.

This may explain why previously I found a send operation resulted in the
receiving end missing files previously.

I understand that during send/receive, a snapshot is taken of the parent
subvolume, then it is modified.  The problem is that if that snapshot is
modified, then these modifications will affect the received subvolumes,
including, in this case, silent data loss.


It would be better for the receive operation to fail, or at least put
out a warning if the parent subvolume it is using has changed or is
different from the reference subvolume used during send.  I'm not sure
whether BTRFS can check this via generation number or some other data,
orbut at the moment, there is no such check and this appears to be a bug.

Is this correct behaviour?  Does BTRFS rely on the user, and user-space
tools, never changing any subvolume in order to avoid silent data loss?

Re: Incremental receive completes succesfully despite missing files

[Andrei Borzenkov]

20.01.2019 13:25, Dennis K пишет:
> Apologies in advance, if the issue I put forward is actually the
> intended behavior of BTRFS.
> 
> I have noted while playing with sub-volumes, and trying to determine
> what exactly are the requirements for a subvolume to act as a legitimate
> parent during a receive operation, that modification of one subvolume,
> can affect children subvolumes that are received.
> 
> It's possible I have noted this before when directories which I though
> should have existed in the destination volume, where not present,
> despite being present in the snapshot at the sending end.  (ie, a
> subvolume is sent incrementally, but the received subvolume is missing
> files that exist on the sent side).
> 
> I can replicate this as follows
> 
> Create the subvolumes and put some files in them.
> # btrfs sub create 1
> # btrfs sub create 2
> # cd 1
> # dd if=/dev/urandom bs=1M count=10 of=test
> # cd ..
> # btrfs sub snap 1 2

Apparently some command is missing here.

> # dd if=/dev/urandom bs=1M count=1 of=test2

This creates test2 outside of subvolumes 1 or 2.

> # cd ..
> 

And this goes one level up so that next commands are invalid (they
assume you are still in direct parent of 1 and 2).

Also I do not see what purpose your "btrfs sub snap" serves. It creates
snapshot 2/1, but it snapshot is not part of replication anyway.

> Now set as read-only to send.  Subvolume 1 has the file "test, and
> subvolume 2 has the files "test" and "test2".
> # btrfs prop set 1 ro true
> # btrfs prop set 2 ro true
> 
> Send, snapshot 2 is an incremental send.  The files created are the
> expected sizes.
> # btrfs send 1 -f /tmp/1
> # btrfs send -p 1 2 -f /tmp 2
> 

That must be a typo, from the following text /tmp/2 is implied. Never
manually type in commands; always copy and paste them (or record using
script or similar and attach exact recording). Otherwise nothing in your
report can be trusted.

> Now we make subvolume one read-write
> # btrfs prop set 1 ro false

At this point all bets are off.

> # rm 1/test
> 

Now subvolume 1 no more matches state that was used to generate
incremental stream.

> Delete subvolume 2 and then recreate it be receiving it.
> # btrfs sub del 2
> # btrfs receive -f /tmp/2 .
> 
> What happens, is that subvolume 2 is created, but it is missing the file
> 'test' which was present in subvolume 1 at the time it was created as a
> snapshot and sent.  It now only contains the file "test2", which is NOT
> the state that it was sent.
> 

That is correct. /tmp/2 contains just the *incremental* replication
stream, which contains instructions to apply changes in subvolume 2
against base subvolume 1. It does *not* contain full content of (replica
of) subvolume 2 because on receiving side btrfs would first have cloned
replica of subvolume 1 and then applied changes in replication stream.

> 
> Note the same results are obtained, if you also delete subvolume 1 and
> then recreate it with btrfs-receive.
> 
> This may explain why previously I found a send operation resulted in the
> receiving end missing files previously.
> 
> I understand that during send/receive, a snapshot is taken of the parent
> subvolume, then it is modified.  The problem is that if that snapshot is
> modified, then these modifications will affect the received subvolumes,
> including, in this case, silent data loss.
> 

Not sure I parse this part correctly, but in your case you intentionally
modified base subvolume and made btrfs apply changes to wrong initial
state. This is classical case of "doctor, it hurts when I stab myself in
the eye".

> 
> It would be better for the receive operation to fail, or at least put
> out a warning if the parent subvolume it is using has changed or is
> different from the reference subvolume used during send. 

I was honestly surprised that btrfs receive did not refuse to apply
changes to read-write subvolume. Otherwise replication stream normally
is applied in receiving side which simply does not have enough
information to check that *source* was not changed. Destination only
knows UUID of parent snapshot and assumes it was not changed.

Personally I consider ability to flip read-only bit major usability
issue which leads to problems you observed.

> I'm not sure
> whether BTRFS can check this via generation number or some other data,
> orbut at the moment, there is no such check and this appears to be a bug.
> 
> Is this correct behaviour?  Does BTRFS rely on the user, and user-space
> tools, never changing any subvolume in order to avoid silent data loss?
> 

Yes.

Re: Incremental receive completes succesfully despite missing files

[Chris Murphy]

On Sun, Jan 20, 2019 at 3:34 AM Dennis K <dennisk@netspace.net.au> wrote:
>
> Apologies in advance, if the issue I put forward is actually the
> intended behavior of BTRFS.
>
> I have noted while playing with sub-volumes, and trying to determine
> what exactly are the requirements for a subvolume to act as a legitimate
> parent during a receive operation, that modification of one subvolume,
> can affect children subvolumes that are received.
>
> It's possible I have noted this before when directories which I though
> should have existed in the destination volume, where not present,
> despite being present in the snapshot at the sending end.  (ie, a
> subvolume is sent incrementally, but the received subvolume is missing
> files that exist on the sent side).
>
> I can replicate this as follows
>
> Create the subvolumes and put some files in them.
> # btrfs sub create 1
> # btrfs sub create 2
> # cd 1
> # dd if=/dev/urandom bs=1M count=10 of=test
> # cd ..
> # btrfs sub snap 1 2
> # dd if=/dev/urandom bs=1M count=1 of=test2
> # cd ..
>
> Now set as read-only to send.  Subvolume 1 has the file "test, and
> subvolume 2 has the files "test" and "test2".
> # btrfs prop set 1 ro true
> # btrfs prop set 2 ro true
>
> Send, snapshot 2 is an incremental send.  The files created are the
> expected sizes.
> # btrfs send 1 -f /tmp/1
> # btrfs send -p 1 2 -f /tmp 2
>
> Now we make subvolume one read-write
> # btrfs prop set 1 ro false
> # rm 1/test
>
> Delete subvolume 2 and then recreate it be receiving it.
> # btrfs sub del 2
> # btrfs receive -f /tmp/2 .

This is an unworkable workflow, for multiple reasons. Your 1 and 2
subvolumes are not related to each other, and incremental send should
fail. So I think you might have stumbled on a bug.

(from man page)
       btrfs send [-ve] [-p <parent>] [-c <clone-src>] [-f <outfile>]
<subvol> [<subvol>...]

(simplifed)

       btrfs send [-p <parent>] [<subvol>...


If <parent> has a UUID of 54321, I expect that <subvol> must have
Parent UUID of 54321, or the send command should fail.

Setting aside the possibility for a bug, let's describe the proper
workflow. Your subvolumes 1 and 2 are separate file trees, they're not
related and shouldn't ever appear together in a send/receive. If you
want to do incremental send, to send only changes that happen in 1 and
2 the workflow looks like this.

btrfs sub snap -r 1 1.20190120
btrfs sub snap -r 2 2.20190120
btrfs send 1.20190120 | btrfs receive /destination/
btrfs send 2.20190120 | btrfs recieve /destination/

First the snapshots are from the outset read only. And these are the
snapshots that you will send to the destination file systems, without
p, to initially populate the destination. Second, you will make
changes to original subvolumes 1 and 2, which are still rw subvolumes.
And then and some later time you snapshot them again, thus:

btrfs sub snap -r 1 1.20190121
btrfs sub snap -r 2 2.20190121

And then send the difference or increment with the command:

btrfs send -p 1.20190120 1.20190121 | btrfs receive /destination/
btrfs send -p 2.20190120 2.20190121 | btrfs receive /destination/

And if the original volume dies and you need to restore these
subvolumes to their most recent state:

btrfs send 1.20190121 | btrfs receive /newdestination/
btrfs send 2.20190121 | btrfs receive /newdestination/
btrfs sub snap 1.20190121 1
btrfs sub snap 2.20190121 2

You do not need to do incremental restore, because the subvolume that
appears as a result of an incremental send is *fully* populated, not
merely with just the incremental data. And in this case I take a
default rw snapshot.

I literally never use the 'property set' feature to set ro and unset
ro because I think it's dangerous.


>
> I understand that during send/receive, a snapshot is taken of the parent
> subvolume, then it is modified.  The problem is that if that snapshot is
> modified, then these modifications will affect the received subvolumes,
> including, in this case, silent data loss.


I think this is user error. And the bug is actually a feature request
for better error handling to account for the user inadvertently trying
to do an incremental send with a subvolume that does not have a
matching parent UUID to the -p specified parent subvolume's UUID. I
thought there was a check for this but I'm virtually certain I've run
into this problem myself with no warning and yes it's an unworkable
result at destination.

-- 
Chris Murphy

Re: Incremental receive completes succesfully despite missing files

[Chris Murphy]

On Mon, Jan 21, 2019 at 3:23 PM Chris Murphy <lists@colorremedies.com> wrote:

> I literally never use the 'property set' feature to set ro and unset
> ro because I think it's dangerous.

Also, there is a distinction between 'btrfs snapshot -r' and 'btrfs
property set ro true' - the former creates a snapshot. The later
creates a read only subvolume. They aren't the same thing. If you try
both methods and then point 'btrfs sub show' to each original
subvolume, you will see only the snapshot is tracked as a snapshot,
and only a snaphot properly has its own unique UUID as well as a
referencing parent UUID.

That is why your workflow is failing. Not that 'property set' is
somehow broken; I consider it dangerous to combine a snapshotting
workflow with one that depends on unsetting the ro flag on a snapshot,
where you make changes to that snapshot outside the snapshotting
regimen. Doing so makes that subvolume/snapshot totally invalid as
either a parent or source for send/receive.


-- 
Chris Murphy

Re: Incremental receive completes succesfully despite missing files

[Chris Murphy]

On Mon, Jan 21, 2019 at 3:23 PM Chris Murphy <lists@colorremedies.com> wrote:



> If <parent> has a UUID of 54321, I expect that <subvol> must have
> Parent UUID of 54321, or the send command should fail.

OK I think the following is a reproducible bug.

# btrfs sub create 1
# btrfs sub create 2
# touch 1/one
# touch 2/two
# btrfs sub snap -r 1 1.ro1
# btrfs sub snap -r 2 2.ro1
# touch 1/oneone
# touch 2/twotwo
# btrfs sub snap -r 1 1.ro2
# btrfs sub snap -r 2 2.ro2
# btrfs send 1.ro1 | ssh chris@fnuc.local "sudo btrfs receive /srv/scratch/"
At subvol 1.ro1
At subvol 1.ro1
# btrfs send 2.ro1 | ssh chris@fnuc.local "sudo btrfs receive /srv/scratch/"
At subvol 2.ro1
At subvol 2.ro1
# btrfs send -p 1.ro1 2.ro2 | ssh chris@fnuc.local "sudo btrfs receive
/srv/scratch/"
At subvol 2.ro2
At snapshot 2.ro2
#

I expect the last command to fail because 1.ro1 is not the parent of
2.ro2. The command completes, and 2.ro2 is on the destination, and at
least in this simple example it contains the expected files. However
'btrfs show' indicates that the "parent UUID" of 2.ro2 is the "UUID"
of 1.ro1, which is definitely wrong. So it's a legit bug, not just a
cosmetic problem due to lack of error checking.

$ ls -l 1.ro1
total 0
-rw-rw-r--. 1 chris chris 0 Jan 21 21:41 one
$ ls -l 2.ro1
total 0
-rw-rw-r--. 1 chris chris 0 Jan 21 21:41 two
$ ls -l 2.ro2
total 0
-rw-rw-r--. 1 chris chris 0 Jan 21 21:41 two
-rw-rw-r--. 1 chris chris 0 Jan 21 21:41 twotwo
$


  fnuc.local  chris  /  srv  scratch  sudo btrfs sub show 1.ro1
scratch/1.ro1
    Name:             1.ro1
    UUID:             eb0bb290-3f7d-d34d-94c3-1ed9da96f2fa
    Parent UUID:         -
    Received UUID:         b965a4b6-e77d-714b-af0e-83e15e734fd5
    Creation time:         2019-01-21 21:43:04 -0700
    Subvolume ID:         3107
    Generation:         25788
    Gen at creation:     25782
    Parent ID:         2039
    Top level ID:         2039
    Flags:             readonly
    Snapshot(s):
                scratch/2.ro2
  fnuc.local  chris  /  srv  scratch  sudo btrfs sub show 1.ro2
ERROR: cannot find real path for '1.ro2': No such file or directory
  fnuc.local  chris  /  srv  scratch  sudo btrfs sub show 2.ro1
scratch/2.ro1
    Name:             2.ro1
    UUID:             ce0d762a-d4ea-6448-aa71-1573d70fe60e
    Parent UUID:         -
    Received UUID:         d382f4f9-23c9-394e-852d-ecbe9a95fa75
    Creation time:         2019-01-21 21:43:15 -0700
    Subvolume ID:         3108
    Generation:         25786
    Gen at creation:     25785
    Parent ID:         2039
    Top level ID:         2039
    Flags:             readonly
    Snapshot(s):
  fnuc.local  chris  /  srv  scratch  sudo btrfs sub show 2.ro2
scratch/2.ro2
    Name:             2.ro2
    UUID:             165a4a66-0600-3b49-80aa-edae8fd850d9
    Parent UUID:         eb0bb290-3f7d-d34d-94c3-1ed9da96f2fa
    Received UUID:         b8343d21-882f-bf47-ac61-b59e53b126a4
    Creation time:         2019-01-21 21:43:38 -0700
    Subvolume ID:         3109
    Generation:         25789
    Gen at creation:     25788
    Parent ID:         2039
    Top level ID:         2039
    Flags:             readonly
    Snapshot(s):
  fnuc.local  chris  /  srv  scratch 





-- 
Chris Murphy

Re: Incremental receive completes succesfully despite missing files

[Remi Gauvin]

[-- Attachment #1.1: Type: text/plain, Size: 919 bytes --]

On 2019-01-21 11:54 p.m., Chris Murphy wrote:
#
> 
> I expect the last command to fail because 1.ro1 is not the parent of
> 2.ro2. The command completes, and 2.ro2 is on the destination, and at
> least in this simple example it contains the expected files. However
> 'btrfs show' indicates that the "parent UUID" of 2.ro2 is the "UUID"
> of 1.ro1, which is definitely wrong. So it's a legit bug, not just a
> cosmetic problem due to lack of error checking.
>


This is, as far as I understand, exactly how it's *supposed* to work, at
least, as documented.  The only relationship needed between a "parent"
snapshot, and the snapshot you're sending, is that the parent already
exists on the destination.  Your example would be completely
counter-productive, since there is no data shared between the two, but
perfectly legitimate.  1.ro1 is the parent of 2.ro2 because *you
explicitly told it to*.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Incremental receive completes succesfully despite missing files

[Chris Murphy]

On Mon, Jan 21, 2019 at 11:00 PM Remi Gauvin <remi@georgianit.com> wrote:
>
> On 2019-01-21 11:54 p.m., Chris Murphy wrote:
> #
> >
> > I expect the last command to fail because 1.ro1 is not the parent of
> > 2.ro2. The command completes, and 2.ro2 is on the destination, and at
> > least in this simple example it contains the expected files. However
> > 'btrfs show' indicates that the "parent UUID" of 2.ro2 is the "UUID"
> > of 1.ro1, which is definitely wrong. So it's a legit bug, not just a
> > cosmetic problem due to lack of error checking.
> >
>
>
> This is, as far as I understand, exactly how it's *supposed* to work, at
> least, as documented.  The only relationship needed between a "parent"
> snapshot, and the snapshot you're sending, is that the parent already
> exists on the destination.

I disagree. From the man page:

       -p <parent>
           send an incremental stream from parent to subvol

You have to understand that the send and receive commands is a
unidirectional stream. The receive command can't talk to the send
side. It can only pass or fail. So to get the correct incremental send
stream, the send command must be correctly formed based on specific
parent child relationship.


>  Your example would be completely
> counter-productive, since there is no data shared between the two, but
> perfectly legitimate.  1.ro1 is the parent of 2.ro2 because *you
> explicitly told it to*.

It's not legitimate, it's a mistake by the user. The actual stream
ends up being a full send of 2.ro2, rather than an incremental, and
then on the destination 2.ro2 ends up showing a parent UUID for a
subvolume that's totally unrelated. It's just not sane.


-- 
Chris Murphy

Re: Incremental receive completes succesfully despite missing files

[Andrei Borzenkov]

22.01.2019 9:28, Chris Murphy пишет:
> On Mon, Jan 21, 2019 at 11:00 PM Remi Gauvin <remi@georgianit.com> wrote:
>>
>> On 2019-01-21 11:54 p.m., Chris Murphy wrote:
>> #
>>>
>>> I expect the last command to fail because 1.ro1 is not the parent of
>>> 2.ro2. The command completes, and 2.ro2 is on the destination, and at
>>> least in this simple example it contains the expected files. However
>>> 'btrfs show' indicates that the "parent UUID" of 2.ro2 is the "UUID"
>>> of 1.ro1, which is definitely wrong. So it's a legit bug, not just a
>>> cosmetic problem due to lack of error checking.
>>>
>>
>>
>> This is, as far as I understand, exactly how it's *supposed* to work, at
>> least, as documented.  The only relationship needed between a "parent"
>> snapshot, and the snapshot you're sending, is that the parent already
>> exists on the destination.
> 
> I disagree. From the man page:
> 
>        -p <parent>
>            send an incremental stream from parent to subvol
> 
> You have to understand that the send and receive commands is a
> unidirectional stream. The receive command can't talk to the send
> side. It can only pass or fail. So to get the correct incremental send
> stream, the send command must be correctly formed based on specific
> parent child relationship.
> 

"Parent" is too ambiguous in btrfs as it is used to denote completely
independent things. Snapshot parent-child relationship is unrelated to
"parent" as used in "btrfs send".

Subvolume referred to by "btrfs send -p" is used as base to compute
changes against. Nothing more nothing less. There is no implication that
subvolume that is being transferred is snapshot of subvolume referred to
by -p argument. Actually the most common scheme is - you have base
subvolume "base"m you periodically create snapshots from these subvolume
as "btrfs sub snap -r base snap1", "btrfs sub snap -r base snap2" etc
and then do incremental transfer between each snapshots as "btrfs send
-p snap1 snap2" etc. Neither of "snapX" is snapshot of another "snapX".
You have fan-out structure on source and linear structure on destination.

> 
>>  Your example would be completely
>> counter-productive, since there is no data shared between the two, but
>> perfectly legitimate.  1.ro1 is the parent of 2.ro2 because *you
>> explicitly told it to*.
> 
> It's not legitimate, it's a mistake by the user. The actual stream
> ends up being a full send of 2.ro2, rather than an incremental, and
> then on the destination 2.ro2 ends up showing a parent UUID for a
> subvolume that's totally unrelated. It's just not sane.
> 
> 

It is still entirely legal operation. The result will be full
replication stream of 2.ro2 + additional instructions to delete content
of (clone of) 1.ro1 on destination.

"Related" is in the eye of the beholder. Clone subvolume, delete content
of clone, reflink content of another volume into clone. Are original
subvolume and clone related now? Clone still have parent UUID ...

Re: Incremental receive completes succesfully despite missing files

[Chris Murphy]

On Tue, Jan 22, 2019 at 10:57 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
> 22.01.2019 9:28, Chris Murphy пишет:
> > On Mon, Jan 21, 2019 at 11:00 PM Remi Gauvin <remi@georgianit.com> wrote:
> >>
> >> On 2019-01-21 11:54 p.m., Chris Murphy wrote:
> >> #
> >>>
> >>> I expect the last command to fail because 1.ro1 is not the parent of
> >>> 2.ro2. The command completes, and 2.ro2 is on the destination, and at
> >>> least in this simple example it contains the expected files. However
> >>> 'btrfs show' indicates that the "parent UUID" of 2.ro2 is the "UUID"
> >>> of 1.ro1, which is definitely wrong. So it's a legit bug, not just a
> >>> cosmetic problem due to lack of error checking.
> >>>
> >>
> >>
> >> This is, as far as I understand, exactly how it's *supposed* to work, at
> >> least, as documented.  The only relationship needed between a "parent"
> >> snapshot, and the snapshot you're sending, is that the parent already
> >> exists on the destination.
> >
> > I disagree. From the man page:
> >
> >        -p <parent>
> >            send an incremental stream from parent to subvol
> >
> > You have to understand that the send and receive commands is a
> > unidirectional stream. The receive command can't talk to the send
> > side. It can only pass or fail. So to get the correct incremental send
> > stream, the send command must be correctly formed based on specific
> > parent child relationship.
> >
>
> "Parent" is too ambiguous in btrfs as it is used to denote completely
> independent things. Snapshot parent-child relationship is unrelated to
> "parent" as used in "btrfs send".
>
> Subvolume referred to by "btrfs send -p" is used as base to compute
> changes against. Nothing more nothing less. There is no implication that
> subvolume that is being transferred is snapshot of subvolume referred to
> by -p argument. Actually the most common scheme is - you have base
> subvolume "base"m you periodically create snapshots from these subvolume
> as "btrfs sub snap -r base snap1", "btrfs sub snap -r base snap2" etc
> and then do incremental transfer between each snapshots as "btrfs send
> -p snap1 snap2" etc. Neither of "snapX" is snapshot of another "snapX".
> You have fan-out structure on source and linear structure on destination.

base is the parent of all snapX

A totally unrelated subvolume "noodle" being used as -p is obviously
user error, it doesn't matter if it doesn't cause corruption. For sure
the resulting subvolume has the wrong "parent UUID" as a result, so
there are at least two unintended consequences.


> >>  Your example would be completely
> >> counter-productive, since there is no data shared between the two, but
> >> perfectly legitimate.  1.ro1 is the parent of 2.ro2 because *you
> >> explicitly told it to*.
> >
> > It's not legitimate, it's a mistake by the user. The actual stream
> > ends up being a full send of 2.ro2, rather than an incremental, and
> > then on the destination 2.ro2 ends up showing a parent UUID for a
> > subvolume that's totally unrelated. It's just not sane.
> >
> >
>
> It is still entirely legal operation. The result will be full
> replication stream of 2.ro2 + additional instructions to delete content
> of (clone of) 1.ro1 on destination.

Which is utterly absurd because the user, by virtue of using -p flag,
is indicating they want an incremental send and yet silently they will
not get an incremental send. They very clearly made a mistake.

> "Related" is in the eye of the beholder. Clone subvolume, delete content
> of clone, reflink content of another volume into clone. Are original
> subvolume and clone related now? Clone still have parent UUID ...

I'm not talking about -c option. Just -p. Conceptually -c is even more
complicated and effectively supports multiple "parents" as it can be
specified more than once.




-- 
Chris Murphy

Re: Incremental receive completes succesfully despite missing files

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

On Tue, Jan 22, 2019 at 12:37:34PM -0700, Chris Murphy wrote:
> On Tue, Jan 22, 2019 at 10:57 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> > "Related" is in the eye of the beholder. Clone subvolume, delete content
> > of clone, reflink content of another volume into clone. Are original
> > subvolume and clone related now? Clone still have parent UUID ...
> 
> I'm not talking about -c option. Just -p. Conceptually -c is even more
> complicated and effectively supports multiple "parents" as it can be
> specified more than once.

   I tend to use the term "reference" for the -p subvolume, and the -c
subvolume(s).

   Hugo.

-- 
Hugo Mills             | The future isn't what it used to be.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Incremental receive completes succesfully despite missing files

[Dennis Katsonis]

On 1/22/19 9:23 AM, Chris Murphy wrote:
> On Sun, Jan 20, 2019 at 3:34 AM Dennis K <dennisk@netspace.net.au> wrote:
>>
>> Apologies in advance, if the issue I put forward is actually the
>> intended behavior of BTRFS.
>>
>> I have noted while playing with sub-volumes, and trying to determine
>> what exactly are the requirements for a subvolume to act as a legitimate
>> parent during a receive operation, that modification of one subvolume,
>> can affect children subvolumes that are received.
>>
>> It's possible I have noted this before when directories which I though
>> should have existed in the destination volume, where not present,
>> despite being present in the snapshot at the sending end.  (ie, a
>> subvolume is sent incrementally, but the received subvolume is missing
>> files that exist on the sent side).
>>
>> I can replicate this as follows
>>
>> Create the subvolumes and put some files in them.
>> # btrfs sub create 1
>> # btrfs sub create 2
>> # cd 1
>> # dd if=/dev/urandom bs=1M count=10 of=test
>> # cd ..
>> # btrfs sub snap 1 2
>> # dd if=/dev/urandom bs=1M count=1 of=test2
>> # cd ..
>>
>> Now set as read-only to send.  Subvolume 1 has the file "test, and
>> subvolume 2 has the files "test" and "test2".
>> # btrfs prop set 1 ro true
>> # btrfs prop set 2 ro true
>>
>> Send, snapshot 2 is an incremental send.  The files created are the
>> expected sizes.
>> # btrfs send 1 -f /tmp/1
>> # btrfs send -p 1 2 -f /tmp 2
>>
>> Now we make subvolume one read-write
>> # btrfs prop set 1 ro false
>> # rm 1/test
>>
>> Delete subvolume 2 and then recreate it be receiving it.
>> # btrfs sub del 2
>> # btrfs receive -f /tmp/2 .
> 
> This is an unworkable workflow, for multiple reasons. Your 1 and 2
> subvolumes are not related to each other, and incremental send should
> fail. So I think you might have stumbled on a bug.
> 
> (from man page)
>        btrfs send [-ve] [-p <parent>] [-c <clone-src>] [-f <outfile>]
> <subvol> [<subvol>...]
> 
> (simplifed)
> 
>        btrfs send [-p <parent>] [<subvol>...
> 
> 
> If <parent> has a UUID of 54321, I expect that <subvol> must have
> Parent UUID of 54321, or the send command should fail.
> 
> Setting aside the possibility for a bug, let's describe the proper
> workflow. Your subvolumes 1 and 2 are separate file trees, they're not
> related and shouldn't ever appear together in a send/receive. If you
> want to do incremental send, to send only changes that happen in 1 and
> 2 the workflow looks like this.
> 
> btrfs sub snap -r 1 1.20190120
> btrfs sub snap -r 2 2.20190120
> btrfs send 1.20190120 | btrfs receive /destination/
> btrfs send 2.20190120 | btrfs recieve /destination/
> 
> First the snapshots are from the outset read only. And these are the
> snapshots that you will send to the destination file systems, without
> p, to initially populate the destination. Second, you will make
> changes to original subvolumes 1 and 2, which are still rw subvolumes.
> And then and some later time you snapshot them again, thus:
> 
> btrfs sub snap -r 1 1.20190121
> btrfs sub snap -r 2 2.20190121
> 
> And then send the difference or increment with the command:
> 
> btrfs send -p 1.20190120 1.20190121 | btrfs receive /destination/
> btrfs send -p 2.20190120 2.20190121 | btrfs receive /destination/
> 
> And if the original volume dies and you need to restore these
> subvolumes to their most recent state:
> 
> btrfs send 1.20190121 | btrfs receive /newdestination/
> btrfs send 2.20190121 | btrfs receive /newdestination/
> btrfs sub snap 1.20190121 1
> btrfs sub snap 2.20190121 2
> 
> You do not need to do incremental restore, because the subvolume that
> appears as a result of an incremental send is *fully* populated, not
> merely with just the incremental data. And in this case I take a
> default rw snapshot.
> 
> I literally never use the 'property set' feature to set ro and unset
> ro because I think it's dangerous.

I think my previous e-mail did not go through.  Basically, if it is
assumed that a btrfs-receive operation will result in a subvolume which
matches the source file for file, then this assumption or expectation
won't be met if one deletes files from the subvolume at the receiving
end which is going to be referred to as the parent.

This can happen inadvertently, or even through filesystem corruption
(which I experienced).


> 
> 
>>
>> I understand that during send/receive, a snapshot is taken of the parent
>> subvolume, then it is modified.  The problem is that if that snapshot is
>> modified, then these modifications will affect the received subvolumes,
>> including, in this case, silent data loss.
> 
> 
> I think this is user error. And the bug is actually a feature request
> for better error handling to account for the user inadvertently trying
> to do an incremental send with a subvolume that does not have a
> matching parent UUID to the -p specified parent subvolume's UUID. I
> thought there was a check for this but I'm virtually certain I've run
> into this problem myself with no warning and yes it's an unworkable
> result at destination.
> 

I do note that btrfs-send man page states "You must not specify clone
sources unless you guarantee that these snapshots are exactly in the
same state on both sides—both for the sender and the receiver.".
Perhaps this could be changed to state "clone or parent", as it could be
interpreted as not being applicable if you specify a parent only.

Re: Incremental receive completes succesfully despite missing files

[Andrei Borzenkov]

On Wed, Jan 23, 2019 at 1:45 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
> I think my previous e-mail did not go through.  Basically, if it is
> assumed that a btrfs-receive operation will result in a subvolume which
> matches the source file for file, then this assumption or expectation
> won't be met if one deletes files from the subvolume at the receiving
> end which is going to be referred to as the parent.
>

This is oxymoron. btrfs send/receive apply to read-only subvolumes.
You are not able to modify them. As soon as you remove read-only bit,
you are fully responsible for consequences.

> This can happen inadvertently,

It cannot. You do not inadvertently make subvolume read-write. And if
you do, you are expected to know what you are doing.

That said, better if btrfs did not allow flipping read-only bit in the
first place.

> or even through filesystem corruption
> (which I experienced).
>

And if corruption happened after applying changes? End result in the
same. Of course it would be perfect if btrfs could notice and warn
you, I just do not see how it can realistically be implemented.

Re: Incremental receive completes succesfully despite missing files

[Dennis Katsonis]

On 1/23/19 10:25 PM, Andrei Borzenkov wrote:
> On Wed, Jan 23, 2019 at 1:45 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>> I think my previous e-mail did not go through.  Basically, if it is
>> assumed that a btrfs-receive operation will result in a subvolume which
>> matches the source file for file, then this assumption or expectation
>> won't be met if one deletes files from the subvolume at the receiving
>> end which is going to be referred to as the parent.
>>
> 
> This is oxymoron. btrfs send/receive apply to read-only subvolumes.
> You are not able to modify them. 


> 
>> This can happen inadvertently,
> 
> It cannot. You do not inadvertently make subvolume read-write. 
It's not the individual steps which happen inadvertently, its that
particular workflow which can happen (i.e.; someone decides to play
around with a replicated snapshot, and flips the ro bit to do so).  Any
subsequent children sent later won't be true replicates on the receiving
end.

They would have to either make a habit of not flipping that ro bit at
all (there is nothing to say don't do it, and there are valid reasons to
do so) or anticipate they may need that subvolume as a parent anytime in
the future and that parents must be the same on both ends (there is
nothing which states that parents have to be identical on both ends for
btrfs-receive to operate).  If they do modify ANY subvolume created by
btrfs-receive, they must remember in the future somehow never to use
them as parents.

BTRFS allows an ad-hoc, unstructured approach to managing snapshots and
replication, but the cost I suppose is there are more corner cases and
more potential for users to use the system in a manner which differs
from the devs.


> 
> That said, better if btrfs did not allow flipping read-only bit in the
> first place.
> 
>> or even through filesystem corruption
>> (which I experienced).
>>
> 
> And if corruption happened after applying changes? End result in the
> same. Of course it would be perfect if btrfs could notice and warn
> you, I just do not see how it can realistically be implemented.
>If the tools can't prevent it, the documentation should, and should also
ideally describe the mode of failure (i.e., if the subvolumes are
different, btrfs-receive won't necessarily fail, but can produce a
subvolume which is not identical to that sent.

At the moment, one must infer that the statement that clones must be
identical at both ends.

Re: Incremental receive completes succesfully despite missing files

[Andrei Borzenkov]

23.01.2019 16:52, Dennis Katsonis пишет:
> On 1/23/19 10:25 PM, Andrei Borzenkov wrote:
>> On Wed, Jan 23, 2019 at 1:45 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>>> I think my previous e-mail did not go through.  Basically, if it is
>>> assumed that a btrfs-receive operation will result in a subvolume which
>>> matches the source file for file, then this assumption or expectation
>>> won't be met if one deletes files from the subvolume at the receiving
>>> end which is going to be referred to as the parent.
>>>
>>
>> This is oxymoron. btrfs send/receive apply to read-only subvolumes.
>> You are not able to modify them. 
> 
> 
>>
>>> This can happen inadvertently,
>>
>> It cannot. You do not inadvertently make subvolume read-write. 
> It's not the individual steps which happen inadvertently, its that
> particular workflow which can happen (i.e.; someone decides to play
> around with a replicated snapshot, and flips the ro bit to do so).  Any
> subsequent children sent later won't be true replicates on the receiving
> end.
> 
> They would have to either make a habit of not flipping that ro bit at
> all (there is nothing to say don't do it, and there are valid reasons to
> do so) or anticipate they may need that subvolume as a parent anytime in
> the future and that parents must be the same on both ends (there is
> nothing which states that parents have to be identical on both ends for
> btrfs-receive to operate).

You cannot have your cake and eat it. Either you want it to be fool
proof or you want extreme flexibility. In the former case it must block
certain dangerous actions. In the latter case it is up to you - you got
what you asked for.

>  If they do modify ANY subvolume created by
> btrfs-receive, they must remember in the future somehow never to use
> them as parents.
> 

Both NetApp and ZFS distinguish between read-only snapshots and
read-write clones. If you want to obtain read-write access to data in
snapshot you clone it. May be we should at least think about why this
workflow did not change in 25+ years.

Problem discussed in this thread simply cannot happen there.

Re: Incremental receive completes succesfully despite missing files

[Hans van Kranenburg]

On 1/23/19 12:25 PM, Andrei Borzenkov wrote:
> On Wed, Jan 23, 2019 at 1:45 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>> I think my previous e-mail did not go through.  Basically, if it is
>> assumed that a btrfs-receive operation will result in a subvolume which
>> matches the source file for file, then this assumption or expectation
>> won't be met if one deletes files from the subvolume at the receiving
>> end which is going to be referred to as the parent.
>>
> 
> This is oxymoron. btrfs send/receive apply to read-only subvolumes.
> You are not able to modify them. As soon as you remove read-only bit,
> you are fully responsible for consequences.
> 
>> This can happen inadvertently,
> 
> It cannot. You do not inadvertently make subvolume read-write. And if
> you do, you are expected to know what you are doing.
> 
> That said, better if btrfs did not allow flipping read-only bit in the
> first place.

+100

But then only disallow if the subvol has a value in the received_uuid
field, I'd say.

And the tooling could display a nice error message, telling the user to
make a rw snapshot of the received subvol instead, which will have an
empty received_uuid field again.

ERROR: subvolume is a received subvolume, which cannot be made read-write
ERROR: suggestion; make a read-write snapshot of it instead

Setting something ro manually (e.g. to prevent accidental deletion of a
static set of data) is still also a use case.

>> or even through filesystem corruption
>> (which I experienced).
>>
> 
> And if corruption happened after applying changes? End result in the
> same. Of course it would be perfect if btrfs could notice and warn
> you, I just do not see how it can realistically be implemented.
> 


-- 
Hans van Kranenburg

Re: Incremental receive completes succesfully despite missing files

[Nikolay Borisov]

On 23.01.19 г. 17:25 ч., Hans van Kranenburg wrote:
> On 1/23/19 12:25 PM, Andrei Borzenkov wrote:
>> On Wed, Jan 23, 2019 at 1:45 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>>> I think my previous e-mail did not go through.  Basically, if it is
>>> assumed that a btrfs-receive operation will result in a subvolume which
>>> matches the source file for file, then this assumption or expectation
>>> won't be met if one deletes files from the subvolume at the receiving
>>> end which is going to be referred to as the parent.
>>>
>>
>> This is oxymoron. btrfs send/receive apply to read-only subvolumes.
>> You are not able to modify them. As soon as you remove read-only bit,
>> you are fully responsible for consequences.
>>
>>> This can happen inadvertently,
>>
>> It cannot. You do not inadvertently make subvolume read-write. And if
>> you do, you are expected to know what you are doing.
>>
>> That said, better if btrfs did not allow flipping read-only bit in the
>> first place.
> 
> +100
> 
> But then only disallow if the subvol has a value in the received_uuid
> field, I'd say.
> 
> And the tooling could display a nice error message, telling the user to
> make a rw snapshot of the received subvol instead, which will have an
> empty received_uuid field again.
> 
> ERROR: subvolume is a received subvolume, which cannot be made read-write
> ERROR: suggestion; make a read-write snapshot of it instead
> 
> Setting something ro manually (e.g. to prevent accidental deletion of a
> static set of data) is still also a use case.

I had a patch which when the ro bit is flipped on a snapshot, then it's
parent snapshot uuid etc is deleted. Because at that point it cannot be
guaranteed that there is any relationship between paren<->snapshot. Ie:

https://lore.kernel.org/linux-btrfs/1520847020-8049-1-git-send-email-nborisov@suse.com/

has anyone tried it?

> 
>>> or even through filesystem corruption
>>> (which I experienced).
>>>
>>
>> And if corruption happened after applying changes? End result in the
>> same. Of course it would be perfect if btrfs could notice and warn
>> you, I just do not see how it can realistically be implemented.
>>
> 
> 

Re: Incremental receive completes succesfully despite missing files

[Hans van Kranenburg]

On 1/23/19 4:32 PM, Nikolay Borisov wrote:
> 
> 
> On 23.01.19 г. 17:25 ч., Hans van Kranenburg wrote:
>> On 1/23/19 12:25 PM, Andrei Borzenkov wrote:
>>> On Wed, Jan 23, 2019 at 1:45 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>>>> I think my previous e-mail did not go through.  Basically, if it is
>>>> assumed that a btrfs-receive operation will result in a subvolume which
>>>> matches the source file for file, then this assumption or expectation
>>>> won't be met if one deletes files from the subvolume at the receiving
>>>> end which is going to be referred to as the parent.
>>>>
>>>
>>> This is oxymoron. btrfs send/receive apply to read-only subvolumes.
>>> You are not able to modify them. As soon as you remove read-only bit,
>>> you are fully responsible for consequences.
>>>
>>>> This can happen inadvertently,
>>>
>>> It cannot. You do not inadvertently make subvolume read-write. And if
>>> you do, you are expected to know what you are doing.
>>>
>>> That said, better if btrfs did not allow flipping read-only bit in the
>>> first place.
>>
>> +100
>>
>> But then only disallow if the subvol has a value in the received_uuid
>> field, I'd say.
>>
>> And the tooling could display a nice error message, telling the user to
>> make a rw snapshot of the received subvol instead, which will have an
>> empty received_uuid field again.
>>
>> ERROR: subvolume is a received subvolume, which cannot be made read-write
>> ERROR: suggestion; make a read-write snapshot of it instead
>>
>> Setting something ro manually (e.g. to prevent accidental deletion of a
>> static set of data) is still also a use case.
> 
> I had a patch which when the ro bit is flipped on a snapshot, then it's
> parent snapshot uuid etc is deleted. Because at that point it cannot be
> guaranteed that there is any relationship between paren<->snapshot. Ie:
> 
> https://lore.kernel.org/linux-btrfs/1520847020-8049-1-git-send-email-nborisov@suse.com/
> 
> has anyone tried it?

Oh, it seems that one never got any reply.

When looking at it from an end user perspective, I'd much more like to
have it explicitly tell me I'm doing something stupid, refuse and
educate me than obeying and adding more implicit behaviour that I cannot
revert any more myself.

E.g. I'm a new btrfs user, I just sent this 1TiB of data for the first
time, and now I'm of course curious and trying more things out.

Trying setting rw ('ooh! that works') and then quickly setting ro back
will destroy the send/receive relationship without mentioning anything
to the user, and in combination with subsequent cryptic errors from
incremental send/receive ("cannot find parent subvolume" or something)
it will end up with smashing keyboards and throwing computers out of the
window.

After asking on IRC and providing output of btrfs sub list etc the user
is told that that subvol was never received because it doesn't have a
received_uuid, and then someone asks "did you turn it rw at some
point?". Now the user remembers, "yes, I was experimenting and I didn't
see an error or warning", and is then told to resend the 1TiB of data.
"But it's there already!" Now someone comes up with a snippet of python
code to reset the received_uuid with the ioctl, and...

:D

>>>> or even through filesystem corruption
>>>> (which I experienced).
>>>>
>>>
>>> And if corruption happened after applying changes? End result in the
>>> same. Of course it would be perfect if btrfs could notice and warn
>>> you, I just do not see how it can realistically be implemented.
>>>
>>
>>


-- 
Hans van Kranenburg

Re: Incremental receive completes succesfully despite missing files

[Dennis K]

On 24/1/19 3:23 am, Hans van Kranenburg wrote:
> On 1/23/19 4:32 PM, Nikolay Borisov wrote:
>>
>>
>> On 23.01.19 г. 17:25 ч., Hans van Kranenburg wrote:
>>> On 1/23/19 12:25 PM, Andrei Borzenkov wrote:
>>>> On Wed, Jan 23, 2019 at 1:45 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>>>>> I think my previous e-mail did not go through.  Basically, if it is
>>>>> assumed that a btrfs-receive operation will result in a subvolume which
>>>>> matches the source file for file, then this assumption or expectation
>>>>> won't be met if one deletes files from the subvolume at the receiving
>>>>> end which is going to be referred to as the parent. 
>>>>>
>>>>
>>>> This is oxymoron. btrfs send/receive apply to read-only subvolumes.
>>>> You are not able to modify them. As soon as you remove read-only bit,
>>>> you are fully responsible for consequences.
>>>>
>>>>> This can happen inadvertently,
>>>>
>>>> It cannot. You do not inadvertently make subvolume read-write. And if
>>>> you do, you are expected to know what you are doing.
>>>>
>>>> That said, better if btrfs did not allow flipping read-only bit in the
>>>> first place.
>>>
>>> +100
>>>
>>> But then only disallow if the subvol has a value in the received_uuid
>>> field, I'd say.
>>>
>>> And the tooling could display a nice error message, telling the user to
>>> make a rw snapshot of the received subvol instead, which will have an
>>> empty received_uuid field again.
>>>
>>> ERROR: subvolume is a received subvolume, which cannot be made read-write
>>> ERROR: suggestion; make a read-write snapshot of it instead
>>>
>>> Setting something ro manually (e.g. to prevent accidental deletion of a
>>> static set of data) is still also a use case.
>>
>> I had a patch which when the ro bit is flipped on a snapshot, then it's
>> parent snapshot uuid etc is deleted. Because at that point it cannot be
>> guaranteed that there is any relationship between paren<->snapshot. Ie:
>>
>> https://lore.kernel.org/linux-btrfs/1520847020-8049-1-git-send-email-nborisov@suse.com/
>>
>> has anyone tried it?
> 
> Oh, it seems that one never got any reply.
> 
> When looking at it from an end user perspective, I'd much more like to
> have it explicitly tell me I'm doing something stupid, refuse and
> educate me than obeying and adding more implicit behaviour that I cannot
> revert any more myself.
> 
It was the new user I had primarily in mind.  Btrfs is the default for
OpenSUSE, and may become the default for more distros.  Thats a lot of
new users, unaware of the vaguaries of COW filesystems who first learn
how to use the tools to solve their problems, before they learn the
"gotchas".

The fact is, this thread is the first time I've seen explicitly written
that parents must be the same at receiving and sending ends, or else
btrfs-send/receive will produce a subvolume which differs from the source.

A user could from the descriptions of how send/receive formats its data
strea, and from interpreting "clone sources" in a manner contrary to how
its used in the man page and the mention of incrementals, conclude that
send/receive is sensitive to changes at the receiving side, but its far,
far more likely that they will start using feature like send/receive and
setting snapshots to ro before they delve into how it works its magic.
Anyway, rsync sends incremental change, and its ability to replicate a
directory tree isn't predicated on the files present at the receiving
end being the same as the last time it was run...




> E.g. I'm a new btrfs user, I just sent this 1TiB of data for the first
> time, and now I'm of course curious and trying more things out.
> 
> Trying setting rw ('ooh! that works') and then quickly setting ro back
> will destroy the send/receive relationship without mentioning anything
> to the user, and in combination with subsequent cryptic errors from
> incremental send/receive ("cannot find parent subvolume" or something)
> it will end up with smashing keyboards and throwing computers out of the
> window.
> 
> After asking on IRC and providing output of btrfs sub list etc the user
> is told that that subvol was never received because it doesn't have a
> received_uuid, and then someone asks "did you turn it rw at some
> point?". Now the user remembers, "yes, I was experimenting and I didn't
> see an error or warning", and is then told to resend the 1TiB of data.
> "But it's there already!" Now someone comes up with a snippet of python
> code to reset the received_uuid with the ioctl, and...
> 
> :D
> 
>>>>> or even through filesystem corruption
>>>>> (which I experienced).
>>>>>
>>>>
>>>> And if corruption happened after applying changes? End result in the
>>>> same. Of course it would be perfect if btrfs could notice and warn
>>>> you, I just do not see how it can realistically be implemented.
>>>>
>>>
>>>
> 
> 

Re: Incremental receive completes succesfully despite missing files

[Chris Murphy]

On Thu, Jan 24, 2019 at 3:40 AM Dennis K <dennisk@netspace.net.au> wrote:
>
> The fact is, this thread is the first time I've seen explicitly written
> that parents must be the same at receiving and sending ends, or else
> btrfs-send/receive will produce a subvolume which differs from the source.

The central user error, as well as btrfs-progs bug is the failure to
meet the requirement that the source(s) be snapshots. Either a full
send, or an incremental send, whether with -p or -c, all of them must
be snapshots. And none of yours were snapshots. They were read-only
subvolumes made using 'btrfs property' to set the read-only flag, and
those are not snapshots.

The btrfs send command should have explicitly failed on that alone.

In my reproducer of a related user error and bug, both were snapshots,
but they were unrelated snapshots, therefore incremental send was
impossible, and the only explanation for such a command is the user is
confused, made a mistake, and therefore the command should have
failed.

>
> A user could from the descriptions of how send/receive formats its data
> strea, and from interpreting "clone sources" in a manner contrary to how
> its used in the man page and the mention of incrementals, conclude that
> send/receive is sensitive to changes at the receiving side, but its far,
> far more likely that they will start using feature like send/receive and
> setting snapshots to ro before they delve into how it works its magic.

Uhhh I think this is a specious statement if you think users are
likely to delve into 'btrfs property' and somehow think setting a
subvolume ro is making a snapshot. You're the first I've encountered
who arrived at this logic. For now almost a decade people arrive at
'btrfs subvolume snapshot' as the way to make snapshots, and the -r
flag to make them read-only snapshots.

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Snapshots

That page only describes snapshots in terms of the 'btrfs subvolume
snapshot' command - the 'btrfs property' command isn't found anywhere
on that page. So I don't actually know how you came to think setting
the ro property on a subvolume to true, makes the subvolume a
snapshot. I think you did it through assumption. And while there might
be a way to fix some things to avoid such confusion in the future, I
do think it's an edge case.


The man 8 btrfs man page could amend:

       subvolume
           Create/delete/list/manage btrfs subvolume.

to

       subvolume
           Create/delete/list/manage btrfs subvolumes and snapshots.

it is something of btrfs geekdom that snapshots are subvolumes; but
still not all subvolumes are snapshots. So...yeah, clarity is a good
thing even if there some redundancy.

> Anyway, rsync sends incremental change, and its ability to replicate a
> directory tree isn't predicated on the files present at the receiving
> end being the same as the last time it was run...

What? Yes it is. By default it compares size and time stamp on both
sides. If you use -c it uses checksums to compare. From the rsync man
page:

>Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in  size  or  in  last-modified
       time.

Rsync does not keep a database on source or destination. It absolutely
is comparing both sides, and if they aren't the same, the file is
synced from the source to the destination.

Also, the man page for btrfs receive very clearly says the receive
side snapshot must be present and must not have been changed since the
last receive, or it will fail.

>       btrfs receive will fail in the following cases:
>       1. receiving subvolume already exists
>       2. previously received subvolume has been changed after it was received

I'm still really not following where your confusion stems from, and
therefore I'm not sure what needs fixing other than the items I've
already mentioned - which itself at least would have stopped you in
your tracks, to go dig deeper or ask questions before arriving at the
understandably confusing results you were getting.

-- 
Chris Murphy

Re: Incremental receive completes succesfully despite missing files

[Dennis Katsonis]

On 1/25/19 4:22 AM, Chris Murphy wrote:
> On Thu, Jan 24, 2019 at 3:40 AM Dennis K <dennisk@netspace.net.au> wrote:
>>
>> The fact is, this thread is the first time I've seen explicitly written
>> that parents must be the same at receiving and sending ends, or else
>> btrfs-send/receive will produce a subvolume which differs from the source.
> 
> The central user error, as well as btrfs-progs bug is the failure to
> meet the requirement that the source(s) be snapshots. Either a full
> send, or an incremental send, whether with -p or -c, all of them must
> be snapshots. And none of yours were snapshots. They were read-only
> subvolumes made using 'btrfs property' to set the read-only flag, and
> those are not snapshots.

Doesn't matter even if you only send snapshots and never set a subvolume
to ro to please btrfs-send.

# btrfs sub create 1
# touch 1/file1
# btrfs sub snap -r 1 2
# btrfs send 2 | btrfs receive /destination
!# sudo btrfs prop set /destination/2/ ro false
!# rm /destination/2/file1
# touch 1/file2
# sudo btrfs sub snap -r 1 3
# btrfs send -p 2 3 | btrfs receive /storage2/
# ls /storage2/3/
file2
# ls 3/
file1 file2

The lines with the '!' are the source of the trouble, but it doesn't
have to be rm.  You could modify file 1 instead.  It is obvious that
users shouldn't do that, AFTER you've run through this sequence.

Note that snapshot 2 at the receiving end was never set back to ro.


> 
> The btrfs send command should have explicitly failed on that alone.
> 
> In my reproducer of a related user error and bug, both were snapshots,
> but they were unrelated snapshots, therefore incremental send was
> impossible, and the only explanation for such a command is the user is
> confused, made a mistake, and therefore the command should have
> failed.
> 
>>
>> A user could from the descriptions of how send/receive formats its data
>> strea, and from interpreting "clone sources" in a manner contrary to how
>> its used in the man page and the mention of incrementals, conclude that
>> send/receive is sensitive to changes at the receiving side, but its far,
>> far more likely that they will start using feature like send/receive and
>> setting snapshots to ro before they delve into how it works its magic.
> 
> Uhhh I think this is a specious statement if you think users are
> likely to delve into 'btrfs property' and somehow think setting a
> subvolume ro is making a snapshot. You're the first I've encountered
> who arrived at this logic. For now almost a decade people arrive at
> 'btrfs subvolume snapshot' as the way to make snapshots, and the -r
> flag to make them read-only snapshots.




> https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Snapshots
> 
> That page only describes snapshots in terms of the 'btrfs subvolume
> snapshot' command - the 'btrfs property' command isn't found anywhere
> on that page. So I don't actually know how you came to think setting
> the ro property on a subvolume to true, makes the subvolume a
> snapshot. I think you did it through assumption. And while there might
> be a way to fix some things to avoid such confusion in the future, I
> do think it's an edge case.
> 
> 
> The man 8 btrfs man page could amend:
> 
>        subvolume
>            Create/delete/list/manage btrfs subvolume.
> 
> to
> 
>        subvolume
>            Create/delete/list/manage btrfs subvolumes and snapshots.
> 
> it is something of btrfs geekdom that snapshots are subvolumes; but
> still not all subvolumes are snapshots. So...yeah, clarity is a good
> thing even if there some redundancy.
> 
>> Anyway, rsync sends incremental change, and its ability to replicate a
>> directory tree isn't predicated on the files present at the receiving
>> end being the same as the last time it was run...
> 
> What? Yes it is. By default it compares size and time stamp on both
> sides. If you use -c it uses checksums to compare. From the rsync man
> page:

# touch file1
# touch file2
# mkdir /destination/snap1
# rsync -r . /destination/snap1
# rm /destination/snap1/file1
# touch file3
# mkdir /destination/snap2
# cp -r /destination/snap1 /destination/snap2
# rsync -r . /destination/snap2
# ls /destination/snap2
file1 file2 file3
# ls .
file1 file2 file3

You can also add an addition "rm /destination/snap2/file1" or even "rm
-rf /destination/snap2/*" before the last rsync, and it still gives you
a full replication.

The expectation after running rsync without any explicit additional
inclusion/exclusion clauses is replication, which is what it provides.
Having deleted files at the destination end in between doesn't cause
rsync to fail meeting that expectation.



> 
>> Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in  size  or  in  last-modified
>        time.
> 
> Rsync does not keep a database on source or destination. It absolutely
> is comparing both sides, and if they aren't the same, the file is
> synced from the source to the destination.

> 
> Also, the man page for btrfs receive very clearly says the receive
> side snapshot must be present and must not have been changed since the
> last receive, or it will fail.
> 
>>       btrfs receive will fail in the following cases:
>>       1. receiving subvolume already exists
>>       2. previously received subvolume has been changed after it was received
> 
> I'm still really not following where your confusion stems from, and
> therefore I'm not sure what needs fixing other than the items I've
> already mentioned - which itself at least would have stopped you in
> your tracks, to go dig deeper or ask questions before arriving at the
> understandably confusing results you were getting.
> 

From the man page

"       btrfs receive will fail in the following cases:

        1. receiving subvolume already exists

        2. previously received subvolume has been changed after it was
received

        3. default subvolume has changed or you didn’t mount the
filesystem at the toplevel subvolume

       A subvolume is made read-only after the receiving process
finishes successfully (see BUGS below).
"

Point 2 is what you are referring to, but it doesn't fail.  It reports
success and doesn't return an error code to the shell.  So the
documentation could be improved by reflecting this fact, which can avoid
users assuming that the snapshots at the receiving end must be the same
because it didn't fail.

Also, the btrfs-send man page could be improved, by including parents as
well as clone sources as being required to be the same on both ends.

That is all.

Re: Incremental receive completes succesfully despite missing files

[Chris Murphy]

On Fri, Jan 25, 2019 at 7:43 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>
> On 1/25/19 4:22 AM, Chris Murphy wrote:
> > On Thu, Jan 24, 2019 at 3:40 AM Dennis K <dennisk@netspace.net.au> wrote:
> >>
> >> The fact is, this thread is the first time I've seen explicitly written
> >> that parents must be the same at receiving and sending ends, or else
> >> btrfs-send/receive will produce a subvolume which differs from the source.
> >
> > The central user error, as well as btrfs-progs bug is the failure to
> > meet the requirement that the source(s) be snapshots. Either a full
> > send, or an incremental send, whether with -p or -c, all of them must
> > be snapshots. And none of yours were snapshots. They were read-only
> > subvolumes made using 'btrfs property' to set the read-only flag, and
> > those are not snapshots.
>
> Doesn't matter even if you only send snapshots and never set a subvolume
> to ro to please btrfs-send.
>
> # btrfs sub create 1
> # touch 1/file1
> # btrfs sub snap -r 1 2
> # btrfs send 2 | btrfs receive /destination
> !# sudo btrfs prop set /destination/2/ ro false
> !# rm /destination/2/file1
> # touch 1/file2
> # sudo btrfs sub snap -r 1 3
> # btrfs send -p 2 3 | btrfs receive /storage2/
> # ls /storage2/3/
> file2
> # ls 3/
> file1 file2
>
> The lines with the '!' are the source of the trouble, but it doesn't
> have to be rm.  You could modify file 1 instead.  It is obvious that
> users shouldn't do that, AFTER you've run through this sequence.
>
> Note that snapshot 2 at the receiving end was never set back to ro.

You're just finding more bugs (technically they are missing features,
as checking for user error is a feature); and I'm noticing there are
zero warnings in the man page for 'btrfs property' about unsetting ro
on snapshots. I agree with others that as soon as the ro flag is unset
on a snapshot, whatever metadata that makes it a snapshot and
references its parent, including both parent UUID and received UUIDs,
should be removed.

> You can also add an addition "rm /destination/snap2/file1" or even "rm
> -rf /destination/snap2/*" before the last rsync, and it still gives you
> a full replication.
>
> The expectation after running rsync without any explicit additional
> inclusion/exclusion clauses is replication, which is what it provides.

Not exactly. From man rsync:
>>>
       Rsync finds files that need to be transferred using a "quick
check" algorithm (by default) that looks for files that have changed
in  size  or  in  last-modified
       time.   Any changes in the other preserved attributes (as
requested by options) are made on the destination file directly when
the quick check indicates that the
       file’s data does not need to be updated.
>>>

If you initially replicate a destination using -r, the destination
files have *current* time stamps. Each additional -r causes
replication because the modification times for the source and
destination are different. If you initially use 'rsync -a' and then
follow it up with rsync -r, you'll see that the destination inodes
remain the same, no copy happens.

Anyway, if you're saying that rsync will at *worst* do an unnecessary
file copy, that's true and hence why -c exists; whereas with btrfs
send receive you can apparently successfully sabotage incremental send
receive by tampering with the receive side snapshot using 'btrfs
property' to unset ro, and remove a file; and a send/receive will not
make a subsequent receive side snapshot identical to the send side
source.



[chris@flap ~]$ ls -li test1/
total 1024
5185860 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
5185861 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
[chris@flap ~]$ ls -li test2/
total 1024
5185866 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
5185867 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
[chris@flap ~]$ rsync -rv test1/ test2/
sending incremental file list
sent 77 bytes  received 12 bytes  178.00 bytes/sec
total size is 1,048,576  speedup is 11,781.75
[chris@flap ~]$ ls -li test1/
total 1024
5185860 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
5185861 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
[chris@flap ~]$ ls -li test2/
total 1024
5185866 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
5185867 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
[chris@flap ~]$



> > I'm still really not following where your confusion stems from, and
> > therefore I'm not sure what needs fixing other than the items I've
> > already mentioned - which itself at least would have stopped you in
> > your tracks, to go dig deeper or ask questions before arriving at the
> > understandably confusing results you were getting.
> >
>
> From the man page
>
> "       btrfs receive will fail in the following cases:
>
>         1. receiving subvolume already exists
>
>         2. previously received subvolume has been changed after it was
> received
>
>         3. default subvolume has changed or you didn’t mount the
> filesystem at the toplevel subvolume
>
>        A subvolume is made read-only after the receiving process
> finishes successfully (see BUGS below).
> "
>
> Point 2 is what you are referring to, but it doesn't fail.  It reports
> success and doesn't return an error code to the shell.  So the
> documentation could be improved by reflecting this fact, which can avoid
> users assuming that the snapshots at the receiving end must be the same
> because it didn't fail.

The bug is that it doesn't fail. I consider the documentation correct,
but the command behavior is wrong.


> Also, the btrfs-send man page could be improved, by including parents as
> well as clone sources as being required to be the same on both ends.

man btrfs send does say this for clones, but not for parents. And I
agree that the documentation needs to be more clear. Someone needs to
volunteer to make the changes and submit a patch.

>>>
       You must not specify clone sources unless you guarantee that
these snapshots are exactly in the same state on both sides—both for
the sender and the receiver.
>>>


-- 
Chris Murphy

Re: Incremental receive completes succesfully despite missing files

[Dennis Katsonis]

On 1/27/19 10:09 AM, Chris Murphy wrote:
> On Fri, Jan 25, 2019 at 7:43 PM Dennis Katsonis <dennisk@netspace.net.au> wrote:
>>
>> On 1/25/19 4:22 AM, Chris Murphy wrote:
>>> On Thu, Jan 24, 2019 at 3:40 AM Dennis K <dennisk@netspace.net.au> wrote:
>>>>
>>>> The fact is, this thread is the first time I've seen explicitly written
>>>> that parents must be the same at receiving and sending ends, or else
>>>> btrfs-send/receive will produce a subvolume which differs from the source.
>>>
>>> The central user error, as well as btrfs-progs bug is the failure to
>>> meet the requirement that the source(s) be snapshots. Either a full
>>> send, or an incremental send, whether with -p or -c, all of them must
>>> be snapshots. And none of yours were snapshots. They were read-only
>>> subvolumes made using 'btrfs property' to set the read-only flag, and
>>> those are not snapshots.
>>
>> Doesn't matter even if you only send snapshots and never set a subvolume
>> to ro to please btrfs-send.
>>
>> # btrfs sub create 1
>> # touch 1/file1
>> # btrfs sub snap -r 1 2
>> # btrfs send 2 | btrfs receive /destination
>> !# sudo btrfs prop set /destination/2/ ro false
>> !# rm /destination/2/file1
>> # touch 1/file2
>> # sudo btrfs sub snap -r 1 3
>> # btrfs send -p 2 3 | btrfs receive /storage2/
>> # ls /storage2/3/
>> file2
>> # ls 3/
>> file1 file2
>>
>> The lines with the '!' are the source of the trouble, but it doesn't
>> have to be rm.  You could modify file 1 instead.  It is obvious that
>> users shouldn't do that, AFTER you've run through this sequence.
>>
>> Note that snapshot 2 at the receiving end was never set back to ro.
> 
> You're just finding more bugs (technically they are missing features,
> as checking for user error is a feature); and I'm noticing there are
> zero warnings in the man page for 'btrfs property' about unsetting ro
> on snapshots. I agree with others that as soon as the ro flag is unset
> on a snapshot, whatever metadata that makes it a snapshot and
> references its parent, including both parent UUID and received UUIDs,
> should be removed.
> 

Early on using btrfs, I assumed that the generation numbers between the
sent and received subvolume were initially the same, and that these
numbers were compared to ensure parents haven't changed relative to each
other.



>> You can also add an addition "rm /destination/snap2/file1" or even "rm
>> -rf /destination/snap2/*" before the last rsync, and it still gives you
>> a full replication.
>>
>> The expectation after running rsync without any explicit additional
>> inclusion/exclusion clauses is replication, which is what it provides.
> 
> Not exactly. From man rsync:
>>>>
>        Rsync finds files that need to be transferred using a "quick
> check" algorithm (by default) that looks for files that have changed
> in  size  or  in  last-modified
>        time.   Any changes in the other preserved attributes (as
> requested by options) are made on the destination file directly when
> the quick check indicates that the
>        file’s data does not need to be updated.
>>>>
> 
> If you initially replicate a destination using -r, the destination
> files have *current* time stamps. Each additional -r causes
> replication because the modification times for the source and
> destination are different. If you initially use 'rsync -a' and then
> follow it up with rsync -r, you'll see that the destination inodes
> remain the same, no copy happens.
> 
> Anyway, if you're saying that rsync will at *worst* do an unnecessary
> file copy, that's true and hence why -c exists; whereas with btrfs
> send receive you can apparently successfully sabotage incremental send
> receive by tampering with the receive side snapshot using 'btrfs
> property' to unset ro, and remove a file; and a send/receive will not
> make a subsequent receive side snapshot identical to the send side
> source.
> 

Yes.  The worst case is, it takes a longer and there is more disk and/or
network activity and what appears to be already sent files sent again.
But no matter what interpretation the user has, or whether this is the
result of accidental or deliberate action, or even a previously aborted
run, the end result after success is returned is a duplicated filetree,
what rsync is 'contracted' to do.

This isn't quite the case with send/receive.


> 
> 
> [chris@flap ~]$ ls -li test1/
> total 1024
> 5185860 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
> 5185861 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
> [chris@flap ~]$ ls -li test2/
> total 1024
> 5185866 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
> 5185867 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
> [chris@flap ~]$ rsync -rv test1/ test2/
> sending incremental file list
> sent 77 bytes  received 12 bytes  178.00 bytes/sec
> total size is 1,048,576  speedup is 11,781.75
> [chris@flap ~]$ ls -li test1/
> total 1024
> 5185860 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
> 5185861 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
> [chris@flap ~]$ ls -li test2/
> total 1024
> 5185866 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file3
> 5185867 -rw-rw-r--. 1 chris chris 524288 Jan 26 15:55 file4
> [chris@flap ~]$
> 
> 
> 
>>> I'm still really not following where your confusion stems from, and
>>> therefore I'm not sure what needs fixing other than the items I've
>>> already mentioned - which itself at least would have stopped you in
>>> your tracks, to go dig deeper or ask questions before arriving at the
>>> understandably confusing results you were getting.
>>>
>>
>> From the man page
>>
>> "       btrfs receive will fail in the following cases:
>>
>>         1. receiving subvolume already exists
>>
>>         2. previously received subvolume has been changed after it was
>> received
>>
>>         3. default subvolume has changed or you didn’t mount the
>> filesystem at the toplevel subvolume
>>
>>        A subvolume is made read-only after the receiving process
>> finishes successfully (see BUGS below).
>> "
>>
>> Point 2 is what you are referring to, but it doesn't fail.  It reports
>> success and doesn't return an error code to the shell.  So the
>> documentation could be improved by reflecting this fact, which can avoid
>> users assuming that the snapshots at the receiving end must be the same
>> because it didn't fail.
> 
> The bug is that it doesn't fail. I consider the documentation correct,
> but the command behavior is wrong.
> 
> 
>> Also, the btrfs-send man page could be improved, by including parents as
>> well as clone sources as being required to be the same on both ends.
> 
> man btrfs send does say this for clones, but not for parents. And I
> agree that the documentation needs to be more clear. Someone needs to
> volunteer to make the changes and submit a patch.
> 
>>>>
>        You must not specify clone sources unless you guarantee that
> these snapshots are exactly in the same state on both sides—both for
> the sender and the receiver.
>>>>
> 
> 

Re: Incremental receive completes succesfully despite missing files

[Remi Gauvin]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

On 2019-01-23 10:25 a.m., Hans van Kranenburg wrote:

> 
> But then only disallow if the subvol has a value in the received_uuid
> field, I'd say.
> 

That would only solve half the self-harm.  User can still make changes
to the parent subvolume on the sending side, and subsequent sends using
the same parent will be inconsistent.


[-- Attachment #2: remi.vcf --]
[-- Type: text/x-vcard, Size: 193 bytes --]

begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:remi@georgianit.com
tel;work:226-256-1545
version:2.1
end:vcard

Re: Incremental receive completes succesfully despite missing files

[Hans van Kranenburg]

On 1/23/19 4:40 PM, Remi Gauvin wrote:
> On 2019-01-23 10:25 a.m., Hans van Kranenburg wrote:
> 
>>
>> But then only disallow if the subvol has a value in the received_uuid
>> field, I'd say.
>>
> 
> That would only solve half the self-harm.  User can still make changes
> to the parent subvolume on the sending side, and subsequent sends using
> the same parent will be inconsistent.

Well, so to say, is the glass half full or is it half empty? \:D/

At the sending side, there's no information currently that indicates the
subvol has been sent. At least we could fix 50% of the issue.

-- 
Hans van Kranenburg