Re: Incremental receive completes succesfully despite missing files

[Chris Murphy <lists@colorremedies.com>]

On Thu, Jan 24, 2019 at 3:40 AM Dennis K <dennisk@netspace.net.au> wrote:
>
> The fact is, this thread is the first time I've seen explicitly written
> that parents must be the same at receiving and sending ends, or else
> btrfs-send/receive will produce a subvolume which differs from the source.

The central user error, as well as btrfs-progs bug is the failure to
meet the requirement that the source(s) be snapshots. Either a full
send, or an incremental send, whether with -p or -c, all of them must
be snapshots. And none of yours were snapshots. They were read-only
subvolumes made using 'btrfs property' to set the read-only flag, and
those are not snapshots.

The btrfs send command should have explicitly failed on that alone.

In my reproducer of a related user error and bug, both were snapshots,
but they were unrelated snapshots, therefore incremental send was
impossible, and the only explanation for such a command is the user is
confused, made a mistake, and therefore the command should have
failed.

>
> A user could from the descriptions of how send/receive formats its data
> strea, and from interpreting "clone sources" in a manner contrary to how
> its used in the man page and the mention of incrementals, conclude that
> send/receive is sensitive to changes at the receiving side, but its far,
> far more likely that they will start using feature like send/receive and
> setting snapshots to ro before they delve into how it works its magic.

Uhhh I think this is a specious statement if you think users are
likely to delve into 'btrfs property' and somehow think setting a
subvolume ro is making a snapshot. You're the first I've encountered
who arrived at this logic. For now almost a decade people arrive at
'btrfs subvolume snapshot' as the way to make snapshots, and the -r
flag to make them read-only snapshots.

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Snapshots

That page only describes snapshots in terms of the 'btrfs subvolume
snapshot' command - the 'btrfs property' command isn't found anywhere
on that page. So I don't actually know how you came to think setting
the ro property on a subvolume to true, makes the subvolume a
snapshot. I think you did it through assumption. And while there might
be a way to fix some things to avoid such confusion in the future, I
do think it's an edge case.


The man 8 btrfs man page could amend:

       subvolume
           Create/delete/list/manage btrfs subvolume.

to

       subvolume
           Create/delete/list/manage btrfs subvolumes and snapshots.

it is something of btrfs geekdom that snapshots are subvolumes; but
still not all subvolumes are snapshots. So...yeah, clarity is a good
thing even if there some redundancy.

> Anyway, rsync sends incremental change, and its ability to replicate a
> directory tree isn't predicated on the files present at the receiving
> end being the same as the last time it was run...

What? Yes it is. By default it compares size and time stamp on both
sides. If you use -c it uses checksums to compare. From the rsync man
page:

>Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in  size  or  in  last-modified
       time.

Rsync does not keep a database on source or destination. It absolutely
is comparing both sides, and if they aren't the same, the file is
synced from the source to the destination.

Also, the man page for btrfs receive very clearly says the receive
side snapshot must be present and must not have been changed since the
last receive, or it will fail.

>       btrfs receive will fail in the following cases:
>       1. receiving subvolume already exists
>       2. previously received subvolume has been changed after it was received

I'm still really not following where your confusion stems from, and
therefore I'm not sure what needs fixing other than the items I've
already mentioned - which itself at least would have stopped you in
your tracks, to go dig deeper or ask questions before arriving at the
understandably confusing results you were getting.

-- 
Chris Murphy