Btrfs send with parent different size depending on source of files.

Btrfs send with parent different size depending on source of files.

[André Malm]

Hello,

I'm not sure this is the right forum to ask on but I'll try and if its 
not I do apologize. I have also created a stack overflow question 
without success ( 
https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files 
) but ill paste the question here too. Thank you.

What i'm trying to achieve is sending only the diff of the parent with 
btrfs send -p

Running this will produce a file 'out' with size 639 bytes, i.e only 
diff sent.

====================================================

btrfs subvolume create A
btrfs subvolume create B
mkdir A/dir

dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K
cp --reflink=always A/dir/server.jar B/server.jar

btrfs subvolume snapshot -r A a
btrfs subvolume snapshot -r B b
btrfs send -p a b > out

====================================================

However if I change the dd command to wget like this:

====================================================

btrfs subvolume create A
btrfs subvolume create B
mkdir A/dir

wget -O A/dir/server.jar 
https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar
cp --reflink=always A/dir/server.jar B/server.jar

btrfs subvolume snapshot -r A a
btrfs subvolume snapshot -r B b
btrfs send -p a b > out

====================================================


The 'out' file is 34M, which is the full size.

What causes this?

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Thu, Feb 14, 2019 at 4:37 AM André Malm <admin@sheepa.org> wrote:
>
> Hello,
>
> I'm not sure this is the right forum to ask on but I'll try and if its
> not I do apologize. I have also created a stack overflow question
> without success (
> https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files
> ) but ill paste the question here too. Thank you.
>
> What i'm trying to achieve is sending only the diff of the parent with
> btrfs send -p
>
> Running this will produce a file 'out' with size 639 bytes, i.e only
> diff sent.
>
> ====================================================
>
> btrfs subvolume create A
> btrfs subvolume create B
> mkdir A/dir
>
> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K
> cp --reflink=always A/dir/server.jar B/server.jar
>
> btrfs subvolume snapshot -r A a
> btrfs subvolume snapshot -r B b
> btrfs send -p a b > out

It doesn't work this way. The snapshots a and b are not based on the
same underlying subvolume. The gist is that you would keep changing A,
and take additional snapshots of A, such as a.1 a.2 a.3, and you can
do incremental send with 'btrfs send -p a.1 a.2' which describes the
difference between those two snapshots of A at their respective
moments in time. You could also do 'btrfs send -p a.2 a.3' or even
'btrfs send -p a.1 a.3'

But as there's no relationship between snapshots a and b, I consider
it a bug/missing error handling feature, that btrfs send doesn't fail
in this case. By using -p you're claiming there is a parent-child
relationship between a and b, but there plainly isn't.

Read this:
https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step

Depending on your use case if you can describe it, might be tolerated
with some adjustments by using the -c clone option instead.

-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[Remi Gauvin]

[-- Attachment #1.1.1: Type: text/plain, Size: 1469 bytes --]


> It doesn't work this way. The snapshots a and b are not based on the
> same underlying subvolume. The gist is that you would keep changing A,
> and take additional snapshots of A, such as a.1 a.2 a.3, and you can
> do incremental send with 'btrfs send -p a.1 a.2' which describes the
> difference between those two snapshots of A at their respective
> moments in time. You could also do 'btrfs send -p a.2 a.3' or even
> 'btrfs send -p a.1 a.3'
> 
> But as there's no relationship between 

snapshots a and b, I consider
> it a bug/missing error handling feature, that btrfs send doesn't fail
> in this case. By using -p you're claiming there is a parent-child
> relationship between a and b, but there plainly isn't.

They kind of are related though, since the two snapshots reference the
same data blocks, and you can see it work in the first example with the
40MB of random data.

However, I can replicate this odd behaviour the conventional way.

btrfs sub create A
mkdir A/dir
dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K
btrfs sub snap -r A a

cp --reflink=always A/dir/server.jar A/server.jar
rm A/dir -rf
btrfs sub snap -r A b

btrfs send -p a b > out


The out file is only 773 bytes.  However, if you repeat all those same
steps, but replace the dd with:
wget -O A/dir/server.jar
https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar

The resulting out file is 34MB.

[-- Attachment #1.1.2: remi.vcf --]
[-- Type: text/x-vcard, Size: 203 bytes --]

begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:remi@georgianit.com
tel;work:226-256-1545
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Thu, Feb 14, 2019 at 9:00 PM Remi Gauvin <remi@georgianit.com> wrote:
>
>
> > It doesn't work this way. The snapshots a and b are not based on the
> > same underlying subvolume. The gist is that you would keep changing A,
> > and take additional snapshots of A, such as a.1 a.2 a.3, and you can
> > do incremental send with 'btrfs send -p a.1 a.2' which describes the
> > difference between those two snapshots of A at their respective
> > moments in time. You could also do 'btrfs send -p a.2 a.3' or even
> > 'btrfs send -p a.1 a.3'
> >
> > But as there's no relationship between
>
> snapshots a and b, I consider
> > it a bug/missing error handling feature, that btrfs send doesn't fail
> > in this case. By using -p you're claiming there is a parent-child
> > relationship between a and b, but there plainly isn't.
>
> They kind of are related though, since the two snapshots reference the
> same data blocks, and you can see it work in the first example with the
> 40MB of random data.

Is it more of a case for clone, -c option? I don't ever use it, and
the man page could use a couple examples to make it more clear what it
can do. And it makes sense this option could be more tolerant, less
error checking, than -p.

But from man page and in particular the wiki, it's clear to me that
the two snapshots following -p option, need to each be derived from
one subvolume. I actually don't like the "parent - child" metaphor
because really the parent is the original source rw subvolume, and its
two children snapshots are the ones used with -p. The first is an
older sibling, the second a younger sibling. And that metaphor fails
too because you'd expect the second sibling to have more information
which plainly is not the case with actual siblings. :P


> The out file is only 773 bytes.  However, if you repeat all those same
> steps, but replace the dd with:
> wget -O A/dir/server.jar
> https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar
>
> The resulting out file is 34MB.

Well I'd say maybe use -vvv and --no-data instead of -f and see what
it's doing. It sounds like the former has no payload, just difference
information, and the latter has a payload. I don't know why.


--
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[Remi Gauvin]

[-- Attachment #1.1.1: Type: text/plain, Size: 467 bytes --]

On 2019-02-15 1:38 p.m., Chris Murphy wrote:
>
> Well I'd say maybe use -vvv and --no-data instead of -f and see what
> it's doing. It sounds like the former has no payload, just difference
> information, and the latter has a payload. I don't know why.
> 
> 


Both have all the payload they need to re-create the subvolumes.  The
first one correctly CoW's the file from subvolume a.. the second one
stuffs it in the output... Why is indeed the question.

[-- Attachment #1.1.2: remi.vcf --]
[-- Type: text/x-vcard, Size: 203 bytes --]

begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:remi@georgianit.com
tel;work:226-256-1545
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Btrfs send with parent different size depending on source of files.

[Andrei Borzenkov]

[-- Attachment #1.1: Type: text/plain, Size: 608 bytes --]

15.02.2019 21:56, Remi Gauvin пишет:
> On 2019-02-15 1:38 p.m., Chris Murphy wrote:
>>
>> Well I'd say maybe use -vvv and --no-data instead of -f and see what
>> it's doing. It sounds like the former has no payload, just difference
>> information, and the latter has a payload. I don't know why.
>>
>>
> 
> 
> Both have all the payload they need to re-create the subvolumes.  The
> first one correctly CoW's the file from subvolume a.. the second one
> stuffs it in the output... Why is indeed the question.
> 

The last extent is incomplete which causes btrfs to not attempt cloning.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Btrfs send with parent different size depending on source of files.

[Andrei Borzenkov]

15.02.2019 1:37, Chris Murphy пишет:
> On Thu, Feb 14, 2019 at 4:37 AM André Malm <admin@sheepa.org> wrote:
>>
>> Hello,
>>
>> I'm not sure this is the right forum to ask on but I'll try and if its
>> not I do apologize. I have also created a stack overflow question
>> without success (
>> https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files
>> ) but ill paste the question here too. Thank you.
>>
>> What i'm trying to achieve is sending only the diff of the parent with
>> btrfs send -p
>>
>> Running this will produce a file 'out' with size 639 bytes, i.e only
>> diff sent.
>>
>> ====================================================
>>
>> btrfs subvolume create A
>> btrfs subvolume create B
>> mkdir A/dir
>>
>> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K
>> cp --reflink=always A/dir/server.jar B/server.jar
>>
>> btrfs subvolume snapshot -r A a
>> btrfs subvolume snapshot -r B b
>> btrfs send -p a b > out
> 
> It doesn't work this way.

It works exactly this way.

> The snapshots a and b are not based on the
> same underlying subvolume.

So what? What prevents you from computing differences between two
subvolumes? That is real question.

> The gist is that you would keep changing A,
> and take additional snapshots of A, such as a.1 a.2 a.3, and you can
> do incremental send with 'btrfs send -p a.1 a.2' which describes the
> difference between those two snapshots of A at their respective
> moments in time. You could also do 'btrfs send -p a.2 a.3' or even
> 'btrfs send -p a.1 a.3'
> 

That it is intended to be used this way does not mean it is restricted
to this way technically. Whether it should have been restricted is
another question.

> But as there's no relationship between snapshots a and b, I consider
> it a bug/missing error handling feature, that btrfs send doesn't fail
> in this case. By using -p you're claiming there is a parent-child
> relationship between a and b, but there plainly isn't.
> 

No. By using "-p" you designate subvolume which must be used as base to
apply differential (I explicitly does not use "incremental") stream on
remote side. Nothing more. Whether it should have different semantic is
subject to discussion, but it does not do what you wish it to do.

> Read this:
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step
> 
> Depending on your use case if you can describe it, might be tolerated
> with some adjustments by using the -c clone option instead.
> 

The only difference is that "-c" in absence of explicit "-p" attempts to
auto-detect the "best" "base" subvolume - but it selects between
subvolumes given with "-c" options so "-c" with single subvolume is
entirely equivalent to "-p".

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Fri, Feb 15, 2019 at 10:45 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
> 15.02.2019 1:37, Chris Murphy пишет:
> > On Thu, Feb 14, 2019 at 4:37 AM André Malm <admin@sheepa.org> wrote:
> >>
> >> Hello,
> >>
> >> I'm not sure this is the right forum to ask on but I'll try and if its
> >> not I do apologize. I have also created a stack overflow question
> >> without success (
> >> https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files
> >> ) but ill paste the question here too. Thank you.
> >>
> >> What i'm trying to achieve is sending only the diff of the parent with
> >> btrfs send -p
> >>
> >> Running this will produce a file 'out' with size 639 bytes, i.e only
> >> diff sent.
> >>
> >> ====================================================
> >>
> >> btrfs subvolume create A
> >> btrfs subvolume create B
> >> mkdir A/dir
> >>
> >> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K
> >> cp --reflink=always A/dir/server.jar B/server.jar
> >>
> >> btrfs subvolume snapshot -r A a
> >> btrfs subvolume snapshot -r B b
> >> btrfs send -p a b > out
> >
> > It doesn't work this way.
>
> It works exactly this way.

Unproven. And in fact we have an example where it does not work, hence
the point of the thread!

The proven way it works, is as I've described, and many emails over a
decade on this list, inferred from the man page, and the step by step
recipe on the Wiki.

> > The gist is that you would keep changing A,
> > and take additional snapshots of A, such as a.1 a.2 a.3, and you can
> > do incremental send with 'btrfs send -p a.1 a.2' which describes the
> > difference between those two snapshots of A at their respective
> > moments in time. You could also do 'btrfs send -p a.2 a.3' or even
> > 'btrfs send -p a.1 a.3'
> >
>
> That it is intended to be used this way does not mean it is restricted
> to this way technically. Whether it should have been restricted is
> another question.

That's fair. But also the alternative cannot be stated to "work
exactly this way" when this very thread starts with two examples
leading to different and therefore unexpected results, differing only
in how the file was created: dd vs wget

And no one can explain that. Which tells me none of us really
understands the full scope and operation of send.

By the way, it is possible for send to succeed in producing a stream,
that receive will reject. So there is a difference in the error
checking when using send with receive, than send without receive.


> > But as there's no relationship between snapshots a and b, I consider
> > it a bug/missing error handling feature, that btrfs send doesn't fail
> > in this case. By using -p you're claiming there is a parent-child
> > relationship between a and b, but there plainly isn't.
> >
>
> No. By using "-p" you designate subvolume which must be used as base to
> apply differential (I explicitly does not use "incremental") stream on
> remote side. Nothing more. Whether it should have different semantic is
> subject to discussion, but it does not do what you wish it to do.

man page and wiki use incremental - and linguistically incremental !=
differential, increment is a subset of difference.

Difference suggests the order does not matter. Increment suggests it
does matter.

btrfs sub create A
cp somestuff A
btrfs sub snap -r A a.1
cp morestuff A
btrfs sub snap -r A a.2

The normal case of send is:

btrfs send -p a.1 a.2

However, is it valid to do?

btrfs send -p a.2 a1

I haven't tried this. Maybe send allows it and it works as expected?
Maybe receive also works. If it's allows *and* also if it works as
expected, then maybe the command is differential and not incremental.
That is a rather fundamental thing to know!


-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[Andrei Borzenkov]

15.02.2019 22:11, Chris Murphy пишет:
...
>>>
>>> It doesn't work this way.
>>
>> It works exactly this way.
> 
> Unproven. And in fact we have an example where it does not work, hence
> the point of the thread!
> 
> The proven way it works, is as I've described, and many emails over a
> decade on this list, inferred from the man page, and the step by step
> recipe on the Wiki.
> 

Oh well.

10:~ # mkfs -t btrfs -f /dev/sdb1
btrfs-progs v4.20.1
See http://btrfs.wiki.kernel.org for more information.

Label:              (null)
UUID:               c8dd61f1-bf32-449b-a154-3380cf4348b5
Node size:          16384
Sector size:        4096
Filesystem size:    1023.00MiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              51.12MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1  1023.00MiB  /dev/sdb1

10:~ # mount -t btrfs /dev/sdb1 /mnt
10:~ # cd /mnt
10:/mnt # btrfs su cr A
Create subvolume './A'
10:/mnt # mkdir A/dir
10:/mnt # dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K
40960+0 records in
40960+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.380302 s, 110 MB/s
10:/mnt # btrfs su sn -r A a1
Create a readonly snapshot of 'A' in './a1'
10:/mnt # cp --reflink=always A/dir/server.jar A/server.jar
10:/mnt # btrfs su sn -r A a2
Create a readonly snapshot of 'A' in './a2'
10:/mnt # btrfs send -p a1 a2 > out
At subvol a2
10:/mnt # ll -sh out
4.0K -rw-r--r-- 1 root root 567 Feb 16 23:15 out
10:/mnt # cd
10:~ # umount /mnt
10:~ # mkfs -t btrfs -f /dev/sdc1
btrfs-progs v4.20.1
See http://btrfs.wiki.kernel.org for more information.

Label:              (null)
UUID:               a2d85827-116a-4dee-874c-d3b180728613
Node size:          16384
Sector size:        4096
Filesystem size:    1023.00MiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              51.12MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1  1023.00MiB  /dev/sdc1

10:~ # mount -t btrfs /dev/sdc1 /mnt
10:~ # cd /mnt
10:/mnt # btrfs su cr A
Create subvolume './A'
10:/mnt # mkdir A/dir
10:/mnt # dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625
34625+0 records in
34625+0 records out
35456000 bytes (35 MB, 34 MiB) copied, 0.31223 s, 114 MB/s
10:/mnt # btrfs su sn -r A a1
Create a readonly snapshot of 'A' in './a1'
10:/mnt # cp --reflink=always A/dir/server.jar A/server.jar
10:/mnt # btrfs su sn -r A a2
Create a readonly snapshot of 'A' in './a2'
10:/mnt # btrfs send -p a1 a2 > out
At subvol a2
10:/mnt # ll -sh out
34M -rw-r--r-- 1 root root 34M Feb 16 23:17 out
10:/mnt #


>>> The gist is that you would keep changing A,
>>> and take additional snapshots of A, such as a.1 a.2 a.3, and you can
>>> do incremental send with 'btrfs send -p a.1 a.2' which describes the
>>> difference between those two snapshots of A at their respective
>>> moments in time. You could also do 'btrfs send -p a.2 a.3' or even
>>> 'btrfs send -p a.1 a.3'
>>>
>>
>> That it is intended to be used this way does not mean it is restricted
>> to this way technically. Whether it should have been restricted is
>> another question.
> 
> That's fair. But also the alternative cannot be stated to "work
> exactly this way" when this very thread starts with two examples
> leading to different and therefore unexpected results, differing only
> in how the file was created: dd vs wget
> 

It is not the *only* difference between two examples.

> And no one can explain that. 

I just sent explanation why it happens.

...
>>
>> No. By using "-p" you designate subvolume which must be used as base to
>> apply differential (I explicitly does not use "incremental") stream on
>> remote side. Nothing more. Whether it should have different semantic is
>> subject to discussion, but it does not do what you wish it to do.
> 
> man page and wiki use incremental - and linguistically incremental !=
> differential, increment is a subset of difference.
> 

btrfs send computes differential stream. Whether it will be
"incremental" or not depends on how you are using it.

> Difference suggests the order does not matter. Increment suggests it
> does matter.
> 
> btrfs sub create A
> cp somestuff A
> btrfs sub snap -r A a.1
> cp morestuff A
> btrfs sub snap -r A a.2
> 
> The normal case of send is:
> 
> btrfs send -p a.1 a.2
> 
> However, is it valid to do?
> 
> btrfs send -p a.2 a1
> 

Of course it is. Stream contains differences between the first and the
second stream, where order does matter. "Incremental" implies temporal
ordering between two snapshots which does not exist on "btrfs send"
level and must be ensured by tools that use "btrfs send/receive". There
is no inherent parent/child or temporal relationship implied by "btrfs
send" itself. "btrfs send" *will* compute differences from a.1 to a.2
although a.2 is newer than a.1; it is up to you to ensure you apply this
stream to correct base on destination.

Re: Btrfs send with parent different size depending on source of files.

[Andrei Borzenkov]

16.02.2019 23:26, Andrei Borzenkov пишет:
...
>> Difference suggests the order does not matter. Increment suggests it
>> does matter.
>>
>> btrfs sub create A
>> cp somestuff A
>> btrfs sub snap -r A a.1
>> cp morestuff A
>> btrfs sub snap -r A a.2
>>
>> The normal case of send is:
>>
>> btrfs send -p a.1 a.2
>>
>> However, is it valid to do?
>>
>> btrfs send -p a.2 a1
>>
> 
> Of course it is. Stream contains differences between the first and the
> second stream, where order does matter. "Incremental" implies temporal
> ordering between two snapshots which does not exist on "btrfs send"
> level and must be ensured by tools that use "btrfs send/receive". There
> is no inherent parent/child or temporal relationship implied by "btrfs
> send" itself. "btrfs send" *will* compute differences from a.1 to a.2
> although a.2 is newer than a.1; it is up to you to ensure you apply this
> stream to correct base on destination.
> 

And BTW your example can actually be useful to revert (roll back)
content of volume to previous snapshot.

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Sat, Feb 16, 2019 at 1:26 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
> 15.02.2019 22:11, Chris Murphy пишет:

> > The proven way it works, is as I've described, and many emails over a
> > decade on this list, inferred from the man page, and the step by step
> > recipe on the Wiki.
> >
>
> Oh well.

OK, you're right, I'm wrong, however your example is the only one I've
seen in ~10 years that describes this alternative.

> btrfs send computes differential stream. Whether it will be
> "incremental" or not depends on how you are using it.

Yes, it's a difference computation, but the 'man btrfs-send' page
explicitly says incremental which is not the same thing.

Also adding 'btrfs receive' is important to test your example. I've
seen 'btrfs send' succeed where 'btrfs receive' complains.


>Stream contains differences between the first and the
> second stream, where order does matter. "Incremental" implies temporal
> ordering between two snapshots which does not exist on "btrfs send"
> level and must be ensured by tools that use "btrfs send/receive".

It's fine to have this alternative explanation. However the existing
'man btrfs-send' page says there are two modes: full and incremental.

-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[Remi Gauvin]

[-- Attachment #1.1.1: Type: text/plain, Size: 662 bytes --]

On 2019-02-14 6:37 a.m., André Malm wrote:

> 
> 
> The 'out' file is 34M, which is the full size.
> 
> What causes this?
> 

Just to narrow it down, the fault is the contents of the server.jar
file, not how it was created.

If I upload a urandom server.jar file to my own website, then download
it with wget as described, btrfs send creates a properly small diff

However, if I upload the actual server.jar file to the same URL, then
repeat the process, I again get a 34MB out

How often does this happen?  Did you just happen to stumble on the one
file in the Universe that does this with server.jar?  Or was there are a
wider context?


[-- Attachment #1.1.2: remi.vcf --]
[-- Type: text/x-vcard, Size: 203 bytes --]

begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:remi@georgianit.com
tel;work:226-256-1545
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Btrfs send with parent different size depending on source of files.

[Remi Gauvin]

[-- Attachment #1.1.1: Type: text/plain, Size: 376 bytes --]

On 2019-02-15 2:29 p.m., Remi Gauvin wrote:

> 
> How often does this happen?  Did you just happen to stumble on the one
> file in the Universe that does this with server.jar?  Or was there are a
> wider context?
> 

I just tried with a random epub file, and again get the full size out,
so it's a complete mystery to me why a file filled with random bytes works.


[-- Attachment #1.1.2: remi.vcf --]
[-- Type: text/x-vcard, Size: 203 bytes --]

begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:remi@georgianit.com
tel;work:226-256-1545
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Btrfs send with parent different size depending on source of files.

[Andrei Borzenkov]

14.02.2019 14:37, André Malm пишет:
> Hello,
> 
> I'm not sure this is the right forum to ask on but I'll try and if its
> not I do apologize. I have also created a stack overflow question
> without success (
> https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files
> ) but ill paste the question here too. Thank you.
> 
> What i'm trying to achieve is sending only the diff of the parent with
> btrfs send -p
> 
> Running this will produce a file 'out' with size 639 bytes, i.e only
> diff sent.
> 
> ====================================================
> 
> btrfs subvolume create A
> btrfs subvolume create B
> mkdir A/dir
> 
> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K
> cp --reflink=always A/dir/server.jar B/server.jar
> 
> btrfs subvolume snapshot -r A a
> btrfs subvolume snapshot -r B b
> btrfs send -p a b > out
> 
> ====================================================
> 
> However if I change the dd command to wget like this:
> 
> ====================================================
> 
> btrfs subvolume create A
> btrfs subvolume create B
> mkdir A/dir
> 
> wget -O A/dir/server.jar
> https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar
> 
> cp --reflink=always A/dir/server.jar B/server.jar
> 
> btrfs subvolume snapshot -r A a
> btrfs subvolume snapshot -r B b
> btrfs send -p a b > out
> 
> ====================================================
> 
> 
> The 'out' file is 34M, which is the full size.
> 
> What causes this?
> 
> 

File size of "real" server.jar is not exact multiple of btrfs block size
(4096) which causes the last extent to be "incomplete". btrfs won't
clone extent in this case. You can trivially reproduce it by using the
same size in the first case (although exact extent distribution may be
different of course, in the worst case you get single extent which is
sent in full):



dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625

Re: Btrfs send with parent different size depending on source of files.

[Remi Gauvin]

[-- Attachment #1.1.1: Type: text/plain, Size: 735 bytes --]

On 2019-02-16 3:08 p.m., Andrei Borzenkov wrote:

> 
> File size of "real" server.jar is not exact multiple of btrfs block size
> (4096) which causes the last extent to be "incomplete". btrfs won't
> clone extent in this case. You can trivially reproduce it by using the
> same size in the first case (although exact extent distribution may be
> different of course, in the worst case you get single extent which is
> sent in full):
> 
> 
> 
> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625
> 

Thank you.  I thought I had tested one where I changed the file size,
(by  altering the bs,) but I must have inadvertently created a file that
still divided by 4096, (or it got split into a second extent).



[-- Attachment #1.1.2: remi.vcf --]
[-- Type: text/x-vcard, Size: 203 bytes --]

begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:remi@georgianit.com
tel;work:226-256-1545
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Btrfs send with parent different size depending on source of files.

[André Malm]

So the takeaway from this is that btrfs send doesn't work this way, I 
can't copy a file with reflink=always and expect it to be diffed 
correctly as in a parent-child situation or?

On 2019-02-17 04:11, Remi Gauvin wrote:
> On 2019-02-16 3:08 p.m., Andrei Borzenkov wrote:
>
>> File size of "real" server.jar is not exact multiple of btrfs block size
>> (4096) which causes the last extent to be "incomplete". btrfs won't
>> clone extent in this case. You can trivially reproduce it by using the
>> same size in the first case (although exact extent distribution may be
>> different of course, in the worst case you get single extent which is
>> sent in full):
>>
>>
>>
>> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625
>>
> Thank you.  I thought I had tested one where I changed the file size,
> (by  altering the bs,) but I must have inadvertently created a file that
> still divided by 4096, (or it got split into a second extent).
>
>

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Mon, Feb 18, 2019 at 6:05 AM André Malm <admin@sheepa.org> wrote:
>
> So the takeaway from this is that btrfs send doesn't work this way, I
> can't copy a file with reflink=always and expect it to be diffed
> correctly as in a parent-child situation or?

Sorry about the confusion, André. What you're doing isn't described or
explained in existing documentation: man page, wiki, or list archive,
it's only explained in this thread. So, all I can say is it's going to
take some testing on your part and possibly documentation and example
improvements, for this alternative usage.

At the moment, based on Andrei's explanation and example, what you're
trying to do is valid. He also explained why you're seeing different
results in send stream output in your dd vs wget example: it's the
files themselves not the application.

"The last extent is incomplete which causes btrfs to not attempt cloning."

-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[André Malm]

What causes the extent to be incomplete? And can I avoid it? I tried 
with my example but with wget with a larger file (1G) and the diff 
produces was only 60MB which indicates that it "mostly" works.

Basically what I'm trying to achieve is having a "reference" / "master" 
btrfs subvolume where i add / edit / remove files continuously. From 
where i can cp --reflink=always some of those files to new "child" 
subvolumes as needed. As long as the files are still in the master 
subvolume I'll only send the diff of the data. The plan is to have this 
master subvolume across multiple remote machines (synced with rsync or 
perhaps btrfs send / receive) and being able to update / change it as 
needed but always only sending the diffs of the child subvolumes. Maybe 
this is a bit optimistic.

On 2019-02-18 19:06, Chris Murphy wrote:
> On Mon, Feb 18, 2019 at 6:05 AM André Malm <admin@sheepa.org> wrote:
>> So the takeaway from this is that btrfs send doesn't work this way, I
>> can't copy a file with reflink=always and expect it to be diffed
>> correctly as in a parent-child situation or?
> Sorry about the confusion, André. What you're doing isn't described or
> explained in existing documentation: man page, wiki, or list archive,
> it's only explained in this thread. So, all I can say is it's going to
> take some testing on your part and possibly documentation and example
> improvements, for this alternative usage.
>
> At the moment, based on Andrei's explanation and example, what you're
> trying to do is valid. He also explained why you're seeing different
> results in send stream output in your dd vs wget example: it's the
> files themselves not the application.
>
> "The last extent is incomplete which causes btrfs to not attempt cloning."
>

Re: Btrfs send with parent different size depending on source of files.

[Graham Cobb]

On 18/02/2019 19:58, André Malm wrote:
> What causes the extent to be incomplete? And can I avoid it? 

Does it matter? I presume the send is working OK, it is just that it
sends a little more data than it needs to. Or have you seen any data loss?

Graham

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Mon, Feb 18, 2019 at 12:58 PM André Malm <admin@sheepa.org> wrote:

> Basically what I'm trying to achieve is having a "reference" / "master"
> btrfs subvolume where i add / edit / remove files continuously. From
> where i can cp --reflink=always some of those files to new "child"
> subvolumes as needed. As long as the files are still in the master
> subvolume I'll only send the diff of the data. The plan is to have this
> master subvolume across multiple remote machines (synced with rsync or
> perhaps btrfs send / receive) and being able to update / change it as
> needed but always only sending the diffs of the child subvolumes. Maybe
> this is a bit optimistic.

I'm not sure what you get out of this method that depends on reflink
rather than just making read only snapshots.

Why don't you create subvolume A as the master, read-write. And once
it's in the "master state" you want, just snapshot it:

btrfs sub snap -r A A.20190218-master

And now continue to make changes to A subvolume, and on whatever
schedule you want:

btrfs sub snap -r A A.20190218-1412
btrfs sub snap -r A A.20190218-1850
btrfs sub snap -r A A.20190219-0920

And now you can "compare" the difference to master at anytime:

btrfs send -p A.20190218-master A.20190218-1850 -f output (or pipe to receive)

*shrug* I'm just not understanding what your use case gains out of
doing a refink copy to another subvolume rather than just make a read
only snapshot.

Is it that subvolume A must accept changes, but itself cannot be the
"master"? And you're selectively reflink copying files from A to B
such that B is the only "master"? That's fine but then I'd say you
don't need any A snapshots, you need B snapshots where:

btrfs sub snap -r B B.20190218-master
btrfs sub snap -r B B.20190219-0920
btrfs send -p B.20190218-master B.20190219-0920 -f output (or pipe to receive)

And so on...

Anyway, my point isn't that you're doing it wrong. I'm jut not
understanding the advantage of doing it the way you're doing it; and
it should be clear by now that most people on the list aren't using
'btrfs send' to compute the difference between two otherwise unrelated
subvolumes.

Part of this *might* be because of how ZFS snapshots work, and Btrfs
users maybe just adopted the same logic due in large part to more
limitations on the ZFS implementation: snapshots are always read only,
incremental send is only done between snapshots of the same dataset,
datasets can't be deleted  until all snapshots are deleted, there are
no reflinks, and no doubt some things I'm forgetting.


-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[André Malm]

The reason why I'm using reflinks instead of snapshots is because the 
master subvolume is large and will contain hundreds of gigabytes worth 
of data. Now if I change / remove, say 10 GB worth of data from the 
master subvolume unrelated to the child subvolume I don't want those 
gigabytes sent down the wire with btrfs send as they are unrelated. Also 
I want to be able to after time, when required, add more data from the 
master subvolume to the child. Thats why I can't split the master 
subvolume into parts of related data.

On 2019-02-18 22:22, Chris Murphy wrote:
> On Mon, Feb 18, 2019 at 12:58 PM André Malm <admin@sheepa.org> wrote:
>
>> Basically what I'm trying to achieve is having a "reference" / "master"
>> btrfs subvolume where i add / edit / remove files continuously. From
>> where i can cp --reflink=always some of those files to new "child"
>> subvolumes as needed. As long as the files are still in the master
>> subvolume I'll only send the diff of the data. The plan is to have this
>> master subvolume across multiple remote machines (synced with rsync or
>> perhaps btrfs send / receive) and being able to update / change it as
>> needed but always only sending the diffs of the child subvolumes. Maybe
>> this is a bit optimistic.
> I'm not sure what you get out of this method that depends on reflink
> rather than just making read only snapshots.
>
> Why don't you create subvolume A as the master, read-write. And once
> it's in the "master state" you want, just snapshot it:
>
> btrfs sub snap -r A A.20190218-master
>
> And now continue to make changes to A subvolume, and on whatever
> schedule you want:
>
> btrfs sub snap -r A A.20190218-1412
> btrfs sub snap -r A A.20190218-1850
> btrfs sub snap -r A A.20190219-0920
>
> And now you can "compare" the difference to master at anytime:
>
> btrfs send -p A.20190218-master A.20190218-1850 -f output (or pipe to receive)
>
> *shrug* I'm just not understanding what your use case gains out of
> doing a refink copy to another subvolume rather than just make a read
> only snapshot.
>
> Is it that subvolume A must accept changes, but itself cannot be the
> "master"? And you're selectively reflink copying files from A to B
> such that B is the only "master"? That's fine but then I'd say you
> don't need any A snapshots, you need B snapshots where:
>
> btrfs sub snap -r B B.20190218-master
> btrfs sub snap -r B B.20190219-0920
> btrfs send -p B.20190218-master B.20190219-0920 -f output (or pipe to receive)
>
> And so on...
>
> Anyway, my point isn't that you're doing it wrong. I'm jut not
> understanding the advantage of doing it the way you're doing it; and
> it should be clear by now that most people on the list aren't using
> 'btrfs send' to compute the difference between two otherwise unrelated
> subvolumes.
>
> Part of this *might* be because of how ZFS snapshots work, and Btrfs
> users maybe just adopted the same logic due in large part to more
> limitations on the ZFS implementation: snapshots are always read only,
> incremental send is only done between snapshots of the same dataset,
> datasets can't be deleted  until all snapshots are deleted, there are
> no reflinks, and no doubt some things I'm forgetting.
>
>

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Mon, Feb 18, 2019 at 2:36 PM André Malm <admin@sheepa.org> wrote:
>
> The reason why I'm using reflinks instead of snapshots is because the
> master subvolume is large and will contain hundreds of gigabytes worth
> of data. Now if I change / remove, say 10 GB worth of data from the
> master subvolume unrelated to the child subvolume I don't want those
> gigabytes sent down the wire with btrfs send as they are unrelated. Also
> I want to be able to after time, when required, add more data from the
> master subvolume to the child. Thats why I can't split the master
> subvolume into parts of related data.

OK, is it your plan to have both master and child on the destination
Btrfs? So you plan to do a 'btrfs send' without -p option to send the
entire master subvolume (snapshot) to destination first? And then you
want to create the child efficiently using a difference between master
and child? Yes I think that might work but I haven't tried it. But of
course, master must already be on the destination.

btrfs sub create master
##populate the master
btrfs sub create childofmaster
cp --reflink master/bunchoffiles childofmaster/
btrfs sub snap -r master master.20190218-initial
btrfs sub snap -r childofmaster childofmaster.20190218-initial
btrfs send master.20190218-initial | btrfs receive /destination/
btrfs send -p master.20190218-initial childofmaster.20190218-initial |
btrfs receive /destination/

However your subsequent incremental send/receive has nothing to do
with master anymore; you indicate above you don't intent to keep
master up to date on the destination. Just child. In that case the
incremental changes look like this:

cp --reflink master/morefiles childofmaster/
btrfs sub snap -r childofmaster childofmaster.20190218T1905
btrfs send child -p childofmaster.20190218-initial
childofmaster.20190218T1905 | btrfs receive /destination/
cp --reflink master/yetmorefiles childofmaster/
btrfs sub snap -r childofmaster childofmaster.20190219T1301
btrfs send child  childofmaster.20190218T1905
childofmaster.20190218T1905  | btrfs receive /destination/


-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[André Malm]

Ok, but I don't want to keep old snapshots of the child volumes. Only 
the latest and then diffing it in regards to the master. Would that be 
possible?

On 2019-02-18 23:28, Chris Murphy wrote:
> On Mon, Feb 18, 2019 at 2:36 PM André Malm <admin@sheepa.org> wrote:
>> The reason why I'm using reflinks instead of snapshots is because the
>> master subvolume is large and will contain hundreds of gigabytes worth
>> of data. Now if I change / remove, say 10 GB worth of data from the
>> master subvolume unrelated to the child subvolume I don't want those
>> gigabytes sent down the wire with btrfs send as they are unrelated. Also
>> I want to be able to after time, when required, add more data from the
>> master subvolume to the child. Thats why I can't split the master
>> subvolume into parts of related data.
> OK, is it your plan to have both master and child on the destination
> Btrfs? So you plan to do a 'btrfs send' without -p option to send the
> entire master subvolume (snapshot) to destination first? And then you
> want to create the child efficiently using a difference between master
> and child? Yes I think that might work but I haven't tried it. But of
> course, master must already be on the destination.
>
> btrfs sub create master
> ##populate the master
> btrfs sub create childofmaster
> cp --reflink master/bunchoffiles childofmaster/
> btrfs sub snap -r master master.20190218-initial
> btrfs sub snap -r childofmaster childofmaster.20190218-initial
> btrfs send master.20190218-initial | btrfs receive /destination/
> btrfs send -p master.20190218-initial childofmaster.20190218-initial |
> btrfs receive /destination/
>
> However your subsequent incremental send/receive has nothing to do
> with master anymore; you indicate above you don't intent to keep
> master up to date on the destination. Just child. In that case the
> incremental changes look like this:
>
> cp --reflink master/morefiles childofmaster/
> btrfs sub snap -r childofmaster childofmaster.20190218T1905
> btrfs send child -p childofmaster.20190218-initial
> childofmaster.20190218T1905 | btrfs receive /destination/
> cp --reflink master/yetmorefiles childofmaster/
> btrfs sub snap -r childofmaster childofmaster.20190219T1301
> btrfs send child  childofmaster.20190218T1905
> childofmaster.20190218T1905  | btrfs receive /destination/
>
>

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Mon, Feb 18, 2019 at 3:58 PM André Malm <admin@sheepa.org> wrote:
>
> Ok, but I don't want to keep old snapshots of the child volumes. Only
> the latest and then diffing it in regards to the master. Would that be
> possible?

In order to do an incremental send/receive you need to have the -p
snapshot on both source and destination file systems. You can delete
snapshots older than that.

I still don't understand why you're diffing in regards to master. It's
going to be inherently inefficient because you've said a large amount
of the contents of master aren't important; but by using it as the -p
reference snapshot, you're asking send to do a comparison that
includes a lot of metadata you don't care about. It's more efficient
to diff the incremental snapshots of the changing state of
"childofmaster" subvolume.

-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[André Malm]

I assume i would have to use rsync (with --inplace possibly) to keep the 
master volume in sync between machines?

Say for example I have a (large) file on master, on machine A, I cp 
reflink it to a child subvolume. I then send -p child subvolume to 
remote machine B (which already have the master volume). Then i change 
parts of the file on the master of machine A. I then rsync (?) the 
master volume so its the same across the machines. Can I then later send 
-p the child volume, either back to the original machine (A) or to a 3rd 
machine (C) given that the master volumes are synced?

About the efficiency I'm not planning to remove large amounts of data 
that is used by child subvolumes (although some will be updated). But 
given the unpredictability of what files will be used by child 
subvolumes i might remove large unused amounts of data.

On 2019-02-19 00:49, Chris Murphy wrote:
> On Mon, Feb 18, 2019 at 3:58 PM André Malm <admin@sheepa.org> wrote:
>> Ok, but I don't want to keep old snapshots of the child volumes. Only
>> the latest and then diffing it in regards to the master. Would that be
>> possible?
> In order to do an incremental send/receive you need to have the -p
> snapshot on both source and destination file systems. You can delete
> snapshots older than that.
>
> I still don't understand why you're diffing in regards to master. It's
> going to be inherently inefficient because you've said a large amount
> of the contents of master aren't important; but by using it as the -p
> reference snapshot, you're asking send to do a comparison that
> includes a lot of metadata you don't care about. It's more efficient
> to diff the incremental snapshots of the changing state of
> "childofmaster" subvolume.
>

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Mon, Feb 18, 2019 at 4:58 PM André Malm <admin@sheepa.org> wrote:
>
> I assume i would have to use rsync (with --inplace possibly) to keep the
> master volume in sync between machines?

Why?

You previously said you didn't want to do that:
" if I change / remove, say 10 GB worth of data from the
master subvolume unrelated to the child subvolume I don't want those
gigabytes sent down the wire with btrfs send as they are unrelated. "

So instead of sending those changes with send you're going to send the
changes with rsync which is even more inefficient?

> Say for example I have a (large) file on master, on machine A, I cp
> reflink it to a child subvolume. I then send -p child subvolume to
> remote machine B (which already have the master volume).

1. The master snapshots must be identical snapshots on source and
destination; it must have been replicated using send/receive or
receive will complain.
2. I don't know what "send -p child subvolume to remove" means because
-p option means two snapshots must be included and I don't know what
two snapshots you're planning on using, so I can't answer the
question.

>Then i change
> parts of the file on the master of machine A. I then rsync (?) the
> master volume so its the same across the machines. Can I then later send
> -p the child volume, either back to the original machine (A) or to a 3rd
> machine (C) given that the master volumes are synced?

rsync'd subvolumes across volumes aren't consistent identical by btrfs
receive. I expect they can't be used for incremental send/receive.
What you're trying to achieve, big picture, isn't clear. You're
describing it in terms of send/receive and confusing it with rsync. I
don't understand what rsync has to do with this.

You realize if you merely move a large file within a subvolume, and
then you using rsync to a remote system, it will delete the original
file and then sync over the nework that same data to the new location?
Rsync has no idea how to just move files around. Btrfs send-receive
does.

> About the efficiency I'm not planning to remove large amounts of data
> that is used by child subvolumes (although some will be updated). But
> given the unpredictability of what files will be used by child
> subvolumes i might remove large unused amounts of data.

OK?


-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Mon, Feb 18, 2019 at 5:16 PM Chris Murphy <lists@colorremedies.com> wrote:

> rsync'd subvolumes across volumes aren't consistent identical by btrfs


s/consistent/considered


-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[André Malm]

Rsync is probably i bad idea yes. I could btrfs send -p the changed 
"new" master subvolume and then delete the old master subvolume and then 
reference the new master subvolume when transferring it later on i guess?

I'll explain the problem I'm trying to solve abit better;

Say i have a program that will run in multiple instances. The program 
requires a dataset of large files to run (say 20GB). The dataset will be 
updated over time, i.e parts of them changes. These changes should only 
apply to new instances for the program. The program will also generate 
new data (both new files and also changing data in the the shared 
dataset) that is unique to the instance of the child subvolume. Finally 
I need to transfer the program together with its generated data to 
another remote machine to continue it's processing there. What i want to 
achieve is avoid having to transfer the entire dataset when only small 
parts of it is changed by the program. I also want to avoid having to 
duplicate copies of the data on the remote machine.

On 2019-02-19 01:16, Chris Murphy wrote:
> On Mon, Feb 18, 2019 at 4:58 PM André Malm <admin@sheepa.org> wrote:
>> I assume i would have to use rsync (with --inplace possibly) to keep the
>> master volume in sync between machines?
> Why?
>
> You previously said you didn't want to do that:
> " if I change / remove, say 10 GB worth of data from the
> master subvolume unrelated to the child subvolume I don't want those
> gigabytes sent down the wire with btrfs send as they are unrelated. "
>
> So instead of sending those changes with send you're going to send the
> changes with rsync which is even more inefficient?
>
>> Say for example I have a (large) file on master, on machine A, I cp
>> reflink it to a child subvolume. I then send -p child subvolume to
>> remote machine B (which already have the master volume).
> 1. The master snapshots must be identical snapshots on source and
> destination; it must have been replicated using send/receive or
> receive will complain.
> 2. I don't know what "send -p child subvolume to remove" means because
> -p option means two snapshots must be included and I don't know what
> two snapshots you're planning on using, so I can't answer the
> question.
>
>> Then i change
>> parts of the file on the master of machine A. I then rsync (?) the
>> master volume so its the same across the machines. Can I then later send
>> -p the child volume, either back to the original machine (A) or to a 3rd
>> machine (C) given that the master volumes are synced?
> rsync'd subvolumes across volumes aren't consistent identical by btrfs
> receive. I expect they can't be used for incremental send/receive.
> What you're trying to achieve, big picture, isn't clear. You're
> describing it in terms of send/receive and confusing it with rsync. I
> don't understand what rsync has to do with this.
>
> You realize if you merely move a large file within a subvolume, and
> then you using rsync to a remote system, it will delete the original
> file and then sync over the nework that same data to the new location?
> Rsync has no idea how to just move files around. Btrfs send-receive
> does.
>
>> About the efficiency I'm not planning to remove large amounts of data
>> that is used by child subvolumes (although some will be updated). But
>> given the unpredictability of what files will be used by child
>> subvolumes i might remove large unused amounts of data.
> OK?
>
>

Re: Btrfs send with parent different size depending on source of files.

[Chris Murphy]

On Mon, Feb 18, 2019 at 5:28 PM André Malm <admin@sheepa.org> wrote:
>
> Rsync is probably i bad idea yes. I could btrfs send -p the changed
> "new" master subvolume and then delete the old master subvolume and then
> reference the new master subvolume when transferring it later on i guess?

I'm not sure how your application reacts to snapshots or reflinks, or
how it updates its files. All of that needs to be tested to see what
the incremental send size is, and if the resulting received snapshot
contains files with the integrity your application expects, and so on.

>
> I'll explain the problem I'm trying to solve abit better;
>
> Say i have a program that will run in multiple instances. The program
> requires a dataset of large files to run (say 20GB). The dataset will be
> updated over time, i.e parts of them changes. These changes should only
> apply to new instances for the program. The program will also generate
> new data (both new files and also changing data in the the shared
> dataset) that is unique to the instance of the child subvolume. Finally
> I need to transfer the program together with its generated data to
> another remote machine to continue it's processing there. What i want to
> achieve is avoid having to transfer the entire dataset when only small
> parts of it is changed by the program. I also want to avoid having to
> duplicate copies of the data on the remote machine.

Yep. Based on this description though, the only time I grok using
'btrfs send -p master.snap child.snap | btrfs receive /destination/'
is for the initial transfer of child. Master must be already fully
replicated. Now you can snapshot master and child on separate
schedules to account for their different use case, and send their
increments independent of each other. Or in fact maybe you'll realize
you do have a use case for clone.

Have you looked at GlusterFS or Ceph for this use case? I kinda wonder
if there's any simplification to just having a clustered file system
make all of the send/receive stuff go away, and you can ensure your
data is replicated pretty much immediately, and is always available
for all computers. *shrug* That's off topic but I'm curious if there
are ways to simplify this for your use case.



-- 
Chris Murphy

Re: Btrfs send with parent different size depending on source of files.

[André Malm]

Okay, I will indeed test the setup thoroughly. I have considered running 
a distributed filesystem such as Ceph but my concern is that it will be 
way to slow as disk IO speed is important. Anyways, thank you for your help!

On 2019-02-19 04:54, Chris Murphy wrote:
> On Mon, Feb 18, 2019 at 5:28 PM André Malm <admin@sheepa.org> wrote:
>> Rsync is probably i bad idea yes. I could btrfs send -p the changed
>> "new" master subvolume and then delete the old master subvolume and then
>> reference the new master subvolume when transferring it later on i guess?
> I'm not sure how your application reacts to snapshots or reflinks, or
> how it updates its files. All of that needs to be tested to see what
> the incremental send size is, and if the resulting received snapshot
> contains files with the integrity your application expects, and so on.
>
>> I'll explain the problem I'm trying to solve abit better;
>>
>> Say i have a program that will run in multiple instances. The program
>> requires a dataset of large files to run (say 20GB). The dataset will be
>> updated over time, i.e parts of them changes. These changes should only
>> apply to new instances for the program. The program will also generate
>> new data (both new files and also changing data in the the shared
>> dataset) that is unique to the instance of the child subvolume. Finally
>> I need to transfer the program together with its generated data to
>> another remote machine to continue it's processing there. What i want to
>> achieve is avoid having to transfer the entire dataset when only small
>> parts of it is changed by the program. I also want to avoid having to
>> duplicate copies of the data on the remote machine.
> Yep. Based on this description though, the only time I grok using
> 'btrfs send -p master.snap child.snap | btrfs receive /destination/'
> is for the initial transfer of child. Master must be already fully
> replicated. Now you can snapshot master and child on separate
> schedules to account for their different use case, and send their
> increments independent of each other. Or in fact maybe you'll realize
> you do have a use case for clone.
>
> Have you looked at GlusterFS or Ceph for this use case? I kinda wonder
> if there's any simplification to just having a clustered file system
> make all of the send/receive stuff go away, and you can ensure your
> data is replicated pretty much immediately, and is always available
> for all computers. *shrug* That's off topic but I'm curious if there
> are ways to simplify this for your use case.
>
>
>