* Btrfs send with parent different size depending on source of files. @ 2019-02-14 11:37 André Malm 2019-02-14 22:37 ` Chris Murphy ` (2 more replies) 0 siblings, 3 replies; 30+ messages in thread From: André Malm @ 2019-02-14 11:37 UTC (permalink / raw) To: linux-btrfs Hello, I'm not sure this is the right forum to ask on but I'll try and if its not I do apologize. I have also created a stack overflow question without success ( https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files ) but ill paste the question here too. Thank you. What i'm trying to achieve is sending only the diff of the parent with btrfs send -p Running this will produce a file 'out' with size 639 bytes, i.e only diff sent. ==================================================== btrfs subvolume create A btrfs subvolume create B mkdir A/dir dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K cp --reflink=always A/dir/server.jar B/server.jar btrfs subvolume snapshot -r A a btrfs subvolume snapshot -r B b btrfs send -p a b > out ==================================================== However if I change the dd command to wget like this: ==================================================== btrfs subvolume create A btrfs subvolume create B mkdir A/dir wget -O A/dir/server.jar https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar cp --reflink=always A/dir/server.jar B/server.jar btrfs subvolume snapshot -r A a btrfs subvolume snapshot -r B b btrfs send -p a b > out ==================================================== The 'out' file is 34M, which is the full size. What causes this? ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-14 11:37 Btrfs send with parent different size depending on source of files André Malm @ 2019-02-14 22:37 ` Chris Murphy 2019-02-15 4:00 ` Remi Gauvin 2019-02-15 17:45 ` Andrei Borzenkov 2019-02-15 19:29 ` Remi Gauvin 2019-02-16 20:08 ` Andrei Borzenkov 2 siblings, 2 replies; 30+ messages in thread From: Chris Murphy @ 2019-02-14 22:37 UTC (permalink / raw) To: André Malm; +Cc: Btrfs BTRFS On Thu, Feb 14, 2019 at 4:37 AM André Malm <admin@sheepa.org> wrote: > > Hello, > > I'm not sure this is the right forum to ask on but I'll try and if its > not I do apologize. I have also created a stack overflow question > without success ( > https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files > ) but ill paste the question here too. Thank you. > > What i'm trying to achieve is sending only the diff of the parent with > btrfs send -p > > Running this will produce a file 'out' with size 639 bytes, i.e only > diff sent. > > ==================================================== > > btrfs subvolume create A > btrfs subvolume create B > mkdir A/dir > > dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K > cp --reflink=always A/dir/server.jar B/server.jar > > btrfs subvolume snapshot -r A a > btrfs subvolume snapshot -r B b > btrfs send -p a b > out It doesn't work this way. The snapshots a and b are not based on the same underlying subvolume. The gist is that you would keep changing A, and take additional snapshots of A, such as a.1 a.2 a.3, and you can do incremental send with 'btrfs send -p a.1 a.2' which describes the difference between those two snapshots of A at their respective moments in time. You could also do 'btrfs send -p a.2 a.3' or even 'btrfs send -p a.1 a.3' But as there's no relationship between snapshots a and b, I consider it a bug/missing error handling feature, that btrfs send doesn't fail in this case. By using -p you're claiming there is a parent-child relationship between a and b, but there plainly isn't. Read this: https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step Depending on your use case if you can describe it, might be tolerated with some adjustments by using the -c clone option instead. -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-14 22:37 ` Chris Murphy @ 2019-02-15 4:00 ` Remi Gauvin 2019-02-15 18:38 ` Chris Murphy 2019-02-15 17:45 ` Andrei Borzenkov 1 sibling, 1 reply; 30+ messages in thread From: Remi Gauvin @ 2019-02-15 4:00 UTC (permalink / raw) To: linux-btrfs [-- Attachment #1.1.1: Type: text/plain, Size: 1469 bytes --] > It doesn't work this way. The snapshots a and b are not based on the > same underlying subvolume. The gist is that you would keep changing A, > and take additional snapshots of A, such as a.1 a.2 a.3, and you can > do incremental send with 'btrfs send -p a.1 a.2' which describes the > difference between those two snapshots of A at their respective > moments in time. You could also do 'btrfs send -p a.2 a.3' or even > 'btrfs send -p a.1 a.3' > > But as there's no relationship between snapshots a and b, I consider > it a bug/missing error handling feature, that btrfs send doesn't fail > in this case. By using -p you're claiming there is a parent-child > relationship between a and b, but there plainly isn't. They kind of are related though, since the two snapshots reference the same data blocks, and you can see it work in the first example with the 40MB of random data. However, I can replicate this odd behaviour the conventional way. btrfs sub create A mkdir A/dir dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K btrfs sub snap -r A a cp --reflink=always A/dir/server.jar A/server.jar rm A/dir -rf btrfs sub snap -r A b btrfs send -p a b > out The out file is only 773 bytes. However, if you repeat all those same steps, but replace the dd with: wget -O A/dir/server.jar https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar The resulting out file is 34MB. [-- Attachment #1.1.2: remi.vcf --] [-- Type: text/x-vcard, Size: 203 bytes --] begin:vcard fn:Remi Gauvin n:Gauvin;Remi org:Georgian Infotech adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada email;internet:remi@georgianit.com tel;work:226-256-1545 version:2.1 end:vcard [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-15 4:00 ` Remi Gauvin @ 2019-02-15 18:38 ` Chris Murphy 2019-02-15 18:56 ` Remi Gauvin 0 siblings, 1 reply; 30+ messages in thread From: Chris Murphy @ 2019-02-15 18:38 UTC (permalink / raw) To: Remi Gauvin; +Cc: linux-btrfs On Thu, Feb 14, 2019 at 9:00 PM Remi Gauvin <remi@georgianit.com> wrote: > > > > It doesn't work this way. The snapshots a and b are not based on the > > same underlying subvolume. The gist is that you would keep changing A, > > and take additional snapshots of A, such as a.1 a.2 a.3, and you can > > do incremental send with 'btrfs send -p a.1 a.2' which describes the > > difference between those two snapshots of A at their respective > > moments in time. You could also do 'btrfs send -p a.2 a.3' or even > > 'btrfs send -p a.1 a.3' > > > > But as there's no relationship between > > snapshots a and b, I consider > > it a bug/missing error handling feature, that btrfs send doesn't fail > > in this case. By using -p you're claiming there is a parent-child > > relationship between a and b, but there plainly isn't. > > They kind of are related though, since the two snapshots reference the > same data blocks, and you can see it work in the first example with the > 40MB of random data. Is it more of a case for clone, -c option? I don't ever use it, and the man page could use a couple examples to make it more clear what it can do. And it makes sense this option could be more tolerant, less error checking, than -p. But from man page and in particular the wiki, it's clear to me that the two snapshots following -p option, need to each be derived from one subvolume. I actually don't like the "parent - child" metaphor because really the parent is the original source rw subvolume, and its two children snapshots are the ones used with -p. The first is an older sibling, the second a younger sibling. And that metaphor fails too because you'd expect the second sibling to have more information which plainly is not the case with actual siblings. :P > The out file is only 773 bytes. However, if you repeat all those same > steps, but replace the dd with: > wget -O A/dir/server.jar > https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar > > The resulting out file is 34MB. Well I'd say maybe use -vvv and --no-data instead of -f and see what it's doing. It sounds like the former has no payload, just difference information, and the latter has a payload. I don't know why. -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-15 18:38 ` Chris Murphy @ 2019-02-15 18:56 ` Remi Gauvin 2019-02-16 20:10 ` Andrei Borzenkov 0 siblings, 1 reply; 30+ messages in thread From: Remi Gauvin @ 2019-02-15 18:56 UTC (permalink / raw) To: linux-btrfs [-- Attachment #1.1.1: Type: text/plain, Size: 467 bytes --] On 2019-02-15 1:38 p.m., Chris Murphy wrote: > > Well I'd say maybe use -vvv and --no-data instead of -f and see what > it's doing. It sounds like the former has no payload, just difference > information, and the latter has a payload. I don't know why. > > Both have all the payload they need to re-create the subvolumes. The first one correctly CoW's the file from subvolume a.. the second one stuffs it in the output... Why is indeed the question. [-- Attachment #1.1.2: remi.vcf --] [-- Type: text/x-vcard, Size: 203 bytes --] begin:vcard fn:Remi Gauvin n:Gauvin;Remi org:Georgian Infotech adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada email;internet:remi@georgianit.com tel;work:226-256-1545 version:2.1 end:vcard [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-15 18:56 ` Remi Gauvin @ 2019-02-16 20:10 ` Andrei Borzenkov 0 siblings, 0 replies; 30+ messages in thread From: Andrei Borzenkov @ 2019-02-16 20:10 UTC (permalink / raw) To: Remi Gauvin, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 608 bytes --] 15.02.2019 21:56, Remi Gauvin пишет: > On 2019-02-15 1:38 p.m., Chris Murphy wrote: >> >> Well I'd say maybe use -vvv and --no-data instead of -f and see what >> it's doing. It sounds like the former has no payload, just difference >> information, and the latter has a payload. I don't know why. >> >> > > > Both have all the payload they need to re-create the subvolumes. The > first one correctly CoW's the file from subvolume a.. the second one > stuffs it in the output... Why is indeed the question. > The last extent is incomplete which causes btrfs to not attempt cloning. [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-14 22:37 ` Chris Murphy 2019-02-15 4:00 ` Remi Gauvin @ 2019-02-15 17:45 ` Andrei Borzenkov 2019-02-15 19:11 ` Chris Murphy 1 sibling, 1 reply; 30+ messages in thread From: Andrei Borzenkov @ 2019-02-15 17:45 UTC (permalink / raw) To: Chris Murphy, André Malm; +Cc: Btrfs BTRFS 15.02.2019 1:37, Chris Murphy пишет: > On Thu, Feb 14, 2019 at 4:37 AM André Malm <admin@sheepa.org> wrote: >> >> Hello, >> >> I'm not sure this is the right forum to ask on but I'll try and if its >> not I do apologize. I have also created a stack overflow question >> without success ( >> https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files >> ) but ill paste the question here too. Thank you. >> >> What i'm trying to achieve is sending only the diff of the parent with >> btrfs send -p >> >> Running this will produce a file 'out' with size 639 bytes, i.e only >> diff sent. >> >> ==================================================== >> >> btrfs subvolume create A >> btrfs subvolume create B >> mkdir A/dir >> >> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K >> cp --reflink=always A/dir/server.jar B/server.jar >> >> btrfs subvolume snapshot -r A a >> btrfs subvolume snapshot -r B b >> btrfs send -p a b > out > > It doesn't work this way. It works exactly this way. > The snapshots a and b are not based on the > same underlying subvolume. So what? What prevents you from computing differences between two subvolumes? That is real question. > The gist is that you would keep changing A, > and take additional snapshots of A, such as a.1 a.2 a.3, and you can > do incremental send with 'btrfs send -p a.1 a.2' which describes the > difference between those two snapshots of A at their respective > moments in time. You could also do 'btrfs send -p a.2 a.3' or even > 'btrfs send -p a.1 a.3' > That it is intended to be used this way does not mean it is restricted to this way technically. Whether it should have been restricted is another question. > But as there's no relationship between snapshots a and b, I consider > it a bug/missing error handling feature, that btrfs send doesn't fail > in this case. By using -p you're claiming there is a parent-child > relationship between a and b, but there plainly isn't. > No. By using "-p" you designate subvolume which must be used as base to apply differential (I explicitly does not use "incremental") stream on remote side. Nothing more. Whether it should have different semantic is subject to discussion, but it does not do what you wish it to do. > Read this: > https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step > > Depending on your use case if you can describe it, might be tolerated > with some adjustments by using the -c clone option instead. > The only difference is that "-c" in absence of explicit "-p" attempts to auto-detect the "best" "base" subvolume - but it selects between subvolumes given with "-c" options so "-c" with single subvolume is entirely equivalent to "-p". ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-15 17:45 ` Andrei Borzenkov @ 2019-02-15 19:11 ` Chris Murphy 2019-02-16 20:26 ` Andrei Borzenkov 0 siblings, 1 reply; 30+ messages in thread From: Chris Murphy @ 2019-02-15 19:11 UTC (permalink / raw) To: Andrei Borzenkov; +Cc: Chris Murphy, André Malm, Btrfs BTRFS On Fri, Feb 15, 2019 at 10:45 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote: > > 15.02.2019 1:37, Chris Murphy пишет: > > On Thu, Feb 14, 2019 at 4:37 AM André Malm <admin@sheepa.org> wrote: > >> > >> Hello, > >> > >> I'm not sure this is the right forum to ask on but I'll try and if its > >> not I do apologize. I have also created a stack overflow question > >> without success ( > >> https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files > >> ) but ill paste the question here too. Thank you. > >> > >> What i'm trying to achieve is sending only the diff of the parent with > >> btrfs send -p > >> > >> Running this will produce a file 'out' with size 639 bytes, i.e only > >> diff sent. > >> > >> ==================================================== > >> > >> btrfs subvolume create A > >> btrfs subvolume create B > >> mkdir A/dir > >> > >> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K > >> cp --reflink=always A/dir/server.jar B/server.jar > >> > >> btrfs subvolume snapshot -r A a > >> btrfs subvolume snapshot -r B b > >> btrfs send -p a b > out > > > > It doesn't work this way. > > It works exactly this way. Unproven. And in fact we have an example where it does not work, hence the point of the thread! The proven way it works, is as I've described, and many emails over a decade on this list, inferred from the man page, and the step by step recipe on the Wiki. > > The gist is that you would keep changing A, > > and take additional snapshots of A, such as a.1 a.2 a.3, and you can > > do incremental send with 'btrfs send -p a.1 a.2' which describes the > > difference between those two snapshots of A at their respective > > moments in time. You could also do 'btrfs send -p a.2 a.3' or even > > 'btrfs send -p a.1 a.3' > > > > That it is intended to be used this way does not mean it is restricted > to this way technically. Whether it should have been restricted is > another question. That's fair. But also the alternative cannot be stated to "work exactly this way" when this very thread starts with two examples leading to different and therefore unexpected results, differing only in how the file was created: dd vs wget And no one can explain that. Which tells me none of us really understands the full scope and operation of send. By the way, it is possible for send to succeed in producing a stream, that receive will reject. So there is a difference in the error checking when using send with receive, than send without receive. > > But as there's no relationship between snapshots a and b, I consider > > it a bug/missing error handling feature, that btrfs send doesn't fail > > in this case. By using -p you're claiming there is a parent-child > > relationship between a and b, but there plainly isn't. > > > > No. By using "-p" you designate subvolume which must be used as base to > apply differential (I explicitly does not use "incremental") stream on > remote side. Nothing more. Whether it should have different semantic is > subject to discussion, but it does not do what you wish it to do. man page and wiki use incremental - and linguistically incremental != differential, increment is a subset of difference. Difference suggests the order does not matter. Increment suggests it does matter. btrfs sub create A cp somestuff A btrfs sub snap -r A a.1 cp morestuff A btrfs sub snap -r A a.2 The normal case of send is: btrfs send -p a.1 a.2 However, is it valid to do? btrfs send -p a.2 a1 I haven't tried this. Maybe send allows it and it works as expected? Maybe receive also works. If it's allows *and* also if it works as expected, then maybe the command is differential and not incremental. That is a rather fundamental thing to know! -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-15 19:11 ` Chris Murphy @ 2019-02-16 20:26 ` Andrei Borzenkov 2019-02-16 20:32 ` Andrei Borzenkov 2019-02-18 18:00 ` Chris Murphy 0 siblings, 2 replies; 30+ messages in thread From: Andrei Borzenkov @ 2019-02-16 20:26 UTC (permalink / raw) To: Chris Murphy; +Cc: André Malm, Btrfs BTRFS 15.02.2019 22:11, Chris Murphy пишет: ... >>> >>> It doesn't work this way. >> >> It works exactly this way. > > Unproven. And in fact we have an example where it does not work, hence > the point of the thread! > > The proven way it works, is as I've described, and many emails over a > decade on this list, inferred from the man page, and the step by step > recipe on the Wiki. > Oh well. 10:~ # mkfs -t btrfs -f /dev/sdb1 btrfs-progs v4.20.1 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: c8dd61f1-bf32-449b-a154-3380cf4348b5 Node size: 16384 Sector size: 4096 Filesystem size: 1023.00MiB Block group profiles: Data: single 8.00MiB Metadata: DUP 51.12MiB System: DUP 8.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 1023.00MiB /dev/sdb1 10:~ # mount -t btrfs /dev/sdb1 /mnt 10:~ # cd /mnt 10:/mnt # btrfs su cr A Create subvolume './A' 10:/mnt # mkdir A/dir 10:/mnt # dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K 40960+0 records in 40960+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.380302 s, 110 MB/s 10:/mnt # btrfs su sn -r A a1 Create a readonly snapshot of 'A' in './a1' 10:/mnt # cp --reflink=always A/dir/server.jar A/server.jar 10:/mnt # btrfs su sn -r A a2 Create a readonly snapshot of 'A' in './a2' 10:/mnt # btrfs send -p a1 a2 > out At subvol a2 10:/mnt # ll -sh out 4.0K -rw-r--r-- 1 root root 567 Feb 16 23:15 out 10:/mnt # cd 10:~ # umount /mnt 10:~ # mkfs -t btrfs -f /dev/sdc1 btrfs-progs v4.20.1 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: a2d85827-116a-4dee-874c-d3b180728613 Node size: 16384 Sector size: 4096 Filesystem size: 1023.00MiB Block group profiles: Data: single 8.00MiB Metadata: DUP 51.12MiB System: DUP 8.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 1023.00MiB /dev/sdc1 10:~ # mount -t btrfs /dev/sdc1 /mnt 10:~ # cd /mnt 10:/mnt # btrfs su cr A Create subvolume './A' 10:/mnt # mkdir A/dir 10:/mnt # dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625 34625+0 records in 34625+0 records out 35456000 bytes (35 MB, 34 MiB) copied, 0.31223 s, 114 MB/s 10:/mnt # btrfs su sn -r A a1 Create a readonly snapshot of 'A' in './a1' 10:/mnt # cp --reflink=always A/dir/server.jar A/server.jar 10:/mnt # btrfs su sn -r A a2 Create a readonly snapshot of 'A' in './a2' 10:/mnt # btrfs send -p a1 a2 > out At subvol a2 10:/mnt # ll -sh out 34M -rw-r--r-- 1 root root 34M Feb 16 23:17 out 10:/mnt # >>> The gist is that you would keep changing A, >>> and take additional snapshots of A, such as a.1 a.2 a.3, and you can >>> do incremental send with 'btrfs send -p a.1 a.2' which describes the >>> difference between those two snapshots of A at their respective >>> moments in time. You could also do 'btrfs send -p a.2 a.3' or even >>> 'btrfs send -p a.1 a.3' >>> >> >> That it is intended to be used this way does not mean it is restricted >> to this way technically. Whether it should have been restricted is >> another question. > > That's fair. But also the alternative cannot be stated to "work > exactly this way" when this very thread starts with two examples > leading to different and therefore unexpected results, differing only > in how the file was created: dd vs wget > It is not the *only* difference between two examples. > And no one can explain that. I just sent explanation why it happens. ... >> >> No. By using "-p" you designate subvolume which must be used as base to >> apply differential (I explicitly does not use "incremental") stream on >> remote side. Nothing more. Whether it should have different semantic is >> subject to discussion, but it does not do what you wish it to do. > > man page and wiki use incremental - and linguistically incremental != > differential, increment is a subset of difference. > btrfs send computes differential stream. Whether it will be "incremental" or not depends on how you are using it. > Difference suggests the order does not matter. Increment suggests it > does matter. > > btrfs sub create A > cp somestuff A > btrfs sub snap -r A a.1 > cp morestuff A > btrfs sub snap -r A a.2 > > The normal case of send is: > > btrfs send -p a.1 a.2 > > However, is it valid to do? > > btrfs send -p a.2 a1 > Of course it is. Stream contains differences between the first and the second stream, where order does matter. "Incremental" implies temporal ordering between two snapshots which does not exist on "btrfs send" level and must be ensured by tools that use "btrfs send/receive". There is no inherent parent/child or temporal relationship implied by "btrfs send" itself. "btrfs send" *will* compute differences from a.1 to a.2 although a.2 is newer than a.1; it is up to you to ensure you apply this stream to correct base on destination. ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-16 20:26 ` Andrei Borzenkov @ 2019-02-16 20:32 ` Andrei Borzenkov 2019-02-18 18:00 ` Chris Murphy 1 sibling, 0 replies; 30+ messages in thread From: Andrei Borzenkov @ 2019-02-16 20:32 UTC (permalink / raw) To: Chris Murphy; +Cc: André Malm, Btrfs BTRFS 16.02.2019 23:26, Andrei Borzenkov пишет: ... >> Difference suggests the order does not matter. Increment suggests it >> does matter. >> >> btrfs sub create A >> cp somestuff A >> btrfs sub snap -r A a.1 >> cp morestuff A >> btrfs sub snap -r A a.2 >> >> The normal case of send is: >> >> btrfs send -p a.1 a.2 >> >> However, is it valid to do? >> >> btrfs send -p a.2 a1 >> > > Of course it is. Stream contains differences between the first and the > second stream, where order does matter. "Incremental" implies temporal > ordering between two snapshots which does not exist on "btrfs send" > level and must be ensured by tools that use "btrfs send/receive". There > is no inherent parent/child or temporal relationship implied by "btrfs > send" itself. "btrfs send" *will* compute differences from a.1 to a.2 > although a.2 is newer than a.1; it is up to you to ensure you apply this > stream to correct base on destination. > And BTW your example can actually be useful to revert (roll back) content of volume to previous snapshot. ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-16 20:26 ` Andrei Borzenkov 2019-02-16 20:32 ` Andrei Borzenkov @ 2019-02-18 18:00 ` Chris Murphy 1 sibling, 0 replies; 30+ messages in thread From: Chris Murphy @ 2019-02-18 18:00 UTC (permalink / raw) To: Andrei Borzenkov; +Cc: Chris Murphy, André Malm, Btrfs BTRFS On Sat, Feb 16, 2019 at 1:26 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote: > > 15.02.2019 22:11, Chris Murphy пишет: > > The proven way it works, is as I've described, and many emails over a > > decade on this list, inferred from the man page, and the step by step > > recipe on the Wiki. > > > > Oh well. OK, you're right, I'm wrong, however your example is the only one I've seen in ~10 years that describes this alternative. > btrfs send computes differential stream. Whether it will be > "incremental" or not depends on how you are using it. Yes, it's a difference computation, but the 'man btrfs-send' page explicitly says incremental which is not the same thing. Also adding 'btrfs receive' is important to test your example. I've seen 'btrfs send' succeed where 'btrfs receive' complains. >Stream contains differences between the first and the > second stream, where order does matter. "Incremental" implies temporal > ordering between two snapshots which does not exist on "btrfs send" > level and must be ensured by tools that use "btrfs send/receive". It's fine to have this alternative explanation. However the existing 'man btrfs-send' page says there are two modes: full and incremental. -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-14 11:37 Btrfs send with parent different size depending on source of files André Malm 2019-02-14 22:37 ` Chris Murphy @ 2019-02-15 19:29 ` Remi Gauvin 2019-02-15 19:41 ` Remi Gauvin 2019-02-16 20:08 ` Andrei Borzenkov 2 siblings, 1 reply; 30+ messages in thread From: Remi Gauvin @ 2019-02-15 19:29 UTC (permalink / raw) To: André Malm, linux-btrfs [-- Attachment #1.1.1: Type: text/plain, Size: 662 bytes --] On 2019-02-14 6:37 a.m., André Malm wrote: > > > The 'out' file is 34M, which is the full size. > > What causes this? > Just to narrow it down, the fault is the contents of the server.jar file, not how it was created. If I upload a urandom server.jar file to my own website, then download it with wget as described, btrfs send creates a properly small diff However, if I upload the actual server.jar file to the same URL, then repeat the process, I again get a 34MB out How often does this happen? Did you just happen to stumble on the one file in the Universe that does this with server.jar? Or was there are a wider context? [-- Attachment #1.1.2: remi.vcf --] [-- Type: text/x-vcard, Size: 203 bytes --] begin:vcard fn:Remi Gauvin n:Gauvin;Remi org:Georgian Infotech adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada email;internet:remi@georgianit.com tel;work:226-256-1545 version:2.1 end:vcard [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-15 19:29 ` Remi Gauvin @ 2019-02-15 19:41 ` Remi Gauvin 0 siblings, 0 replies; 30+ messages in thread From: Remi Gauvin @ 2019-02-15 19:41 UTC (permalink / raw) To: André Malm, linux-btrfs [-- Attachment #1.1.1: Type: text/plain, Size: 376 bytes --] On 2019-02-15 2:29 p.m., Remi Gauvin wrote: > > How often does this happen? Did you just happen to stumble on the one > file in the Universe that does this with server.jar? Or was there are a > wider context? > I just tried with a random epub file, and again get the full size out, so it's a complete mystery to me why a file filled with random bytes works. [-- Attachment #1.1.2: remi.vcf --] [-- Type: text/x-vcard, Size: 203 bytes --] begin:vcard fn:Remi Gauvin n:Gauvin;Remi org:Georgian Infotech adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada email;internet:remi@georgianit.com tel;work:226-256-1545 version:2.1 end:vcard [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-14 11:37 Btrfs send with parent different size depending on source of files André Malm 2019-02-14 22:37 ` Chris Murphy 2019-02-15 19:29 ` Remi Gauvin @ 2019-02-16 20:08 ` Andrei Borzenkov 2019-02-17 3:11 ` Remi Gauvin 2 siblings, 1 reply; 30+ messages in thread From: Andrei Borzenkov @ 2019-02-16 20:08 UTC (permalink / raw) To: André Malm, linux-btrfs 14.02.2019 14:37, André Malm пишет: > Hello, > > I'm not sure this is the right forum to ask on but I'll try and if its > not I do apologize. I have also created a stack overflow question > without success ( > https://stackoverflow.com/questions/54634703/btrfs-send-with-parent-different-size-depending-on-source-of-files > ) but ill paste the question here too. Thank you. > > What i'm trying to achieve is sending only the diff of the parent with > btrfs send -p > > Running this will produce a file 'out' with size 639 bytes, i.e only > diff sent. > > ==================================================== > > btrfs subvolume create A > btrfs subvolume create B > mkdir A/dir > > dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=40K > cp --reflink=always A/dir/server.jar B/server.jar > > btrfs subvolume snapshot -r A a > btrfs subvolume snapshot -r B b > btrfs send -p a b > out > > ==================================================== > > However if I change the dd command to wget like this: > > ==================================================== > > btrfs subvolume create A > btrfs subvolume create B > mkdir A/dir > > wget -O A/dir/server.jar > https://launcher.mojang.com/v1/objects/20c069d373e77265aaeeedb733f7051e294325a3/server.jar > > cp --reflink=always A/dir/server.jar B/server.jar > > btrfs subvolume snapshot -r A a > btrfs subvolume snapshot -r B b > btrfs send -p a b > out > > ==================================================== > > > The 'out' file is 34M, which is the full size. > > What causes this? > > File size of "real" server.jar is not exact multiple of btrfs block size (4096) which causes the last extent to be "incomplete". btrfs won't clone extent in this case. You can trivially reproduce it by using the same size in the first case (although exact extent distribution may be different of course, in the worst case you get single extent which is sent in full): dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-16 20:08 ` Andrei Borzenkov @ 2019-02-17 3:11 ` Remi Gauvin 2019-02-18 13:05 ` André Malm 0 siblings, 1 reply; 30+ messages in thread From: Remi Gauvin @ 2019-02-17 3:11 UTC (permalink / raw) To: linux-btrfs [-- Attachment #1.1.1: Type: text/plain, Size: 735 bytes --] On 2019-02-16 3:08 p.m., Andrei Borzenkov wrote: > > File size of "real" server.jar is not exact multiple of btrfs block size > (4096) which causes the last extent to be "incomplete". btrfs won't > clone extent in this case. You can trivially reproduce it by using the > same size in the first case (although exact extent distribution may be > different of course, in the worst case you get single extent which is > sent in full): > > > > dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625 > Thank you. I thought I had tested one where I changed the file size, (by altering the bs,) but I must have inadvertently created a file that still divided by 4096, (or it got split into a second extent). [-- Attachment #1.1.2: remi.vcf --] [-- Type: text/x-vcard, Size: 203 bytes --] begin:vcard fn:Remi Gauvin n:Gauvin;Remi org:Georgian Infotech adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada email;internet:remi@georgianit.com tel;work:226-256-1545 version:2.1 end:vcard [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-17 3:11 ` Remi Gauvin @ 2019-02-18 13:05 ` André Malm 2019-02-18 18:06 ` Chris Murphy 0 siblings, 1 reply; 30+ messages in thread From: André Malm @ 2019-02-18 13:05 UTC (permalink / raw) To: linux-btrfs So the takeaway from this is that btrfs send doesn't work this way, I can't copy a file with reflink=always and expect it to be diffed correctly as in a parent-child situation or? On 2019-02-17 04:11, Remi Gauvin wrote: > On 2019-02-16 3:08 p.m., Andrei Borzenkov wrote: > >> File size of "real" server.jar is not exact multiple of btrfs block size >> (4096) which causes the last extent to be "incomplete". btrfs won't >> clone extent in this case. You can trivially reproduce it by using the >> same size in the first case (although exact extent distribution may be >> different of course, in the worst case you get single extent which is >> sent in full): >> >> >> >> dd if=/dev/urandom of=A/dir/server.jar bs=1024 count=34625 >> > Thank you. I thought I had tested one where I changed the file size, > (by altering the bs,) but I must have inadvertently created a file that > still divided by 4096, (or it got split into a second extent). > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 13:05 ` André Malm @ 2019-02-18 18:06 ` Chris Murphy 2019-02-18 19:58 ` André Malm 0 siblings, 1 reply; 30+ messages in thread From: Chris Murphy @ 2019-02-18 18:06 UTC (permalink / raw) To: André Malm; +Cc: linux-btrfs On Mon, Feb 18, 2019 at 6:05 AM André Malm <admin@sheepa.org> wrote: > > So the takeaway from this is that btrfs send doesn't work this way, I > can't copy a file with reflink=always and expect it to be diffed > correctly as in a parent-child situation or? Sorry about the confusion, André. What you're doing isn't described or explained in existing documentation: man page, wiki, or list archive, it's only explained in this thread. So, all I can say is it's going to take some testing on your part and possibly documentation and example improvements, for this alternative usage. At the moment, based on Andrei's explanation and example, what you're trying to do is valid. He also explained why you're seeing different results in send stream output in your dd vs wget example: it's the files themselves not the application. "The last extent is incomplete which causes btrfs to not attempt cloning." -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 18:06 ` Chris Murphy @ 2019-02-18 19:58 ` André Malm 2019-02-18 20:59 ` Graham Cobb 2019-02-18 21:22 ` Chris Murphy 0 siblings, 2 replies; 30+ messages in thread From: André Malm @ 2019-02-18 19:58 UTC (permalink / raw) To: Chris Murphy; +Cc: linux-btrfs What causes the extent to be incomplete? And can I avoid it? I tried with my example but with wget with a larger file (1G) and the diff produces was only 60MB which indicates that it "mostly" works. Basically what I'm trying to achieve is having a "reference" / "master" btrfs subvolume where i add / edit / remove files continuously. From where i can cp --reflink=always some of those files to new "child" subvolumes as needed. As long as the files are still in the master subvolume I'll only send the diff of the data. The plan is to have this master subvolume across multiple remote machines (synced with rsync or perhaps btrfs send / receive) and being able to update / change it as needed but always only sending the diffs of the child subvolumes. Maybe this is a bit optimistic. On 2019-02-18 19:06, Chris Murphy wrote: > On Mon, Feb 18, 2019 at 6:05 AM André Malm <admin@sheepa.org> wrote: >> So the takeaway from this is that btrfs send doesn't work this way, I >> can't copy a file with reflink=always and expect it to be diffed >> correctly as in a parent-child situation or? > Sorry about the confusion, André. What you're doing isn't described or > explained in existing documentation: man page, wiki, or list archive, > it's only explained in this thread. So, all I can say is it's going to > take some testing on your part and possibly documentation and example > improvements, for this alternative usage. > > At the moment, based on Andrei's explanation and example, what you're > trying to do is valid. He also explained why you're seeing different > results in send stream output in your dd vs wget example: it's the > files themselves not the application. > > "The last extent is incomplete which causes btrfs to not attempt cloning." > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 19:58 ` André Malm @ 2019-02-18 20:59 ` Graham Cobb 2019-02-18 21:22 ` Chris Murphy 1 sibling, 0 replies; 30+ messages in thread From: Graham Cobb @ 2019-02-18 20:59 UTC (permalink / raw) To: André Malm; +Cc: linux-btrfs On 18/02/2019 19:58, André Malm wrote: > What causes the extent to be incomplete? And can I avoid it? Does it matter? I presume the send is working OK, it is just that it sends a little more data than it needs to. Or have you seen any data loss? Graham ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 19:58 ` André Malm 2019-02-18 20:59 ` Graham Cobb @ 2019-02-18 21:22 ` Chris Murphy 2019-02-18 21:36 ` André Malm 1 sibling, 1 reply; 30+ messages in thread From: Chris Murphy @ 2019-02-18 21:22 UTC (permalink / raw) To: André Malm; +Cc: Chris Murphy, linux-btrfs On Mon, Feb 18, 2019 at 12:58 PM André Malm <admin@sheepa.org> wrote: > Basically what I'm trying to achieve is having a "reference" / "master" > btrfs subvolume where i add / edit / remove files continuously. From > where i can cp --reflink=always some of those files to new "child" > subvolumes as needed. As long as the files are still in the master > subvolume I'll only send the diff of the data. The plan is to have this > master subvolume across multiple remote machines (synced with rsync or > perhaps btrfs send / receive) and being able to update / change it as > needed but always only sending the diffs of the child subvolumes. Maybe > this is a bit optimistic. I'm not sure what you get out of this method that depends on reflink rather than just making read only snapshots. Why don't you create subvolume A as the master, read-write. And once it's in the "master state" you want, just snapshot it: btrfs sub snap -r A A.20190218-master And now continue to make changes to A subvolume, and on whatever schedule you want: btrfs sub snap -r A A.20190218-1412 btrfs sub snap -r A A.20190218-1850 btrfs sub snap -r A A.20190219-0920 And now you can "compare" the difference to master at anytime: btrfs send -p A.20190218-master A.20190218-1850 -f output (or pipe to receive) *shrug* I'm just not understanding what your use case gains out of doing a refink copy to another subvolume rather than just make a read only snapshot. Is it that subvolume A must accept changes, but itself cannot be the "master"? And you're selectively reflink copying files from A to B such that B is the only "master"? That's fine but then I'd say you don't need any A snapshots, you need B snapshots where: btrfs sub snap -r B B.20190218-master btrfs sub snap -r B B.20190219-0920 btrfs send -p B.20190218-master B.20190219-0920 -f output (or pipe to receive) And so on... Anyway, my point isn't that you're doing it wrong. I'm jut not understanding the advantage of doing it the way you're doing it; and it should be clear by now that most people on the list aren't using 'btrfs send' to compute the difference between two otherwise unrelated subvolumes. Part of this *might* be because of how ZFS snapshots work, and Btrfs users maybe just adopted the same logic due in large part to more limitations on the ZFS implementation: snapshots are always read only, incremental send is only done between snapshots of the same dataset, datasets can't be deleted until all snapshots are deleted, there are no reflinks, and no doubt some things I'm forgetting. -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 21:22 ` Chris Murphy @ 2019-02-18 21:36 ` André Malm 2019-02-18 22:28 ` Chris Murphy 0 siblings, 1 reply; 30+ messages in thread From: André Malm @ 2019-02-18 21:36 UTC (permalink / raw) To: Chris Murphy; +Cc: linux-btrfs The reason why I'm using reflinks instead of snapshots is because the master subvolume is large and will contain hundreds of gigabytes worth of data. Now if I change / remove, say 10 GB worth of data from the master subvolume unrelated to the child subvolume I don't want those gigabytes sent down the wire with btrfs send as they are unrelated. Also I want to be able to after time, when required, add more data from the master subvolume to the child. Thats why I can't split the master subvolume into parts of related data. On 2019-02-18 22:22, Chris Murphy wrote: > On Mon, Feb 18, 2019 at 12:58 PM André Malm <admin@sheepa.org> wrote: > >> Basically what I'm trying to achieve is having a "reference" / "master" >> btrfs subvolume where i add / edit / remove files continuously. From >> where i can cp --reflink=always some of those files to new "child" >> subvolumes as needed. As long as the files are still in the master >> subvolume I'll only send the diff of the data. The plan is to have this >> master subvolume across multiple remote machines (synced with rsync or >> perhaps btrfs send / receive) and being able to update / change it as >> needed but always only sending the diffs of the child subvolumes. Maybe >> this is a bit optimistic. > I'm not sure what you get out of this method that depends on reflink > rather than just making read only snapshots. > > Why don't you create subvolume A as the master, read-write. And once > it's in the "master state" you want, just snapshot it: > > btrfs sub snap -r A A.20190218-master > > And now continue to make changes to A subvolume, and on whatever > schedule you want: > > btrfs sub snap -r A A.20190218-1412 > btrfs sub snap -r A A.20190218-1850 > btrfs sub snap -r A A.20190219-0920 > > And now you can "compare" the difference to master at anytime: > > btrfs send -p A.20190218-master A.20190218-1850 -f output (or pipe to receive) > > *shrug* I'm just not understanding what your use case gains out of > doing a refink copy to another subvolume rather than just make a read > only snapshot. > > Is it that subvolume A must accept changes, but itself cannot be the > "master"? And you're selectively reflink copying files from A to B > such that B is the only "master"? That's fine but then I'd say you > don't need any A snapshots, you need B snapshots where: > > btrfs sub snap -r B B.20190218-master > btrfs sub snap -r B B.20190219-0920 > btrfs send -p B.20190218-master B.20190219-0920 -f output (or pipe to receive) > > And so on... > > Anyway, my point isn't that you're doing it wrong. I'm jut not > understanding the advantage of doing it the way you're doing it; and > it should be clear by now that most people on the list aren't using > 'btrfs send' to compute the difference between two otherwise unrelated > subvolumes. > > Part of this *might* be because of how ZFS snapshots work, and Btrfs > users maybe just adopted the same logic due in large part to more > limitations on the ZFS implementation: snapshots are always read only, > incremental send is only done between snapshots of the same dataset, > datasets can't be deleted until all snapshots are deleted, there are > no reflinks, and no doubt some things I'm forgetting. > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 21:36 ` André Malm @ 2019-02-18 22:28 ` Chris Murphy 2019-02-18 22:58 ` André Malm 0 siblings, 1 reply; 30+ messages in thread From: Chris Murphy @ 2019-02-18 22:28 UTC (permalink / raw) To: André Malm; +Cc: Chris Murphy, linux-btrfs On Mon, Feb 18, 2019 at 2:36 PM André Malm <admin@sheepa.org> wrote: > > The reason why I'm using reflinks instead of snapshots is because the > master subvolume is large and will contain hundreds of gigabytes worth > of data. Now if I change / remove, say 10 GB worth of data from the > master subvolume unrelated to the child subvolume I don't want those > gigabytes sent down the wire with btrfs send as they are unrelated. Also > I want to be able to after time, when required, add more data from the > master subvolume to the child. Thats why I can't split the master > subvolume into parts of related data. OK, is it your plan to have both master and child on the destination Btrfs? So you plan to do a 'btrfs send' without -p option to send the entire master subvolume (snapshot) to destination first? And then you want to create the child efficiently using a difference between master and child? Yes I think that might work but I haven't tried it. But of course, master must already be on the destination. btrfs sub create master ##populate the master btrfs sub create childofmaster cp --reflink master/bunchoffiles childofmaster/ btrfs sub snap -r master master.20190218-initial btrfs sub snap -r childofmaster childofmaster.20190218-initial btrfs send master.20190218-initial | btrfs receive /destination/ btrfs send -p master.20190218-initial childofmaster.20190218-initial | btrfs receive /destination/ However your subsequent incremental send/receive has nothing to do with master anymore; you indicate above you don't intent to keep master up to date on the destination. Just child. In that case the incremental changes look like this: cp --reflink master/morefiles childofmaster/ btrfs sub snap -r childofmaster childofmaster.20190218T1905 btrfs send child -p childofmaster.20190218-initial childofmaster.20190218T1905 | btrfs receive /destination/ cp --reflink master/yetmorefiles childofmaster/ btrfs sub snap -r childofmaster childofmaster.20190219T1301 btrfs send child childofmaster.20190218T1905 childofmaster.20190218T1905 | btrfs receive /destination/ -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 22:28 ` Chris Murphy @ 2019-02-18 22:58 ` André Malm 2019-02-18 23:49 ` Chris Murphy 0 siblings, 1 reply; 30+ messages in thread From: André Malm @ 2019-02-18 22:58 UTC (permalink / raw) To: Chris Murphy; +Cc: linux-btrfs Ok, but I don't want to keep old snapshots of the child volumes. Only the latest and then diffing it in regards to the master. Would that be possible? On 2019-02-18 23:28, Chris Murphy wrote: > On Mon, Feb 18, 2019 at 2:36 PM André Malm <admin@sheepa.org> wrote: >> The reason why I'm using reflinks instead of snapshots is because the >> master subvolume is large and will contain hundreds of gigabytes worth >> of data. Now if I change / remove, say 10 GB worth of data from the >> master subvolume unrelated to the child subvolume I don't want those >> gigabytes sent down the wire with btrfs send as they are unrelated. Also >> I want to be able to after time, when required, add more data from the >> master subvolume to the child. Thats why I can't split the master >> subvolume into parts of related data. > OK, is it your plan to have both master and child on the destination > Btrfs? So you plan to do a 'btrfs send' without -p option to send the > entire master subvolume (snapshot) to destination first? And then you > want to create the child efficiently using a difference between master > and child? Yes I think that might work but I haven't tried it. But of > course, master must already be on the destination. > > btrfs sub create master > ##populate the master > btrfs sub create childofmaster > cp --reflink master/bunchoffiles childofmaster/ > btrfs sub snap -r master master.20190218-initial > btrfs sub snap -r childofmaster childofmaster.20190218-initial > btrfs send master.20190218-initial | btrfs receive /destination/ > btrfs send -p master.20190218-initial childofmaster.20190218-initial | > btrfs receive /destination/ > > However your subsequent incremental send/receive has nothing to do > with master anymore; you indicate above you don't intent to keep > master up to date on the destination. Just child. In that case the > incremental changes look like this: > > cp --reflink master/morefiles childofmaster/ > btrfs sub snap -r childofmaster childofmaster.20190218T1905 > btrfs send child -p childofmaster.20190218-initial > childofmaster.20190218T1905 | btrfs receive /destination/ > cp --reflink master/yetmorefiles childofmaster/ > btrfs sub snap -r childofmaster childofmaster.20190219T1301 > btrfs send child childofmaster.20190218T1905 > childofmaster.20190218T1905 | btrfs receive /destination/ > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 22:58 ` André Malm @ 2019-02-18 23:49 ` Chris Murphy 2019-02-18 23:58 ` André Malm 0 siblings, 1 reply; 30+ messages in thread From: Chris Murphy @ 2019-02-18 23:49 UTC (permalink / raw) To: André Malm; +Cc: Chris Murphy, linux-btrfs On Mon, Feb 18, 2019 at 3:58 PM André Malm <admin@sheepa.org> wrote: > > Ok, but I don't want to keep old snapshots of the child volumes. Only > the latest and then diffing it in regards to the master. Would that be > possible? In order to do an incremental send/receive you need to have the -p snapshot on both source and destination file systems. You can delete snapshots older than that. I still don't understand why you're diffing in regards to master. It's going to be inherently inefficient because you've said a large amount of the contents of master aren't important; but by using it as the -p reference snapshot, you're asking send to do a comparison that includes a lot of metadata you don't care about. It's more efficient to diff the incremental snapshots of the changing state of "childofmaster" subvolume. -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 23:49 ` Chris Murphy @ 2019-02-18 23:58 ` André Malm 2019-02-19 0:16 ` Chris Murphy 0 siblings, 1 reply; 30+ messages in thread From: André Malm @ 2019-02-18 23:58 UTC (permalink / raw) To: Chris Murphy; +Cc: linux-btrfs I assume i would have to use rsync (with --inplace possibly) to keep the master volume in sync between machines? Say for example I have a (large) file on master, on machine A, I cp reflink it to a child subvolume. I then send -p child subvolume to remote machine B (which already have the master volume). Then i change parts of the file on the master of machine A. I then rsync (?) the master volume so its the same across the machines. Can I then later send -p the child volume, either back to the original machine (A) or to a 3rd machine (C) given that the master volumes are synced? About the efficiency I'm not planning to remove large amounts of data that is used by child subvolumes (although some will be updated). But given the unpredictability of what files will be used by child subvolumes i might remove large unused amounts of data. On 2019-02-19 00:49, Chris Murphy wrote: > On Mon, Feb 18, 2019 at 3:58 PM André Malm <admin@sheepa.org> wrote: >> Ok, but I don't want to keep old snapshots of the child volumes. Only >> the latest and then diffing it in regards to the master. Would that be >> possible? > In order to do an incremental send/receive you need to have the -p > snapshot on both source and destination file systems. You can delete > snapshots older than that. > > I still don't understand why you're diffing in regards to master. It's > going to be inherently inefficient because you've said a large amount > of the contents of master aren't important; but by using it as the -p > reference snapshot, you're asking send to do a comparison that > includes a lot of metadata you don't care about. It's more efficient > to diff the incremental snapshots of the changing state of > "childofmaster" subvolume. > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-18 23:58 ` André Malm @ 2019-02-19 0:16 ` Chris Murphy 2019-02-19 0:17 ` Chris Murphy 2019-02-19 0:28 ` André Malm 0 siblings, 2 replies; 30+ messages in thread From: Chris Murphy @ 2019-02-19 0:16 UTC (permalink / raw) To: André Malm; +Cc: Chris Murphy, linux-btrfs On Mon, Feb 18, 2019 at 4:58 PM André Malm <admin@sheepa.org> wrote: > > I assume i would have to use rsync (with --inplace possibly) to keep the > master volume in sync between machines? Why? You previously said you didn't want to do that: " if I change / remove, say 10 GB worth of data from the master subvolume unrelated to the child subvolume I don't want those gigabytes sent down the wire with btrfs send as they are unrelated. " So instead of sending those changes with send you're going to send the changes with rsync which is even more inefficient? > Say for example I have a (large) file on master, on machine A, I cp > reflink it to a child subvolume. I then send -p child subvolume to > remote machine B (which already have the master volume). 1. The master snapshots must be identical snapshots on source and destination; it must have been replicated using send/receive or receive will complain. 2. I don't know what "send -p child subvolume to remove" means because -p option means two snapshots must be included and I don't know what two snapshots you're planning on using, so I can't answer the question. >Then i change > parts of the file on the master of machine A. I then rsync (?) the > master volume so its the same across the machines. Can I then later send > -p the child volume, either back to the original machine (A) or to a 3rd > machine (C) given that the master volumes are synced? rsync'd subvolumes across volumes aren't consistent identical by btrfs receive. I expect they can't be used for incremental send/receive. What you're trying to achieve, big picture, isn't clear. You're describing it in terms of send/receive and confusing it with rsync. I don't understand what rsync has to do with this. You realize if you merely move a large file within a subvolume, and then you using rsync to a remote system, it will delete the original file and then sync over the nework that same data to the new location? Rsync has no idea how to just move files around. Btrfs send-receive does. > About the efficiency I'm not planning to remove large amounts of data > that is used by child subvolumes (although some will be updated). But > given the unpredictability of what files will be used by child > subvolumes i might remove large unused amounts of data. OK? -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-19 0:16 ` Chris Murphy @ 2019-02-19 0:17 ` Chris Murphy 2019-02-19 0:28 ` André Malm 1 sibling, 0 replies; 30+ messages in thread From: Chris Murphy @ 2019-02-19 0:17 UTC (permalink / raw) To: Chris Murphy; +Cc: André Malm, linux-btrfs On Mon, Feb 18, 2019 at 5:16 PM Chris Murphy <lists@colorremedies.com> wrote: > rsync'd subvolumes across volumes aren't consistent identical by btrfs s/consistent/considered -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-19 0:16 ` Chris Murphy 2019-02-19 0:17 ` Chris Murphy @ 2019-02-19 0:28 ` André Malm 2019-02-19 3:54 ` Chris Murphy 1 sibling, 1 reply; 30+ messages in thread From: André Malm @ 2019-02-19 0:28 UTC (permalink / raw) To: Chris Murphy; +Cc: linux-btrfs Rsync is probably i bad idea yes. I could btrfs send -p the changed "new" master subvolume and then delete the old master subvolume and then reference the new master subvolume when transferring it later on i guess? I'll explain the problem I'm trying to solve abit better; Say i have a program that will run in multiple instances. The program requires a dataset of large files to run (say 20GB). The dataset will be updated over time, i.e parts of them changes. These changes should only apply to new instances for the program. The program will also generate new data (both new files and also changing data in the the shared dataset) that is unique to the instance of the child subvolume. Finally I need to transfer the program together with its generated data to another remote machine to continue it's processing there. What i want to achieve is avoid having to transfer the entire dataset when only small parts of it is changed by the program. I also want to avoid having to duplicate copies of the data on the remote machine. On 2019-02-19 01:16, Chris Murphy wrote: > On Mon, Feb 18, 2019 at 4:58 PM André Malm <admin@sheepa.org> wrote: >> I assume i would have to use rsync (with --inplace possibly) to keep the >> master volume in sync between machines? > Why? > > You previously said you didn't want to do that: > " if I change / remove, say 10 GB worth of data from the > master subvolume unrelated to the child subvolume I don't want those > gigabytes sent down the wire with btrfs send as they are unrelated. " > > So instead of sending those changes with send you're going to send the > changes with rsync which is even more inefficient? > >> Say for example I have a (large) file on master, on machine A, I cp >> reflink it to a child subvolume. I then send -p child subvolume to >> remote machine B (which already have the master volume). > 1. The master snapshots must be identical snapshots on source and > destination; it must have been replicated using send/receive or > receive will complain. > 2. I don't know what "send -p child subvolume to remove" means because > -p option means two snapshots must be included and I don't know what > two snapshots you're planning on using, so I can't answer the > question. > >> Then i change >> parts of the file on the master of machine A. I then rsync (?) the >> master volume so its the same across the machines. Can I then later send >> -p the child volume, either back to the original machine (A) or to a 3rd >> machine (C) given that the master volumes are synced? > rsync'd subvolumes across volumes aren't consistent identical by btrfs > receive. I expect they can't be used for incremental send/receive. > What you're trying to achieve, big picture, isn't clear. You're > describing it in terms of send/receive and confusing it with rsync. I > don't understand what rsync has to do with this. > > You realize if you merely move a large file within a subvolume, and > then you using rsync to a remote system, it will delete the original > file and then sync over the nework that same data to the new location? > Rsync has no idea how to just move files around. Btrfs send-receive > does. > >> About the efficiency I'm not planning to remove large amounts of data >> that is used by child subvolumes (although some will be updated). But >> given the unpredictability of what files will be used by child >> subvolumes i might remove large unused amounts of data. > OK? > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-19 0:28 ` André Malm @ 2019-02-19 3:54 ` Chris Murphy 2019-02-19 12:05 ` André Malm 0 siblings, 1 reply; 30+ messages in thread From: Chris Murphy @ 2019-02-19 3:54 UTC (permalink / raw) To: André Malm; +Cc: Chris Murphy, linux-btrfs On Mon, Feb 18, 2019 at 5:28 PM André Malm <admin@sheepa.org> wrote: > > Rsync is probably i bad idea yes. I could btrfs send -p the changed > "new" master subvolume and then delete the old master subvolume and then > reference the new master subvolume when transferring it later on i guess? I'm not sure how your application reacts to snapshots or reflinks, or how it updates its files. All of that needs to be tested to see what the incremental send size is, and if the resulting received snapshot contains files with the integrity your application expects, and so on. > > I'll explain the problem I'm trying to solve abit better; > > Say i have a program that will run in multiple instances. The program > requires a dataset of large files to run (say 20GB). The dataset will be > updated over time, i.e parts of them changes. These changes should only > apply to new instances for the program. The program will also generate > new data (both new files and also changing data in the the shared > dataset) that is unique to the instance of the child subvolume. Finally > I need to transfer the program together with its generated data to > another remote machine to continue it's processing there. What i want to > achieve is avoid having to transfer the entire dataset when only small > parts of it is changed by the program. I also want to avoid having to > duplicate copies of the data on the remote machine. Yep. Based on this description though, the only time I grok using 'btrfs send -p master.snap child.snap | btrfs receive /destination/' is for the initial transfer of child. Master must be already fully replicated. Now you can snapshot master and child on separate schedules to account for their different use case, and send their increments independent of each other. Or in fact maybe you'll realize you do have a use case for clone. Have you looked at GlusterFS or Ceph for this use case? I kinda wonder if there's any simplification to just having a clustered file system make all of the send/receive stuff go away, and you can ensure your data is replicated pretty much immediately, and is always available for all computers. *shrug* That's off topic but I'm curious if there are ways to simplify this for your use case. -- Chris Murphy ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Btrfs send with parent different size depending on source of files. 2019-02-19 3:54 ` Chris Murphy @ 2019-02-19 12:05 ` André Malm 0 siblings, 0 replies; 30+ messages in thread From: André Malm @ 2019-02-19 12:05 UTC (permalink / raw) To: Chris Murphy; +Cc: linux-btrfs Okay, I will indeed test the setup thoroughly. I have considered running a distributed filesystem such as Ceph but my concern is that it will be way to slow as disk IO speed is important. Anyways, thank you for your help! On 2019-02-19 04:54, Chris Murphy wrote: > On Mon, Feb 18, 2019 at 5:28 PM André Malm <admin@sheepa.org> wrote: >> Rsync is probably i bad idea yes. I could btrfs send -p the changed >> "new" master subvolume and then delete the old master subvolume and then >> reference the new master subvolume when transferring it later on i guess? > I'm not sure how your application reacts to snapshots or reflinks, or > how it updates its files. All of that needs to be tested to see what > the incremental send size is, and if the resulting received snapshot > contains files with the integrity your application expects, and so on. > >> I'll explain the problem I'm trying to solve abit better; >> >> Say i have a program that will run in multiple instances. The program >> requires a dataset of large files to run (say 20GB). The dataset will be >> updated over time, i.e parts of them changes. These changes should only >> apply to new instances for the program. The program will also generate >> new data (both new files and also changing data in the the shared >> dataset) that is unique to the instance of the child subvolume. Finally >> I need to transfer the program together with its generated data to >> another remote machine to continue it's processing there. What i want to >> achieve is avoid having to transfer the entire dataset when only small >> parts of it is changed by the program. I also want to avoid having to >> duplicate copies of the data on the remote machine. > Yep. Based on this description though, the only time I grok using > 'btrfs send -p master.snap child.snap | btrfs receive /destination/' > is for the initial transfer of child. Master must be already fully > replicated. Now you can snapshot master and child on separate > schedules to account for their different use case, and send their > increments independent of each other. Or in fact maybe you'll realize > you do have a use case for clone. > > Have you looked at GlusterFS or Ceph for this use case? I kinda wonder > if there's any simplification to just having a clustered file system > make all of the send/receive stuff go away, and you can ensure your > data is replicated pretty much immediately, and is always available > for all computers. *shrug* That's off topic but I'm curious if there > are ways to simplify this for your use case. > > > ^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2019-02-19 12:05 UTC | newest] Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-02-14 11:37 Btrfs send with parent different size depending on source of files André Malm 2019-02-14 22:37 ` Chris Murphy 2019-02-15 4:00 ` Remi Gauvin 2019-02-15 18:38 ` Chris Murphy 2019-02-15 18:56 ` Remi Gauvin 2019-02-16 20:10 ` Andrei Borzenkov 2019-02-15 17:45 ` Andrei Borzenkov 2019-02-15 19:11 ` Chris Murphy 2019-02-16 20:26 ` Andrei Borzenkov 2019-02-16 20:32 ` Andrei Borzenkov 2019-02-18 18:00 ` Chris Murphy 2019-02-15 19:29 ` Remi Gauvin 2019-02-15 19:41 ` Remi Gauvin 2019-02-16 20:08 ` Andrei Borzenkov 2019-02-17 3:11 ` Remi Gauvin 2019-02-18 13:05 ` André Malm 2019-02-18 18:06 ` Chris Murphy 2019-02-18 19:58 ` André Malm 2019-02-18 20:59 ` Graham Cobb 2019-02-18 21:22 ` Chris Murphy 2019-02-18 21:36 ` André Malm 2019-02-18 22:28 ` Chris Murphy 2019-02-18 22:58 ` André Malm 2019-02-18 23:49 ` Chris Murphy 2019-02-18 23:58 ` André Malm 2019-02-19 0:16 ` Chris Murphy 2019-02-19 0:17 ` Chris Murphy 2019-02-19 0:28 ` André Malm 2019-02-19 3:54 ` Chris Murphy 2019-02-19 12:05 ` André Malm
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).