received uuid not set btrfs send/receive

received uuid not set btrfs send/receive

[Anders Halman]

Hello,

I try to backup my laptop over an unreliable slow internet connection to 
a even slower Raspberry Pi.

To bootstrap the backup I used the following:

# local
btrfs send root.send.ro | pigz | split --verbose -d -b 1G
rsync -aHAXxv --numeric-ids --partial --progress -e "ssh -T -o 
Compression=no -x" x* remote-host:/mnt/backup/btrfs-backup/

# remote
cat x* > split.gz
pigz -d split.gz
btrfs receive -f split

worked nicely. But I don't understand why the "received uuid" on the 
remote site in blank.
I tried it locally with smaller volumes and it worked.

The 'split' file contains the correct uuid, but it is not set (remote).

remote$ btrfs receive --dump -f split | head
subvol          ./root.send.ro uuid=99a34963-3506-7e4c-a82d-93e337191684 
transid=1232187

local$ sudo btrfs sub show root.send.ro| grep -i uuid:
     UUID:             99a34963-3506-7e4c-a82d-93e337191684


Questions:

- Is there a way to set the "received uuid"?
- Is it a matter of btrfs-progs version difference?
- What whould be a better approach?


Thank you


----

# local

root@fos ~$ uname -a
Linux fos 5.9.16-200.fc33.x86_64 #1 SMP Mon Dec 21 14:08:22 UTC 2020 
x86_64 x86_64 x86_64 GNU/Linux

root@fos ~$   btrfs --version
btrfs-progs v5.9

root@fos ~$   btrfs fi show
Label: 'DATA'  uuid: b6e675b3-84e3-4869-b858-218c5f0ac5ad
     Total devices 1 FS bytes used 402.17GiB
     devid    1 size 464.27GiB used 414.06GiB path 
/dev/mapper/luks-e4e69cfa-faae-4af8-93f5-7b21b25ab4e6

root@fos ~$   btrfs fi df /btrfs-root/
Data, single: total=404.00GiB, used=397.80GiB
System, DUP: total=32.00MiB, used=64.00KiB
Metadata, DUP: total=5.00GiB, used=4.38GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# remote
root@pih:~# uname -a
Linux pih 5.4.72+ #1356 Thu Oct 22 13:56:00 BST 2020 armv6l GNU/Linux

root@pih:~#   btrfs --version
btrfs-progs v4.20.1

root@pih:~#   btrfs fi show
Label: 'DATA'  uuid: 6be1e09c-d1a5-469d-932b-a8d1c339afae
     Total devices 1 FS bytes used 377.57GiB
     devid    2 size 931.51GiB used 383.06GiB path 
/dev/mapper/luks_open_backup0

root@pih:~#   btrfs fi df /mnt/backup
Data, single: total=375.00GiB, used=374.25GiB
System, DUP: total=32.00MiB, used=64.00KiB
Metadata, DUP: total=4.00GiB, used=3.32GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


dmesg is empty for the time of import/btrfs receive.

Re: received uuid not set btrfs send/receive

[Hugo Mills]

On Sun, Jan 17, 2021 at 10:49:26AM -0800, Anders Halman wrote:
> Hello,
> 
> I try to backup my laptop over an unreliable slow internet connection to a
> even slower Raspberry Pi.
> 
> To bootstrap the backup I used the following:
> 
> # local
> btrfs send root.send.ro | pigz | split --verbose -d -b 1G
> rsync -aHAXxv --numeric-ids --partial --progress -e "ssh -T -o
> Compression=no -x" x* remote-host:/mnt/backup/btrfs-backup/
> 
> # remote
> cat x* > split.gz
> pigz -d split.gz
> btrfs receive -f split

> worked nicely. But I don't understand why the "received uuid" on the remote
> site in blank.

   Are you doing the receive as root?

   Hugo.

> I tried it locally with smaller volumes and it worked.
> 
> The 'split' file contains the correct uuid, but it is not set (remote).
> 
> remote$ btrfs receive --dump -f split | head
> subvol          ./root.send.ro uuid=99a34963-3506-7e4c-a82d-93e337191684
> transid=1232187
> 
> local$ sudo btrfs sub show root.send.ro| grep -i uuid:
>     UUID:             99a34963-3506-7e4c-a82d-93e337191684
> 
> 
> Questions:
> 
> - Is there a way to set the "received uuid"?
> - Is it a matter of btrfs-progs version difference?
> - What whould be a better approach?
> 
> 
> Thank you
> 
> 
> ----
> 
> # local
> 
> root@fos ~$ uname -a
> Linux fos 5.9.16-200.fc33.x86_64 #1 SMP Mon Dec 21 14:08:22 UTC 2020 x86_64
> x86_64 x86_64 GNU/Linux
> 
> root@fos ~$   btrfs --version
> btrfs-progs v5.9
> 
> root@fos ~$   btrfs fi show
> Label: 'DATA'  uuid: b6e675b3-84e3-4869-b858-218c5f0ac5ad
>     Total devices 1 FS bytes used 402.17GiB
>     devid    1 size 464.27GiB used 414.06GiB path
> /dev/mapper/luks-e4e69cfa-faae-4af8-93f5-7b21b25ab4e6
> 
> root@fos ~$   btrfs fi df /btrfs-root/
> Data, single: total=404.00GiB, used=397.80GiB
> System, DUP: total=32.00MiB, used=64.00KiB
> Metadata, DUP: total=5.00GiB, used=4.38GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 
> # remote
> root@pih:~# uname -a
> Linux pih 5.4.72+ #1356 Thu Oct 22 13:56:00 BST 2020 armv6l GNU/Linux
> 
> root@pih:~#   btrfs --version
> btrfs-progs v4.20.1
> 
> root@pih:~#   btrfs fi show
> Label: 'DATA'  uuid: 6be1e09c-d1a5-469d-932b-a8d1c339afae
>     Total devices 1 FS bytes used 377.57GiB
>     devid    2 size 931.51GiB used 383.06GiB path
> /dev/mapper/luks_open_backup0
> 
> root@pih:~#   btrfs fi df /mnt/backup
> Data, single: total=375.00GiB, used=374.25GiB
> System, DUP: total=32.00MiB, used=64.00KiB
> Metadata, DUP: total=4.00GiB, used=3.32GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 
> dmesg is empty for the time of import/btrfs receive.

-- 
Hugo Mills             | If it ain't broke, hit it again.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                                  Foon

Re: received uuid not set btrfs send/receive

[Chris Murphy]

On Sun, Jan 17, 2021 at 11:51 AM Anders Halman <anders.halman@gmail.com> wrote:
>
> Hello,
>
> I try to backup my laptop over an unreliable slow internet connection to
> a even slower Raspberry Pi.
>
> To bootstrap the backup I used the following:
>
> # local
> btrfs send root.send.ro | pigz | split --verbose -d -b 1G
> rsync -aHAXxv --numeric-ids --partial --progress -e "ssh -T -o
> Compression=no -x" x* remote-host:/mnt/backup/btrfs-backup/
>
> # remote
> cat x* > split.gz
> pigz -d split.gz
> btrfs receive -f split
>
> worked nicely. But I don't understand why the "received uuid" on the
> remote site in blank.
> I tried it locally with smaller volumes and it worked.

I suggest using -v or -vv on the receive side to dig into why the
receive is failing. Setting the received uuid is one of the last
things performed on receive, so if it's not set it suggests the
receive isn't finished.

-- 
Chris Murphy

Re: received uuid not set btrfs send/receive

[Andrei Borzenkov]

17.01.2021 21:49, Anders Halman пишет:
> Hello,
> 
> I try to backup my laptop over an unreliable slow internet connection to
> a even slower Raspberry Pi.
> 
> To bootstrap the backup I used the following:
> 
> # local
> btrfs send root.send.ro | pigz | split --verbose -d -b 1G
> rsync -aHAXxv --numeric-ids --partial --progress -e "ssh -T -o
> Compression=no -x" x* remote-host:/mnt/backup/btrfs-backup/
> 
> # remote
> cat x* > split.gz
> pigz -d split.gz
> btrfs receive -f split
> 

This command is missing argument.

> worked nicely.

It is impossible, you did not tell "btrfs receive" where to store
incoming data.

bor@bor-Latitude-E5450:~/src/btrfs-progs$ btrfs receive
btrfs receive: exactly 1 argument expected, 0 given

> But I don't understand why the "received uuid" on the
> remote site in blank.

Where is it blank? Show commands you used to check it.

> I tried it locally with smaller volumes and it worked.
> 
> The 'split' file contains the correct uuid, but it is not set (remote).
> 
> remote$ btrfs receive --dump -f split | head
> subvol          ./root.send.ro uuid=99a34963-3506-7e4c-a82d-93e337191684
> transid=1232187
> 
> local$ sudo btrfs sub show root.send.ro| grep -i uuid:
>     UUID:             99a34963-3506-7e4c-a82d-93e337191684
> 
> 
> Questions:
> 
> - Is there a way to set the "received uuid"?
> - Is it a matter of btrfs-progs version difference?
> - What whould be a better approach?
> 
> 
> Thank you
> 
> 
> ----
> 
> # local
> 
> root@fos ~$ uname -a
> Linux fos 5.9.16-200.fc33.x86_64 #1 SMP Mon Dec 21 14:08:22 UTC 2020
> x86_64 x86_64 x86_64 GNU/Linux
> 
> root@fos ~$   btrfs --version
> btrfs-progs v5.9
> 
> root@fos ~$   btrfs fi show
> Label: 'DATA'  uuid: b6e675b3-84e3-4869-b858-218c5f0ac5ad
>     Total devices 1 FS bytes used 402.17GiB
>     devid    1 size 464.27GiB used 414.06GiB path
> /dev/mapper/luks-e4e69cfa-faae-4af8-93f5-7b21b25ab4e6
> 
> root@fos ~$   btrfs fi df /btrfs-root/
> Data, single: total=404.00GiB, used=397.80GiB
> System, DUP: total=32.00MiB, used=64.00KiB
> Metadata, DUP: total=5.00GiB, used=4.38GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 
> # remote
> root@pih:~# uname -a
> Linux pih 5.4.72+ #1356 Thu Oct 22 13:56:00 BST 2020 armv6l GNU/Linux
> 
> root@pih:~#   btrfs --version
> btrfs-progs v4.20.1
> 
> root@pih:~#   btrfs fi show
> Label: 'DATA'  uuid: 6be1e09c-d1a5-469d-932b-a8d1c339afae
>     Total devices 1 FS bytes used 377.57GiB
>     devid    2 size 931.51GiB used 383.06GiB path
> /dev/mapper/luks_open_backup0
> 
> root@pih:~#   btrfs fi df /mnt/backup
> Data, single: total=375.00GiB, used=374.25GiB
> System, DUP: total=32.00MiB, used=64.00KiB
> Metadata, DUP: total=4.00GiB, used=3.32GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 
> dmesg is empty for the time of import/btrfs receive.

Re: received uuid not set btrfs send/receive

[Anders Halman]

Sorry for answering so late. I compiled a more recent (v5.9) version of 
btrfs-progs first.
The unpacking/receiving took about 2 days on the raspberry.

$ nohup btrfs receive -v -f splitter . &
$ tail -n1 nohup.out
ERROR: short read from stream: expected 49233 read 44997

thanks to your -v suggestions I got a little bit more information.

first I figured the files on the local and remote host are not the same, 
by md5sum.

So I will solve this issue first. Is there any recommended way to 
transfer btrfs subvolumes over an unstable connection?


Am 17.01.21 um 13:07 schrieb Chris Murphy:
> On Sun, Jan 17, 2021 at 11:51 AM Anders Halman <anders.halman@gmail.com> wrote:
>> Hello,
>>
>> I try to backup my laptop over an unreliable slow internet connection to
>> a even slower Raspberry Pi.
>>
>> To bootstrap the backup I used the following:
>>
>> # local
>> btrfs send root.send.ro | pigz | split --verbose -d -b 1G
>> rsync -aHAXxv --numeric-ids --partial --progress -e "ssh -T -o
>> Compression=no -x" x* remote-host:/mnt/backup/btrfs-backup/
>>
>> # remote
>> cat x* > split.gz
>> pigz -d split.gz
>> btrfs receive -f split
>>
>> worked nicely. But I don't understand why the "received uuid" on the
>> remote site in blank.
>> I tried it locally with smaller volumes and it worked.
> I suggest using -v or -vv on the receive side to dig into why the
> receive is failing. Setting the received uuid is one of the last
> things performed on receive, so if it's not set it suggests the
> receive isn't finished.
>

Re: [SOLVED] received uuid not set btrfs send/receive

[Anders Halman]

The problem was a corrupt "btrfs send file" or to be more specific, the 
file got corrupt somewhere on the line of transport
To recap the problem for further reference:

check the import (btrfs receive) with the -v option like so:

$ nohup btrfs receive -v -f root.receive.file . &

the import is successful, thus the received uuid is set when you get 
something like ....

$ tail -n1 nohup.out
BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=99a34963-3506-7e4c-a82d-93e337191684, 
stransid=1232187

... after "btrfs receive" is done.

make sure to double check the file size and checksum (md5sum) of "btrfs 
send file" and "btrfs receive file".

I don't know exactly where the corruption happened, but in the second 
attempt I successful combined the import like so:

$ cat x* | pigz -d > root.receive.file


Thanks for the support


Am 17.01.21 um 13:07 schrieb Chris Murphy:
> On Sun, Jan 17, 2021 at 11:51 AM Anders Halman <anders.halman@gmail.com> wrote:
>> Hello,
>>
>> I try to backup my laptop over an unreliable slow internet connection to
>> a even slower Raspberry Pi.
>>
>> To bootstrap the backup I used the following:
>>
>> # local
>> btrfs send root.send.ro | pigz | split --verbose -d -b 1G
>> rsync -aHAXxv --numeric-ids --partial --progress -e "ssh -T -o
>> Compression=no -x" x* remote-host:/mnt/backup/btrfs-backup/
>>
>> # remote
>> cat x* > split.gz
>> pigz -d split.gz
>> btrfs receive -f split
>>
>> worked nicely. But I don't understand why the "received uuid" on the
>> remote site in blank.
>> I tried it locally with smaller volumes and it worked.
> I suggest using -v or -vv on the receive side to dig into why the
> receive is failing. Setting the received uuid is one of the last
> things performed on receive, so if it's not set it suggests the
> receive isn't finished.
>