Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
From: Eric Mesa <eric@ericmesa.com>
To: Chris Murphy <lists@colorremedies.com>
Cc: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: Re: Issues with btrfs send/receive with parents
Date: Mon, 10 Jun 2019 23:47:11 -0400
Message-ID: <2339083.MuzE7HVQJs@supermario.mushroomkingdom> (raw)
In-Reply-To: <CAJCQCtSLVxVtArLHrx0XD6J1oWOowqAnOPzeWVx3J25O7vxFQw@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 14330 bytes --]

On Monday, June 10, 2019 9:10:17 PM EDT Chris Murphy wrote:
> On Mon, Jun 10, 2019 at 6:03 PM Eric Mesa <eric@ericmesa.com> wrote:
> > When I did the btrfs send / receive for SSDHome it took 2.5 days to send
> > ~500GiB over a 1GBps cable to the 3GBps drive in the server. It also had
> > the error:
> > ERROR: failed to clone extents to ermesa/.cache/krunner/
> > bookmarkrunnerfirefoxfavdbfile.sqlite: Invalid argument
> 
> While there are distinct send and receive errors possible, I'm not
> familiar with recognizing them. You can get a better idea what the
> problem is with -vv or -vvv to get a more verbose error on the side
> that's having the problem. My guess is this is a send error message.
> 
> > Let's say that snapshot A is a snapshot sent to the server without -p. It
> > sends the entire 500GB for 18 hours.
> > 
> > Then I do snapshot B. I send it with -p - takes 15 minutes or so depending
> > on how much data I've added.
> > 
> > Then I do snapshot C - and here I always get an error.
> 
> It's most useful if you show exact commands because actually it's not
> always obvious to everyone what the logic should be and the error
> handling doesn't always stop a user from doing something that doesn't
> make a lot of sense. We need to know the name of the rw subvolume; the
> command to snapshot it; the full send/receive command for that first
> snapshot; the command for a subsequent snapshot; and the command to
> incrementally send/receive it.
> 

OK, here's an example from when it failed, although I didn't save the error - 
I'm just looking at my history for a time it failed:

#btrfs sub snap -r /home/ /home/.snapshots/2019-06-08-1437
#cd /home/.snapshots/
#btrfs send -p /home/.snapshots/2019-06-05-postdefrag/ 2019-06-08-1437/ | ssh 
#root@tanukimario btrfs receive /media/backups/backups1/supermario-home
#btrfs sub snap -r /home/ /home/.snapshots/2019-06-09-1415
#btrfs send -p 2019-06-08-1437/ 2019-06-09-1415/ | ssh root@tanukimario btrfs 
receive /media/backups/backups1/supermario-home



> > And it always is something like:
> > 
> > ERROR: link ermesa/.mozilla/firefox/n35gu0fb.default/bookmarkbackups/
> > bookmarks-2019-06-09_679_I1bs5PtgsPwtyXvcvcRdSg==.jsonlz4 ->
> > ermesa/.mozilla/ firefox/n35gu0fb.default/bookmarkbackups/
> > bookmarks-2019-06-08_679_I1bs5PtgsPwtyXvcvcRdSg==.jsonlz4 failed: No such
> > file or directory
> > 
> > 
> > 
> > It always involves either .cache or .mozilla - the types of files that are
> > constantly changing.
> > 
> > It doesn't matter if I do a defrag before snapshot C followed by the sync
> > command. It seems that for SSDHome I can only do one full snap send and
> > then one parent send.
> 
> I don't actually know the status of snapshot aware defragmentation. It
> wasn't there, then it was there, then there were problems, and I think
> it was pulled rather than fixed. But I don't remember really. I also
> don't know if there's a difference between manual defragging and
> autodefrag, because I don't use either one. I do use reflinks. And I
> have done deduplication. And I don't have any send/receive failures. I
> do sometimes see slow sections of send/receive.
> 
> > Again, so far it seems to be working fine with the other drives which
> > seems to suggest to me that it's maybe not the version of my kernel or
> > btrfs progs or anything else.
> 
> Do you remember the mkfs command for this file system? Or also helpful would
> be:
> 
> # btrfs insp dump-s -f /dev/X  ## for both send and receive side file
> system (only one device from each Btrfs volume is needed), this will
> give us an idea what the mkfs options were including feature flags.
> 

for the sender:
btrfs insp dump-s -f /dev/sdb1 
superblock: bytenr=65536, device=/dev/sdb1
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0xa7282fe2 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    1080b060-4f1d-4b72-8544-ada43ec3cb54
metadata_uuid           1080b060-4f1d-4b72-8544-ada43ec3cb54
label                   SSDHome
generation              3034047
root                    373805891584
sys_array_size          194
chunk_root_generation   3008316
root_level              1
chunk_root              1010940559360
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             1000203091968
bytes_used              648933253120
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x161
                        ( MIXED_BACKREF |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
cache_generation        3034047
uuid_tree_generation    3034047
dev_item.uuid           61b1e688-5aad-4dff-95b7-2500c08aefb4
dev_item.fsid           1080b060-4f1d-4b72-8544-ada43ec3cb54 [match]
dev_item.type           0
dev_item.total_bytes    1000203091968
dev_item.bytes_used     864416694272
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1048576)
                length 4194304 owner 2 stripe_len 65536 type SYSTEM
                io_align 4096 io_width 4096 sector_size 4096
                num_stripes 1 sub_stripes 0
                        stripe 0 devid 1 offset 1048576
                        dev_uuid 61b1e688-5aad-4dff-95b7-2500c08aefb4
        item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 1010940510208)
                length 33554432 owner 2 stripe_len 65536 type SYSTEM
                io_align 65536 io_width 65536 sector_size 4096
                num_stripes 1 sub_stripes 1
                        stripe 0 devid 1 offset 710839107584
                        dev_uuid 61b1e688-5aad-4dff-95b7-2500c08aefb4
backup_roots[4]:
        backup 0:
                backup_tree_root:       373802975232    gen: 3034046    level: 
1
                backup_chunk_root:      1010940559360   gen: 3008316    level: 
1
                backup_extent_root:     373801369600    gen: 3034046    level: 
2
                backup_fs_root:         982774349824    gen: 2278090    level: 
0
                backup_dev_root:        38486016        gen: 3033103    level: 
1
                backup_csum_root:       373801058304    gen: 3034046    level: 
2
                backup_total_bytes:     1000203091968
                backup_bytes_used:      648933277696
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       373805891584    gen: 3034047    level: 
1
                backup_chunk_root:      1010940559360   gen: 3008316    level: 
1
                backup_extent_root:     373805121536    gen: 3034047    level: 
2
                backup_fs_root:         982774349824    gen: 2278090    level: 
0
                backup_dev_root:        38486016        gen: 3033103    level: 
1
                backup_csum_root:       373804859392    gen: 3034047    level: 
2
                backup_total_bytes:     1000203091968
                backup_bytes_used:      648933253120
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       373804662784    gen: 3034044    level: 
1
                backup_chunk_root:      1010940559360   gen: 3008316    level: 
1
                backup_extent_root:     373803008000    gen: 3034044    level: 
2
                backup_fs_root:         982774349824    gen: 2278090    level: 
0
                backup_dev_root:        38486016        gen: 3033103    level: 
1
                backup_csum_root:       373796159488    gen: 3034044    level: 
2
                backup_total_bytes:     1000203091968
                backup_bytes_used:      648933285888
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       373800550400    gen: 3034045    level: 
1
                backup_chunk_root:      1010940559360   gen: 3008316    level: 
1
                backup_extent_root:     373800271872    gen: 3034045    level: 
2
                backup_fs_root:         982774349824    gen: 2278090    level: 
0
                backup_dev_root:        38486016        gen: 3033103    level: 
1
                backup_csum_root:       373799960576    gen: 3034045    level: 
2
                backup_total_bytes:     1000203091968
                backup_bytes_used:      648933298176
                backup_num_devices:     1


for receiver:
superblock: bytenr=65536, device=/dev/sdc1
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0x47edc62b [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    e1bc05b0-6fad-4f1a-9586-26a461b65d57
metadata_uuid           e1bc05b0-6fad-4f1a-9586-26a461b65d57
label                   Backups1
generation              15013
root                    368023306240
sys_array_size          129
chunk_root_generation   14915
root_level              1
chunk_root              22020096
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             4000785104896
bytes_used              2350642929664
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x161
                        ( MIXED_BACKREF |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
cache_generation        15013
uuid_tree_generation    15013
dev_item.uuid           6640d884-bcf6-4f0c-8d97-a31d72133a76
dev_item.fsid           e1bc05b0-6fad-4f1a-9586-26a461b65d57 [match]
dev_item.type           0
dev_item.total_bytes    4000785104896
dev_item.bytes_used     2509351419904
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
                length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
                io_align 65536 io_width 65536 sector_size 4096
                num_stripes 2 sub_stripes 0
                        stripe 0 devid 1 offset 22020096
                        dev_uuid 6640d884-bcf6-4f0c-8d97-a31d72133a76
                        stripe 1 devid 1 offset 30408704
                        dev_uuid 6640d884-bcf6-4f0c-8d97-a31d72133a76
backup_roots[4]:
        backup 0:
                backup_tree_root:       368023109632    gen: 15010      level: 
1
                backup_chunk_root:      22020096        gen: 14915      level: 
1
                backup_extent_root:     368023158784    gen: 15010      level: 
2
                backup_fs_root:         30621696        gen: 7  level: 0
                backup_dev_root:        499089408       gen: 14915      level: 
1
                backup_csum_root:       368016621568    gen: 15010      level: 
3
                backup_total_bytes:     4000785104896
                backup_bytes_used:      2350286446592
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       368024518656    gen: 15011      level: 
1
                backup_chunk_root:      22020096        gen: 14915      level: 
1
                backup_extent_root:     368024551424    gen: 15011      level: 
2
                backup_fs_root:         30621696        gen: 7  level: 0
                backup_dev_root:        499089408       gen: 14915      level: 
1
                backup_csum_root:       368024682496    gen: 15011      level: 
3
                backup_total_bytes:     4000785104896

                backup_total_bytes:     4000785104896
                backup_bytes_used:      2350642929664
                backup_num_devices:     1



> > And dmesg.log is attached
> 
> [    6.949347] BTRFS info (device sdb1): enabling auto defrag
> 
> Could be related. And then also
> 
> [    9.906695] usb-storage 8-1.3:1.0: USB Mass Storage device detected
> [    9.907006] scsi host7: usb-storage 8-1.3:1.0
> [   10.950446] scsi 7:0:0:0: Direct-Access     B&N      NOOK
>   0322 PQ: 0 ANSI: 2
> [   10.951110] sd 7:0:0:0: Attached scsi generic sg7 type 0
> [   10.951161] sd 7:0:0:0: Power-on or device reset occurred
> [   10.952880] sd 7:0:0:0: [sdg] Attached SCSI removable disk
> snip
> [  267.794434] usb 9-1.1: reset SuperSpeed Gen 1 USB device number 3
> using xhci_hcd
> [  272.838054] usb 9-1.1: device descriptor read/8, error -110
> [  272.941832] usb 9-1.1: reset SuperSpeed Gen 1 USB device number 3
> using xhci_hcd
> [  277.958049] usb 9-1.1: device descriptor read/8, error -110
> [  278.236339] usb 9-1.1: reset SuperSpeed Gen 1 USB device number 3
> using xhci_hcd
> 
> 
> USB enclosed drives can be troublesome with any file system, expressly
> because of these seemingly random reset that happen. I had the same
> thing in an early form of my setup, and it did cause me problems that
> Btrfs worked around. But I considered it untenable and fixed it with a
> good quality self-powered USB hub (don't rely on bus power), or
> perhaps more specifically one that comes with a high amp power
> adapter. It needs to be able to drive all the drives in their
> read/write usage, which for laptop drives is ~0.35A each.
> 
> You really shouldn't be getting link resets like the above, even
> though I suspect it's unrelated to the current problem report.

the drives in question aren't USB. That's for a USB3 hub that I use for my SD 
cards.
-- 
--
Eric Mesa

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  parent reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-10 23:56 Eric Mesa
2019-06-11  1:10 ` Chris Murphy
2019-06-11  3:39   ` Andrei Borzenkov
2019-06-11  4:19     ` Eric Mesa
2019-06-12  5:43       ` Andrei Borzenkov
2019-06-12 23:03         ` Eric Mesa
2019-06-12 23:11           ` Eric Mesa
2019-06-13  6:26           ` Andrei Borzenkov
2019-06-13 21:22             ` Eric Mesa
2019-06-14  3:55               ` Chris Murphy
2019-07-14 20:59                 ` Eric Mesa
2019-06-18  4:15               ` Andrei Borzenkov
2019-06-11  3:47   ` Eric Mesa [this message]
2019-06-11  1:11 ` Remi Gauvin

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2339083.MuzE7HVQJs@supermario.mushroomkingdom \
    --to=eric@ericmesa.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=lists@colorremedies.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox