Access Beyond End of Device & Input/Output Errors

Access Beyond End of Device & Input/Output Errors

[Justin Brown]

Hello,

I've run into a strange problem that I haven't seen before, and I need
some help. I started getting generic "input/output" errors on a couple
of files, and when I looked deeper, the kernel logs are full of
messages like:

    sd 5:0:0:0: [sdf] tag#29 access beyond end of device

I've never seen anything like this before with any FS, so I figured it
was worth asking before I consider running the standard btrfs tools.
(I briefly started a scrub, but it was going crazy with uncorrectable
errors, so I cancelled it.)

Here's my system info:

Fedora 32, kernel 5.7.7-200.fc32.x86_64
btrfs-progs v5.7

/etc/fstab entry:
LABEL=media /var/media btrfs subvol=media,discard 0 2

btrfs fi show /var/media/
Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
Total devices 6 FS bytes used 4.68TiB
devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1

btrfs fi df /var/media/
Data, RAID5: total=4.69TiB, used=4.68TiB
System, RAID1C3: total=32.00MiB, used=304.00KiB
Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

I can only mount -o degraded now. Here are the logs when mounting:

Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
degraded /dev/sda1 /var/media/
Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
access beyond end of device
Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
class 0
Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
sdf1, logical block 16, async page read
Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
sde1): allowing degraded mounts
Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
sde1): disk space caching is enabled
Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
gen 0

It seems like only relatively recently written files are encountering
I/O errors. If I `cat` one of the problematic files when the FS is
mounted normally, I see a ton of this:

Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
access beyond end of device
Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
access beyond end of device

Now that I'm remounted in -o degraded, I'm getting more comprehensible
warnings, but it still results in I/O read failures:

Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
expected csum 0xbe3f80a4 mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
expected csum 0x9c36a6b4 mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
expected csum 0x44d30ca2 mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
expected csum 0xc0f08acc mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
expected csum 0xcb11db59 mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
expected csum 0x8a4ee0aa mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
expected csum 0xdfb79e85 mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
expected csum 0xc14921a0 mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
expected csum 0xf2fe8774 mirror 2
Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
expected csum 0xae1cafd6 mirror 2

Why trying to research this problem, I came across a Github issue
https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
prevent access beyond device boundary). I do use the discard mount
option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
drives with the 2x8TB drives about 1 month ago, which involved a
conversion to -d raid5 -m raid1c3, which I suppose could hit the same
code paths that resize2fs would?

Any advice on how to proceed would be greatly appreciated.

Thanks,
Justin

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 6999 bytes --]



On 2020/8/1 下午2:51, Justin Brown wrote:
> Hello,
> 
> I've run into a strange problem that I haven't seen before, and I need
> some help. I started getting generic "input/output" errors on a couple
> of files, and when I looked deeper, the kernel logs are full of
> messages like:
> 
>     sd 5:0:0:0: [sdf] tag#29 access beyond end of device

We had a new fix for trim. But according to your kernel message, it
doesn't look like the case.

(No obvious tag showing it's trim/discard)

> 
> I've never seen anything like this before with any FS, so I figured it
> was worth asking before I consider running the standard btrfs tools.
> (I briefly started a scrub, but it was going crazy with uncorrectable
> errors, so I cancelled it.)
> 
> Here's my system info:
> 
> Fedora 32, kernel 5.7.7-200.fc32.x86_64
> btrfs-progs v5.7
> 
> /etc/fstab entry:
> LABEL=media /var/media btrfs subvol=media,discard 0 2
> 
> btrfs fi show /var/media/
> Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
> Total devices 6 FS bytes used 4.68TiB
> devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
> devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
> devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
> devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
> devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
> devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1
> 
> btrfs fi df /var/media/
> Data, RAID5: total=4.69TiB, used=4.68TiB
> System, RAID1C3: total=32.00MiB, used=304.00KiB
> Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> I can only mount -o degraded now. Here are the logs when mounting:
> 
> Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
> ; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
> degraded /dev/sda1 /var/media/
> Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> access beyond end of device
> Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
> error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
> class 0

OK, it's read, not DISCARD, thus a completely different problem.


> Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
> sdf1, logical block 16, async page read
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> sde1): allowing degraded mounts
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> sde1): disk space caching is enabled
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
> gen 0
> 
> It seems like only relatively recently written files are encountering
> I/O errors. If I `cat` one of the problematic files when the FS is
> mounted normally, I see a ton of this:
> 
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
> access beyond end of device
> 
> Now that I'm remounted in -o degraded, I'm getting more comprehensible
> warnings, but it still results in I/O read failures:
> 
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
> expected csum 0xbe3f80a4 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
> expected csum 0x9c36a6b4 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
> expected csum 0x44d30ca2 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
> expected csum 0xc0f08acc mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
> expected csum 0xcb11db59 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
> expected csum 0x8a4ee0aa mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
> expected csum 0xdfb79e85 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
> expected csum 0xc14921a0 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
> expected csum 0xf2fe8774 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
> expected csum 0xae1cafd6 mirror 2
> 
> Why trying to research this problem, I came across a Github issue
> https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
> from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
> prevent access beyond device boundary). I do use the discard mount
> option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
> drives with the 2x8TB drives about 1 month ago, which involved a
> conversion to -d raid5 -m raid1c3, which I suppose could hit the same
> code paths that resize2fs would?

The problem doesn't look like a trim one, but more likely some device
boundary bug.

Would you please provide the following info?
- btrfs ins dump-tree -t chunk /dev/sde1
  This contains the device info and chunk tree dump. Doesn't contain
  any confidential info.
  We can use this info to determine if there is some chunk really beyond
  device boundary.
  I guess some chunks are already beyond device boundary by somehow.

Thanks,
Qu

> 
> Any advice on how to proceed would be greatly appreciated.
> 
> Thanks,
> Justin
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 7331 bytes --]



On 2020/8/1 下午2:58, Qu Wenruo wrote:
> 
> 
> On 2020/8/1 下午2:51, Justin Brown wrote:
>> Hello,
>>
>> I've run into a strange problem that I haven't seen before, and I need
>> some help. I started getting generic "input/output" errors on a couple
>> of files, and when I looked deeper, the kernel logs are full of
>> messages like:
>>
>>     sd 5:0:0:0: [sdf] tag#29 access beyond end of device
> 
> We had a new fix for trim. But according to your kernel message, it
> doesn't look like the case.
> 
> (No obvious tag showing it's trim/discard)
> 
>>
>> I've never seen anything like this before with any FS, so I figured it
>> was worth asking before I consider running the standard btrfs tools.
>> (I briefly started a scrub, but it was going crazy with uncorrectable
>> errors, so I cancelled it.)
>>
>> Here's my system info:
>>
>> Fedora 32, kernel 5.7.7-200.fc32.x86_64
>> btrfs-progs v5.7
>>
>> /etc/fstab entry:
>> LABEL=media /var/media btrfs subvol=media,discard 0 2
>>
>> btrfs fi show /var/media/
>> Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
>> Total devices 6 FS bytes used 4.68TiB
>> devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
>> devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
>> devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
>> devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
>> devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
>> devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1
>>
>> btrfs fi df /var/media/
>> Data, RAID5: total=4.69TiB, used=4.68TiB
>> System, RAID1C3: total=32.00MiB, used=304.00KiB
>> Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> I can only mount -o degraded now. Here are the logs when mounting:
>>
>> Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
>> ; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
>> degraded /dev/sda1 /var/media/
>> Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
>> access beyond end of device
>> Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
>> error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
>> class 0
> 
> OK, it's read, not DISCARD, thus a completely different problem.
> 
> 
>> Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
>> sdf1, logical block 16, async page read
>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>> sde1): allowing degraded mounts
>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>> sde1): disk space caching is enabled
>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>> sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
>> gen 0
>>
>> It seems like only relatively recently written files are encountering
>> I/O errors. If I `cat` one of the problematic files when the FS is
>> mounted normally, I see a ton of this:
>>
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
>> access beyond end of device
>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
>> access beyond end of device
>>
>> Now that I'm remounted in -o degraded, I'm getting more comprehensible
>> warnings, but it still results in I/O read failures:
>>
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
>> expected csum 0xbe3f80a4 mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
>> expected csum 0x9c36a6b4 mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
>> expected csum 0x44d30ca2 mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
>> expected csum 0xc0f08acc mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
>> expected csum 0xcb11db59 mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
>> expected csum 0x8a4ee0aa mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
>> expected csum 0xdfb79e85 mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
>> expected csum 0xc14921a0 mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
>> expected csum 0xf2fe8774 mirror 2
>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>> sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
>> expected csum 0xae1cafd6 mirror 2
>>
>> Why trying to research this problem, I came across a Github issue
>> https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
>> from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
>> prevent access beyond device boundary). I do use the discard mount
>> option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
>> drives with the 2x8TB drives about 1 month ago, which involved a
>> conversion to -d raid5 -m raid1c3, which I suppose could hit the same
>> code paths that resize2fs would?
> 
> The problem doesn't look like a trim one, but more likely some device
> boundary bug.
> 
> Would you please provide the following info?
> - btrfs ins dump-tree -t chunk /dev/sde1
>   This contains the device info and chunk tree dump. Doesn't contain
>   any confidential info.
>   We can use this info to determine if there is some chunk really beyond
>   device boundary.
>   I guess some chunks are already beyond device boundary by somehow.

And `lsblk -b` output.

It may be possible that device size in btrfs doesn't match with the real
device...
> 
> Thanks,
> Qu
> 
>>
>> Any advice on how to proceed would be greatly appreciated.
>>
>> Thanks,
>> Justin
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 9437 bytes --]



On 2020/8/1 下午4:30, Justin Brown wrote:
> Hi Qu,
> 
> Thanks for the help.
> 
> Here's is the lsblk -b:
> 
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 2000398934016 0 disk
> └─sda1 8:1 0 2000397868544 0 part
> sdb 8:16 0 8001563222016 0 disk
> └─sdb1 8:17 0 8001562156544 0 part
> sdc 8:32 0 120034123776 0 disk
> ├─sdc1 8:33 0 1048576 0 part
> ├─sdc2 8:34 0 524288000 0 part /boot
> └─sdc3 8:35 0 119507255296 0 part /home
> sdd 8:48 0 8001563222016 0 disk
> └─sdd1 8:49 0 8001562156544 0 part
> sde 8:64 0 2000398934016 0 disk
> └─sde1 8:65 0 2000397868544 0 part
> sdf 8:80 0 2000398934016 0 disk
> └─sdf1 8:81 0 2000397868544 0 part /var/media
> sdg 8:96 1 2000398934016 0 disk
> └─sdg1 8:97 1 2000397868544 0 part
> 
> The `btrfs ins...` output is quite long. I've attached it as a txt and
> also uploaded it at
> https://gist.github.com/fandingo/aa345d6c6fa97162f810e86c9ab20d6a


Thanks, this already shows some device size difference.

But all of them are in fact just a little smaller than device size, thus
it should be fine.

Another problem I found is, it looks like either size or start of some
partitions are not aligned to 4K.

It may be a problem for 4K aligned hard disks, so it may worthy some
concern after solving the btrfs problem.

Would you please also provide some extra dump?
- btrfs check /dev/sda1
  It should detect any problems I missed

- btrfs ins dump-super <device> | grep dev_item.uuid
  It's a little hard to find which device owns to which device id.
  So we need this dump of each btrfs device to make sure.

Thanks,
Qu


> 
> Thanks,
> Justin
> 
> On Sat, Aug 1, 2020 at 2:02 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/8/1 下午2:58, Qu Wenruo wrote:
>>>
>>>
>>> On 2020/8/1 下午2:51, Justin Brown wrote:
>>>> Hello,
>>>>
>>>> I've run into a strange problem that I haven't seen before, and I need
>>>> some help. I started getting generic "input/output" errors on a couple
>>>> of files, and when I looked deeper, the kernel logs are full of
>>>> messages like:
>>>>
>>>>     sd 5:0:0:0: [sdf] tag#29 access beyond end of device
>>>
>>> We had a new fix for trim. But according to your kernel message, it
>>> doesn't look like the case.
>>>
>>> (No obvious tag showing it's trim/discard)
>>>
>>>>
>>>> I've never seen anything like this before with any FS, so I figured it
>>>> was worth asking before I consider running the standard btrfs tools.
>>>> (I briefly started a scrub, but it was going crazy with uncorrectable
>>>> errors, so I cancelled it.)
>>>>
>>>> Here's my system info:
>>>>
>>>> Fedora 32, kernel 5.7.7-200.fc32.x86_64
>>>> btrfs-progs v5.7
>>>>
>>>> /etc/fstab entry:
>>>> LABEL=media /var/media btrfs subvol=media,discard 0 2
>>>>
>>>> btrfs fi show /var/media/
>>>> Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
>>>> Total devices 6 FS bytes used 4.68TiB
>>>> devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
>>>> devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
>>>> devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
>>>> devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
>>>> devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
>>>> devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1
>>>>
>>>> btrfs fi df /var/media/
>>>> Data, RAID5: total=4.69TiB, used=4.68TiB
>>>> System, RAID1C3: total=32.00MiB, used=304.00KiB
>>>> Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>
>>>> I can only mount -o degraded now. Here are the logs when mounting:
>>>>
>>>> Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
>>>> ; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
>>>> degraded /dev/sda1 /var/media/
>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
>>>> access beyond end of device
>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
>>>> error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
>>>> class 0
>>>
>>> OK, it's read, not DISCARD, thus a completely different problem.
>>>
>>>
>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
>>>> sdf1, logical block 16, async page read
>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>>>> sde1): allowing degraded mounts
>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>>>> sde1): disk space caching is enabled
>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>>>> sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
>>>> gen 0
>>>>
>>>> It seems like only relatively recently written files are encountering
>>>> I/O errors. If I `cat` one of the problematic files when the FS is
>>>> mounted normally, I see a ton of this:
>>>>
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
>>>> access beyond end of device
>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
>>>> access beyond end of device
>>>>
>>>> Now that I'm remounted in -o degraded, I'm getting more comprehensible
>>>> warnings, but it still results in I/O read failures:
>>>>
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
>>>> expected csum 0xbe3f80a4 mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
>>>> expected csum 0x9c36a6b4 mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
>>>> expected csum 0x44d30ca2 mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
>>>> expected csum 0xc0f08acc mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
>>>> expected csum 0xcb11db59 mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
>>>> expected csum 0x8a4ee0aa mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
>>>> expected csum 0xdfb79e85 mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
>>>> expected csum 0xc14921a0 mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
>>>> expected csum 0xf2fe8774 mirror 2
>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>> sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
>>>> expected csum 0xae1cafd6 mirror 2
>>>>
>>>> Why trying to research this problem, I came across a Github issue
>>>> https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
>>>> from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
>>>> prevent access beyond device boundary). I do use the discard mount
>>>> option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
>>>> drives with the 2x8TB drives about 1 month ago, which involved a
>>>> conversion to -d raid5 -m raid1c3, which I suppose could hit the same
>>>> code paths that resize2fs would?
>>>
>>> The problem doesn't look like a trim one, but more likely some device
>>> boundary bug.
>>>
>>> Would you please provide the following info?
>>> - btrfs ins dump-tree -t chunk /dev/sde1
>>>   This contains the device info and chunk tree dump. Doesn't contain
>>>   any confidential info.
>>>   We can use this info to determine if there is some chunk really beyond
>>>   device boundary.
>>>   I guess some chunks are already beyond device boundary by somehow.
>>
>> And `lsblk -b` output.
>>
>> It may be possible that device size in btrfs doesn't match with the real
>> device...
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> Any advice on how to proceed would be greatly appreciated.
>>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Access Beyond End of Device & Input/Output Errors

[Justin Brown]

Hi Qu,

Thanks for your continued help.

dump-super:

for i in a b d e f g; do x=$(sudo btrfs ins dump-super /dev/sd${i}1 |
grep dev_item.uuid | cut -f 3); echo "/dev/sd${i}1 $x"; done
/dev/sda1 cc3f9a00-bd69-4ceb-b6e5-4fb874be2aaf
/dev/sdb1 27e1cf24-9349-4f72-a23b-86668b2a9e78
/dev/sdd1 601d409e-8ffd-489c-91af-daf3e0cc9bd2
/dev/sde1 2908ebfb-e6b5-4991-b25d-32d1487ff6a4
/dev/sdf1 cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0

btrfs check:

sudo btrfs check /dev/sda1
Opening filesystem to check...
Checking filesystem on /dev/sda1
UUID: 51eef0c7-2977-4037-b271-3270ea22c7d9
[1/7] checking root items
[2/7] checking extents
WARNING: unaligned total_bytes detected for devid 2, have
2000397868544 should be aligned to 4096
WARNING: this is OK for older kernel, but may cause kernel warning for
newer kernels
WARNING: this can be fixed by 'btrfs rescue fix-device-size'
WARNING: unaligned total_bytes detected for devid 4, have
2000397868544 should be aligned to 4096
WARNING: this is OK for older kernel, but may cause kernel warning for
newer kernels
WARNING: this can be fixed by 'btrfs rescue fix-device-size'
WARNING: unaligned total_bytes detected for devid 6, have
2000397868544 should be aligned to 4096
WARNING: this is OK for older kernel, but may cause kernel warning for
newer kernels
WARNING: this can be fixed by 'btrfs rescue fix-device-size'
WARNING: minor unaligned/mismatch device size detected
WARNING: recommended to use 'btrfs rescue fix-device-size' to fix it
failed to load free space cache for block group 92568662507520
failed to load free space cache for block group 92574031216640
...
failed to load free space cache for block group 97722656817152
failed to load free space cache for block group 97728025526272
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 5148381876224 bytes used, no error found
total csum bytes: 4998903140
total tree bytes: 5301813248
total fs tree bytes: 96894976
total extent tree bytes: 41910272
btree space waste bytes: 135561977
file data blocks allocated: 8972043898880
referenced 5113155596288

The alignment issue would be confined to performance, correct?

Thanks,
Justin

/dev/sdg1 1b938c84-eafd-4396-b06c-8a5bf1339840On Sat, Aug 1, 2020 at
4:31 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/1 下午4:30, Justin Brown wrote:
> > Hi Qu,
> >
> > Thanks for the help.
> >
> > Here's is the lsblk -b:
> >
> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > sda 8:0 0 2000398934016 0 disk
> > └─sda1 8:1 0 2000397868544 0 part
> > sdb 8:16 0 8001563222016 0 disk
> > └─sdb1 8:17 0 8001562156544 0 part
> > sdc 8:32 0 120034123776 0 disk
> > ├─sdc1 8:33 0 1048576 0 part
> > ├─sdc2 8:34 0 524288000 0 part /boot
> > └─sdc3 8:35 0 119507255296 0 part /home
> > sdd 8:48 0 8001563222016 0 disk
> > └─sdd1 8:49 0 8001562156544 0 part
> > sde 8:64 0 2000398934016 0 disk
> > └─sde1 8:65 0 2000397868544 0 part
> > sdf 8:80 0 2000398934016 0 disk
> > └─sdf1 8:81 0 2000397868544 0 part /var/media
> > sdg 8:96 1 2000398934016 0 disk
> > └─sdg1 8:97 1 2000397868544 0 part
> >
> > The `btrfs ins...` output is quite long. I've attached it as a txt and
> > also uploaded it at
> > https://gist.github.com/fandingo/aa345d6c6fa97162f810e86c9ab20d6a
>
>
> Thanks, this already shows some device size difference.
>
> But all of them are in fact just a little smaller than device size, thus
> it should be fine.
>
> Another problem I found is, it looks like either size or start of some
> partitions are not aligned to 4K.
>
> It may be a problem for 4K aligned hard disks, so it may worthy some
> concern after solving the btrfs problem.
>
> Would you please also provide some extra dump?
> - btrfs check /dev/sda1
>   It should detect any problems I missed
>
> - btrfs ins dump-super <device> | grep dev_item.uuid
>   It's a little hard to find which device owns to which device id.
>   So we need this dump of each btrfs device to make sure.
>
> Thanks,
> Qu
>
>
> >
> > Thanks,
> > Justin
> >
> > On Sat, Aug 1, 2020 at 2:02 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/8/1 下午2:58, Qu Wenruo wrote:
> >>>
> >>>
> >>> On 2020/8/1 下午2:51, Justin Brown wrote:
> >>>> Hello,
> >>>>
> >>>> I've run into a strange problem that I haven't seen before, and I need
> >>>> some help. I started getting generic "input/output" errors on a couple
> >>>> of files, and when I looked deeper, the kernel logs are full of
> >>>> messages like:
> >>>>
> >>>>     sd 5:0:0:0: [sdf] tag#29 access beyond end of device
> >>>
> >>> We had a new fix for trim. But according to your kernel message, it
> >>> doesn't look like the case.
> >>>
> >>> (No obvious tag showing it's trim/discard)
> >>>
> >>>>
> >>>> I've never seen anything like this before with any FS, so I figured it
> >>>> was worth asking before I consider running the standard btrfs tools.
> >>>> (I briefly started a scrub, but it was going crazy with uncorrectable
> >>>> errors, so I cancelled it.)
> >>>>
> >>>> Here's my system info:
> >>>>
> >>>> Fedora 32, kernel 5.7.7-200.fc32.x86_64
> >>>> btrfs-progs v5.7
> >>>>
> >>>> /etc/fstab entry:
> >>>> LABEL=media /var/media btrfs subvol=media,discard 0 2
> >>>>
> >>>> btrfs fi show /var/media/
> >>>> Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
> >>>> Total devices 6 FS bytes used 4.68TiB
> >>>> devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
> >>>> devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
> >>>> devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
> >>>> devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
> >>>> devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
> >>>> devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1
> >>>>
> >>>> btrfs fi df /var/media/
> >>>> Data, RAID5: total=4.69TiB, used=4.68TiB
> >>>> System, RAID1C3: total=32.00MiB, used=304.00KiB
> >>>> Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
> >>>> GlobalReserve, single: total=512.00MiB, used=0.00B
> >>>>
> >>>> I can only mount -o degraded now. Here are the logs when mounting:
> >>>>
> >>>> Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
> >>>> ; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
> >>>> degraded /dev/sda1 /var/media/
> >>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> >>>> access beyond end of device
> >>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
> >>>> error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
> >>>> class 0
> >>>
> >>> OK, it's read, not DISCARD, thus a completely different problem.
> >>>
> >>>
> >>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
> >>>> sdf1, logical block 16, async page read
> >>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> >>>> sde1): allowing degraded mounts
> >>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> >>>> sde1): disk space caching is enabled
> >>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
> >>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> >>>> sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
> >>>> gen 0
> >>>>
> >>>> It seems like only relatively recently written files are encountering
> >>>> I/O errors. If I `cat` one of the problematic files when the FS is
> >>>> mounted normally, I see a ton of this:
> >>>>
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
> >>>> access beyond end of device
> >>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
> >>>> access beyond end of device
> >>>>
> >>>> Now that I'm remounted in -o degraded, I'm getting more comprehensible
> >>>> warnings, but it still results in I/O read failures:
> >>>>
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
> >>>> expected csum 0xbe3f80a4 mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
> >>>> expected csum 0x9c36a6b4 mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
> >>>> expected csum 0x44d30ca2 mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
> >>>> expected csum 0xc0f08acc mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
> >>>> expected csum 0xcb11db59 mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
> >>>> expected csum 0x8a4ee0aa mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
> >>>> expected csum 0xdfb79e85 mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
> >>>> expected csum 0xc14921a0 mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
> >>>> expected csum 0xf2fe8774 mirror 2
> >>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>> sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
> >>>> expected csum 0xae1cafd6 mirror 2
> >>>>
> >>>> Why trying to research this problem, I came across a Github issue
> >>>> https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
> >>>> from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
> >>>> prevent access beyond device boundary). I do use the discard mount
> >>>> option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
> >>>> drives with the 2x8TB drives about 1 month ago, which involved a
> >>>> conversion to -d raid5 -m raid1c3, which I suppose could hit the same
> >>>> code paths that resize2fs would?
> >>>
> >>> The problem doesn't look like a trim one, but more likely some device
> >>> boundary bug.
> >>>
> >>> Would you please provide the following info?
> >>> - btrfs ins dump-tree -t chunk /dev/sde1
> >>>   This contains the device info and chunk tree dump. Doesn't contain
> >>>   any confidential info.
> >>>   We can use this info to determine if there is some chunk really beyond
> >>>   device boundary.
> >>>   I guess some chunks are already beyond device boundary by somehow.
> >>
> >> And `lsblk -b` output.
> >>
> >> It may be possible that device size in btrfs doesn't match with the real
> >> device...
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>>>
> >>>> Any advice on how to proceed would be greatly appreciated.
> >>>>
> >>>> Thanks,
> >>>> Justin
> >>>>
> >>>
> >>
>

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 12562 bytes --]



On 2020/8/1 下午7:56, Justin Brown wrote:
> Hi Qu,
> 
> Thanks for your continued help.
> 
> dump-super:
> 
> for i in a b d e f g; do x=$(sudo btrfs ins dump-super /dev/sd${i}1 |
> grep dev_item.uuid | cut -f 3); echo "/dev/sd${i}1 $x"; done
> /dev/sda1 cc3f9a00-bd69-4ceb-b6e5-4fb874be2aaf
> /dev/sdb1 27e1cf24-9349-4f72-a23b-86668b2a9e78
> /dev/sdd1 601d409e-8ffd-489c-91af-daf3e0cc9bd2
> /dev/sde1 2908ebfb-e6b5-4991-b25d-32d1487ff6a4
> /dev/sdf1 cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0

They match with the device size. So no chunk item beyond device boundary.

> 
> btrfs check:
> 
> sudo btrfs check /dev/sda1
> Opening filesystem to check...
> Checking filesystem on /dev/sda1
> UUID: 51eef0c7-2977-4037-b271-3270ea22c7d9
> [1/7] checking root items
> [2/7] checking extents
...
> failed to load free space cache for block group 92568662507520
> failed to load free space cache for block group 92574031216640
> ...
> failed to load free space cache for block group 97722656817152
> failed to load free space cache for block group 97728025526272

This is interesting. Maybe that's related to the problem?

> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)

Great that all metadata are fine.

> found 5148381876224 bytes used, no error found
> total csum bytes: 4998903140
> total tree bytes: 5301813248
> total fs tree bytes: 96894976
> total extent tree bytes: 41910272
> btree space waste bytes: 135561977
> file data blocks allocated: 8972043898880
> referenced 5113155596288
> 
> The alignment issue would be confined to performance, correct?

Yep, only related to performance and some noisy warning for newer kernel.
Not a big problem yet.

Since btrfs-check reports no obvious problem but free space cache
problems, maybe btrfs repair --clear-space-cache v1 is worthy trying.

BTW, since current kernel and btrfs-progs doesn't do restrict chunk
check against device boundary, I'll add such checks to both kernel and
progs soon.

In the mean time, I also see the following dmesg showing that kernel
failed to detect one device:

  Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
  sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing

Can you reproduce that problem? And if so, maybe try "btrfs device scan"
and then mount again?

Thanks,
Qu

> 
> Thanks,
> Justin
> 
> /dev/sdg1 1b938c84-eafd-4396-b06c-8a5bf1339840On Sat, Aug 1, 2020 at
> 4:31 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/8/1 下午4:30, Justin Brown wrote:
>>> Hi Qu,
>>>
>>> Thanks for the help.
>>>
>>> Here's is the lsblk -b:
>>>
>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>>> sda 8:0 0 2000398934016 0 disk
>>> └─sda1 8:1 0 2000397868544 0 part
>>> sdb 8:16 0 8001563222016 0 disk
>>> └─sdb1 8:17 0 8001562156544 0 part
>>> sdc 8:32 0 120034123776 0 disk
>>> ├─sdc1 8:33 0 1048576 0 part
>>> ├─sdc2 8:34 0 524288000 0 part /boot
>>> └─sdc3 8:35 0 119507255296 0 part /home
>>> sdd 8:48 0 8001563222016 0 disk
>>> └─sdd1 8:49 0 8001562156544 0 part
>>> sde 8:64 0 2000398934016 0 disk
>>> └─sde1 8:65 0 2000397868544 0 part
>>> sdf 8:80 0 2000398934016 0 disk
>>> └─sdf1 8:81 0 2000397868544 0 part /var/media
>>> sdg 8:96 1 2000398934016 0 disk
>>> └─sdg1 8:97 1 2000397868544 0 part
>>>
>>> The `btrfs ins...` output is quite long. I've attached it as a txt and
>>> also uploaded it at
>>> https://gist.github.com/fandingo/aa345d6c6fa97162f810e86c9ab20d6a
>>
>>
>> Thanks, this already shows some device size difference.
>>
>> But all of them are in fact just a little smaller than device size, thus
>> it should be fine.
>>
>> Another problem I found is, it looks like either size or start of some
>> partitions are not aligned to 4K.
>>
>> It may be a problem for 4K aligned hard disks, so it may worthy some
>> concern after solving the btrfs problem.
>>
>> Would you please also provide some extra dump?
>> - btrfs check /dev/sda1
>>   It should detect any problems I missed
>>
>> - btrfs ins dump-super <device> | grep dev_item.uuid
>>   It's a little hard to find which device owns to which device id.
>>   So we need this dump of each btrfs device to make sure.
>>
>> Thanks,
>> Qu
>>
>>
>>>
>>> Thanks,
>>> Justin
>>>
>>> On Sat, Aug 1, 2020 at 2:02 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2020/8/1 下午2:58, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2020/8/1 下午2:51, Justin Brown wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I've run into a strange problem that I haven't seen before, and I need
>>>>>> some help. I started getting generic "input/output" errors on a couple
>>>>>> of files, and when I looked deeper, the kernel logs are full of
>>>>>> messages like:
>>>>>>
>>>>>>     sd 5:0:0:0: [sdf] tag#29 access beyond end of device
>>>>>
>>>>> We had a new fix for trim. But according to your kernel message, it
>>>>> doesn't look like the case.
>>>>>
>>>>> (No obvious tag showing it's trim/discard)
>>>>>
>>>>>>
>>>>>> I've never seen anything like this before with any FS, so I figured it
>>>>>> was worth asking before I consider running the standard btrfs tools.
>>>>>> (I briefly started a scrub, but it was going crazy with uncorrectable
>>>>>> errors, so I cancelled it.)
>>>>>>
>>>>>> Here's my system info:
>>>>>>
>>>>>> Fedora 32, kernel 5.7.7-200.fc32.x86_64
>>>>>> btrfs-progs v5.7
>>>>>>
>>>>>> /etc/fstab entry:
>>>>>> LABEL=media /var/media btrfs subvol=media,discard 0 2
>>>>>>
>>>>>> btrfs fi show /var/media/
>>>>>> Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
>>>>>> Total devices 6 FS bytes used 4.68TiB
>>>>>> devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
>>>>>> devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
>>>>>> devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
>>>>>> devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
>>>>>> devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
>>>>>> devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1
>>>>>>
>>>>>> btrfs fi df /var/media/
>>>>>> Data, RAID5: total=4.69TiB, used=4.68TiB
>>>>>> System, RAID1C3: total=32.00MiB, used=304.00KiB
>>>>>> Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
>>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>>>
>>>>>> I can only mount -o degraded now. Here are the logs when mounting:
>>>>>>
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
>>>>>> ; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
>>>>>> degraded /dev/sda1 /var/media/
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
>>>>>> access beyond end of device
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
>>>>>> error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
>>>>>> class 0
>>>>>
>>>>> OK, it's read, not DISCARD, thus a completely different problem.
>>>>>
>>>>>
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
>>>>>> sdf1, logical block 16, async page read
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>>>>>> sde1): allowing degraded mounts
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>>>>>> sde1): disk space caching is enabled
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
>>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
>>>>>> sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
>>>>>> gen 0
>>>>>>
>>>>>> It seems like only relatively recently written files are encountering
>>>>>> I/O errors. If I `cat` one of the problematic files when the FS is
>>>>>> mounted normally, I see a ton of this:
>>>>>>
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
>>>>>> access beyond end of device
>>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
>>>>>> access beyond end of device
>>>>>>
>>>>>> Now that I'm remounted in -o degraded, I'm getting more comprehensible
>>>>>> warnings, but it still results in I/O read failures:
>>>>>>
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
>>>>>> expected csum 0xbe3f80a4 mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
>>>>>> expected csum 0x9c36a6b4 mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
>>>>>> expected csum 0x44d30ca2 mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
>>>>>> expected csum 0xc0f08acc mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
>>>>>> expected csum 0xcb11db59 mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
>>>>>> expected csum 0x8a4ee0aa mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
>>>>>> expected csum 0xdfb79e85 mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
>>>>>> expected csum 0xc14921a0 mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
>>>>>> expected csum 0xf2fe8774 mirror 2
>>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
>>>>>> sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
>>>>>> expected csum 0xae1cafd6 mirror 2
>>>>>>
>>>>>> Why trying to research this problem, I came across a Github issue
>>>>>> https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
>>>>>> from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
>>>>>> prevent access beyond device boundary). I do use the discard mount
>>>>>> option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
>>>>>> drives with the 2x8TB drives about 1 month ago, which involved a
>>>>>> conversion to -d raid5 -m raid1c3, which I suppose could hit the same
>>>>>> code paths that resize2fs would?
>>>>>
>>>>> The problem doesn't look like a trim one, but more likely some device
>>>>> boundary bug.
>>>>>
>>>>> Would you please provide the following info?
>>>>> - btrfs ins dump-tree -t chunk /dev/sde1
>>>>>   This contains the device info and chunk tree dump. Doesn't contain
>>>>>   any confidential info.
>>>>>   We can use this info to determine if there is some chunk really beyond
>>>>>   device boundary.
>>>>>   I guess some chunks are already beyond device boundary by somehow.
>>>>
>>>> And `lsblk -b` output.
>>>>
>>>> It may be possible that device size in btrfs doesn't match with the real
>>>> device...
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>
>>>>>> Any advice on how to proceed would be greatly appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Justin
>>>>>>
>>>>>
>>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Access Beyond End of Device & Input/Output Errors

[Justin Brown]

Hi Qu,

Sorry for the late reply. I've had this system powered off since we
last talked, so no actions taken.

Yes, /dev/sde is dropping out occasionally, but these errors happen
regardless of whether it's in the array or not. Once the disk drops
out, it's completely gone until a reboot (no response from fdisk -l
info, brtfs dev scan, etc.).

The disk was manufactured in 2014, so it's quite old, and the
motherboard/cpu/integrated SATA controller) are about a year older
than that. SMART data on that disk don't indicate any serious
failures. I should probably replace that disk, or maybe just drop it
from the array . However, I'm concerned about the migration path. Any
sort of btrfs remove and btrfs add for new disks will require a btrfs
balance to maintain redundancy. The "access beyond end of device"
errors have shown different disks, not just /dev/sde (most kernel
messages are about sdf, but maybe that's just how messages are
logged), which makes me concerned my problem isn't related to a single
disk and any attempt at a balance could be catastrophic.

What's the best way to get this FS back to a healthy state?

Thanks,
Justin


On Sat, Aug 1, 2020 at 6:30 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/8/1 下午7:56, Justin Brown wrote:
> > Hi Qu,
> >
> > Thanks for your continued help.
> >
> > dump-super:
> >
> > for i in a b d e f g; do x=$(sudo btrfs ins dump-super /dev/sd${i}1 |
> > grep dev_item.uuid | cut -f 3); echo "/dev/sd${i}1 $x"; done
> > /dev/sda1 cc3f9a00-bd69-4ceb-b6e5-4fb874be2aaf
> > /dev/sdb1 27e1cf24-9349-4f72-a23b-86668b2a9e78
> > /dev/sdd1 601d409e-8ffd-489c-91af-daf3e0cc9bd2
> > /dev/sde1 2908ebfb-e6b5-4991-b25d-32d1487ff6a4
> > /dev/sdf1 cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0
>
> They match with the device size. So no chunk item beyond device boundary.
>
> >
> > btrfs check:
> >
> > sudo btrfs check /dev/sda1
> > Opening filesystem to check...
> > Checking filesystem on /dev/sda1
> > UUID: 51eef0c7-2977-4037-b271-3270ea22c7d9
> > [1/7] checking root items
> > [2/7] checking extents
> ...
> > failed to load free space cache for block group 92568662507520
> > failed to load free space cache for block group 92574031216640
> > ...
> > failed to load free space cache for block group 97722656817152
> > failed to load free space cache for block group 97728025526272
>
> This is interesting. Maybe that's related to the problem?
>
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
>
> Great that all metadata are fine.
>
> > found 5148381876224 bytes used, no error found
> > total csum bytes: 4998903140
> > total tree bytes: 5301813248
> > total fs tree bytes: 96894976
> > total extent tree bytes: 41910272
> > btree space waste bytes: 135561977
> > file data blocks allocated: 8972043898880
> > referenced 5113155596288
> >
> > The alignment issue would be confined to performance, correct?
>
> Yep, only related to performance and some noisy warning for newer kernel.
> Not a big problem yet.
>
> Since btrfs-check reports no obvious problem but free space cache
> problems, maybe btrfs repair --clear-space-cache v1 is worthy trying.
>
> BTW, since current kernel and btrfs-progs doesn't do restrict chunk
> check against device boundary, I'll add such checks to both kernel and
> progs soon.
>
> In the mean time, I also see the following dmesg showing that kernel
> failed to detect one device:
>
>   Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
>   sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
>
> Can you reproduce that problem? And if so, maybe try "btrfs device scan"
> and then mount again?
>
> Thanks,
> Qu
>
> >
> > Thanks,
> > Justin
> >
> > /dev/sdg1 1b938c84-eafd-4396-b06c-8a5bf1339840On Sat, Aug 1, 2020 at
> > 4:31 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2020/8/1 下午4:30, Justin Brown wrote:
> >>> Hi Qu,
> >>>
> >>> Thanks for the help.
> >>>
> >>> Here's is the lsblk -b:
> >>>
> >>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> >>> sda 8:0 0 2000398934016 0 disk
> >>> └─sda1 8:1 0 2000397868544 0 part
> >>> sdb 8:16 0 8001563222016 0 disk
> >>> └─sdb1 8:17 0 8001562156544 0 part
> >>> sdc 8:32 0 120034123776 0 disk
> >>> ├─sdc1 8:33 0 1048576 0 part
> >>> ├─sdc2 8:34 0 524288000 0 part /boot
> >>> └─sdc3 8:35 0 119507255296 0 part /home
> >>> sdd 8:48 0 8001563222016 0 disk
> >>> └─sdd1 8:49 0 8001562156544 0 part
> >>> sde 8:64 0 2000398934016 0 disk
> >>> └─sde1 8:65 0 2000397868544 0 part
> >>> sdf 8:80 0 2000398934016 0 disk
> >>> └─sdf1 8:81 0 2000397868544 0 part /var/media
> >>> sdg 8:96 1 2000398934016 0 disk
> >>> └─sdg1 8:97 1 2000397868544 0 part
> >>>
> >>> The `btrfs ins...` output is quite long. I've attached it as a txt and
> >>> also uploaded it at
> >>> https://gist.github.com/fandingo/aa345d6c6fa97162f810e86c9ab20d6a
> >>
> >>
> >> Thanks, this already shows some device size difference.
> >>
> >> But all of them are in fact just a little smaller than device size, thus
> >> it should be fine.
> >>
> >> Another problem I found is, it looks like either size or start of some
> >> partitions are not aligned to 4K.
> >>
> >> It may be a problem for 4K aligned hard disks, so it may worthy some
> >> concern after solving the btrfs problem.
> >>
> >> Would you please also provide some extra dump?
> >> - btrfs check /dev/sda1
> >>   It should detect any problems I missed
> >>
> >> - btrfs ins dump-super <device> | grep dev_item.uuid
> >>   It's a little hard to find which device owns to which device id.
> >>   So we need this dump of each btrfs device to make sure.
> >>
> >> Thanks,
> >> Qu
> >>
> >>
> >>>
> >>> Thanks,
> >>> Justin
> >>>
> >>> On Sat, Aug 1, 2020 at 2:02 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2020/8/1 下午2:58, Qu Wenruo wrote:
> >>>>>
> >>>>>
> >>>>> On 2020/8/1 下午2:51, Justin Brown wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> I've run into a strange problem that I haven't seen before, and I need
> >>>>>> some help. I started getting generic "input/output" errors on a couple
> >>>>>> of files, and when I looked deeper, the kernel logs are full of
> >>>>>> messages like:
> >>>>>>
> >>>>>>     sd 5:0:0:0: [sdf] tag#29 access beyond end of device
> >>>>>
> >>>>> We had a new fix for trim. But according to your kernel message, it
> >>>>> doesn't look like the case.
> >>>>>
> >>>>> (No obvious tag showing it's trim/discard)
> >>>>>
> >>>>>>
> >>>>>> I've never seen anything like this before with any FS, so I figured it
> >>>>>> was worth asking before I consider running the standard btrfs tools.
> >>>>>> (I briefly started a scrub, but it was going crazy with uncorrectable
> >>>>>> errors, so I cancelled it.)
> >>>>>>
> >>>>>> Here's my system info:
> >>>>>>
> >>>>>> Fedora 32, kernel 5.7.7-200.fc32.x86_64
> >>>>>> btrfs-progs v5.7
> >>>>>>
> >>>>>> /etc/fstab entry:
> >>>>>> LABEL=media /var/media btrfs subvol=media,discard 0 2
> >>>>>>
> >>>>>> btrfs fi show /var/media/
> >>>>>> Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
> >>>>>> Total devices 6 FS bytes used 4.68TiB
> >>>>>> devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
> >>>>>> devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
> >>>>>> devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
> >>>>>> devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
> >>>>>> devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
> >>>>>> devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1
> >>>>>>
> >>>>>> btrfs fi df /var/media/
> >>>>>> Data, RAID5: total=4.69TiB, used=4.68TiB
> >>>>>> System, RAID1C3: total=32.00MiB, used=304.00KiB
> >>>>>> Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
> >>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
> >>>>>>
> >>>>>> I can only mount -o degraded now. Here are the logs when mounting:
> >>>>>>
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
> >>>>>> ; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
> >>>>>> degraded /dev/sda1 /var/media/
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
> >>>>>> error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
> >>>>>> class 0
> >>>>>
> >>>>> OK, it's read, not DISCARD, thus a completely different problem.
> >>>>>
> >>>>>
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
> >>>>>> sdf1, logical block 16, async page read
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> >>>>>> sde1): allowing degraded mounts
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> >>>>>> sde1): disk space caching is enabled
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
> >>>>>> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> >>>>>> sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
> >>>>>> gen 0
> >>>>>>
> >>>>>> It seems like only relatively recently written files are encountering
> >>>>>> I/O errors. If I `cat` one of the problematic files when the FS is
> >>>>>> mounted normally, I see a ton of this:
> >>>>>>
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
> >>>>>> access beyond end of device
> >>>>>> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
> >>>>>> access beyond end of device
> >>>>>>
> >>>>>> Now that I'm remounted in -o degraded, I'm getting more comprehensible
> >>>>>> warnings, but it still results in I/O read failures:
> >>>>>>
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
> >>>>>> expected csum 0xbe3f80a4 mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
> >>>>>> expected csum 0x9c36a6b4 mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
> >>>>>> expected csum 0x44d30ca2 mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
> >>>>>> expected csum 0xc0f08acc mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
> >>>>>> expected csum 0xcb11db59 mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
> >>>>>> expected csum 0x8a4ee0aa mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
> >>>>>> expected csum 0xdfb79e85 mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
> >>>>>> expected csum 0xc14921a0 mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
> >>>>>> expected csum 0xf2fe8774 mirror 2
> >>>>>> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> >>>>>> sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
> >>>>>> expected csum 0xae1cafd6 mirror 2
> >>>>>>
> >>>>>> Why trying to research this problem, I came across a Github issue
> >>>>>> https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
> >>>>>> from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
> >>>>>> prevent access beyond device boundary). I do use the discard mount
> >>>>>> option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
> >>>>>> drives with the 2x8TB drives about 1 month ago, which involved a
> >>>>>> conversion to -d raid5 -m raid1c3, which I suppose could hit the same
> >>>>>> code paths that resize2fs would?
> >>>>>
> >>>>> The problem doesn't look like a trim one, but more likely some device
> >>>>> boundary bug.
> >>>>>
> >>>>> Would you please provide the following info?
> >>>>> - btrfs ins dump-tree -t chunk /dev/sde1
> >>>>>   This contains the device info and chunk tree dump. Doesn't contain
> >>>>>   any confidential info.
> >>>>>   We can use this info to determine if there is some chunk really beyond
> >>>>>   device boundary.
> >>>>>   I guess some chunks are already beyond device boundary by somehow.
> >>>>
> >>>> And `lsblk -b` output.
> >>>>
> >>>> It may be possible that device size in btrfs doesn't match with the real
> >>>> device...
> >>>>>
> >>>>> Thanks,
> >>>>> Qu
> >>>>>
> >>>>>>
> >>>>>> Any advice on how to proceed would be greatly appreciated.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Justin
> >>>>>>
> >>>>>
> >>>>
> >>
>

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

[-- Attachment #1: Type: text/plain, Size: 3755 bytes --]

Hi all,
Hi Qu,

I am also getting this very same error on my system.
Actually, I am experiencing this since the following old bug was introduced AND also even after it has been fixed:
https://lore.kernel.org/linux-btrfs/20190521190023.GA68070@glet/T/

That (dm-related) bug was claimed to have been fixed and users confirmed that their btrfs partitions were working correctly again, but I am still experiencing some issues from time to time - and obviously only on SSD devices.

Just to clarify: I am using btrfs volumes on encrypted LUKS partitions, where every partition is encrypted individually.
I am *NOT* using LVM at all: just btrfs directly on top of LUKS (which is different from the users' setup in the above-mentioned bug reports).
And I am trimming the partitions only via fstrim, have configured the mount points with the "nodiscard" option and the LUKS volumes in /etc/crypttab with "discard", so to have the pass-through when I use fstrim.

Opposite to Justin, my partitions are all aligned.

When this happens on my root partition, I cannot launch any command anymore because the file system is not responding (e.g.: I get "ls: command not found"). I cannot actually do anything in reality, because the system console is flooded with messages like:

 "sd <....> [sdX] tag#29 access beyond end of device"

and that "clogs" the system. Since the root fs is unusable, the system log cannot store those messages, so I can't find them at the next reboot.
I can only soft-reset (CTRL-ALT-DEL), it's the "cleanest" (and only) way I can get back to a working system.

When the system restarts, it takes 2 seconds longer than usual to mount the file systems, and then I can use the PC again.
Immediately after login, if I run btrfs scrub, I get no errors (I scrub ALL of my partitions: they're all fine). So, it seems that at least the auto-recovery capability of btrfs works fine, thanks to you devs :-)

Then, if I boot from an external device and run btrfs check on the unmounted file systems, it also reports NO errors - opposite to what was happening when the dm bug was still open: to me, this really means that btrfs today is able to heal itself from this issue (it was not always the case in 2019, when the dm bug was opened).
I have not tried to boot from external device directly after this issue occurs - I mean, performing btrfs check without going first through the btrfs scrub step. I will do that next time and see what output I get.

All my partitions are snapshotted, and surely this could help with auto-recovery.

What I have noticed is that when this bug happens, it ALWAYS happens after I have purged the old snapshots: that is, when the root partition only has one "fresh" (read-only) snapshot. This is never happening when I have more than one snapshot - maybe it means nothing, but it seems to me to be systematic.

I have attached a file with my setup.
Could you maybe spot anything weird there? It looks fine to me. The USER and SCRATCH volumes are in RAID-0.

I am unable to provide any dmesg output or system log because, as said, when it happens it does not write anything to the SYS partition (where /var/log is). I will move at least /var/log/journal to another device, so hopefully next time I will be able to provide some useful info.

Another info: of course, I have tried (twice!) to reconstruct the system SSD from scratch, because I wanted to be sure that it was not depending on some exotic issue. And each time I used a brand new device.
So, this issue has been happening with a SanDisk Ultra II and with two different Samsung EVO 860.

Is it possible that what we are experiencing is still an effect of that dm bug, that it was not completely fixed?


Thanks for your help, and for reading till here :)

(c)

[-- Attachment #2: system-setup.txt --]
[-- Type: text/plain, Size: 25144 bytes --]

-------------
# uname -a
Linux <****> 5.10.7-3-MANJARO #1 SMP PREEMPT Fri Jan 15 21:11:34 UTC 2021 x86_64 GNU/Linux
-------------
# btrfs fi show
Label: 'SYS'  uuid: 8c63fc14-a0a8-4a6a-9e2a-45d8735a6cb8
        Total devices 1 FS bytes used 39.44GiB
        devid    1 size 440.57GiB used 58.05GiB path /dev/mapper/SYS

Label: 'HOME'  uuid: 9362ac9d-c280-451d-9c8a-c09798e1c887
        Total devices 1 FS bytes used 129.38GiB
        devid    1 size 232.88GiB used 179.05GiB path /dev/mapper/OMO

Label: 'USER'  uuid: b0b9b664-6629-43c0-b89e-e2db1fe0f39b
        Total devices 2 FS bytes used 2.94TiB
        devid    1 size 1.77TiB used 1.49TiB path /dev/mapper/user1
        devid    2 size 1.77TiB used 1.49TiB path /dev/mapper/user2

Label: 'SCRATCH'  uuid: cdb788b8-665b-44db-a6f0-ee743fd67b4f
        Total devices 2 FS bytes used 76.61GiB
        devid    1 size 46.55GiB used 40.01GiB path /dev/mapper/duro1
        devid    2 size 46.55GiB used 40.01GiB path /dev/mapper/duro2
-------------
# btrfs fi df /
Data, single: total=56.01GiB, used=38.48GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.01GiB, used=976.75MiB
GlobalReserve, single: total=107.81MiB, used=0.00B
-------------
# blkid
/dev/sda1: UUID="ee18ef63-d156-4c61-a4d3-09ffc6a82659" TYPE="crypto_LUKS" PARTLABEL="BACKUP" PARTUUID="11c718e6-d06e-4d42-bfd0-2325d7da14dc"
/dev/sdb1: UUID="dd0a3dc5-c8dc-4c0d-86e8-294e6c76cb30" TYPE="crypto_LUKS" PARTLABEL="SCRATCH" PARTUUID="0af5e58d-c057-4972-96f6-d12cf1005180"
/dev/sdb2: UUID="99e588a1-5f1d-43f9-bace-d5dc0d251c0f" TYPE="crypto_LUKS" PARTLABEL="USER" PARTUUID="79794d70-6cb8-4e84-b6db-32b6ddefb398"
/dev/sdc1: UUID="0a51adb7-b8dd-4cc2-816f-13ed478391a7" TYPE="crypto_LUKS" PARTLABEL="HOME" PARTUUID="2a43d162-ea4f-40a9-a5ab-44e922472b80"
/dev/sdd1: UUID="1af89040-20e8-43f9-a4b4-0359ab938c37" TYPE="crypto_LUKS" PARTLABEL="SCRATCH" PARTUUID="adf428b0-93ba-4bdc-bba9-e12c4dbaa18a"
/dev/sdd2: UUID="9c78d9bb-8830-4576-91f6-1425b85616c9" TYPE="crypto_LUKS" PARTLABEL="USER" PARTUUID="bb10bd57-53d8-40fe-8705-9bae6893e892"
/dev/sde2: UUID="e821b81b-2cc2-4b51-84a7-d5a9e459e108" TYPE="crypto_LUKS" PARTLABEL="BOOT" PARTUUID="349bb8d4-ef92-4d3f-95fc-7cf3029bc8f2"
/dev/sde3: UUID="bb95b6d5-6567-4837-8f66-eac5de4aa7ba" TYPE="crypto_LUKS" PARTLABEL="SYS" PARTUUID="653ae11a-4c13-4797-8f86-2807a961420d"
/dev/sde4: UUID="58e5764f-6522-4dad-86f2-696c3c87847c" TYPE="crypto_LUKS" PARTLABEL="SWAP" PARTUUID="943befb3-ccd4-440a-a4e1-ad98874e1aa3"
/dev/sdh1: UUID="fb23d5c4-1cec-4a2b-b5c6-701b15436bfc" TYPE="crypto_LUKS" PARTLABEL="Entertainment" PARTUUID="9228f216-982a-4c32-9996-9df66893d71b"
/dev/sdg1: UUID="ec9deffe-47d8-4776-9e2e-0a74ad93cd29" TYPE="crypto_LUKS" PARTLABEL="orfano" PARTUUID="1d1dd982-f810-4993-a705-61d356da39b0"
/dev/mapper/SYS: LABEL="SYS" UUID="8c63fc14-a0a8-4a6a-9e2a-45d8735a6cb8" UUID_SUB="1e5141ca-474c-4144-a1e3-cd1810739d91" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/mapper/suoppo: LABEL="SWAP" UUID="762cf066-660f-4a2b-9ecb-b91220d23868" TYPE="swap"
/dev/mapper/OMO: LABEL="HOME" UUID="9362ac9d-c280-451d-9c8a-c09798e1c887" UUID_SUB="eab5d229-1328-483e-9ad8-43e570e4e223" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/mapper/duro2: LABEL="SCRATCH" UUID="cdb788b8-665b-44db-a6f0-ee743fd67b4f" UUID_SUB="fde44687-d8e8-4603-bb1c-dd3b77482d9f" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/mapper/buttalo: LABEL="BOOT" UUID="afba68e5-56da-4af0-bab4-6c05025ba0d8" BLOCK_SIZE="1024" TYPE="ext4"
/dev/mapper/duro1: LABEL="SCRATCH" UUID="cdb788b8-665b-44db-a6f0-ee743fd67b4f" UUID_SUB="371e9000-94f5-4a15-a74f-c60ffb6d8a7c" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/mapper/user1: LABEL="USER" UUID="b0b9b664-6629-43c0-b89e-e2db1fe0f39b" UUID_SUB="9c0d7301-6318-48c8-9507-bf36e0056e55" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/mapper/user2: LABEL="USER" UUID="b0b9b664-6629-43c0-b89e-e2db1fe0f39b" UUID_SUB="065f20e5-8ab4-4361-a377-7eae43f90eb9" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sde1: PARTLABEL="GRUB" PARTUUID="3c12ac5f-275c-40c0-914b-73c1b728dea0"
-------------
# lsblk -b
NAME        MAJ:MIN RM          SIZE RO TYPE  MOUNTPOINT
sda           8:0    1 2000398934016  0 disk  
└─sda1        8:1    1 2000397795328  0 part  
sdb           8:16   1 2000398934016  0 disk  
├─sdb1        8:17   1   49999249408  0 part  
│ └─duro2   254:3    0   49982472192  0 crypt 
└─sdb2        8:18   1 1950398545920  0 part  
  └─user2   254:7    0 1950381768704  0 crypt 
sdc           8:32   1  250059350016  0 disk  
└─sdc1        8:33   1  250058113024  0 part  
  └─OMO     254:2    0  250056015872  0 crypt /home
sdd           8:48   1 2000398934016  0 disk  
├─sdd1        8:49   1   49999249408  0 part  
│ └─duro1   254:5    0   49982472192  0 crypt /var/lib/libvirt/images
└─sdd2        8:50   1 1950398545920  0 part  
  └─user1   254:6    0 1950381768704  0 crypt /PATTUME/Entertainment
sde           8:64   1  500107862016  0 disk  
├─sde1        8:65   1       1048576  0 part  
├─sde2        8:66   1     198180864  0 part  
│ └─buttalo 254:4    0     196083712  0 crypt /boot
├─sde3        8:67   1  473064038400  0 part  
│ └─SYS     254:0    0  473061941248  0 crypt /
└─sde4        8:68   1   26842497024  0 part  
  └─suoppo  254:1    0   26840399872  0 crypt [SWAP]
sdg           8:96   0 2000398934016  0 disk  
└─sdg1        8:97   0 2000397795328  0 part  
sdh           8:112  0 3000558944256  0 disk  
└─sdh1        8:113  0 3000556847104  0 part  
sr0          11:0    1    1073741312  0 rom   
sr1          11:1    1    1073741312  0 rom   
-------------
# btrfs ins dump-tree -t chunk /dev/mapper/SYS
btrfs-progs v5.9 
chunk tree
leaf 120415338496 items 62 free space 9755 generation 312763 owner CHUNK_TREE
leaf 120415338496 flags 0x1(WRITTEN) backref revision 1
fs uuid 8c63fc14-a0a8-4a6a-9e2a-45d8735a6cb8
chunk uuid f24f036f-ad1a-43b0-bba9-83ca72d633be
	item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
		devid 1 total_bytes 473061941248 bytes_used 62327357440
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
		fsid 8c63fc14-a0a8-4a6a-9e2a-45d8735a6cb8
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 5242880) itemoff 16105 itemsize 80
		length 8388608 owner 2 stripe_len 65536 type METADATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 0
			stripe 0 devid 1 offset 5242880
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 13631488) itemoff 16025 itemsize 80
		length 8388608 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 0
			stripe 0 devid 1 offset 13631488
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15945 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type METADATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 22020096
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 1095761920) itemoff 15865 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 1095761920
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 2169503744) itemoff 15785 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 2169503744
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 3243245568) itemoff 15705 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 3243245568
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 4316987392) itemoff 15625 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 4316987392
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 5390729216) itemoff 15545 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 5390729216
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 6464471040) itemoff 15465 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 6464471040
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 10 key (FIRST_CHUNK_TREE CHUNK_ITEM 7538212864) itemoff 15385 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 7538212864
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 11 key (FIRST_CHUNK_TREE CHUNK_ITEM 8611954688) itemoff 15305 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 8611954688
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 12 key (FIRST_CHUNK_TREE CHUNK_ITEM 9685696512) itemoff 15225 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 9685696512
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 13 key (FIRST_CHUNK_TREE CHUNK_ITEM 12906921984) itemoff 15145 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 12906921984
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 14 key (FIRST_CHUNK_TREE CHUNK_ITEM 13980663808) itemoff 15065 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 13980663808
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 15 key (FIRST_CHUNK_TREE CHUNK_ITEM 15054405632) itemoff 14985 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 15054405632
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 16 key (FIRST_CHUNK_TREE CHUNK_ITEM 16128147456) itemoff 14905 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 16128147456
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 17 key (FIRST_CHUNK_TREE CHUNK_ITEM 17201889280) itemoff 14825 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 17201889280
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 18 key (FIRST_CHUNK_TREE CHUNK_ITEM 18275631104) itemoff 14745 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 18275631104
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 19 key (FIRST_CHUNK_TREE CHUNK_ITEM 19349372928) itemoff 14665 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 19349372928
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 20 key (FIRST_CHUNK_TREE CHUNK_ITEM 20423114752) itemoff 14585 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 20423114752
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 21 key (FIRST_CHUNK_TREE CHUNK_ITEM 21496856576) itemoff 14505 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 21496856576
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 22 key (FIRST_CHUNK_TREE CHUNK_ITEM 22570598400) itemoff 14425 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 22570598400
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 23 key (FIRST_CHUNK_TREE CHUNK_ITEM 23644340224) itemoff 14345 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 23644340224
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 24 key (FIRST_CHUNK_TREE CHUNK_ITEM 24718082048) itemoff 14265 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 24718082048
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 25 key (FIRST_CHUNK_TREE CHUNK_ITEM 25791823872) itemoff 14185 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 25791823872
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 26 key (FIRST_CHUNK_TREE CHUNK_ITEM 26865565696) itemoff 14105 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 26865565696
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 27 key (FIRST_CHUNK_TREE CHUNK_ITEM 27939307520) itemoff 14025 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 27939307520
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 28 key (FIRST_CHUNK_TREE CHUNK_ITEM 29013049344) itemoff 13945 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 29013049344
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 29 key (FIRST_CHUNK_TREE CHUNK_ITEM 30086791168) itemoff 13865 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 30086791168
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 30 key (FIRST_CHUNK_TREE CHUNK_ITEM 31160532992) itemoff 13785 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 31160532992
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 31 key (FIRST_CHUNK_TREE CHUNK_ITEM 32234274816) itemoff 13705 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 32234274816
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 32 key (FIRST_CHUNK_TREE CHUNK_ITEM 33308016640) itemoff 13625 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 33308016640
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 33 key (FIRST_CHUNK_TREE CHUNK_ITEM 34381758464) itemoff 13545 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 34381758464
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 34 key (FIRST_CHUNK_TREE CHUNK_ITEM 35455500288) itemoff 13465 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 35455500288
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 35 key (FIRST_CHUNK_TREE CHUNK_ITEM 36529242112) itemoff 13385 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type METADATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 36529242112
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 36 key (FIRST_CHUNK_TREE CHUNK_ITEM 37602983936) itemoff 13305 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 37602983936
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 37 key (FIRST_CHUNK_TREE CHUNK_ITEM 38676725760) itemoff 13225 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 38676725760
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 38 key (FIRST_CHUNK_TREE CHUNK_ITEM 39817576448) itemoff 13145 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 39817576448
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 39 key (FIRST_CHUNK_TREE CHUNK_ITEM 40891318272) itemoff 13065 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 40891318272
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 40 key (FIRST_CHUNK_TREE CHUNK_ITEM 43038801920) itemoff 12985 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 43038801920
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 41 key (FIRST_CHUNK_TREE CHUNK_ITEM 49481252864) itemoff 12905 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 49481252864
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 42 key (FIRST_CHUNK_TREE CHUNK_ITEM 54883516416) itemoff 12825 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 47333769216
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 43 key (FIRST_CHUNK_TREE CHUNK_ITEM 55990812672) itemoff 12745 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 10759438336
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 44 key (FIRST_CHUNK_TREE CHUNK_ITEM 57064554496) itemoff 12665 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 11833180160
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 45 key (FIRST_CHUNK_TREE CHUNK_ITEM 117194096640) itemoff 12585 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 104242085888
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 46 key (FIRST_CHUNK_TREE CHUNK_ITEM 118267838464) itemoff 12505 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 41965060096
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 47 key (FIRST_CHUNK_TREE CHUNK_ITEM 119341580288) itemoff 12425 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 44112543744
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 48 key (FIRST_CHUNK_TREE CHUNK_ITEM 120415322112) itemoff 12345 itemsize 80
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 39750467584
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 49 key (FIRST_CHUNK_TREE CHUNK_ITEM 154808614912) itemoff 12265 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 52702478336
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 50 key (FIRST_CHUNK_TREE CHUNK_ITEM 166619774976) itemoff 12185 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 61292412928
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 51 key (FIRST_CHUNK_TREE CHUNK_ITEM 179504676864) itemoff 12105 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 63439896576
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 52 key (FIRST_CHUNK_TREE CHUNK_ITEM 180578418688) itemoff 12025 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 64513638400
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 53 key (FIRST_CHUNK_TREE CHUNK_ITEM 182725902336) itemoff 11945 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 66661122048
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 54 key (FIRST_CHUNK_TREE CHUNK_ITEM 381368139776) itemoff 11865 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 62366154752
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 55 key (FIRST_CHUNK_TREE CHUNK_ITEM 396400525312) itemoff 11785 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 67734863872
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 56 key (FIRST_CHUNK_TREE CHUNK_ITEM 397474267136) itemoff 11705 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 45186285568
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 57 key (FIRST_CHUNK_TREE CHUNK_ITEM 398548008960) itemoff 11625 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 46260027392
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 58 key (FIRST_CHUNK_TREE CHUNK_ITEM 399621750784) itemoff 11545 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 48407511040
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 59 key (FIRST_CHUNK_TREE CHUNK_ITEM 400695492608) itemoff 11465 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 50554994688
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 60 key (FIRST_CHUNK_TREE CHUNK_ITEM 631549984768) itemoff 11385 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 56997445632
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91
	item 61 key (FIRST_CHUNK_TREE CHUNK_ITEM 704564428800) itemoff 11305 itemsize 80
		length 1073741824 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 107463311360
			dev_uuid 1e5141ca-474c-4144-a1e3-cd1810739d91

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

On 2021/1/18 上午7:38, chainofflowers wrote:
> Hi all,
> Hi Qu,
>
> I am also getting this very same error on my system.
> Actually, I am experiencing this since the following old bug was introduced AND also even after it has been fixed:
> https://lore.kernel.org/linux-btrfs/20190521190023.GA68070@glet/T/
>
> That (dm-related) bug was claimed to have been fixed and users confirmed that their btrfs partitions were working correctly again, but I am still experiencing some issues from time to time - and obviously only on SSD devices.
>
> Just to clarify: I am using btrfs volumes on encrypted LUKS partitions, where every partition is encrypted individually.
> I am *NOT* using LVM at all: just btrfs directly on top of LUKS (which is different from the users' setup in the above-mentioned bug reports).
> And I am trimming the partitions only via fstrim, have configured the mount points with the "nodiscard" option and the LUKS volumes in /etc/crypttab with "discard", so to have the pass-through when I use fstrim.
>
> Opposite to Justin, my partitions are all aligned.
>
> When this happens on my root partition, I cannot launch any command anymore because the file system is not responding (e.g.: I get "ls: command not found"). I cannot actually do anything in reality, because the system console is flooded with messages like:
>
>   "sd <....> [sdX] tag#29 access beyond end of device"

The best way to debug such problem is to recompile the kernel adding
some debug outputs.
(Maybe it can be done with bpftrace, but not yet familiar with that)

If you're able to recompile the kenerl (using abs + makepkg for Arch
based kernel), please try the following diff.

This will add extra debugging to show where the offending length
happens, either extent discard or unallocated space discard.
And from that output we can continue our investigation.

Thanks,
Qu

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 30b1a630dc2f..7451fa0b14b9 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -5776,6 +5776,7 @@ static int btrfs_trim_free_extents(struct
btrfs_device *device, u64 *trimmed)

         ret = 0;

+       pr_info("%s: enter devid=%llu\n", __func__, device->devid);
         while (1) {
                 struct btrfs_fs_info *fs_info = device->fs_info;
                 u64 bytes;
@@ -5820,6 +5821,8 @@ static int btrfs_trim_free_extents(struct
btrfs_device *device, u64 *trimmed)
                         break;
                 }

+               pr_info("%s: devid=%llu start=%llu len=%llu\n",
+                       __func__, device->devid, start, len);
                 ret = btrfs_issue_discard(device->bdev, start, len,
                                           &bytes);
                 if (!ret)
@@ -5842,6 +5845,7 @@ static int btrfs_trim_free_extents(struct
btrfs_device *device, u64 *trimmed)
                 cond_resched();
         }

+       pr_info("%s: done devid=%llu ret=%d\n", __func__, device->devid,
ret);
         return ret;
  }

diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 379bef967e1d..03046fca53a2 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -3772,6 +3772,8 @@ int btrfs_trim_block_group(struct
btrfs_block_group *block_group,
                 spin_unlock(&block_group->lock);
                 return 0;
         }
+       pr_info("%s: enter bg start=%llu start=%llu end=%llu minlen=%llu\n",
+               __func__, block_group->start, start, end, minlen);
         btrfs_freeze_block_group(block_group);
         spin_unlock(&block_group->lock);

@@ -3786,6 +3788,8 @@ int btrfs_trim_block_group(struct
btrfs_block_group *block_group,
                 reset_trimming_bitmap(ctl, offset_to_bitmap(ctl, end));
  out:
         btrfs_unfreeze_block_group(block_group);
+       pr_info("%s: enter bg start=%llu ret=%d\n",
+               __func__, block_group->start, ret);
         return ret;
  }


>
> and that "clogs" the system. Since the root fs is unusable, the system log cannot store those messages, so I can't find them at the next reboot.
> I can only soft-reset (CTRL-ALT-DEL), it's the "cleanest" (and only) way I can get back to a working system.
>
> When the system restarts, it takes 2 seconds longer than usual to mount the file systems, and then I can use the PC again.
> Immediately after login, if I run btrfs scrub, I get no errors (I scrub ALL of my partitions: they're all fine). So, it seems that at least the auto-recovery capability of btrfs works fine, thanks to you devs :-)
>
> Then, if I boot from an external device and run btrfs check on the unmounted file systems, it also reports NO errors - opposite to what was happening when the dm bug was still open: to me, this really means that btrfs today is able to heal itself from this issue (it was not always the case in 2019, when the dm bug was opened).
> I have not tried to boot from external device directly after this issue occurs - I mean, performing btrfs check without going first through the btrfs scrub step. I will do that next time and see what output I get.
>
> All my partitions are snapshotted, and surely this could help with auto-recovery.
>
> What I have noticed is that when this bug happens, it ALWAYS happens after I have purged the old snapshots: that is, when the root partition only has one "fresh" (read-only) snapshot. This is never happening when I have more than one snapshot - maybe it means nothing, but it seems to me to be systematic.
>
> I have attached a file with my setup.
> Could you maybe spot anything weird there? It looks fine to me. The USER and SCRATCH volumes are in RAID-0.
>
> I am unable to provide any dmesg output or system log because, as said, when it happens it does not write anything to the SYS partition (where /var/log is). I will move at least /var/log/journal to another device, so hopefully next time I will be able to provide some useful info.
>
> Another info: of course, I have tried (twice!) to reconstruct the system SSD from scratch, because I wanted to be sure that it was not depending on some exotic issue. And each time I used a brand new device.
> So, this issue has been happening with a SanDisk Ultra II and with two different Samsung EVO 860.
>
> Is it possible that what we are experiencing is still an effect of that dm bug, that it was not completely fixed?
>
>
> Thanks for your help, and for reading till here :)
>
> (c)
>

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

Thanks, I put everything in place... I am now waiting for the next
occurrence...
> The best way to debug such problem is to recompile the kernel adding
> some debug outputs.
> (Maybe it can be done with bpftrace, but not yet familiar with that)
> 
> If you're able to recompile the kenerl (using abs + makepkg for Arch
> based kernel), please try the following diff.
> 
> This will add extra debugging to show where the offending length
> happens, either extent discard or unallocated space discard.
> And from that output we can continue our investigation.
> 
> Thanks,
> Qu

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

[-- Attachment #1: Type: text/plain, Size: 2384 bytes --]

Hi Qu,

it happened again. This time on my /home partition.
I rebooted from an external disk and ran btrfs check without first going
through btrfs scrub, and this is the output, no errors:

------------------------------------------
[manjaro oc]# btrfs check /dev/mapper/OMO
Opening filesystem to check…
Checking filesystem on /dev/mapper/OMO
UUID: 9362ac9d-c280-451d-9c8a-c09798e1c887
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 137523740672 bytes used, no error found
total csum bytes: 113842816
total tree bytes: 1740537856
total fs tree bytes: 1444249600
total extent tree bytes: 143835136
btree space waste bytes: 325744995
file data blocks allocated: 210346024960
 referenced 172314374144
------------------------------------------

Then, I rebooted from my internal disk, everything went well. I ran
btrfs scrub and also got no errors.

I have dumped the messages from journalctl, and the debug ones related
to btrfs were only the ones from btrfs_trim_block_group - so, the issue
is related to free space extents I guess?

I have attached the logs.
You can see that the last line:

------------------------------------------
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg
start=26864517120 start=26864517120 end=27938258944 minlen=512
------------------------------------------

does not have a second matching line with "ret=0", because the kernel
stopped storing messages in the log. So, I guess the issue occurred
while btrfs_trim_block_group was working on 26864517120..27938258944.

Unfortunately I did not dump the output of dmesg directly in that
moment, so all I could get is what was available in the journal after
the reboot.

In the log you can also see that some time before BTRFS detected that
the space cache for dm-3 needed to be rebuilt:
------------------------------------------
Jan 21 19:29:17 <***> kernel: BTRFS warning (device dm-3): block group
82699091968 has wrong amount of free space
Jan 21 19:29:17 <***> kernel: BTRFS warning (device dm-3): failed to
load free space cache for block group 82699091968, rebuilding it now
------------------------------------------

Any hint about what I could do now?

Thanks! :-)



(c)

[-- Attachment #2: btrfs_trim.log --]
[-- Type: text/x-log, Size: 11181 bytes --]

Jan 21 19:25:47 <***> kernel: Linux version 5.10.7-4-MANJARO-BTRFS-DEBUG (chainofflowers@luna) (gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35.1) #1 SMP PREEMPT Mon Jan 18 02:47:41 CET 2021
Jan 21 19:25:47 <***> kernel: usb usb1: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG ehci_hcd
Jan 21 19:25:47 <***> kernel: usb usb2: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG ehci_hcd
Jan 21 19:25:47 <***> kernel: usb usb3: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG uhci_hcd
Jan 21 19:25:47 <***> kernel: usb usb4: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG uhci_hcd
Jan 21 19:25:47 <***> kernel: usb usb5: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG uhci_hcd
Jan 21 19:25:47 <***> kernel: usb usb6: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG uhci_hcd
Jan 21 19:25:47 <***> kernel: usb usb7: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG uhci_hcd
Jan 21 19:25:47 <***> kernel: usb usb8: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG uhci_hcd
Jan 21 19:25:47 <***> kernel: usb usb9: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG xhci-hcd
Jan 21 19:25:47 <***> kernel: usb usb10: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG xhci-hcd
Jan 21 19:25:47 <***> kernel: usb usb11: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG xhci-hcd
Jan 21 19:25:47 <***> kernel: usb usb12: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG xhci-hcd
Jan 21 19:25:47 <***> kernel: usb usb13: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG xhci-hcd
Jan 21 19:25:47 <***> kernel: usb usb14: Manufacturer: Linux 5.10.7-4-MANJARO-BTRFS-DEBUG xhci-hcd
Jan 21 19:25:47 <***> kernel: Btrfs loaded, crc32c=crc32c-intel
Jan 21 19:25:47 <***> kernel: BTRFS: device label SYS devid 1 transid 320418 /dev/dm-0 scanned by systemd-udevd (212)
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): disk space caching is enabled
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): has skinny extents
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): enabling ssd optimizations
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): disk space caching is enabled
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): has skinny extents
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): enabling ssd optimizations
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): use zstd compression, level 3
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): enabling auto defrag
Jan 21 19:25:47 <***> kernel: BTRFS info (device dm-0): disk space caching is enabled
Jan 21 19:25:49 <***> kernel: BTRFS info (device dm-0): devid 1 device path /dev/mapper/SYS changed to /dev/dm-0 scanned by systemd-udevd (432)
Jan 21 19:25:49 <***> kernel: BTRFS info (device dm-0): devid 1 device path /dev/dm-0 changed to /dev/mapper/SYS scanned by systemd-udevd (432)
Jan 21 19:25:51 <***> kernel: BTRFS: device label SCRATCH devid 1 transid 36967 /dev/dm-2 scanned by systemd-udevd (427)
Jan 21 19:25:51 <***> kernel: BTRFS: device label HOME devid 1 transid 4316351 /dev/dm-3 scanned by systemd-udevd (427)
Jan 21 19:25:51 <***> kernel: BTRFS info (device dm-3): use zstd compression, level 3
Jan 21 19:25:51 <***> kernel: BTRFS info (device dm-3): enabling ssd optimizations
Jan 21 19:25:51 <***> kernel: BTRFS info (device dm-3): enabling auto defrag
Jan 21 19:25:51 <***> kernel: BTRFS info (device dm-3): disk space caching is enabled
Jan 21 19:25:51 <***> kernel: BTRFS info (device dm-3): has skinny extents
Jan 21 19:25:53 <***> kernel: BTRFS: device label USER devid 1 transid 8387 /dev/dm-5 scanned by systemd-udevd (427)
Jan 21 19:25:55 <***> kernel: BTRFS: device label USER devid 2 transid 8387 /dev/dm-6 scanned by systemd-udevd (427)
Jan 21 19:25:55 <***> kernel: BTRFS info (device dm-5): use zstd compression, level 3
Jan 21 19:25:55 <***> kernel: BTRFS info (device dm-5): enabling ssd optimizations
Jan 21 19:25:55 <***> kernel: BTRFS info (device dm-5): enabling auto defrag
Jan 21 19:25:55 <***> kernel: BTRFS info (device dm-5): disk space caching is enabled
Jan 21 19:25:55 <***> kernel: BTRFS info (device dm-5): has skinny extents
Jan 21 19:25:55 <***> kernel: BTRFS info (device dm-5): devid 1 device path /dev/mapper/user1 changed to /dev/dm-5 scanned by systemd-udevd (440)
Jan 21 19:25:55 <***> kernel: BTRFS info (device dm-5): devid 1 device path /dev/dm-5 changed to /dev/mapper/user1 scanned by systemd-udevd (440)
Jan 21 19:25:57 <***> kernel: BTRFS: device label SCRATCH devid 2 transid 36967 /dev/dm-7 scanned by systemd-udevd (427)
Jan 21 19:26:00 <***> kernel: BTRFS info (device dm-2): use no compression
Jan 21 19:26:00 <***> kernel: BTRFS info (device dm-2): enabling ssd optimizations
Jan 21 19:26:00 <***> kernel: BTRFS info (device dm-2): setting nodatacow, compression disabled
Jan 21 19:26:00 <***> kernel: BTRFS info (device dm-2): disk space caching is enabled
Jan 21 19:26:00 <***> kernel: BTRFS info (device dm-2): has skinny extents
Jan 21 19:29:17 <***> kernel: BTRFS warning (device dm-3): block group 82699091968 has wrong amount of free space
Jan 21 19:29:17 <***> kernel: BTRFS warning (device dm-3): failed to load free space cache for block group 82699091968, rebuilding it now
Jan 21 22:00:23 <***> kernel: BTRFS: device label BACKUP devid 1 transid 407 /dev/dm-8 scanned by systemd-udevd (26929)
Jan 21 22:00:23 <***> kernel: BTRFS info (device dm-8): use zstd compression, level 3
Jan 21 22:00:23 <***> kernel: BTRFS info (device dm-8): enabling auto defrag
Jan 21 22:00:23 <***> kernel: BTRFS info (device dm-8): disk space caching is enabled
Jan 21 22:00:23 <***> kernel: BTRFS info (device dm-8): has skinny extents
Jan 21 23:56:48 <***> kernel: btrfs_trim_block_group: enter bg start=4194304 start=4194304 end=12582912 minlen=512
Jan 21 23:56:48 <***> kernel: btrfs_trim_block_group: enter bg start=4194304 ret=0
Jan 21 23:56:48 <***> kernel: btrfs_trim_block_group: enter bg start=12582912 start=12582912 end=20971520 minlen=512
Jan 21 23:56:48 <***> kernel: btrfs_trim_block_group: enter bg start=12582912 ret=0
Jan 21 23:56:48 <***> kernel: btrfs_trim_block_group: enter bg start=20971520 start=20971520 end=1094713344 minlen=512
Jan 21 23:56:54 <***> kernel: btrfs_trim_block_group: enter bg start=20971520 ret=0
Jan 21 23:56:54 <***> kernel: btrfs_trim_block_group: enter bg start=1094713344 start=1094713344 end=2168455168 minlen=512
Jan 21 23:56:55 <***> kernel: btrfs_trim_block_group: enter bg start=1094713344 ret=0
Jan 21 23:56:55 <***> kernel: btrfs_trim_block_group: enter bg start=3242196992 start=3242196992 end=4315938816 minlen=512
Jan 21 23:56:58 <***> kernel: btrfs_trim_block_group: enter bg start=3242196992 ret=0
Jan 21 23:56:58 <***> kernel: btrfs_trim_block_group: enter bg start=4315938816 start=4315938816 end=5389680640 minlen=512
Jan 21 23:57:02 <***> kernel: btrfs_trim_block_group: enter bg start=4315938816 ret=0
Jan 21 23:57:02 <***> kernel: btrfs_trim_block_group: enter bg start=5389680640 start=5389680640 end=6463422464 minlen=512
Jan 21 23:57:08 <***> kernel: btrfs_trim_block_group: enter bg start=5389680640 ret=0
Jan 21 23:57:08 <***> kernel: btrfs_trim_block_group: enter bg start=6463422464 start=6463422464 end=7537164288 minlen=512
Jan 21 23:57:12 <***> kernel: btrfs_trim_block_group: enter bg start=6463422464 ret=0
Jan 21 23:57:12 <***> kernel: btrfs_trim_block_group: enter bg start=7537164288 start=7537164288 end=8610906112 minlen=512
Jan 21 23:57:15 <***> kernel: btrfs_trim_block_group: enter bg start=7537164288 ret=0
Jan 21 23:57:15 <***> kernel: btrfs_trim_block_group: enter bg start=8610906112 start=8610906112 end=9684647936 minlen=512
Jan 21 23:57:18 <***> kernel: btrfs_trim_block_group: enter bg start=8610906112 ret=0
Jan 21 23:57:18 <***> kernel: btrfs_trim_block_group: enter bg start=9684647936 start=9684647936 end=10758389760 minlen=512
Jan 21 23:57:21 <***> kernel: btrfs_trim_block_group: enter bg start=9684647936 ret=0
Jan 21 23:57:21 <***> kernel: btrfs_trim_block_group: enter bg start=10758389760 start=10758389760 end=11832131584 minlen=512
Jan 21 23:57:23 <***> kernel: btrfs_trim_block_group: enter bg start=10758389760 ret=0
Jan 21 23:57:23 <***> kernel: btrfs_trim_block_group: enter bg start=11832131584 start=11832131584 end=12905873408 minlen=512
Jan 21 23:57:23 <***> kernel: btrfs_trim_block_group: enter bg start=11832131584 ret=0
Jan 21 23:57:23 <***> kernel: btrfs_trim_block_group: enter bg start=12905873408 start=12905873408 end=13979615232 minlen=512
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=12905873408 ret=0
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=13979615232 start=13979615232 end=15053357056 minlen=512
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=13979615232 ret=0
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=15053357056 start=15053357056 end=16127098880 minlen=512
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=15053357056 ret=0
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=16127098880 start=16127098880 end=17200840704 minlen=512
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=16127098880 ret=0
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=17200840704 start=17200840704 end=18274582528 minlen=512
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=17200840704 ret=0
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=18274582528 start=18274582528 end=19348324352 minlen=512
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=18274582528 ret=0
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=19348324352 start=19348324352 end=20422066176 minlen=512
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=19348324352 ret=0
Jan 21 23:57:24 <***> kernel: btrfs_trim_block_group: enter bg start=20422066176 start=20422066176 end=21495808000 minlen=512
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=20422066176 ret=0
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=21495808000 start=21495808000 end=22569549824 minlen=512
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=21495808000 ret=0
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=22569549824 start=22569549824 end=23643291648 minlen=512
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=22569549824 ret=0
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=23643291648 start=23643291648 end=24717033472 minlen=512
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=23643291648 ret=0
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=24717033472 start=24717033472 end=25790775296 minlen=512
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=24717033472 ret=0
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=25790775296 start=25790775296 end=26864517120 minlen=512
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=25790775296 ret=0
Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg start=26864517120 start=26864517120 end=27938258944 minlen=512

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

On 2021/1/22 上午7:55, chainofflowers wrote:
> Hi Qu,
>
> it happened again. This time on my /home partition.
> I rebooted from an external disk and ran btrfs check without first going
> through btrfs scrub, and this is the output, no errors:
>
> ------------------------------------------
> [manjaro oc]# btrfs check /dev/mapper/OMO
> Opening filesystem to check…
> Checking filesystem on /dev/mapper/OMO
> UUID: 9362ac9d-c280-451d-9c8a-c09798e1c887
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 137523740672 bytes used, no error found
> total csum bytes: 113842816
> total tree bytes: 1740537856
> total fs tree bytes: 1444249600
> total extent tree bytes: 143835136
> btree space waste bytes: 325744995
> file data blocks allocated: 210346024960
>   referenced 172314374144
> ------------------------------------------
>
> Then, I rebooted from my internal disk, everything went well. I ran
> btrfs scrub and also got no errors.

So far so good.

>
> I have dumped the messages from journalctl, and the debug ones related
> to btrfs were only the ones from btrfs_trim_block_group - so, the issue
> is related to free space extents I guess?

Unfortunately, without the crash output, it can be anything.

>
> I have attached the logs.
> You can see that the last line:
>
> ------------------------------------------
> Jan 21 23:57:25 <***> kernel: btrfs_trim_block_group: enter bg
> start=26864517120 start=26864517120 end=27938258944 minlen=512
> ------------------------------------------
>
> does not have a second matching line with "ret=0", because the kernel
> stopped storing messages in the log. So, I guess the issue occurred
> while btrfs_trim_block_group was working on 26864517120..27938258944.

If you have some machine running 24x7, like a RPi, I would recommend to
setup netconsole to catch the full dying message to be extra safe.

Or setup kdump, to catch the dying message.

Personally speaking, netconsole would be much easier to setup though.

Currently with truncated journal it's really hard to say.

Thanks,
Qu
>
> Unfortunately I did not dump the output of dmesg directly in that
> moment, so all I could get is what was available in the journal after
> the reboot.
>
> In the log you can also see that some time before BTRFS detected that
> the space cache for dm-3 needed to be rebuilt:
> ------------------------------------------
> Jan 21 19:29:17 <***> kernel: BTRFS warning (device dm-3): block group
> 82699091968 has wrong amount of free space
> Jan 21 19:29:17 <***> kernel: BTRFS warning (device dm-3): failed to
> load free space cache for block group 82699091968, rebuilding it now
> ------------------------------------------
>
> Any hint about what I could do now?
>
> Thanks! :-)
>
>
>
> (c)
>

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

[-- Attachment #1: Type: text/plain, Size: 344 bytes --]

Hi Qu!

It happened again, and this time I've been able to dump the dmesg.
Also this time it happened on my home drive, please see the attached dump.

What can I do to fix it?
btrfs scrub reports no error, neither does brfs check.
I have also remounted the partition with -oclear_cache,space_cache, I
hoped that could fix it...

Thanks...

(c)

[-- Attachment #2: btrfs_error.log --]
[-- Type: text/x-log, Size: 122564 bytes --]

[  771.710488] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.710530] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.710587] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.710621] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.710665] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.710690] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.710704] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.710752] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.710775] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.710840] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.710896] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.710932] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.710985] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.711008] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.711052] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.711141] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.711205] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.711282] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.711315] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.711354] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.711400] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.711429] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.711449] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.711489] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.711528] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.711579] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.711595] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.711635] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.711686] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.711748] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.711807] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.711834] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.711865] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.711904] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.711962] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.712020] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.712035] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.712063] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.712123] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.712141] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.712193] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.712238] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.712269] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.712289] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.712322] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.712352] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.712435] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.712499] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.712534] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.712600] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.712625] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.712658] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.712729] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.712785] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.712828] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.712890] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.712909] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.712953] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.713014] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.713029] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.713045] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.713064] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.713096] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.713146] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.713200] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.713220] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.713283] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.713305] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.713367] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.713400] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.713454] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.713482] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.713557] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.713583] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.713660] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.713738] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.713815] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.713894] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.713940] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.713959] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.714017] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.714053] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.714097] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.714150] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.714163] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.714181] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.714192] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.714212] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.714267] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.714315] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.714331] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.714346] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.714397] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.714456] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.714478] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.714564] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.714578] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.714597] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.714618] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.714642] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.714659] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.714680] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.714692] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.714733] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.714768] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.714863] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.714957] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.715012] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.715040] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.715083] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.715140] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.715204] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.715244] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.715260] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.715309] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.715328] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.715400] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.715452] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.715510] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.715558] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.715575] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.715634] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.715658] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.715735] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.716103] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.716162] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.716504] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.716562] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.716832] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.716919] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.716945] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.717319] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.717421] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.718159] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.718416] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.718600] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.718844] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.719401] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.719685] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.720020] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.720466] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.720647] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.720675] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.721858] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.722889] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.722978] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.723101] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.723426] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.724108] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.724632] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.724708] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.725260] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.725335] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.725439] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.725986] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.726741] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.726876] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.727175] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.727570] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.727939] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.728461] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.729109] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.729269] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.729558] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.729596] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.729903] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.730056] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.730216] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.730444] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.730802] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.731386] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.732641] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.732714] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.733711] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.733879] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.734162] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.734346] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.735030] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.735370] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.735409] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.735901] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.736836] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.737073] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.737251] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.737955] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.739579] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.739696] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.739777] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.739832] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.741405] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.741544] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.741576] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.741695] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.741835] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.742639] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.742654] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.744030] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.744123] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.744415] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.745244] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.746028] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.746593] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.746658] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.746754] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.747882] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.748103] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.748703] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.749444] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.750446] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.750677] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.751133] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.751189] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.751414] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.753086] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.753150] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.753231] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.753954] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.754065] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.754305] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.755960] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.756163] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.756430] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.756804] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.756868] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.757232] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.757883] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.758767] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.759196] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.759385] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.761444] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.761554] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.761745] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.761898] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.763324] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.764097] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.764487] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.765107] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.765701] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.766176] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.766191] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.767557] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.769063] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.769482] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.769497] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.769923] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.769998] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.770047] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.770119] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.770363] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.770889] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.771391] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.771645] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.771908] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.771979] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.772341] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.774102] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.774152] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.774668] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.774683] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.775237] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.775685] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.775878] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.775951] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.776899] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.777412] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.777621] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.777867] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.778823] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.779501] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.779902] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.782026] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.783046] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.784466] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.787817] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.787834] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.788306] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.788320] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.790374] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.790894] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.790909] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.791544] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.791857] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.792223] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.792861] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.793572] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.794162] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.794768] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.795448] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.796643] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.798025] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.798041] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.798988] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.799244] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.800316] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.800622] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.800682] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.800698] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.800758] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.801408] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.801423] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.802803] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.803032] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.803525] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.804989] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.805102] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.805626] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.806137] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.806802] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.809208] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.809991] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.810458] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.811343] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.811869] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.812906] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.813485] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.813502] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.815410] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.815429] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.818343] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.818361] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.819200] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.819364] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.820218] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.820772] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.821896] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.822760] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.823403] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.824757] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.825007] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.825408] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.827449] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.827577] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.828851] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.829452] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.831838] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.832266] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.832281] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.832709] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.832976] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.833048] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.833079] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.833127] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.833163] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.833203] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.833359] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.833470] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.833504] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.833590] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.833670] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.833708] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.833726] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.833817] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.833834] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.833907] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.833946] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.833997] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.834064] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.834082] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.834168] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.834226] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.834256] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.834297] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.834375] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.834392] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.834416] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.834499] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.834541] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.834559] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.834579] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.834619] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.834679] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.834708] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.834753] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.834815] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.834844] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.834887] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.834910] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.834938] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.834982] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.835046] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.835070] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.835120] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.835143] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.835168] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.835182] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.835209] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.835227] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.835320] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.835382] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.835411] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.835637] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.835702] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.835789] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.835948] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.835985] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.836153] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.836312] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.836528] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.836555] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.836792] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.836969] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.837064] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.837126] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.837238] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.837259] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.837345] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.837422] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.837436] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.837507] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.837648] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.837718] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.837732] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.837911] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.838006] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.838054] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.838119] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.840056] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.840147] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.840173] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.840187] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.840274] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.840630] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.840651] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.840683] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.840690] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.840704] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.840713] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.840727] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.840808] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.840819] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.840827] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.840835] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.840843] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.840850] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.840858] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.840865] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.840873] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.840880] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.840903] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.840911] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.840919] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.840927] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.840937] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.840945] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.840952] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.840961] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.840979] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.840988] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.840997] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.841005] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.841021] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.841032] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.841040] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.841056] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.841066] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.841081] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.841110] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.841121] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.841142] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.841159] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.841195] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.841234] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.841298] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.841312] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.841353] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.841394] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.841420] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.841459] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.841473] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.841488] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.841507] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.841516] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.841555] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.841565] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.841581] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.841590] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.841601] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.841642] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.841651] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.841664] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.841678] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.841686] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.841712] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.841726] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.841747] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.841762] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.841771] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.841783] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.841820] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.841832] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.841844] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.841861] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.841869] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.841877] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.841889] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.841901] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.841921] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.841934] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.841957] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.841974] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.841983] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.841995] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.842012] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.842049] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.842065] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.842073] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.842085] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.842097] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.842122] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.842133] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.842150] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.842162] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.842175] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.842187] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.842229] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.842243] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.842284] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.842292] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.842327] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.842344] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.842360] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.842389] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.842405] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.842431] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.842442] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.842459] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.842474] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.842488] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.842506] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.842516] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.842531] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.842546] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.842584] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.842596] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.842611] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.842621] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.842640] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.842651] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.842668] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.842690] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.842700] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.842710] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.842718] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.842732] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.842783] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.842949] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.842959] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.843139] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.843727] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.843872] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.843874] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.843996] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.844773] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.845570] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.847032] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.847301] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.847786] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.847963] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.848712] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.849317] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.857587] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.857615] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.857718] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.857814] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.857885] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.857944] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.858029] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.858173] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.858248] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.858258] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.858265] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.858271] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.858278] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.858284] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.858292] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.858303] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.858310] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.858316] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.858323] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.858329] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.858340] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.858351] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.858358] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.858364] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.858402] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.858416] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.858425] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.858436] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.858442] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.858449] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.858455] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.858461] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.858468] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.858474] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.858480] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.858487] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.858494] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.858506] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.858515] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.858521] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.858553] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.858565] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.858574] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.858581] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.858587] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.858593] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.858599] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.858606] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.858612] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.858618] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.858624] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.858631] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.858637] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.858643] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.858649] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.858655] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.859508] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.859853] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.860156] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.861871] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.862000] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.864366] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.864972] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.865124] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.866754] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.875541] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.875628] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.877119] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.877178] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.878286] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.878444] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.879202] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.879238] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.879691] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.879895] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.880173] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.881407] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.881468] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.882706] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.884430] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.887019] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.895844] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.895888] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.895997] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.896158] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.896287] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.896315] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.896355] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.896466] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.896528] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.896598] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.896623] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.896690] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.896749] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.896789] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.896843] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.896874] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.897052] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.897074] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.897086] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.897160] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.897182] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.897318] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.897349] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.897368] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.897468] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.897620] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.897659] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.897687] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.897779] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.897821] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.897935] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.898335] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.898397] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.898553] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.898634] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.898689] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.898712] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.898849] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.898928] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.899008] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.899088] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.899245] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.899322] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.899364] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.899404] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.900474] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.900507] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.900518] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.900538] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.900549] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.900569] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.900579] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.900599] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.900609] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.900629] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.900639] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.900654] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.900665] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.900685] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.900696] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.900748] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.900951] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.901107] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.901181] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.901457] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.902331] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.902345] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.902917] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.902972] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.902995] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.903012] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.903025] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.903035] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.903051] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.903068] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.903078] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.903111] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.903121] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.903393] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.904977] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.905047] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.905067] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.905081] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.905102] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.905115] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.905127] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.905139] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.905154] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.905166] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.905180] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.905194] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.905205] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.905297] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.905448] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.905449] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.905538] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.905694] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.905778] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.905921] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.905923] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.905999] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.906005] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.906401] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.906919] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.907474] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.909011] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.910096] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.914264] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.914918] systemd-journald[381]: /var/log/journal/dd4394eec244453fbacba5f15e14141e/system.journal: IO error, rotating.
[  771.916324] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.916389] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.920095] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.920302] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.920407] BTRFS: error (device dm-3) in btrfs_commit_transaction:2376: errno=-5 IO failure (Error while writing out transaction)
[  771.920414] BTRFS info (device dm-3): forced readonly
[  771.920417] BTRFS warning (device dm-3): Skipping commit of aborted transaction.
[  771.920418] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.920422] BTRFS: error (device dm-3) in cleanup_transaction:1941: errno=-5 IO failure
[  771.920876] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.925074] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.925171] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.925801] systemd-journald[381]: Failed to create new system journal: Read-only file system
[  771.925833] systemd-journald[381]: Failed to rotate /var/log/journal/dd4394eec244453fbacba5f15e14141e/user-1000.journal: Read-only file system
[  771.930792] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.930928] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.931029] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.931042] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.931053] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.931063] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.931073] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.931083] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.931093] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.931103] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.931113] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.931123] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.931134] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.931142] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.931153] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.931163] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.931173] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.931218] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.931233] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.931244] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.931254] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.931265] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.931274] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.931283] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.931294] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.931312] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.931322] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.931333] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.931342] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.931353] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.931363] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.931372] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.931380] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.931423] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.931436] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.931447] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.931458] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.931468] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.931479] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.931491] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.931501] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.931515] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.931525] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.931535] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.931545] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.931556] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.931566] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.931577] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.931586] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.931633] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.931645] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.931655] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.931666] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.931675] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.931686] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.931696] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.931706] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.931715] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.931726] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.931736] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.931747] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.931758] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  771.931767] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  771.931777] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  771.931793] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.931817] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.931835] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.931836] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.932001] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.932084] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.932104] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.932267] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.932365] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.932367] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.932622] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.932877] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.933131] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.933295] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.933387] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.933400] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.933565] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.933660] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.933666] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.933831] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.933918] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.933933] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.934098] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.934196] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.934199] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.934536] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.934555] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.934568] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.934589] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.934650] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.934669] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.934681] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  771.934693] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  771.934705] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  771.934716] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  771.934728] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  771.934739] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  771.934750] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  771.934762] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  771.934774] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  771.934785] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  771.934797] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  771.934808] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  771.934820] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  771.934835] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  771.934875] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  771.934885] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  771.934894] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  771.934910] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  771.934926] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  771.934941] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  771.934959] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  771.934974] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  771.934991] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.935006] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.935022] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.935038] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  771.935054] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  771.935528] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  771.940192] systemd-journald[381]: /dev/kmsg buffer overrun, some messages lost.
[  771.945560] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  771.945636] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  771.950056] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  771.950900] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  771.954308] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  772.817534] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  773.837334] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  773.837493] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  773.837598] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  773.837699] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  773.837839] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  773.837937] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  773.838020] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  773.838734] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  773.838923] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  773.839083] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  773.839190] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  773.839445] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  773.839653] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  773.839767] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  773.839876] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  773.839981] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  773.840086] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  773.840285] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  773.840547] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  773.840742] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  774.793659] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  774.794378] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  774.794595] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  774.795843] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  774.863744] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  774.863879] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  774.887741] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  774.887890] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  774.902379] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  774.902948] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  774.903219] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  774.903353] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  774.903435] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  774.903619] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  774.903986] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  774.904272] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  774.904408] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  774.904598] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  774.905472] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  777.531321] kauditd_printk_skb: 24 callbacks suppressed
[  780.129334] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  780.129540] print_req_error: 4763 callbacks suppressed
[  780.129541] blk_update_request: I/O error, dev sdc, sector 230776800 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.129715] btrfs_dev_stat_print_on_error: 4811 callbacks suppressed
[  780.129717] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 92, flush 0, corrupt 0, gen 0
[  780.129909] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  780.130144] blk_update_request: I/O error, dev sdc, sector 230776800 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.130252] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 93, flush 0, corrupt 0, gen 0
[  780.130519] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  780.130711] blk_update_request: I/O error, dev sdc, sector 230776800 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.130814] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 94, flush 0, corrupt 0, gen 0
[  780.131398] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  780.131605] blk_update_request: I/O error, dev sdc, sector 230776800 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.131710] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 95, flush 0, corrupt 0, gen 0
[  780.131988] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  780.132210] blk_update_request: I/O error, dev sdc, sector 346395072 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.132314] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 96, flush 0, corrupt 0, gen 0
[  780.132600] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  780.132789] blk_update_request: I/O error, dev sdc, sector 230776800 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.132909] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 97, flush 0, corrupt 0, gen 0
[  780.133101] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  780.133329] blk_update_request: I/O error, dev sdc, sector 230776800 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.133497] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 98, flush 0, corrupt 0, gen 0
[  780.133663] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  780.133840] blk_update_request: I/O error, dev sdc, sector 230776800 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.133953] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 99, flush 0, corrupt 0, gen 0
[  780.137622] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  780.141155] blk_update_request: I/O error, dev sdc, sector 346395072 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.144828] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 100, flush 0, corrupt 0, gen 0
[  780.148704] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  780.153188] blk_update_request: I/O error, dev sdc, sector 346395072 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  780.157383] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4730, rd 101, flush 0, corrupt 0, gen 0
[  780.251725] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  780.263834] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  780.269445] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  780.282070] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  780.287687] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  780.294056] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  780.300512] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  780.304346] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  780.308136] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  780.312361] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  780.315750] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  780.319684] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  780.324010] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  780.330465] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  780.339199] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  780.346294] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  780.352903] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  780.358975] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  780.368066] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  780.375362] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  780.380704] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  780.384891] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  780.390468] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  780.395668] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  780.400222] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  780.407228] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  780.411947] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  780.416601] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  780.422605] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  780.426600] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[  780.431167] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  780.727581] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  780.731536] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  780.769379] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  780.776270] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  780.782321] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  780.788429] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  780.848186] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  780.856979] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  780.868208] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  780.872293] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  780.885137] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  780.890074] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  780.895599] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  780.902187] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  780.908122] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  780.914657] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  780.968965] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  780.975216] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  781.106952] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  781.111217] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  781.744274] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  781.812357] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  781.816228] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  781.820423] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  781.840382] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  781.845510] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  781.867744] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  781.875318] btrfs_lookup_bio_sums: 1 callbacks suppressed
[  781.875322] BTRFS info (device dm-3): no csum found for inode 19081943 start 1179648
[  781.875363] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  781.883143] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  781.889918] BTRFS info (device dm-3): no csum found for inode 19081943 start 1179648
[  781.889959] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  781.895847] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  781.924197] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  781.931555] BTRFS info (device dm-3): no csum found for inode 19081943 start 1179648
[  781.931597] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  781.939546] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  781.945833] BTRFS info (device dm-3): no csum found for inode 19081943 start 1179648
[  781.945872] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  781.951474] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  786.766371] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  786.772117] print_req_error: 67 callbacks suppressed
[  786.772120] blk_update_request: I/O error, dev sdc, sector 222703088 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
[  786.778903] btrfs_dev_stat_print_on_error: 53 callbacks suppressed
[  786.778907] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4747, rd 138, flush 0, corrupt 0, gen 0
[  786.786684] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  786.793005] blk_update_request: I/O error, dev sdc, sector 222703088 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
[  786.799584] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4747, rd 139, flush 0, corrupt 0, gen 0
[  788.044429] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  788.048534] blk_update_request: I/O error, dev sdc, sector 184094304 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  788.054028] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4747, rd 140, flush 0, corrupt 0, gen 0
[  788.058464] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  788.062803] blk_update_request: I/O error, dev sdc, sector 184094304 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  788.067205] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4747, rd 141, flush 0, corrupt 0, gen 0
[  793.261282] ata7: softreset failed (1st FIS failed)
[  793.266990] ata7: limiting SATA link speed to 3.0 Gbps
[  793.266992] ata7: hard resetting link
[  798.268514] ata7: softreset failed (1st FIS failed)
[  798.274056] ata7: reset failed, giving up
[  798.279756] ata7.00: disabled
[  798.279812] ata7: EH complete
[  798.279828] scsi_io_completion_action: 85 callbacks suppressed
[  798.279833] sd 6:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[  798.279837] sd 6:0:0:0: [sdb] tag#1 CDB: Write(10) 2a 00 2d 22 52 80 00 00 80 00
[  798.279840] blk_update_request: I/O error, dev sdb, sector 757224064 op 0x1:(WRITE) flags 0x800 phys_seg 9 prio class 0
[  798.285431] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[  798.290183] sd 6:0:0:0: [sdb] tag#11 access beyond end of device
[  798.290997] sd 6:0:0:0: [sdb] tag#3 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[  798.295176] blk_update_request: I/O error, dev sdb, sector 3907028992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[  798.295177] sd 6:0:0:0: [sdb] tag#3 CDB: Write(10) 2a 00 2d 22 53 00 00 00 80 00
[  798.295181] blk_update_request: I/O error, dev sdb, sector 757224192 op 0x1:(WRITE) flags 0x800 phys_seg 5 prio class 0
[  798.299652] sd 6:0:0:0: [sdb] tag#12 access beyond end of device
[  798.305002] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[  798.309542] blk_update_request: I/O error, dev sdb, sector 3907028992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  798.309543] Buffer I/O error on dev sdb, logical block 488378624, async page read
[  798.327660] sd 6:0:0:0: [sdb] tag#30 access beyond end of device
[  798.331947] blk_update_request: I/O error, dev sdb, sector 757224064 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 0
[  798.336467] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
[  798.340539] sd 6:0:0:0: [sdb] tag#0 access beyond end of device
[  798.345108] blk_update_request: I/O error, dev sdb, sector 757224192 op 0x1:(WRITE) flags 0x800 phys_seg 80 prio class 0
[  798.349650] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
[  798.353962] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
[  798.358406] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
[  798.362510] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
[  798.367004] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[  798.371182] sd 6:0:0:0: [sdb] tag#2 access beyond end of device
[  798.375244] blk_update_request: I/O error, dev sdb, sector 757224064 op 0x1:(WRITE) flags 0x800 phys_seg 32 prio class 0
[  798.379415] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
[  798.383613] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
[  798.387803] sd 6:0:0:0: [sdb] tag#4 access beyond end of device
[  798.392010] blk_update_request: I/O error, dev sdb, sector 757224320 op 0x1:(WRITE) flags 0x800 phys_seg 63 prio class 0
[  798.397163] sd 6:0:0:0: [sdb] tag#5 access beyond end of device
[  798.401216] blk_update_request: I/O error, dev sdb, sector 97814144 op 0x1:(WRITE) flags 0x1800 phys_seg 32 prio class 0
[  798.405364] sd 6:0:0:0: [sdb] tag#6 access beyond end of device
[  798.409563] blk_update_request: I/O error, dev sdb, sector 97814400 op 0x1:(WRITE) flags 0x1800 phys_seg 40 prio class 0
[  798.413817] sd 6:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[  798.413820] sd 6:0:0:0: [sdb] tag#10 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[  798.415448] BTRFS warning (device dm-5): chunk 1104150528 missing 1 devices, max tolerance is 0 for writable mount
[  798.415450] BTRFS: error (device dm-5) in write_all_supers:3845: errno=-5 IO failure (errors while submitting device barriers.)
[  798.419382] BTRFS info (device dm-5): forced readonly
[  798.419384] BTRFS warning (device dm-5): Skipping commit of aborted transaction.
[  798.419387] BTRFS: error (device dm-5) in cleanup_transaction:1941: errno=-5 IO failure
[  817.373624] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  817.377747] print_req_error: 1 callbacks suppressed
[  817.377749] blk_update_request: I/O error, dev sdc, sector 220065696 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
[  817.381943] btrfs_dev_stat_print_on_error: 10 callbacks suppressed
[  817.381946] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4748, rd 141, flush 0, corrupt 0, gen 0
[  869.759176] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  869.763236] blk_update_request: I/O error, dev sdc, sector 203049640 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  869.767312] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4748, rd 142, flush 0, corrupt 0, gen 0
[  918.275654] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  918.279358] blk_update_request: I/O error, dev sdc, sector 217107352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  918.283270] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4748, rd 143, flush 0, corrupt 0, gen 0
[  918.287044] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  918.290990] blk_update_request: I/O error, dev sdc, sector 217107352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  918.294740] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4748, rd 144, flush 0, corrupt 0, gen 0
[  948.165247] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  948.172497] blk_update_request: I/O error, dev sdc, sector 371644064 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  948.179791] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4748, rd 145, flush 0, corrupt 0, gen 0
[  948.187015] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[  948.194560] blk_update_request: I/O error, dev sdc, sector 203070592 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  948.201735] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4748, rd 146, flush 0, corrupt 0, gen 0
[  948.209473] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[  948.216882] blk_update_request: I/O error, dev sdc, sector 371644064 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[  948.224781] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4748, rd 147, flush 0, corrupt 0, gen 0
[  949.009801] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  949.017481] blk_update_request: I/O error, dev sdc, sector 220065704 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
[  949.024952] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 147, flush 0, corrupt 0, gen 0
[  952.081804] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  952.089599] blk_update_request: I/O error, dev sdc, sector 219679872 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  952.097224] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 148, flush 0, corrupt 0, gen 0
[  952.105440] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[  952.112994] blk_update_request: I/O error, dev sdc, sector 219679872 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  952.120277] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 149, flush 0, corrupt 0, gen 0
[  953.203409] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  953.204901] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[  953.209185] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  953.209189] blk_update_request: I/O error, dev sdc, sector 336333320 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[  953.209281] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[  953.209284] blk_update_request: I/O error, dev sdc, sector 336333320 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  953.209289] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 150, flush 0, corrupt 0, gen 0
[  953.209451] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[  953.209454] blk_update_request: I/O error, dev sdc, sector 352191936 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  953.209458] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 151, flush 0, corrupt 0, gen 0
[  953.211079] blk_update_request: I/O error, dev sdc, sector 352191808 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  953.218686] blk_update_request: I/O error, dev sdc, sector 352191816 op 0x0:(READ) flags 0x0 phys_seg 5 prio class 0
[  953.226158] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 152, flush 0, corrupt 0, gen 0
[  953.226179] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[  953.226182] blk_update_request: I/O error, dev sdc, sector 352191936 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  953.226186] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 153, flush 0, corrupt 0, gen 0
[  953.233796] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 154, flush 0, corrupt 0, gen 0
[  953.301215] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[  953.336593] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  953.337365] blk_update_request: I/O error, dev sdc, sector 352191944 op 0x0:(READ) flags 0x0 phys_seg 5 prio class 0
[  953.344265] blk_update_request: I/O error, dev sdc, sector 352191808 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  953.344350] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 155, flush 0, corrupt 0, gen 0
[  953.350838] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  953.351625] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 156, flush 0, corrupt 0, gen 0
[  953.356688] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[  953.356691] blk_update_request: I/O error, dev sdc, sector 352858392 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
[  953.356695] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 157, flush 0, corrupt 0, gen 0
[  953.356767] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[  953.356769] blk_update_request: I/O error, dev sdc, sector 352858392 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0
[  953.356772] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 158, flush 0, corrupt 0, gen 0
[  953.358898] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 159, flush 0, corrupt 0, gen 0
[  953.369344] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[  953.373465] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[  953.376947] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[  953.379253] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[  953.379707] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[  953.384611] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[  953.392996] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[  953.443439] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[  953.446286] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[  953.452022] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[  953.452026] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[  953.470224] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[  953.476810] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[  953.478919] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[  953.486755] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[  953.493344] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[  953.494438] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[  953.494662] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[  953.498240] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[  953.548027] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[  953.554966] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[  953.561259] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1068.171396] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1068.177879] print_req_error: 23 callbacks suppressed
[ 1068.177881] blk_update_request: I/O error, dev sdc, sector 371644064 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1068.184060] btrfs_dev_stat_print_on_error: 23 callbacks suppressed
[ 1068.184063] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 183, flush 0, corrupt 0, gen 0
[ 1068.190491] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1068.196726] blk_update_request: I/O error, dev sdc, sector 203070592 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1068.203275] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 184, flush 0, corrupt 0, gen 0
[ 1068.210025] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1068.216016] blk_update_request: I/O error, dev sdc, sector 371644064 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1068.222299] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 185, flush 0, corrupt 0, gen 0
[ 1072.476152] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1072.482583] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1072.488537] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 186, flush 0, corrupt 0, gen 0
[ 1072.494562] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1072.501247] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1072.507677] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 187, flush 0, corrupt 0, gen 0
[ 1072.635774] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1072.641883] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1072.648310] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 188, flush 0, corrupt 0, gen 0
[ 1072.655078] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1072.661621] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1072.668481] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 189, flush 0, corrupt 0, gen 0
[ 1072.674820] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1072.681451] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1072.688075] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 190, flush 0, corrupt 0, gen 0
[ 1072.694535] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1072.700809] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1072.707325] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 191, flush 0, corrupt 0, gen 0
[ 1072.714080] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1072.720651] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1072.727002] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 192, flush 0, corrupt 0, gen 0
[ 1072.735847] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1072.742617] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1072.762344] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1072.768685] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1072.782360] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1072.789348] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1072.796105] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1072.802993] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1072.809996] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[ 1072.816881] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1072.823626] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1072.833573] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1072.840285] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1072.846717] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1072.853251] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1072.859845] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1072.867679] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1072.873853] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1072.880009] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1072.886520] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1072.892869] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1072.932561] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1072.949033] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1072.958305] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1072.964886] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1072.971575] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1072.979181] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1072.986384] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1072.992815] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1072.999116] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1073.005746] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1073.013144] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1073.023178] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1073.029514] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1073.035653] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1073.042277] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1073.049009] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1073.055898] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1073.062775] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1073.074060] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1073.080719] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1073.087574] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1073.094124] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1073.100243] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1073.106485] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1073.113707] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1073.120246] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1073.126381] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1073.132829] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1073.138586] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1073.145336] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1073.151005] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1073.157793] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1073.163445] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1073.169406] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1073.175263] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1073.180771] print_req_error: 55 callbacks suppressed
[ 1073.180855] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.186339] btrfs_dev_stat_print_on_error: 55 callbacks suppressed
[ 1073.186343] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 248, flush 0, corrupt 0, gen 0
[ 1073.191978] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1073.197365] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.202803] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 249, flush 0, corrupt 0, gen 0
[ 1073.208073] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1073.213759] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.219458] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 250, flush 0, corrupt 0, gen 0
[ 1073.225196] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1073.229923] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.235745] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 251, flush 0, corrupt 0, gen 0
[ 1073.241893] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1073.247572] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.253478] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 252, flush 0, corrupt 0, gen 0
[ 1073.259362] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1073.265101] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.270618] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 253, flush 0, corrupt 0, gen 0
[ 1073.276654] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1073.282584] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.288366] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 254, flush 0, corrupt 0, gen 0
[ 1073.294334] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[ 1073.300304] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.306603] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 255, flush 0, corrupt 0, gen 0
[ 1073.311697] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1073.315878] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.320452] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 256, flush 0, corrupt 0, gen 0
[ 1073.324837] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[ 1073.329157] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1073.333754] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 257, flush 0, corrupt 0, gen 0
[ 1073.337951] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1073.343633] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1073.349967] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1073.354768] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1073.359293] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1073.363761] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1073.371275] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1073.377983] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1073.384858] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1073.391323] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1073.397685] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1073.404512] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1073.410889] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1073.417069] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1073.421606] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1073.426049] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1073.430237] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1073.434367] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1073.440724] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1073.445190] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1073.449347] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1073.456053] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1073.460440] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1073.465781] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1073.471829] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1073.478929] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1073.485126] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1073.491964] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1073.497935] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1073.503777] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1073.508956] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1073.514811] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1073.522250] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1073.528316] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1073.534439] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1073.540717] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1073.545797] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1073.552694] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1073.558493] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1073.564326] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1073.570925] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1073.577068] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1073.581788] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1073.586305] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1073.593209] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1073.597792] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1073.602454] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1073.608494] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[ 1073.613203] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1073.617123] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1073.622007] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1073.627075] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[ 1073.630937] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1073.635465] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1073.640149] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1073.644350] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1073.650584] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1073.655120] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1073.659502] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1073.664816] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1073.670070] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1073.676575] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1073.681844] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1073.686379] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1073.690992] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1073.694327] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1073.699299] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1073.704155] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1073.709189] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1073.713551] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1073.718421] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1073.723499] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1073.728964] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1073.735083] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1073.739825] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1073.743547] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1073.748806] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1073.754072] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1073.759566] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1073.764183] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1073.770070] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1073.774514] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1073.781261] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1073.785559] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1073.790914] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1073.795778] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1073.801148] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1073.807352] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1073.812359] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[ 1073.817600] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1073.822615] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[ 1073.827757] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1073.833326] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1073.837397] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1073.842414] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1073.847463] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1073.852785] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1073.858132] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1073.862891] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1073.867247] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1073.871806] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1073.876736] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1073.882118] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1073.888526] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1073.893467] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1073.898773] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1073.904052] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1073.909278] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1073.914720] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1073.919758] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1073.924874] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1073.929942] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1073.934495] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1073.939040] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1073.943523] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1073.950047] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1073.954908] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1073.960253] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1073.965475] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1073.970126] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1073.975448] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1073.980514] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1073.985539] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1073.990506] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1073.995589] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1074.000571] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1074.005987] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1074.011094] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1074.016406] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1074.022581] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1074.027866] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1074.033544] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1074.038999] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1074.046295] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1074.051562] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1074.057168] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1074.062310] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1074.068134] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1074.073059] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1074.078293] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1074.083714] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1074.088921] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1074.094411] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1074.099401] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1074.105171] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[ 1074.111181] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1074.116350] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1074.121461] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1074.126697] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1074.132865] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[ 1074.137939] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1074.143032] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1074.148314] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1074.153608] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1074.158689] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1074.164070] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1074.169886] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1074.174917] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1074.180135] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1074.185324] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1074.190206] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1074.195375] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1074.200543] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1074.206205] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1074.210914] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1074.215894] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1074.220961] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1074.225942] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1074.233735] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1074.239060] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1074.244487] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1074.249751] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1074.254966] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1074.260917] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1074.266148] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1074.271327] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1074.276509] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1074.281545] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1074.291800] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1074.296616] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1074.302151] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1074.307163] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1074.312245] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1084.547610] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1084.552885] print_req_error: 183 callbacks suppressed
[ 1084.552887] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.558126] btrfs_dev_stat_print_on_error: 183 callbacks suppressed
[ 1084.558129] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 441, flush 0, corrupt 0, gen 0
[ 1084.564888] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1084.570106] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.575347] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 442, flush 0, corrupt 0, gen 0
[ 1084.585982] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[ 1084.591609] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.597412] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 443, flush 0, corrupt 0, gen 0
[ 1084.602918] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1084.608928] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.614508] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 444, flush 0, corrupt 0, gen 0
[ 1084.622877] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1084.628428] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.633682] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 445, flush 0, corrupt 0, gen 0
[ 1084.639174] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1084.644946] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.650861] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 446, flush 0, corrupt 0, gen 0
[ 1084.657510] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[ 1084.663460] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.669284] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 447, flush 0, corrupt 0, gen 0
[ 1084.675188] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1084.680949] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.686757] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 448, flush 0, corrupt 0, gen 0
[ 1084.692964] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1084.698859] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.705103] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 449, flush 0, corrupt 0, gen 0
[ 1084.710735] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1084.716924] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1084.723089] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 450, flush 0, corrupt 0, gen 0
[ 1084.729718] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1084.736046] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1084.742219] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1084.749796] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1084.755922] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1084.763467] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1084.769998] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1084.776654] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1084.783773] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1084.790390] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1084.796685] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1084.802960] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1084.810186] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1084.818467] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1084.824803] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1084.831650] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1084.838134] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1084.844549] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1084.963203] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1084.969988] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1084.987830] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1084.994525] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1085.001363] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[ 1085.007644] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1085.014006] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1085.026489] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1085.033079] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1085.040891] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1085.047237] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1085.054046] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1085.059907] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1085.066925] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1085.073366] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1085.079870] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1085.086827] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1085.093753] sd 7:0:0:0: [sdc] tag#26 access beyond end of device
[ 1085.100579] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1085.107114] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1085.113725] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1085.120097] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1085.126379] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1085.132692] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1085.139081] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1085.146470] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1085.152624] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1085.159023] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1085.164933] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1085.171298] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1085.178194] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1085.184392] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1085.190820] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1085.196850] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1085.203279] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1085.208651] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1085.214612] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1085.220284] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1085.226199] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1085.232584] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1085.238428] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1085.244167] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1085.249702] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1085.254621] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1085.260200] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1085.265493] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1085.270885] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1085.276291] sd 7:0:0:0: [sdc] tag#24 access beyond end of device
[ 1085.281681] sd 7:0:0:0: [sdc] tag#25 access beyond end of device
[ 1085.286990] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1085.292079] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1085.373049] sd 7:0:0:0: [sdc] tag#8 access beyond end of device
[ 1085.378076] sd 7:0:0:0: [sdc] tag#9 access beyond end of device
[ 1085.383771] sd 7:0:0:0: [sdc] tag#10 access beyond end of device
[ 1085.388889] sd 7:0:0:0: [sdc] tag#27 access beyond end of device
[ 1085.394379] sd 7:0:0:0: [sdc] tag#28 access beyond end of device
[ 1085.400119] sd 7:0:0:0: [sdc] tag#7 access beyond end of device
[ 1085.405906] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1085.411001] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1085.416294] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1085.421488] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1085.426533] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1085.431703] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1085.436933] sd 7:0:0:0: [sdc] tag#11 access beyond end of device
[ 1085.442627] sd 7:0:0:0: [sdc] tag#12 access beyond end of device
[ 1085.447833] sd 7:0:0:0: [sdc] tag#13 access beyond end of device
[ 1085.452916] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1085.457683] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1085.462723] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1085.468653] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1085.473663] sd 7:0:0:0: [sdc] tag#29 access beyond end of device
[ 1085.478816] sd 7:0:0:0: [sdc] tag#30 access beyond end of device
[ 1085.483988] sd 7:0:0:0: [sdc] tag#0 access beyond end of device
[ 1085.489056] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1085.494513] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1085.499625] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1085.504550] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1085.509328] sd 7:0:0:0: [sdc] tag#1 access beyond end of device
[ 1085.514560] sd 7:0:0:0: [sdc] tag#2 access beyond end of device
[ 1085.917555] sd 7:0:0:0: [sdc] tag#22 access beyond end of device
[ 1085.922903] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1085.928028] sd 7:0:0:0: [sdc] tag#23 access beyond end of device
[ 1086.032997] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1086.051987] sd 7:0:0:0: [sdc] tag#3 access beyond end of device
[ 1086.063824] sd 7:0:0:0: [sdc] tag#4 access beyond end of device
[ 1086.076722] sd 7:0:0:0: [sdc] tag#5 access beyond end of device
[ 1086.095344] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1086.112788] sd 7:0:0:0: [sdc] tag#6 access beyond end of device
[ 1188.832794] sd 7:0:0:0: [sdc] tag#14 access beyond end of device
[ 1188.837882] print_req_error: 106 callbacks suppressed
[ 1188.837885] blk_update_request: I/O error, dev sdc, sector 352487904 op 0x0:(READ) flags 0x84700 phys_seg 168 prio class 0
[ 1188.843077] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1188.848382] blk_update_request: I/O error, dev sdc, sector 352489248 op 0x0:(READ) flags 0x80700 phys_seg 88 prio class 0
[ 1188.853907] sd 7:0:0:0: [sdc] tag#16 access beyond end of device
[ 1188.859579] blk_update_request: I/O error, dev sdc, sector 352490088 op 0x0:(READ) flags 0x84700 phys_seg 157 prio class 0
[ 1188.864990] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1188.870788] blk_update_request: I/O error, dev sdc, sector 352492648 op 0x0:(READ) flags 0x84700 phys_seg 99 prio class 0
[ 1188.876025] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1188.881527] blk_update_request: I/O error, dev sdc, sector 352494344 op 0x0:(READ) flags 0x80700 phys_seg 92 prio class 0
[ 1188.887385] sd 7:0:0:0: [sdc] tag#19 access beyond end of device
[ 1188.893126] blk_update_request: I/O error, dev sdc, sector 352487904 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1188.899159] btrfs_dev_stat_print_on_error: 106 callbacks suppressed
[ 1188.899162] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 557, flush 0, corrupt 0, gen 0
[ 1188.905456] sd 7:0:0:0: [sdc] tag#31 access beyond end of device
[ 1188.911324] blk_update_request: I/O error, dev sdc, sector 352487904 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1188.917329] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 558, flush 0, corrupt 0, gen 0
[ 1188.944702] sd 7:0:0:0: [sdc] tag#20 access beyond end of device
[ 1188.951160] blk_update_request: I/O error, dev sdc, sector 352487904 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1188.957485] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 559, flush 0, corrupt 0, gen 0
[ 1188.964137] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1188.970445] blk_update_request: I/O error, dev sdc, sector 352487904 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1188.976758] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 560, flush 0, corrupt 0, gen 0
[ 1232.404774] sd 7:0:0:0: [sdc] tag#30 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 1232.404779] sd 7:0:0:0: [sdc] tag#30 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 1232.466693] sd 6:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 1232.466697] sd 6:0:0:0: [sdb] tag#1 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 1356.067319] sd 7:0:0:0: [sdc] tag#21 access beyond end of device
[ 1356.073702] blk_update_request: I/O error, dev sdc, sector 491744 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1356.080151] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 561, flush 0, corrupt 0, gen 0
[ 1614.398936] uvcvideo: Found UVC 1.00 device HD Pro Webcam C920 (046d:082d)
[ 1619.838144] kauditd_printk_skb: 1 callbacks suppressed
[ 1624.028278] sd 7:0:0:0: [sdc] tag#15 access beyond end of device
[ 1624.034877] blk_update_request: I/O error, dev sdc, sector 216970512 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[ 1624.041012] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 562, flush 0, corrupt 0, gen 0
[ 1624.119043] sd 6:0:0:0: [sdb] tag#14 access beyond end of device
[ 1624.125494] blk_update_request: I/O error, dev sdb, sector 98110368 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1624.132491] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 19, rd 1, flush 1, corrupt 0, gen 0
[ 1624.141791] sd 6:0:0:0: [sdb] tag#15 access beyond end of device
[ 1624.148997] blk_update_request: I/O error, dev sdb, sector 98110272 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[ 1624.156211] BTRFS error (device dm-5): bdev /dev/mapper/user2 errs: wr 19, rd 2, flush 1, corrupt 0, gen 0
[ 1624.164881] sd 7:0:0:0: [sdc] tag#17 access beyond end of device
[ 1624.172006] blk_update_request: I/O error, dev sdc, sector 216970512 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[ 1624.179201] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 563, flush 0, corrupt 0, gen 0
[ 1624.188543] sd 7:0:0:0: [sdc] tag#18 access beyond end of device
[ 1624.195911] blk_update_request: I/O error, dev sdc, sector 216970512 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[ 1624.203350] BTRFS error (device dm-3): bdev /dev/mapper/OMO errs: wr 4749, rd 564, flush 0, corrupt 0, gen 0

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

Hi Qu!

Is there any chance to find some hints of the issue in the log I attached?

On 08.02.21 22:05, chainofflowers wrote:
> Hi Qu!
> 
> It happened again, and this time I've been able to dump the dmesg.
> Also this time it happened on my home drive, please see the attached dump.
> 
> What can I do to fix it?
> btrfs scrub reports no error, neither does brfs check.
> I have also remounted the partition with -oclear_cache,space_cache, I
> hoped that could fix it...
> 
> Thanks...
> 
> (c)
> 

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

On 2021/2/20 下午7:26, chainofflowers wrote:
> Hi Qu!
>
> Is there any chance to find some hints of the issue in the log I attached?

Sorry for the late reply.

The weird part is, the access beyond boundary happens for read, no
wonder why previously added debug for discard doesn't work.

Currently I have no idea at all.

BTW, have you tried "btrfs check --check-data-csum"?
Regular btrfs check only check metadata but not data.

Considering how many the beyond boundary read there are, it looks like
some thing wrong related to data, thus "btrfs check --check-data-csum"
may help.

Another idea is to rule out memory corruption by running memtest, but I
doubt if it's the case.

Thanks,
Qu
>
> On 08.02.21 22:05, chainofflowers wrote:
>> Hi Qu!
>>
>> It happened again, and this time I've been able to dump the dmesg.
>> Also this time it happened on my home drive, please see the attached dump.
>>
>> What can I do to fix it?
>> btrfs scrub reports no error, neither does brfs check.
>> I have also remounted the partition with -oclear_cache,space_cache, I
>> hoped that could fix it...
>>
>> Thanks...
>>
>> (c)
>>

Re: Access Beyond End of Device & Input/Output Errors

[Forza]

On 2021-02-08 22:05, chainofflowers wrote:
> Hi Qu!
> 
> It happened again, and this time I've been able to dump the dmesg.
> Also this time it happened on my home drive, please see the attached dump.
> 
> What can I do to fix it?
> btrfs scrub reports no error, neither does brfs check.
> I have also remounted the partition with -oclear_cache,space_cache, I
> hoped that could fix it...
> 
> Thanks...
> 
> (c)
> 

Are you using fstrim by any chance? Could the problem be related to 
https://patchwork.kernel.org/project/fstests/patch/20200730121735.55389-1-wqu@suse.com/

/Forza

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

On 20.02.21 12:46, Forza wrote:

> Are you using fstrim by any chance? Could the problem be related to
> https://patchwork.kernel.org/project/fstests/patch/20200730121735.55389-1-wqu@suse.com/

Yes, that's what I mentioned in my first post.
Actually, it all started with the bug with dm, but some similar
behaviour persists even after that bug was fixed:
https://lore.kernel.org/linux-btrfs/20190521190023.GA68070@glet/T/

The only "maybe unusual" thing in my setup is that I use btrfs on the
top of dmcrypt directly, without lvm in-between, but I am not the only
one...

@Qu:
My RAM looks OK so far, I also thought of that, and I actually ran
memtest for 12+ hours and more than once. I would exclude that case.

I will do a "btrfs check --check-data-csum" and let you know.

In the meantime, I thought of a related question:
-> When a data-csum is corrupted (for whatever reason), is there a
chance that the corruption persists when I copy the whole file system
over to a new one?

As I said previously, I copied the whole fs to new, virgin SSDs more
than once with "rsync -avAHX", and I couldn't spot any issue related to
the copy itself...

(please help! ;-))

(c)

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

On 2021/2/20 下午8:07, chainofflowers wrote:
> On 20.02.21 12:46, Forza wrote:
>
>> Are you using fstrim by any chance? Could the problem be related to
>> https://patchwork.kernel.org/project/fstests/patch/20200730121735.55389-1-wqu@suse.com/
>
> Yes, that's what I mentioned in my first post.
> Actually, it all started with the bug with dm, but some similar
> behaviour persists even after that bug was fixed:
> https://lore.kernel.org/linux-btrfs/20190521190023.GA68070@glet/T/
>
> The only "maybe unusual" thing in my setup is that I use btrfs on the
> top of dmcrypt directly, without lvm in-between, but I am not the only
> one...
>
> @Qu:
> My RAM looks OK so far, I also thought of that, and I actually ran
> memtest for 12+ hours and more than once. I would exclude that case.
>
> I will do a "btrfs check --check-data-csum" and let you know.
>
> In the meantime, I thought of a related question:
> -> When a data-csum is corrupted (for whatever reason), is there a
> chance that the corruption persists when I copy the whole file system
> over to a new one?

You can rule out the possibility that some data checksum itself is
corrupted.
Data checksum is stored in btrfs trees, and all tree blocks have their
own checksum.

>
> As I said previously, I copied the whole fs to new, virgin SSDs more
> than once with "rsync -avAHX", and I couldn't spot any issue related to
> the copy itself...

Then you can rule out the checksum problem.

Thanks,
Qu
>
> (please help! ;-))
>
> (c)
>

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

Hi Qu!

So far, I am not experiencing the access-beyond-end-of-device issue
anymore since some weeks.

The new thing I did was:
1. Booted from an external disk
2. Ran:
       btrfs check --clear-space-cache v1
       btrfs check --clear-ino-cache
   on all my volumes.

I had somehow missed this (from man btrfs-check):
> --clear-space-cache v1|v2
>     completely wipe all free space cache of given type
>
>     For free space cache v1, the clear_cache kernel mount option only
>     rebuilds the free space cache for block groups that are modified
>     while the filesystem is mounted with that option. Thus, using this
>     option with v1 makes it possible to actually clear the entire free
>     space cache.

I think, in all of this I had missed that I needed to "clear-space-cache
v1" on the non-mounted volume, as I have always done it while the volume
was mounted.

Maybe this info could help Justin too.


BTW: Could "--clear-ino-cache" have also made any difference? Or was it
just about cleaning unused left-overs?


Thanks a lot for your help, Qu! :-)

Have a nice weekend,

(c)



On 20.02.21 13:13, Qu Wenruo wrote:
> 
> 
> On 2021/2/20 下午8:07, chainofflowers wrote:
>> On 20.02.21 12:46, Forza wrote:
>>
>>> Are you using fstrim by any chance? Could the problem be related to
>>> https://patchwork.kernel.org/project/fstests/patch/20200730121735.55389-1-wqu@suse.com/
>>>
>>
>> Yes, that's what I mentioned in my first post.
>> Actually, it all started with the bug with dm, but some similar
>> behaviour persists even after that bug was fixed:
>> https://lore.kernel.org/linux-btrfs/20190521190023.GA68070@glet/T/
>>
>> The only "maybe unusual" thing in my setup is that I use btrfs on the
>> top of dmcrypt directly, without lvm in-between, but I am not the only
>> one...
>>
>> @Qu:
>> My RAM looks OK so far, I also thought of that, and I actually ran
>> memtest for 12+ hours and more than once. I would exclude that case.
>>
>> I will do a "btrfs check --check-data-csum" and let you know.
>>
>> In the meantime, I thought of a related question:
>> -> When a data-csum is corrupted (for whatever reason), is there a
>> chance that the corruption persists when I copy the whole file system
>> over to a new one?
> 
> You can rule out the possibility that some data checksum itself is
> corrupted.
> Data checksum is stored in btrfs trees, and all tree blocks have their
> own checksum.
> 
>>
>> As I said previously, I copied the whole fs to new, virgin SSDs more
>> than once with "rsync -avAHX", and I couldn't spot any issue related to
>> the copy itself...
> 
> Then you can rule out the checksum problem.
> 
> Thanks,
> Qu

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

On 2021/4/24 上午7:36, chainofflowers wrote:
> Hi Qu!
> 
> So far, I am not experiencing the access-beyond-end-of-device issue
> anymore since some weeks.
> 
> The new thing I did was:
> 1. Booted from an external disk
> 2. Ran:
>         btrfs check --clear-space-cache v1
>         btrfs check --clear-ino-cache
>     on all my volumes.
> 
> I had somehow missed this (from man btrfs-check):
>> --clear-space-cache v1|v2
>>      completely wipe all free space cache of given type
>>
>>      For free space cache v1, the clear_cache kernel mount option only
>>      rebuilds the free space cache for block groups that are modified
>>      while the filesystem is mounted with that option. Thus, using this
>>      option with v1 makes it possible to actually clear the entire free
>>      space cache.
> 
> I think, in all of this I had missed that I needed to "clear-space-cache
> v1" on the non-mounted volume, as I have always done it while the volume
> was mounted.

That's interesting, this means we're lacking sanity check against free 
space cache.

But if that's the case, btrfs-check should be able to report such 
problem from the very beginning.

Anyway, it's great to know the problem no longer bothers you.

Thanks,
Qu
> 
> Maybe this info could help Justin too.
> 
> 
> BTW: Could "--clear-ino-cache" have also made any difference? Or was it
> just about cleaning unused left-overs?
> 
> 
> Thanks a lot for your help, Qu! :-)
> 
> Have a nice weekend,
> 
> (c)
> 
> 
> 
> On 20.02.21 13:13, Qu Wenruo wrote:
>>
>>
>> On 2021/2/20 下午8:07, chainofflowers wrote:
>>> On 20.02.21 12:46, Forza wrote:
>>>
>>>> Are you using fstrim by any chance? Could the problem be related to
>>>> https://patchwork.kernel.org/project/fstests/patch/20200730121735.55389-1-wqu@suse.com/
>>>>
>>>
>>> Yes, that's what I mentioned in my first post.
>>> Actually, it all started with the bug with dm, but some similar
>>> behaviour persists even after that bug was fixed:
>>> https://lore.kernel.org/linux-btrfs/20190521190023.GA68070@glet/T/
>>>
>>> The only "maybe unusual" thing in my setup is that I use btrfs on the
>>> top of dmcrypt directly, without lvm in-between, but I am not the only
>>> one...
>>>
>>> @Qu:
>>> My RAM looks OK so far, I also thought of that, and I actually ran
>>> memtest for 12+ hours and more than once. I would exclude that case.
>>>
>>> I will do a "btrfs check --check-data-csum" and let you know.
>>>
>>> In the meantime, I thought of a related question:
>>> -> When a data-csum is corrupted (for whatever reason), is there a
>>> chance that the corruption persists when I copy the whole file system
>>> over to a new one?
>>
>> You can rule out the possibility that some data checksum itself is
>> corrupted.
>> Data checksum is stored in btrfs trees, and all tree blocks have their
>> own checksum.
>>
>>>
>>> As I said previously, I copied the whole fs to new, virgin SSDs more
>>> than once with "rsync -avAHX", and I couldn't spot any issue related to
>>> the copy itself...
>>
>> Then you can rule out the checksum problem.
>>
>> Thanks,
>> Qu
> 

Re: Access Beyond End of Device & Input/Output Errors

[chainofflowers]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On Samstag, 24. April 2021 02:25 Qu Wenruo wrote:
> But if that's the case, btrfs-check should be able to report such
> problem from the very beginning.

btrfs check was only reporting that "the cache would have been invalidated", I 
don't know if that was enough to suggest that a cache clear was needed. I 
assumed it would have done it automatically the next time the volume was 
mounted, that's why I didn't clean it... "pro-actively".

Could "clear-ino-cache" have helped, or not at all?


(c)


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Access Beyond End of Device & Input/Output Errors

[Qu Wenruo]

On 2021/4/24 下午10:13, chainofflowers wrote:
> On Samstag, 24. April 2021 02:25 Qu Wenruo wrote:
>> But if that's the case, btrfs-check should be able to report such
>> problem from the very beginning.
>
> btrfs check was only reporting that "the cache would have been invalidated", I
> don't know if that was enough to suggest that a cache clear was needed.

That means the cache generation is not matching super block generation.
Kernel is good enough to detect such case and ignore the cache.

  I
> assumed it would have done it automatically the next time the volume was
> mounted, that's why I didn't clean it... "pro-actively".
>
> Could "clear-ino-cache" have helped, or not at all?

ino-cache is a feature that is mostly deprecated, and no known
real-world usage.

Thanks,
Qu

>
>
> (c)
>