btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

Was attempting my first btrfs send receive over ssh and continually
received ioctl error at different points but always in the first 3
minutes. The volume consists of three devices with only metadata
duplication. I narrowed down the error to the send command by
recreating the error while redirecting to /dev/null. Sometime it would
happen after ~12Gib, or ~7.6Gib, right now rerunning multiple times it
has stopped on exactly 3.76 multiple times.

$ sudo btrfs send /mnt/dataroot.2017.10.21/ | pv -i5 > /dev/null
At subvol /mnt/dataroot.2017.10.21/
ERROR: send ioctl failed with -5: Input/output error                    ]
3.76GiB 0:00:13 [ 290MiB/s] [      <=>                                  ]


First I checked the btrfs device stats, each of the 3 drives appear clean:
$ sudo btrfs device stats /mnt
[/dev/sdi].write_io_errs    0
[/dev/sdi].read_io_errs     0
[/dev/sdi].flush_io_errs    0
[/dev/sdi].corruption_errs  0
[/dev/sdi].generation_errs  0
[/dev/sdh].write_io_errs    0
[/dev/sdh].read_io_errs     0
[/dev/sdh].flush_io_errs    0
[/dev/sdh].corruption_errs  0
[/dev/sdh].generation_errs  0
[/dev/sdn].write_io_errs    0
[/dev/sdn].read_io_errs     0
[/dev/sdn].flush_io_errs    0
[/dev/sdn].corruption_errs  0
[/dev/sdn].generation_errs  0

The next thing I tried was running and checking that SMART short
selftest passed on each of three drives with no error.
$ sudo smartctl -l selftest /dev/sdh
# 1  Short offline       Completed without error


I read somewhere to check dmesg, which yielded some info:
BTRFS warning (device sdn): csum failed ino 6407 off 7683907584 csum
1745651892 expected csum 3952841867

But when I when to see if scrub could detect the errors, nothing was found:
$ sudo btrfs scrub status -d /mnt
scrub status for 88406942-e3e1-42c6-ad71-e23bb315caa7
scrub device /dev/sdi (id 1) history
        scrub started at Sun Oct 22 14:43:20 2017 and finished after 01:57:00
        total bytes scrubbed: 677.69GiB with 0 errors
scrub device /dev/sdh (id 2) history
        scrub started at Sun Oct 22 14:43:20 2017 and finished after 01:56:38
        total bytes scrubbed: 677.44GiB with 0 errors
scrub device /dev/sdn (id 3) history
        scrub started at Sun Oct 22 14:43:20 2017 and finished after 01:56:38
        total bytes scrubbed: 677.36GiB with 0 errors


After all that scrubbing I still receive the ioctl error.


Does anyone have any ideas of what to try next? Right now I am running
the SMART 'long' self test.

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

All three devices completed the 'long' SMART selftest without error:

# 1  Extended offline    Completed without error       00%


Here is the standard data that I forgot to include in my first message:
Running Arch linux

$ uname -a
Linux HOSTNAME 4.9.56-1-lts #1 SMP Thu Oct 12 22:34:15 CEST 2017
x86_64 GNU/Linux

$ btrfs --version
btrfs-progs v4.13

$ sudo btrfs fi show
Label: 'CRUCIAL116'  uuid: 31c38558-c8c7-49c4-8fea-9d0730ee58a7
        Total devices 1 FS bytes used 7.77GiB
        devid    1 size 59.62GiB used 59.62GiB path /dev/sda2

Label: 'OfflineJ'  uuid: 88406942-e3e1-42c6-ad71-e23bb315caa7
        Total devices 3 FS bytes used 1.98TiB
        devid    1 size 1.82TiB used 679.00GiB path /dev/sdi
        devid    2 size 1.82TiB used 679.01GiB path /dev/sdh
        devid    3 size 1.82TiB used 679.01GiB path /dev/sdn

$ sudo btrfs fi df /mnt
Data, RAID0: total=1.98TiB, used=1.98TiB
System, RAID1: total=8.00MiB, used=144.00KiB
Metadata, RAID1: total=3.00GiB, used=2.44GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

$ dmesg | grep BTRFS
[    5.262090] BTRFS: device label CRUCIAL116 devid 1 transid 98407 /dev/sda2
[   15.636475] BTRFS: device label OfflineJ devid 2 transid 612 /dev/sdh
[   15.646343] BTRFS: device label OfflineJ devid 1 transid 612 /dev/sdi
[   15.647194] BTRFS: device label OfflineJ devid 3 transid 612 /dev/sdn
[   15.754204] BTRFS info (device sda2): disk space caching is enabled
[   15.754206] BTRFS info (device sda2): has skinny extents
[   15.778659] BTRFS info (device sda2): detected SSD devices, enabling SSD mode
[   58.492530] BTRFS info (device sdn): disk space caching is enabled
[   58.492532] BTRFS info (device sdn): has skinny extents
[   61.243226] BTRFS info (device sdn): checking UUID tree
[  114.437424] BTRFS warning (device sdn): csum failed ino 6407 off
7683907584 csum 1745651892 expected csum 3952841867
[  114.450699] BTRFS warning (device sdn): csum failed ino 6407 off
7683907584 csum 1745651892 expected csum 3952841867
[38494.978379] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 876064455 expected csum 874979996
[38494.989301] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 2615801759 expected csum 874979996
[38541.079264] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 876064455 expected csum 874979996
[38571.245421] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 2615801759 expected csum 874979996
[39434.215600] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 2615801759 expected csum 874979996
[73132.653297] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 2615801759 expected csum 874979996
[73167.897106] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 2615801759 expected csum 874979996


One thing I notice is that ino 4708 keeps returns a few different
'wrong' csums, I can also
confirm that one of those 'csum failed' messages gets written each
time I run '$ sudo btrfs send /mnt/dataroot.2017.10.21/ | pv -i5 > /dev/null'

Does anyone know why scrub did not catch these errors that show up in dmesg?



On Mon, Oct 23, 2017 at 12:25 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
> Was attempting my first btrfs send receive over ssh and continually
> received ioctl error at different points but always in the first 3
> minutes. The volume consists of three devices with only metadata
> duplication. I narrowed down the error to the send command by
> recreating the error while redirecting to /dev/null. Sometime it would
> happen after ~12Gib, or ~7.6Gib, right now rerunning multiple times it
> has stopped on exactly 3.76 multiple times.
>
> $ sudo btrfs send /mnt/dataroot.2017.10.21/ | pv -i5 > /dev/null
> At subvol /mnt/dataroot.2017.10.21/
> ERROR: send ioctl failed with -5: Input/output error                    ]
> 3.76GiB 0:00:13 [ 290MiB/s] [      <=>                                  ]
>
>
> First I checked the btrfs device stats, each of the 3 drives appear clean:
> $ sudo btrfs device stats /mnt
> [/dev/sdi].write_io_errs    0
> [/dev/sdi].read_io_errs     0
> [/dev/sdi].flush_io_errs    0
> [/dev/sdi].corruption_errs  0
> [/dev/sdi].generation_errs  0
> [/dev/sdh].write_io_errs    0
> [/dev/sdh].read_io_errs     0
> [/dev/sdh].flush_io_errs    0
> [/dev/sdh].corruption_errs  0
> [/dev/sdh].generation_errs  0
> [/dev/sdn].write_io_errs    0
> [/dev/sdn].read_io_errs     0
> [/dev/sdn].flush_io_errs    0
> [/dev/sdn].corruption_errs  0
> [/dev/sdn].generation_errs  0
>
> The next thing I tried was running and checking that SMART short
> selftest passed on each of three drives with no error.
> $ sudo smartctl -l selftest /dev/sdh
> # 1  Short offline       Completed without error
>
>
> I read somewhere to check dmesg, which yielded some info:
> BTRFS warning (device sdn): csum failed ino 6407 off 7683907584 csum
> 1745651892 expected csum 3952841867
>
> But when I when to see if scrub could detect the errors, nothing was found:
> $ sudo btrfs scrub status -d /mnt
> scrub status for 88406942-e3e1-42c6-ad71-e23bb315caa7
> scrub device /dev/sdi (id 1) history
>         scrub started at Sun Oct 22 14:43:20 2017 and finished after 01:57:00
>         total bytes scrubbed: 677.69GiB with 0 errors
> scrub device /dev/sdh (id 2) history
>         scrub started at Sun Oct 22 14:43:20 2017 and finished after 01:56:38
>         total bytes scrubbed: 677.44GiB with 0 errors
> scrub device /dev/sdn (id 3) history
>         scrub started at Sun Oct 22 14:43:20 2017 and finished after 01:56:38
>         total bytes scrubbed: 677.36GiB with 0 errors
>
>
> After all that scrubbing I still receive the ioctl error.
>
>
> Does anyone have any ideas of what to try next? Right now I am running
> the SMART 'long' self test.

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Lakshmipathi.G]

> Does anyone know why scrub did not catch these errors that show up in dmesg?

Can you try offline scrub from this repo
https://github.com/gujx2017/btrfs-progs/tree/offline_scrub and see
whether it
detects the issue?  "btrfs scrub start --offline <dev>"


----
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

Yes, it is finding much more than just one error.

>From dmesg
[89520.441354] BTRFS warning (device sdn): csum failed ino 4708 off
27529216 csum 2615801759 expected csum 874979996

$ sudo btrfs scrub start --offline --progress /dev/sdn
ERROR: data at bytenr 68431499264 mirror 1 csum mismatch, have
0x5aa0d40f expect 0xd4a15873
ERROR: extent 68431474688 len 14467072 CORRUPTED, all mirror(s)
corrupted, can't be repaired
ERROR: data at bytenr 83646357504 mirror 1 csum mismatch, have
0xfc0baabe expect 0x7f9cb681
ERROR: extent 83519741952 len 134217728 CORRUPTED, all mirror(s)
corrupted, can't be repaired
ERROR: data at bytenr 121936633856 mirror 1 csum mismatch, have
0x507016a5 expect 0x50609afe
ERROR: extent 121858334720 len 134217728 CORRUPTED, all mirror(s)
corrupted, can't be repaired
ERROR: data at bytenr 144872591360 mirror 1 csum mismatch, have
0x33964d73 expect 0xf9937032
ERROR: extent 144822386688 len 61231104 CORRUPTED, all mirror(s)
corrupted, can't be repaired
ERROR: data at bytenr 167961075712 mirror 1 csum mismatch, have
0xf43bd0e3 expect 0x5be589bb
ERROR: extent 167950999552 len 27537408 CORRUPTED, all mirror(s)
corrupted, can't be repaired
ERROR: data at bytenr 175643619328 mirror 1 csum mismatch, have
0x1e168ca1 expect 0xd413b1e0
ERROR: data at bytenr 175643754496 mirror 1 csum mismatch, have
0x6cfdc8ae expect 0xa6f8f5ef
ERROR: extent 175640539136 len 6381568 CORRUPTED, all mirror(s)
corrupted, can't be repaired
ERROR: data at bytenr 183316750336 mirror 1 csum mismatch, have
0x145bdf76 expect 0x7390565e
.....
and the list goes on.


Questions:
1. Using "find /mnt -inum 4708" I can link the dmesg to a specific
file. Is there a
way link the the --offline ERRORs above to the inode?

2. How could do "btrfs device stats /mnt" and normal full scrub fail
to detect the csum errors?

3. Do these errors appear to be hardware failure (despite pristine
SMART), user error on
volume creation/mounting, or an actual btrfs issue? I feel that the
need for question #1
indicates a problem with btrfs regardless of whether there is a real
hardware failure or not.


Next I will try an online scrub of only the sdn device, as before I
was running the full filesystem scrub.

On Tue, Oct 24, 2017 at 12:52 AM, Lakshmipathi.G
<lakshmipathi.g@gmail.com> wrote:
>> Does anyone know why scrub did not catch these errors that show up in dmesg?
>
> Can you try offline scrub from this repo
> https://github.com/gujx2017/btrfs-progs/tree/offline_scrub and see
> whether it
> detects the issue?  "btrfs scrub start --offline <dev>"
>
>
> ----
> Cheers,
> Lakshmipathi.G
> http://www.giis.co.in http://www.webminal.org

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

I apologize for the bad line wrapping on the last post...will be
setting up mutt soon.

This is the final result for the offline scrub:
Doing offline scrub [O] [681/683]
Scrub result:
Tree bytes scrubbed: 5234491392
Tree extents scrubbed: 638975
Data bytes scrubbed: 4353723572224
Data extents scrubbed: 374300
Data bytes without csum: 533200896
Read error: 0
Verify error: 0
Csum error: 175

The offline scrub apparently corrected some metadata extents while
scanning /dev/sdn


I also ran the online scrub directly on the /dev/sdn, "0 errors":

$ btrfs scrub status /dev/sdn
scrub status for 88406942-e3e1-42c6-ad71-e23bb315caa7
        scrub started at Tue Oct 24 06:55:12 2017 and finished after 01:52:44
        total bytes scrubbed: 677.35GiB with 0 errors

The csum mismatches are still missed by the online scrub when choosing
a single <device>. Now I am doing offline scrub on the other devices
to see if they are clean.

$ lsblk -o +SERIAL
NAME      MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT SERIAL
sdh         8:112  0  1.8T  0 disk             WD-WMAZA370XXXX
sdi         8:128  0  1.8T  0 disk             WD-WCAZA569XXXX
sdn         8:208  0  1.8T  0 disk             WD-WCAZA580XXXX

$ btrfs scrub start --offline --progress /dev/sdh
ERROR: data at bytenr 5365456896 ...
ERROR: extent 5341712384 ...
...

One thing to note is that a /dev/sdh is also having csum errors
detected despite it having never been mentioned dmesg. I understand
that you may have the ability to run two offline checks at once but
the error message I get is slightly misleading.

$ btrfs scrub start --offline --progress /dev/sdi
ERROR: cannot open device '/dev/sdn': Device or resource busy
ERROR: cannot open file system

I get an error about sdn when the device I am trying to scan is sdi,
and the device that is currently being scanned is sdh.

On Tue, Oct 24, 2017 at 2:00 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
> Yes, it is finding much more than just one error.
>
> From dmesg
> [89520.441354] BTRFS warning (device sdn): csum failed ino 4708 off
> 27529216 csum 2615801759 expected csum 874979996
>
> $ sudo btrfs scrub start --offline --progress /dev/sdn
> ERROR: data at bytenr 68431499264 mirror 1 csum mismatch, have
> 0x5aa0d40f expect 0xd4a15873
> ERROR: extent 68431474688 len 14467072 CORRUPTED, all mirror(s)
> corrupted, can't be repaired
> ERROR: data at bytenr 83646357504 mirror 1 csum mismatch, have
> 0xfc0baabe expect 0x7f9cb681
> ERROR: extent 83519741952 len 134217728 CORRUPTED, all mirror(s)
> corrupted, can't be repaired
> ERROR: data at bytenr 121936633856 mirror 1 csum mismatch, have
> 0x507016a5 expect 0x50609afe
> ERROR: extent 121858334720 len 134217728 CORRUPTED, all mirror(s)
> corrupted, can't be repaired
> ERROR: data at bytenr 144872591360 mirror 1 csum mismatch, have
> 0x33964d73 expect 0xf9937032
> ERROR: extent 144822386688 len 61231104 CORRUPTED, all mirror(s)
> corrupted, can't be repaired
> ERROR: data at bytenr 167961075712 mirror 1 csum mismatch, have
> 0xf43bd0e3 expect 0x5be589bb
> ERROR: extent 167950999552 len 27537408 CORRUPTED, all mirror(s)
> corrupted, can't be repaired
> ERROR: data at bytenr 175643619328 mirror 1 csum mismatch, have
> 0x1e168ca1 expect 0xd413b1e0
> ERROR: data at bytenr 175643754496 mirror 1 csum mismatch, have
> 0x6cfdc8ae expect 0xa6f8f5ef
> ERROR: extent 175640539136 len 6381568 CORRUPTED, all mirror(s)
> corrupted, can't be repaired
> ERROR: data at bytenr 183316750336 mirror 1 csum mismatch, have
> 0x145bdf76 expect 0x7390565e
> .....
> and the list goes on.
>
>
> Questions:
> 1. Using "find /mnt -inum 4708" I can link the dmesg to a specific
> file. Is there a
> way link the the --offline ERRORs above to the inode?
>
> 2. How could do "btrfs device stats /mnt" and normal full scrub fail
> to detect the csum errors?
>
> 3. Do these errors appear to be hardware failure (despite pristine
> SMART), user error on
> volume creation/mounting, or an actual btrfs issue? I feel that the
> need for question #1
> indicates a problem with btrfs regardless of whether there is a real
> hardware failure or not.
>
>
> Next I will try an online scrub of only the sdn device, as before I
> was running the full filesystem scrub.
>
> On Tue, Oct 24, 2017 at 12:52 AM, Lakshmipathi.G
> <lakshmipathi.g@gmail.com> wrote:
>>> Does anyone know why scrub did not catch these errors that show up in dmesg?
>>
>> Can you try offline scrub from this repo
>> https://github.com/gujx2017/btrfs-progs/tree/offline_scrub and see
>> whether it
>> detects the issue?  "btrfs scrub start --offline <dev>"
>>
>>
>> ----
>> Cheers,
>> Lakshmipathi.G
>> http://www.giis.co.in http://www.webminal.org

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Lakshmipathi.G]

1. I guess you should be able to dump tree details via
'btrfs-debug-tree' and then map the extent/data (from scrub
offline output) and track it back to inode-object. Store output of
both btrfs-debug-tree and scrub-offline in different files and then
play around with grep to extract required data.

2. I think normal scrub(online) fails to detect these csum errors for
some reason,I don't have much idea about online scrub.

3. I assume, the issue is not related to hardware. Since the offline
scrub able to get available (corrupted) csum.

Yes, offline scrub will try to fix corruption whenever it is possible.
And also you have quite lot of "all mirror(s) corrupted, can't be repaired",
which will be hard to recovery.

I suggest running offline scrub on all devices. Then online scrub
and finally track those corrupted files with the help of extent info.
----
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Wed, Oct 25, 2017 at 7:22 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
> I apologize for the bad line wrapping on the last post...will be
> setting up mutt soon.
>
> This is the final result for the offline scrub:
> Doing offline scrub [O] [681/683]
> Scrub result:
> Tree bytes scrubbed: 5234491392
> Tree extents scrubbed: 638975
> Data bytes scrubbed: 4353723572224
> Data extents scrubbed: 374300
> Data bytes without csum: 533200896
> Read error: 0
> Verify error: 0
> Csum error: 175
>
> The offline scrub apparently corrected some metadata extents while
> scanning /dev/sdn
>
>
> I also ran the online scrub directly on the /dev/sdn, "0 errors":
>
> $ btrfs scrub status /dev/sdn
> scrub status for 88406942-e3e1-42c6-ad71-e23bb315caa7
>         scrub started at Tue Oct 24 06:55:12 2017 and finished after 01:52:44
>         total bytes scrubbed: 677.35GiB with 0 errors
>
> The csum mismatches are still missed by the online scrub when choosing
> a single <device>. Now I am doing offline scrub on the other devices
> to see if they are clean.
>
> $ lsblk -o +SERIAL
> NAME      MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT SERIAL
> sdh         8:112  0  1.8T  0 disk             WD-WMAZA370XXXX
> sdi         8:128  0  1.8T  0 disk             WD-WCAZA569XXXX
> sdn         8:208  0  1.8T  0 disk             WD-WCAZA580XXXX
>
> $ btrfs scrub start --offline --progress /dev/sdh
> ERROR: data at bytenr 5365456896 ...
> ERROR: extent 5341712384 ...
> ...
>
> One thing to note is that a /dev/sdh is also having csum errors
> detected despite it having never been mentioned dmesg. I understand
> that you may have the ability to run two offline checks at once but
> the error message I get is slightly misleading.
>
> $ btrfs scrub start --offline --progress /dev/sdi
> ERROR: cannot open device '/dev/sdn': Device or resource busy
> ERROR: cannot open file system
>
> I get an error about sdn when the device I am trying to scan is sdi,
> and the device that is currently being scanned is sdh.
>
> On Tue, Oct 24, 2017 at 2:00 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
>> Yes, it is finding much more than just one error.
>>
>> From dmesg
>> [89520.441354] BTRFS warning (device sdn): csum failed ino 4708 off
>> 27529216 csum 2615801759 expected csum 874979996
>>
>> $ sudo btrfs scrub start --offline --progress /dev/sdn
>> ERROR: data at bytenr 68431499264 mirror 1 csum mismatch, have
>> 0x5aa0d40f expect 0xd4a15873
>> ERROR: extent 68431474688 len 14467072 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 83646357504 mirror 1 csum mismatch, have
>> 0xfc0baabe expect 0x7f9cb681
>> ERROR: extent 83519741952 len 134217728 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 121936633856 mirror 1 csum mismatch, have
>> 0x507016a5 expect 0x50609afe
>> ERROR: extent 121858334720 len 134217728 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 144872591360 mirror 1 csum mismatch, have
>> 0x33964d73 expect 0xf9937032
>> ERROR: extent 144822386688 len 61231104 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 167961075712 mirror 1 csum mismatch, have
>> 0xf43bd0e3 expect 0x5be589bb
>> ERROR: extent 167950999552 len 27537408 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 175643619328 mirror 1 csum mismatch, have
>> 0x1e168ca1 expect 0xd413b1e0
>> ERROR: data at bytenr 175643754496 mirror 1 csum mismatch, have
>> 0x6cfdc8ae expect 0xa6f8f5ef
>> ERROR: extent 175640539136 len 6381568 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 183316750336 mirror 1 csum mismatch, have
>> 0x145bdf76 expect 0x7390565e
>> .....
>> and the list goes on.
>>
>>
>> Questions:
>> 1. Using "find /mnt -inum 4708" I can link the dmesg to a specific
>> file. Is there a
>> way link the the --offline ERRORs above to the inode?
>>
>> 2. How could do "btrfs device stats /mnt" and normal full scrub fail
>> to detect the csum errors?
>>
>> 3. Do these errors appear to be hardware failure (despite pristine
>> SMART), user error on
>> volume creation/mounting, or an actual btrfs issue? I feel that the
>> need for question #1
>> indicates a problem with btrfs regardless of whether there is a real
>> hardware failure or not.
>>
>>
>> Next I will try an online scrub of only the sdn device, as before I
>> was running the full filesystem scrub.
>>
>> On Tue, Oct 24, 2017 at 12:52 AM, Lakshmipathi.G
>> <lakshmipathi.g@gmail.com> wrote:
>>>> Does anyone know why scrub did not catch these errors that show up in dmesg?
>>>
>>> Can you try offline scrub from this repo
>>> https://github.com/gujx2017/btrfs-progs/tree/offline_scrub and see
>>> whether it
>>> detects the issue?  "btrfs scrub start --offline <dev>"
>>>
>>>
>>> ----
>>> Cheers,
>>> Lakshmipathi.G
>>> http://www.giis.co.in http://www.webminal.org

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

I don't need to recover in this case. I can just remake the filesystem. I'm
just very concerned that this corruption was able to happen. Here is the entire
history of the filesystem:

2017.10.18 create btrfs from 3 drives aka OfflineJ and rsync
data from old madam raid5
---------------------------------------
# badblocks run on all three drives
$ badblocks -wsv /dev/disk/by-id/WD-XXX
Pass completed, 0 bad blocks found. (0/0/0 errors)

$ mkfs.btrfs -L OfflineJ /dev/disk/by-id/WD-XX1 /dev/disk/by-id/WD-XX2 /dev/disk/by-id/WD-XX3

$ mount -t btrfs UUID=88406942-e3e1-42c6-ad71-e23bb315caa7 /mnt/

$ btrfs subvolume create /mnt/dataroot

$ mkdir /media/OfflineJ

/etc/fstab
----------
UUID=XXX       /media/OfflineJ         btrfs   rw,relatime,subvol=/dataroot,noauto 0 0

$ mount /media/OfflineJ/

$ btrfs filesystem df /media/OfflineJ/
Data, RAID0: total=3.00GiB, used=1.00MiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=128.00KiB
GlobalReserve, single: total=16.00MiB, used=0.00B

$ btrfs filesystem usage /media/OfflineJ/
Overall:
   Device size:                   5.46TiB
   Device allocated:              5.02GiB
   Device unallocated:            5.45TiB
   Device missing:                5.46TiB
   Used:                          1.28MiB
   Free (estimated):              5.46TiB      (min: 2.73TiB)
   Data ratio:                       1.00
   Metadata ratio:                   2.00
   Global reserve:               16.00MiB      (used: 0.00B)

$ sudo mount -o noatime -o ro /media/oldmdadmraid5/

$ rsync -aAXh --progress --stats /media/oldmdadmraid5/ /media/OfflineJ


I will gladly repeat this process, but I am very concerned why this
corruption happened in the first place.

More tests:

scrub start --offline
    All devices had errors in differing amounts
    I will verify that these counts are repeatable.
    Csum error: 150
    Csum error: 238
    Csum error: 175

btrfs check
    found 2179745955840 bytes used, no error found

btrfs check --check-data-csum
    mirror 0 bytenr 13348855808 csum 2387937020 expected csum 562782116
    mirror 0 bytenr 23398821888 csum 3602081170 expected csum 1963854755
    ...

The only thing I could think of is that the btrfs version that I used to mkfs
was not up to date. Is there a way to determine which version was used to
create the filesystem?

Anything else I can do to help determine the cause?

> On October 24, 2017 at 11:43 PM "Lakshmipathi.G" <lakshmipathi.g@gmail.com> wrote:
> 
> 1.  I guess you should be able to dump tree details via
> 'btrfs-debug-tree' and then map the extent/data (from scrub
> offline output) and track it back to inode-object. Store output of
> both btrfs-debug-tree and scrub-offline in different files and then
> play around with grep to extract required data.
> 
> 2.  I think normal scrub(online) fails to detect these csum errors for
> some reason,I don't have much idea about online scrub.
> 
> 3.  I assume, the issue is not related to hardware. Since the offline
> scrub able to get available (corrupted) csum.
> 
> Yes, offline scrub will try to fix corruption whenever it is possible.
> And also you have quite lot of "all mirror(s) corrupted, can't be repaired",
> which will be hard to recovery.
> 
> I suggest running offline scrub on all devices. Then online scrub
> 
> and finally track those corrupted files with the help of extent info.
> 
> ----
> Cheers,
> Lakshmipathi.G
> http://www.giis.co.in http://www.webminal.org
> 
> On Wed, Oct 25, 2017 at 7:22 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
> 
> > I apologize for the bad line wrapping on the last post...will be
> > setting up mutt soon.
> > 
> > This is the final result for the offline scrub:
> > Doing offline scrub [O] [681/683]
> > Scrub result:
> > Tree bytes scrubbed: 5234491392
> > Tree extents scrubbed: 638975
> > Data bytes scrubbed: 4353723572224
> > Data extents scrubbed: 374300
> > Data bytes without csum: 533200896
> > Read error: 0
> > Verify error: 0
> > Csum error: 175
> > 
> > The offline scrub apparently corrected some metadata extents while
> > scanning /dev/sdn
> > 
> > I also ran the online scrub directly on the /dev/sdn, "0 errors":
> > 
> > $ btrfs scrub status /dev/sdn
> > scrub status for 88406942-e3e1-42c6-ad71-e23bb315caa7
> >  scrub started at Tue Oct 24 06:55:12 2017 and finished after 01:52:44
> >  total bytes scrubbed: 677.35GiB with 0 errors
> > 
> > The csum mismatches are still missed by the online scrub when choosing
> > a single . Now I am doing offline scrub on the other devices
> > to see if they are clean.
> > 
> > $ lsblk -o +SERIAL
> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT SERIAL
> > sdh 8:112 0 1.8T 0 disk WD-WMAZA370XXXX
> > sdi 8:128 0 1.8T 0 disk WD-WCAZA569XXXX
> > sdn 8:208 0 1.8T 0 disk WD-WCAZA580XXXX
> > 
> > $ btrfs scrub start --offline --progress /dev/sdh
> > ERROR: data at bytenr 5365456896 ...
> > ERROR: extent 5341712384 ...
> > ...
> > 
> > One thing to note is that a /dev/sdh is also having csum errors
> > detected despite it having never been mentioned dmesg. I understand
> > that you may have the ability to run two offline checks at once but
> > the error message I get is slightly misleading.
> > 
> > $ btrfs scrub start --offline --progress /dev/sdi
> > ERROR: cannot open device '/dev/sdn': Device or resource busy
> > ERROR: cannot open file system
> > 
> > I get an error about sdn when the device I am trying to scan is sdi,
> > and the device that is currently being scanned is sdh.
> > 
> > On Tue, Oct 24, 2017 at 2:00 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
> > 
> > > Yes, it is finding much more than just one error.
> > > 
> > > From dmesg
> > > [89520.441354] BTRFS warning (device sdn): csum failed ino 4708 off
> > > 27529216 csum 2615801759 expected csum 874979996
> > > 
> > > $ sudo btrfs scrub start --offline --progress /dev/sdn
> > > ERROR: data at bytenr 68431499264 mirror 1 csum mismatch, have
> > > 0x5aa0d40f expect 0xd4a15873
> > > ERROR: extent 68431474688 len 14467072 CORRUPTED, all mirror(s)
> > > corrupted, can't be repaired
> > > ERROR: data at bytenr 83646357504 mirror 1 csum mismatch, have
> > > 0xfc0baabe expect 0x7f9cb681
> > > ERROR: extent 83519741952 len 134217728 CORRUPTED, all mirror(s)
> > > corrupted, can't be repaired
> > > ERROR: data at bytenr 121936633856 mirror 1 csum mismatch, have
> > > 0x507016a5 expect 0x50609afe
> > > ERROR: extent 121858334720 len 134217728 CORRUPTED, all mirror(s)
> > > corrupted, can't be repaired
> > > ERROR: data at bytenr 144872591360 mirror 1 csum mismatch, have
> > > 0x33964d73 expect 0xf9937032
> > > ERROR: extent 144822386688 len 61231104 CORRUPTED, all mirror(s)
> > > corrupted, can't be repaired
> > > ERROR: data at bytenr 167961075712 mirror 1 csum mismatch, have
> > > 0xf43bd0e3 expect 0x5be589bb
> > > ERROR: extent 167950999552 len 27537408 CORRUPTED, all mirror(s)
> > > corrupted, can't be repaired
> > > ERROR: data at bytenr 175643619328 mirror 1 csum mismatch, have
> > > 0x1e168ca1 expect 0xd413b1e0
> > > ERROR: data at bytenr 175643754496 mirror 1 csum mismatch, have
> > > 0x6cfdc8ae expect 0xa6f8f5ef
> > > ERROR: extent 175640539136 len 6381568 CORRUPTED, all mirror(s)
> > > corrupted, can't be repaired
> > > ERROR: data at bytenr 183316750336 mirror 1 csum mismatch, have
> > > 0x145bdf76 expect 0x7390565e
> > > .....
> > > and the list goes on.
> > > 
> > > Questions:
> > > 
> > > 1.  Using "find /mnt -inum 4708" I can link the dmesg to a specific
> > > file. Is there a
> > > way link the the --offline ERRORs above to the inode?
> > > 
> > > 2.  How could do "btrfs device stats /mnt" and normal full scrub fail
> > > to detect the csum errors?
> > > 
> > > 3.  Do these errors appear to be hardware failure (despite pristine
> > > SMART), user error on
> > > volume creation/mounting, or an actual btrfs issue? I feel that the
> > > need for question #1
> > > indicates a problem with btrfs regardless of whether there is a real
> > > hardware failure or not.
> > > 
> > > Next I will try an online scrub of only the sdn device, as before I
> > > was running the full filesystem scrub.
> > > 
> > > On Tue, Oct 24, 2017 at 12:52 AM, Lakshmipathi.G
> > > 
> > > <lakshmipathi.g@gmail.com> wrote:
> > > 
> > > > > Does anyone know why scrub did not catch these errors that show up in dmesg?
> > > > 
> > > > Can you try offline scrub from this repo
> > > > https://github.com/gujx2017/btrfs-progs/tree/offline_scrub and see
> > > > whether it
> > > > detects the issue? "btrfs scrub start --offline "
> > > > 
> > > > ----
> > > > Cheers,
> > > > Lakshmipathi.G
> > > > 
> > > > http://www.giis.co.in http://www.webminal.org
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Chris Murphy]

On Thu, Oct 26, 2017 at 4:34 AM, Zak Kohler <y2k@y2kbugger.com> wrote:

> I will gladly repeat this process, but I am very concerned why this
> corruption happened in the first place.

Right. So everyone who can, needs to run the three scrubs on all
available Btrfs volumes/devices and see if they get any discrepancies.
I only ever use the online scrub so I have no idea if --offline or the
older check --check-data-csum differ from it.

scrub start
scrub start --offline
btrfs check --check-data-csum

I think you've hit a software bug if those three methods don't exactly
agree with each other. And it's a question which one is correct, or if
they all have different bugs in them?



>
> More tests:
>
> scrub start --offline
>     All devices had errors in differing amounts
>     I will verify that these counts are repeatable.
>     Csum error: 150
>     Csum error: 238
>     Csum error: 175
>
> btrfs check
>     found 2179745955840 bytes used, no error found
>
> btrfs check --check-data-csum
>     mirror 0 bytenr 13348855808 csum 2387937020 expected csum 562782116
>     mirror 0 bytenr 23398821888 csum 3602081170 expected csum 1963854755


Offhand that sounds like three different results, which is sorta fakaked.


>     ...
>
> The only thing I could think of is that the btrfs version that I used to mkfs
> was not up to date. Is there a way to determine which version was used to
> create the filesystem?

That information isn't in the superblock. I think it could be added to
the device tree as a PERSISTENT_ITEM, although I'm not sure how useful
it is.

Anyway, I don't think it's related. mkfs.btrfs writes a tiny amount to
the drive, and almost certainly the problem is happening later.



-- 
Chris Murphy

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

On Sun, Oct 29, 2017 at 3:05 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Thu, Oct 26, 2017 at 4:34 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
>
>> I will gladly repeat this process, but I am very concerned why this
>> corruption happened in the first place.
>
> Right. So everyone who can, needs to run the three scrubs on all
> available Btrfs volumes/devices and see if they get any discrepancies.
> I only ever use the online scrub so I have no idea if --offline or the
> older check --check-data-csum differ from it.
>
> scrub start
> scrub start --offline
> btrfs check --check-data-csum
>
> I think you've hit a software bug if those three methods don't exactly
> agree with each other. And it's a question which one is correct, or if
> they all have different bugs in them?
>
>
>
>>
>> More tests:
>>
>> scrub start --offline
>>     All devices had errors in differing amounts
>>     I will verify that these counts are repeatable.
>>     Csum error: 150
>>     Csum error: 238
>>     Csum error: 175
>>
>> btrfs check
>>     found 2179745955840 bytes used, no error found
>>
>> btrfs check --check-data-csum
>>     mirror 0 bytenr 13348855808 csum 2387937020 expected csum 562782116
>>     mirror 0 bytenr 23398821888 csum 3602081170 expected csum 1963854755
>
>
> Offhand that sounds like three different results, which is sorta fakaked.
Those results for passing each of the three devices:

$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX1
Scrub result:
Tree bytes scrubbed: 5234425856
Tree extents scrubbed: 638968
Data bytes scrubbed: 4353723670528
Data extents scrubbed: 374300
Data bytes without csum: 533200896
Read error: 0
Verify error: 0
Csum error: 150

$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX2
Scrub result:
Tree bytes scrubbed: 5234425856
Tree extents scrubbed: 638967
Data bytes scrubbed: 4353723314176
Data extents scrubbed: 374300
Data bytes without csum: 533200896
Read error: 0
Verify error: 0
Csum error: 238

$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX3
Scrub result:
Tree bytes scrubbed: 5234491392
Tree extents scrubbed: 638975
Data bytes scrubbed: 4353723572224
Data extents scrubbed: 374300
Data bytes without csum: 533200896
Read error: 0
Verify error: 0
Csum error: 175 #first run
Csum error: 112 #second run...
Csum error: 285 #third run...

But I ran the /dev/disk/by-id/WD-XX3 device three times and you can
see the result...


So I ran memtest86+ 5.01 for >4 days:
Pass: 39 Errors: 0


Only other think I can think to try is transferring those drives to
another system.

I'm running '$ sudo btrfs scrub start --offline --progress
/dev/disk/by-id/WD-XX3' one more time to just to make sure that I
wasn't reading something wrong.

I just noticed that all three drives are blinking so I can assume that
$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX1
$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX2
$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX3
are all equivalent and are checking the entire filesystem?

The original problem was noticed during a btrfs send, and it would
stop and send a dmesg after an intermittent amount of data. You could
run it 3 times and it would stop at the same spot and then next time
it would make it further.
Given these dmesg error, I'll tabularize the info to more clearly show
the intermittent nature

[  114.450699] BTRFS warning (device sdn): csum failed ino 6407 \
off 7683907584 csum 1745651892 expected csum 3952841867

   time     ino        off          csum    expected
[  114.43]  6407    7683907584  1745651892  3952841867
[  114.45]  6407    7683907584  1745651892  3952841867
[38494.97]  4708    27529216    876064455   874979996
[38494.98]  4708    27529216    2615801759  874979996
[38541.07]  4708    27529216    876064455   874979996
[38571.24]  4708    27529216    2615801759  874979996
[39434.21]  4708    27529216    2615801759  874979996
[73132.65]  4708    27529216    2615801759  874979996
[73167.89]  4708    27529216    2615801759  874979996

Sometimes it gets stuck at the same ino, sometimes not. Sometime the
actual incorrect csum repeats, sometimes it doesn't.



>
>>     ...
>>
>> The only thing I could think of is that the btrfs version that I used to mkfs
>> was not up to date. Is there a way to determine which version was used to
>> create the filesystem?
>
> That information isn't in the superblock. I think it could be added to
> the device tree as a PERSISTENT_ITEM, although I'm not sure how useful
> it is.
>
> Anyway, I don't think it's related. mkfs.btrfs writes a tiny amount to
> the drive, and almost certainly the problem is happening later.
>
>
>
> --
> Chris Murphy

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Lakshmipathi.G]

> I will gladly repeat this process, but I am very concerned why this
> corruption happened in the first place.
>
Can you share these steps as simple bash script? Currently I'm running
few tests, I can check your script and share the results.


> The only thing I could think of is that the btrfs version that I used to mkfs
> was not up to date. Is there a way to determine which version was used to
> create the filesystem?
>

If I'm not wrong, i don't think we can get mkfs version from existing file
system layout. what was your kernel version?

----
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Duncan]

Zak Kohler posted on Sun, 29 Oct 2017 21:57:00 -0400 as excerpted:

> So I ran memtest86+ 5.01 for >4 days:
> Pass: 39 Errors: 0

Be aware that memtest86+ will detect some types of errors but not others.

In particular, some years ago I had some memory (DDR1/3-digit-Opteron 
era), actually registered as required by Opterons and ECC, that passed 
that sort of memory test because what the test /tests/ is memory cell 
retention (if you put a value in does it verify on read-back?), that was 
none-the-less bad memory in that at its rated speed it was unreliable at 
memory /transfers/.

Eventually that mobo got a BIOS update that could adjust memory clocking, 
and I downclocked it a notch (from its rated pc3200 to 3000, IIRC).  At 
the lower clock it was rock stable, even with reduced wait-states to make 
up a bit of the performance I was losing to the lower clock.  But I had 
to get a BIOS that could do it, first, and as I said, in the mean time 
memtest86, etc, reported it was fine, because it was testing stable-
state, not stress-testing memory transfers at full rated bandwidth.

Eventually I did a memory upgrade, and the new memory worked fine at full 
rated speed.

As for actual symptoms, that was well before I switched to btrfs 
(actually probably about time btrfs development started, I'd guess), but 
the most frequent early symptoms were untar/bunzip2 checksum failures, 
with some system crashes during builds (gentoo so I do a lot of 
untarballing of sources and builds).  Later, when the kernel got support 
for the amd/opteron system-check error detection hardware, that would 
sometimes, but not always, trigger as well, but the most frequent symptom 
really was checksum verification failures untarring tbz2s.

Of course if btrfs had been around for me to try to run back then I 
expect I'd have been triggering enough checksum failures that I'd have 
given up on running it.  But FWIW, tho the reiserfs I was running wasn't 
checksummed, it didn't ever seem to significantly corrupt, with this or 
other hardware problems I had.  

That's what really amazed me about reiserfs, that it remained stable thru 
not only that hardware problem but various others I've had over the years 
that would have killed or made unworkable other filesystems.  Once it got 
ordered journaling (as opposed to the original writeback journaling that 
gave it the bad rep... and later poked holes in ext3 reliability when 
they tried it there for a few kernel versions too) and switched to 
ordered by default, it really /was/ remarkably stable, even in the face 
of hardware issues that would take down many filesystems.

Of course one of the things that attracted me to btrfs years later was 
that it was the same Chris Mason that helped reiserfs get that ordered 
journaling back when he was working for SuSE and they were using reiserfs 
by default, that began btrfs.

Tho btrfs is far more complex, and isn't yet as stable as reiserfs ended 
up being for me.  And on bad hardware it may never be, even if eventually 
it's simply due to checksum failures making it effectively unusable.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

Lakshmipathi.G posted on Mon, Oct 30, 2017 at 12:07 AM as excerpted:

> Can you share these steps as simple bash script? Currently I'm running
> few tests, I can check your script and share the results.

sure, but it is pretty trivial:

device1=/dev/disk/by-id/XX1
device2=/dev/disk/by-id/XX2
device3=/dev/disk/by-id/XX3

mkfs.btrfs -L OfflineJ $device1 $device2 $device3
mount -t btrfs $device1 /mnt
btrfs subvolume create /mnt/dataroot
mkdir -p /media/OfflineJ

umount /mnt
mount -o subvol=/dataroot $device1 /media/OfflineJ

> > The only thing I could think of is that the btrfs version that I used to mkfs
> > was not up to date. Is there a way to determine which version was used to
> > create the filesystem?
> >
> If I'm not wrong, i don't think we can get mkfs version from existing file
> system layout. what was your kernel version?

Not quite sure.
Oldest: archlinux linux-lts as of 2017.06.08
Most likely: archlinux linux-lts as of 2017.10.18

It may be possible that I updated and created the fs without
rebooting or reloading the kernel modules. But normally in that case
any interaction, even mounting, just fails loudly.


Duncan posted on Mon, Oct 30, 2017 at 12:09 AM as excerpted:

> and I downclocked it a notch (from its rated pc3200 to 3000, IIRC).  At

Hmm, I did have a mild overclock on this system when I used it as a desktop
but I thought I reverted it when I made it into a server. I'll double
check this.

> sometimes, but not always, trigger as well, but the most frequent symptom
> really was checksum verification failures untarring tbz2s.

I'll have to try and recreate this using my EXT4 system disk.


Zak Kohler posted on Sun, Oct 29, 2017 at 9:57 PM as excerpted:

> I'm running '$ sudo btrfs scrub start --offline --progress
> /dev/disk/by-id/WD-XX3' one more time to just to make sure that I
> wasn't reading something wrong.

Well even stranger news on the fourth run:

$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX3
Csum error: 175 # first run
Csum error: 112 # second run...
Csum error: 285 # third run...
Csum error: 0 # fourth run

Also can anyone confirm that the above --offline command checks the whole
filesystem regardless which device is specified on the cli?

> Only other think I can think to try is transferring those drives to
> another system.

I still need to try this also.

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Chris Murphy]

On Mon, Oct 30, 2017 at 2:57 AM, Zak Kohler <y2k@y2kbugger.com> wrote:

> $ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX1
> Scrub result:
> Tree bytes scrubbed: 5234425856
> Tree extents scrubbed: 638968
> Data bytes scrubbed: 4353723670528
> Data extents scrubbed: 374300
> Data bytes without csum: 533200896
> Read error: 0
> Verify error: 0
> Csum error: 150
>
> $ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX2
> Scrub result:
> Tree bytes scrubbed: 5234425856
> Tree extents scrubbed: 638967
> Data bytes scrubbed: 4353723314176
> Data extents scrubbed: 374300
> Data bytes without csum: 533200896
> Read error: 0
> Verify error: 0
> Csum error: 238
>
> $ sudo btrfs scrub start --offline --progress /dev/disk/by-id/WD-XX3
> Scrub result:
> Tree bytes scrubbed: 5234491392
> Tree extents scrubbed: 638975
> Data bytes scrubbed: 4353723572224
> Data extents scrubbed: 374300
> Data bytes without csum: 533200896
> Read error: 0
> Verify error: 0
> Csum error: 175 #first run
> Csum error: 112 #second run...
> Csum error: 285 #third run...
>
> But I ran the /dev/disk/by-id/WD-XX3 device three times and you can
> see the result...


I expect these commands are the same, and involve all three drives in
the offline scrub each time. So you have five different results, but
all five involve csum errors. So the errors have a certain transience
to them, hence inconsistent results.

But the online scrub consistently reports zero errors. That to me
sounds like a bug in the offline scrub code. Maybe it's confused, and
reports data without csums (nodatacow) as csum errors? That does not
explain the inconsistency though.

And then you're getting an consistent failure, but at an inconsistent
location, with Btrfs send, ostensibly due to IO error, which sounds
like it's hitting a bad csum check.

It is entirely possible to get transient errors like this somewhere in
a storage stack that's otherwise not reported by the error detection
code in that layer. The thing I really don't understand is how you're
getting zero errors with conventional online scrub, every time.

On my tiny 23G installation I'm traveling with, I get the same results
with all three scrub methods on an NVMe drive. Zero errors. The
slighly larger spinning rust drives are not with me so I can't check
them for a while.



-- 
Chris Murphy

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Duncan]

Duncan posted on Mon, 30 Oct 2017 04:09:58 +0000 as excerpted:

> Zak Kohler posted on Sun, 29 Oct 2017 21:57:00 -0400 as excerpted:
> 
>> So I ran memtest86+ 5.01 for >4 days:
>> Pass: 39 Errors: 0
> 
> Be aware that memtest86+ will detect some types of errors but not
> others.
> 
> In particular, some years ago I had some memory (DDR1/3-digit-Opteron
> era), actually registered as required by Opterons and ECC, that passed
> that sort of memory test because what the test /tests/ is memory cell
> retention (if you put a value in does it verify on read-back?), that was
> none-the-less bad memory in that at its rated speed it was unreliable at
> memory /transfers/.
> 
> Eventually that mobo got a BIOS update that could adjust memory
> clocking, and I downclocked it a notch[.] At
> the lower clock it was rock stable, even with reduced wait-states to
> make up a bit of the performance I was losing to the lower clock.  But I
> had to get a BIOS that could do it, first [...]

> That's what really amazed me about reiserfs, that it remained stable
> thru not only that hardware problem but various others I've had over the
> years that would have killed or made unworkable other filesystems.

BTW, one of those other hardware problems I had, the one that ultimately 
did in my old server-clase mobo, was leaky capacitors on the then 8-ish 
years old system.  It was of the generation that had problem capacitors, 
and it eventually succumbed...

The reason this is relevant is that it was the storage path that had the 
worst problems.  Before I figured out what the problem actually was I did 
try btrfs, with around kernel 3.6 at the time, and it really /was/ 
unusable due to checksum errors.  But I could still limp along with 
reiserfs...

One thing about the behavior I noticed, however, was that as the problem 
was developing, the system was more usable if I kept it reasonably cool.  
By the time I gave up on it, it was early summer here in Phoenix, and 
temperatures were climbing.  But I was sitting at home with the AC on, in 
a heavy winter jacket, wearing sweats under my pants and trying to type 
with gloves on my hands to keep warm, in ordered to cool down the 
computer so it'd work.  That's when I decided enough was enough and gave 
up on it.  I only found the burst capacitors, however, once I got the new 
mobo and was switching out the old one.  No WONDER it wasn't working 
right any more!

The rest of the system was actually reasonably stable, however.  I guess 
the worst caps were in the storage path.

But as I said, reiserfs was amazing.  I bought a SATA addon board with 
the same chipset as on the old mobo so I could boot the old monolithic 
kernel with those drivers builtin on the new mobo, and I didn't notice 
any corruption or anything.  But as I said, btrfs was entirely unusable 
on the old hardware, due to both checksum errors and large transactions 
(like trying to copy files over from reiserfs) ending up entirely 
reverted when I'd crash, instead of the partial completion I'd get on 
reiserfs, so I could at least reboot and start where it has crashed on 
reiserfs, instead of having to start over entirely, thus making no 
progress at all, which is what I was seeing on btrfs.

That reiserfs continued to work well enough to keep going so long under 
those conditions, while btrfs was entirely unworkable, and even more that 
once I was running good hardware again, I didn't see massive corruption 
on reiserfs as a result of trying to run it on so long on the bad 
hardware, was really /really/ amazing!

But like I said, I don't expect that btrfs, with its checksumming, with 
/ever/ be really workable on that sort of defective hardware.  It was 
just really amazing to me that reiserfs wasn't screwed up by it as well, 
as it had every right to be given the screwed up hardware I was trying to 
run it on.  No filesystem can be expected to go thru that and end up 
still usable, but somehow, reiserfs did.

Bottom line, it could be the storage path, not the memory or cpu.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Zak Kohler]

Well it looks like things have stabilized....for the
moment at least.

$ btrfs scrub start --offline --progress /dev/disk/by-id/XX3
Doing offline scrub [o] [681/681]
Scrub result:
Tree bytes scrubbed: 5234425856
Tree extents scrubbed: 638968
Data bytes scrubbed: 4353724284928
Data extents scrubbed: 374300
Data bytes without csum: 533200896
Read error: 0
Verify error: 0
Csum error: 0

$ sudo btrfs scrub start --offline --progress /dev/disk/by-id/XX3
Doing offline scrub [o] [681/681]
Scrub result:
Tree bytes scrubbed: 5234425856
Tree extents scrubbed: 638968
Data bytes scrubbed: 4353724284928
Data extents scrubbed: 374300
Data bytes without csum: 533200896
Read error: 0
Verify error: 0
Csum error: 0

$ sudo btrfs send /mnt/dataroot.2017.10.21 | pv -i2 > /dev/null
At subvol /mnt/dataroot.2017.10.21
1.55TiB 1:38:46 [ 283MiB/s] [         <=>                               ]

One interesting note is that when the --offline scrub came back with
Csum errors, sometimes the Tree bytes scrubbed were different:

Tree bytes scrubbed: 5234491392 #bad
vs
Tree bytes scrubbed: 5234425856 #good

The hardware is a Q6600 (the first Core2 Quad @2.4GHz) and a dell PERC
6/i card flashed with IT mode.

*** 2 days have past since I wrote the above
I checked my overclock and sure enough I had the FSB boosted, CPU
reaching ~2.9 GHz. The PCI were held at a constant freq but I bet
there was some bad interaction with the PERC. I don't know why the
system chose to be stable 2 days ago before resetting the overclock,
but I am very confident it will stay that way now.

Takeaways:

1. I came to btrfs because upon manual hash comparison I noticed bit
flips occurring. Now I have very likely found the source of the issues
thanks to btrfs and I can also be more confident against those issues
in the future.

2. A stable memtest86+ doesn't necessarily mean a stable storage stack

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Chris Murphy]

On Sun, Oct 29, 2017 at 1:05 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Thu, Oct 26, 2017 at 4:34 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
>
>> I will gladly repeat this process, but I am very concerned why this
>> corruption happened in the first place.
>
> Right. So everyone who can, needs to run the three scrubs on all
> available Btrfs volumes/devices and see if they get any discrepancies.

I've just done this test with five Btrfs volumes of various ages, and
a total of a few TB. There were no errors, and no discrepancies
between the three scrubs. What is needed for more thorough testing is
corrupt metadata and data (unrelated), and see how the three scrubs
report those corruptions. More information in the new thread "feedback
on three different scrub methods"


-- 
Chris Murphy

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Chris Murphy]

On Sat, Nov 11, 2017 at 11:27 AM, Zak Kohler <y2k@y2kbugger.com> wrote:
> It seems that since the errors were due to a very slight instability in
> computer due to overclock, the online and offline scrub's causing different
> stresses or paths. Could it be that one is telling the drives to use some
> level of caching where the offline is explicitly tells it not to(or one of
> many similar scenarios)? There must be some line to draw about supporting
> faulty Hardware but I think having all forms of scrubbing in agreement must
> be within scope.

I think once you're in some non-deterministic state, all bets are off.
But I don't know anything about what physical parts of a CPU could
cause more or less error, or more or less determinism, based on code
differences.


-- 
Chris Murphy