Unable to delete directory: input/output error and corrupt leaf

Unable to delete directory: input/output error and corrupt leaf

[Barnes, Samuel]

I have a btrfs drive that started have problems, I first noticed some of my old backup directories couldn't be deleted, the automated delete commands removed all the files, but got stuck on the directories. Now if I try to delete I get input/output error:

rm -rf 20190806-020001-412/
rm: cannot remove '20190806-020001-412/backup/media/network_mriphysics/HAT_data/Phantoms/Penn State Coll Time1': Input/output error

dmesg shows the following errors:

[11655.022724] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027629] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027990] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.028311] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3

I have tried a scrub, no errors:

sudo ./btrfs scrub status /dev/sdc1
UUID:             d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
Scrub started:    Tue Sep 17 10:43:20 2019
Status:           finished
Duration:         1:46:29
Total to scrub:   1.18TiB
Rate:             193.46MiB/s
Error summary:    no errors found

a btrfs check gives a very very long list of errors most include "link count wrong" such as:

root 5 inode 5202109 errors 2000, link count wrong
    unresolved ref dir 7424552 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
    unresolved ref dir 8454942 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
    unresolved ref dir 8616651 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index

another example:

root 5 inode 7422709 errors 2001, no inode item, link count wrong
    unresolved ref dir 261 index 182 namelen 19 name 20190630-020001-715 filetype 2 errors 4, no inode ref

I ran a balance, that didn't help, and interestingly if I run check --repair it aborts:

sudo ./btrfs check --repair /dev/sdc1
enabling repair mode
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
incorrect offsets 9191 67118055
incorrect offsets 9191 67118055
incorrect offsets 9191 67118055
incorrect offsets 9191 67118055
items overlap, can't fix
check/main.c:4265: fix_item_offset: BUG_ON `ret` triggered, value -5 ./btrfs(+0x5e2f0)[0x557b1e3c62f0] ./btrfs(+0x5e396)[0x557b1e3c6396] ./btrfs(+0x6951b)[0x557b1e3d151b] ./btrfs(+0x6a45a)[0x557b1e3d245a] ./btrfs(+0x6cf09)[0x557b1e3d4f09] ./btrfs(main+0x95)[0x557b1e37c690]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7aec452b97]
./btrfs(_start+0x2a)[0x557b1e37c1ea]
Aborted

dev stats doesn't show any corruption errors:

sudo ./btrfs dev stats /dev/sdc1
[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0

SMART doesn't show any problems. I updated my kernel to 5.0 and built the latest version of btrfs-prog (5.2.2) and have been using that. I can't find anything else to try, these are incremental backups so it would be nice to save them if possible. Any help much appreciated!! Version and other info listed below:

uname -a
Linux mrispec 5.0.0-27-generic #28~18.04.1-Ubuntu SMP Thu Aug 22 03:00:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

./btrfs --version
btrfs-progs v5.2.2 

sudo ./btrfs fi show
Label: 'backup'  uuid: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
    Total devices 1 FS bytes used 1.17TiB
    devid    1 size 9.09TiB used 1.18TiB path /dev/sdc1

sudo ./btrfs fi df /media/mrispec/backup/ Data, single: total=1.16TiB, used=1.16TiB System, DUP: total=8.00MiB, used=160.00KiB Metadata, DUP: total=11.00GiB, used=8.33GiB GlobalReserve, single: total=512.00MiB, used=0.00B





CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain confidential and privileged information for the use of the designated recipients named above. If you are not the intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please notify me immediately by replying to this message and destroy all copies of this communication and any attachments. Thank you.

Re: Unable to delete directory: input/output error and corrupt leaf

[Chris Murphy]

On Thu, Sep 19, 2019 at 11:38 AM Barnes, Samuel <SABarnes@llu.edu> wrote:
>
> I have a btrfs drive that started have problems, I first noticed some of my old backup directories couldn't be deleted, the automated delete commands removed all the files, but got stuck on the directories. Now if I try to delete I get input/output error:
>
> rm -rf 20190806-020001-412/
> rm: cannot remove '20190806-020001-412/backup/media/network_mriphysics/HAT_data/Phantoms/Penn State Coll Time1': Input/output error
>
> dmesg shows the following errors:
>
>
[11655.022724] BTRFS critical (device sdc1): corrupt leaf: root=5
block=1399280648192 slot=69 ino=8904454, name hash mismatch with key,
have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027629] BTRFS
critical (device sdc1): corrupt leaf: root=5 block=1399280648192
slot=69 ino=8904454, name hash mismatch with key, have
0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027990] BTRFS
critical (device sdc1): corrupt leaf: root=5 block=1399280648192
slot=69 ino=8904454, name hash mismatch with key, have
0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.028311] BTRFS
critical (device sdc1): corrupt leaf: root=5 block=1399280648192
slot=69 ino=8904454, name hash mismatch with key, have
0x00000000d8649173 expect 0x00000000f88d2ac3
>

What do you get for:
# btrfs insp dump-t -b 1399280648192 /dev/sdc1

This likely exposes filenames, it's OK to remove them.

> I have tried a scrub, no errors:
>
> sudo ./btrfs scrub status /dev/sdc1
> UUID:             d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
> Scrub started:    Tue Sep 17 10:43:20 2019
> Status:           finished
> Duration:         1:46:29
> Total to scrub:   1.18TiB
> Rate:             193.46MiB/s
> Error summary:    no errors found
>
> a btrfs check gives a very very long list of errors most include "link count wrong" such as:
>
> root 5 inode 5202109 errors 2000, link count wrong
>     unresolved ref dir 7424552 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
>     unresolved ref dir 8454942 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
>     unresolved ref dir 8616651 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
>
> another example:
>
> root 5 inode 7422709 errors 2001, no inode item, link count wrong
>     unresolved ref dir 261 index 182 namelen 19 name 20190630-020001-715 filetype 2 errors 4, no inode ref
>
> I ran a balance, that didn't help, and interestingly if I run check --repair it aborts:
>
> sudo ./btrfs check --repair /dev/sdc1
> enabling repair mode
> Opening filesystem to check...
> Checking filesystem on /dev/sdc1
> UUID: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055
> items overlap, can't fix
> check/main.c:4265: fix_item_offset: BUG_ON `ret` triggered, value -5 ./btrfs(+0x5e2f0)[0x557b1e3c62f0] ./btrfs(+0x5e396)[0x557b1e3c6396] ./btrfs(+0x6951b)[0x557b1e3d151b] ./btrfs(+0x6a45a)[0x557b1e3d245a] ./btrfs(+0x6cf09)[0x557b1e3d4f09] ./btrfs(main+0x95)[0x557b1e37c690]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7aec452b97]
> ./btrfs(_start+0x2a)[0x557b1e37c1ea]
> Aborted

That's a bug.

I think overall the file system is probably fine, but there's an older
kernel bug that caused this leaf to become corrupt at some point, and
increasingly stronger checking is what's exposing it now. So the trick
is really how to delete the offending file, because right now it looks
like that isn't being allowed.

Qu might have an idea.



-- 
Chris Murphy

Re: Unable to delete directory: input/output error and corrupt leaf

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 5617 bytes --]



On 2019/9/20 上午1:38, Barnes, Samuel wrote:
> I have a btrfs drive that started have problems, I first noticed some of my old backup directories couldn't be deleted, the automated delete commands removed all the files, but got stuck on the directories. Now if I try to delete I get input/output error:
> 
> rm -rf 20190806-020001-412/
> rm: cannot remove '20190806-020001-412/backup/media/network_mriphysics/HAT_data/Phantoms/Penn State Coll Time1': Input/output error
> 
> dmesg shows the following errors:
> 
> [11655.022724] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027629] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027990] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.028311] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3

Some old kernel screwed up the leaf.
One ITEM_DIR has bad hash. Normally it should be fixed by btrfs check,
but...

> 
> I have tried a scrub, no errors:
> 
> sudo ./btrfs scrub status /dev/sdc1
> UUID:             d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
> Scrub started:    Tue Sep 17 10:43:20 2019
> Status:           finished
> Duration:         1:46:29
> Total to scrub:   1.18TiB
> Rate:             193.46MiB/s
> Error summary:    no errors found
> 
> a btrfs check gives a very very long list of errors most include "link count wrong" such as:

That's in fact not the biggest problem.

> 
> root 5 inode 5202109 errors 2000, link count wrong
>     unresolved ref dir 7424552 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
>     unresolved ref dir 8454942 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
>     unresolved ref dir 8616651 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index
> 
> another example:
> 
> root 5 inode 7422709 errors 2001, no inode item, link count wrong
>     unresolved ref dir 261 index 182 namelen 19 name 20190630-020001-715 filetype 2 errors 4, no inode ref
> 
> I ran a balance, that didn't help, and interestingly if I run check --repair it aborts:
> 
> sudo ./btrfs check --repair /dev/sdc1
> enabling repair mode
> Opening filesystem to check...
> Checking filesystem on /dev/sdc1
> UUID: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055

And this is the biggest problem.

You have bad extent tree offsets, that's a bigger problem.
In that case, the corruption is pretty beyond repair, although I'm not
sure how this happened, it's definitely not a good sign to your fs.

I'd recommend to salvage your data and rebuild the fs.

Thanks,
Qu

> items overlap, can't fix
> check/main.c:4265: fix_item_offset: BUG_ON `ret` triggered, value -5 ./btrfs(+0x5e2f0)[0x557b1e3c62f0] ./btrfs(+0x5e396)[0x557b1e3c6396] ./btrfs(+0x6951b)[0x557b1e3d151b] ./btrfs(+0x6a45a)[0x557b1e3d245a] ./btrfs(+0x6cf09)[0x557b1e3d4f09] ./btrfs(main+0x95)[0x557b1e37c690]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7aec452b97]
> ./btrfs(_start+0x2a)[0x557b1e37c1ea]
> Aborted
> 
> dev stats doesn't show any corruption errors:
> 
> sudo ./btrfs dev stats /dev/sdc1
> [/dev/sdc1].write_io_errs    0
> [/dev/sdc1].read_io_errs     0
> [/dev/sdc1].flush_io_errs    0
> [/dev/sdc1].corruption_errs  0
> [/dev/sdc1].generation_errs  0
> 
> SMART doesn't show any problems. I updated my kernel to 5.0 and built the latest version of btrfs-prog (5.2.2) and have been using that. I can't find anything else to try, these are incremental backups so it would be nice to save them if possible. Any help much appreciated!! Version and other info listed below:
> 
> uname -a
> Linux mrispec 5.0.0-27-generic #28~18.04.1-Ubuntu SMP Thu Aug 22 03:00:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> 
> ./btrfs --version
> btrfs-progs v5.2.2 
> 
> sudo ./btrfs fi show
> Label: 'backup'  uuid: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
>     Total devices 1 FS bytes used 1.17TiB
>     devid    1 size 9.09TiB used 1.18TiB path /dev/sdc1
> 
> sudo ./btrfs fi df /media/mrispec/backup/ Data, single: total=1.16TiB, used=1.16TiB System, DUP: total=8.00MiB, used=160.00KiB Metadata, DUP: total=11.00GiB, used=8.33GiB GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 
> 
> 
> 
> CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain confidential and privileged information for the use of the designated recipients named above. If you are not the intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please notify me immediately by replying to this message and destroy all copies of this communication and any attachments. Thank you.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Unable to delete directory: input/output error and corrupt leaf

[Barnes, Samuel]

Bummer, thanks for the help.

Samuel Barnes


-----Original Message-----
From: Qu Wenruo [mailto:quwenruo.btrfs@gmx.com] 
Sent: Thursday, September 19, 2019 3:37 PM
To: Barnes, Samuel; linux-btrfs@vger.kernel.org
Subject: [EXTERNAL] Re: Unable to delete directory: input/output error and corrupt leaf



On 2019/9/20 上午1:38, Barnes, Samuel wrote:
> I have a btrfs drive that started have problems, I first noticed some of my old backup directories couldn't be deleted, the automated delete commands removed all the files, but got stuck on the directories. Now if I try to delete I get input/output error:
> 
> rm -rf 20190806-020001-412/
> rm: cannot remove 
> '20190806-020001-412/backup/media/network_mriphysics/HAT_data/Phantoms
> /Penn State Coll Time1': Input/output error
> 
> dmesg shows the following errors:
> 
> [11655.022724] BTRFS critical (device sdc1): corrupt leaf: root=5 
> block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, 
> have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027629] BTRFS 
> critical (device sdc1): corrupt leaf: root=5 block=1399280648192 
> slot=69 ino=8904454, name hash mismatch with key, have 
> 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027990] BTRFS 
> critical (device sdc1): corrupt leaf: root=5 block=1399280648192 
> slot=69 ino=8904454, name hash mismatch with key, have 
> 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.028311] BTRFS 
> critical (device sdc1): corrupt leaf: root=5 block=1399280648192 
> slot=69 ino=8904454, name hash mismatch with key, have 
> 0x00000000d8649173 expect 0x00000000f88d2ac3

Some old kernel screwed up the leaf.
One ITEM_DIR has bad hash. Normally it should be fixed by btrfs check, but...

> 
> I have tried a scrub, no errors:
> 
> sudo ./btrfs scrub status /dev/sdc1
> UUID:             d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
> Scrub started:    Tue Sep 17 10:43:20 2019
> Status:           finished
> Duration:         1:46:29
> Total to scrub:   1.18TiB
> Rate:             193.46MiB/s
> Error summary:    no errors found
> 
> a btrfs check gives a very very long list of errors most include "link count wrong" such as:

That's in fact not the biggest problem.

> 
> root 5 inode 5202109 errors 2000, link count wrong
>     unresolved ref dir 7424552 index 19 namelen 9 name 12-25.dcm 
> filetype 0 errors 3, no dir item, no dir index
>     unresolved ref dir 8454942 index 19 namelen 9 name 12-25.dcm 
> filetype 0 errors 3, no dir item, no dir index
>     unresolved ref dir 8616651 index 19 namelen 9 name 12-25.dcm 
> filetype 0 errors 3, no dir item, no dir index
> 
> another example:
> 
> root 5 inode 7422709 errors 2001, no inode item, link count wrong
>     unresolved ref dir 261 index 182 namelen 19 name 
> 20190630-020001-715 filetype 2 errors 4, no inode ref
> 
> I ran a balance, that didn't help, and interestingly if I run check --repair it aborts:
> 
> sudo ./btrfs check --repair /dev/sdc1
> enabling repair mode
> Opening filesystem to check...
> Checking filesystem on /dev/sdc1
> UUID: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055
> incorrect offsets 9191 67118055

And this is the biggest problem.

You have bad extent tree offsets, that's a bigger problem.
In that case, the corruption is pretty beyond repair, although I'm not sure how this happened, it's definitely not a good sign to your fs.

I'd recommend to salvage your data and rebuild the fs.

Thanks,
Qu

> items overlap, can't fix
> check/main.c:4265: fix_item_offset: BUG_ON `ret` triggered, value -5 
> ./btrfs(+0x5e2f0)[0x557b1e3c62f0] ./btrfs(+0x5e396)[0x557b1e3c6396] 
> ./btrfs(+0x6951b)[0x557b1e3d151b] ./btrfs(+0x6a45a)[0x557b1e3d245a] 
> ./btrfs(+0x6cf09)[0x557b1e3d4f09] ./btrfs(main+0x95)[0x557b1e37c690]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7aec452b97
> ]
> ./btrfs(_start+0x2a)[0x557b1e37c1ea]
> Aborted
> 
> dev stats doesn't show any corruption errors:
> 
> sudo ./btrfs dev stats /dev/sdc1
> [/dev/sdc1].write_io_errs    0
> [/dev/sdc1].read_io_errs     0
> [/dev/sdc1].flush_io_errs    0
> [/dev/sdc1].corruption_errs  0
> [/dev/sdc1].generation_errs  0
> 
> SMART doesn't show any problems. I updated my kernel to 5.0 and built the latest version of btrfs-prog (5.2.2) and have been using that. I can't find anything else to try, these are incremental backups so it would be nice to save them if possible. Any help much appreciated!! Version and other info listed below:
> 
> uname -a
> Linux mrispec 5.0.0-27-generic #28~18.04.1-Ubuntu SMP Thu Aug 22 
> 03:00:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> 
> ./btrfs --version
> btrfs-progs v5.2.2
> 
> sudo ./btrfs fi show
> Label: 'backup'  uuid: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a
>     Total devices 1 FS bytes used 1.17TiB
>     devid    1 size 9.09TiB used 1.18TiB path /dev/sdc1
> 
> sudo ./btrfs fi df /media/mrispec/backup/ Data, single: total=1.16TiB, 
> used=1.16TiB System, DUP: total=8.00MiB, used=160.00KiB Metadata, DUP: 
> total=11.00GiB, used=8.33GiB GlobalReserve, single: total=512.00MiB, 
> used=0.00B
> 
> 
> 
> 
> 


CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain confidential and privileged information for the use of the designated recipients named above. If you are not the intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please notify me immediately by replying to this message and destroy all copies of this communication and any attachments. Thank you.