btrfs becomes read-only

btrfs becomes read-only

[Alexey Isaev]

Hello!

BTRFS volume becomes read-only with this messages in dmesg.
What can i do to repair btrfs partition?

[Jan25 08:18] BTRFS error (device sdg): parent transid verify failed on 
52180048330752 wanted 132477 found 132432
[  +0.007587] BTRFS error (device sdg): parent transid verify failed on 
52180048330752 wanted 132477 found 132432
[  +0.000132] BTRFS error (device sdg): qgroup scan failed with -5

[Jan25 19:52] BTRFS error (device sdg): parent transid verify failed on 
52180048330752 wanted 132477 found 132432
[  +0.009783] BTRFS error (device sdg): parent transid verify failed on 
52180048330752 wanted 132477 found 132432
[  +0.000132] BTRFS: error (device sdg) in __btrfs_cow_block:1176: 
errno=-5 IO failure
[  +0.000060] BTRFS info (device sdg): forced readonly
[  +0.000004] BTRFS info (device sdg): failed to delete reference to 
ftrace.h, inode 2986197 parent 2989315
[  +0.000002] BTRFS: error (device sdg) in __btrfs_unlink_inode:4220: 
errno=-5 IO failure
[  +0.006071] BTRFS error (device sdg): pending csums is 430080

-- 
Best Regards,
Aleksey Isaev,
RQC

Re: btrfs becomes read-only

[Chris Murphy]

On Wed, Jan 27, 2021 at 12:22 AM Alexey Isaev <a.isaev@rqc.ru> wrote:
>
> Hello!
>
> BTRFS volume becomes read-only with this messages in dmesg.
> What can i do to repair btrfs partition?
>
> [Jan25 08:18] BTRFS error (device sdg): parent transid verify failed on
> 52180048330752 wanted 132477 found 132432
> [  +0.007587] BTRFS error (device sdg): parent transid verify failed on
> 52180048330752 wanted 132477 found 132432
> [  +0.000132] BTRFS error (device sdg): qgroup scan failed with -5
>
> [Jan25 19:52] BTRFS error (device sdg): parent transid verify failed on
> 52180048330752 wanted 132477 found 132432
> [  +0.009783] BTRFS error (device sdg): parent transid verify failed on
> 52180048330752 wanted 132477 found 132432
> [  +0.000132] BTRFS: error (device sdg) in __btrfs_cow_block:1176:
> errno=-5 IO failure
> [  +0.000060] BTRFS info (device sdg): forced readonly
> [  +0.000004] BTRFS info (device sdg): failed to delete reference to
> ftrace.h, inode 2986197 parent 2989315
> [  +0.000002] BTRFS: error (device sdg) in __btrfs_unlink_inode:4220:
> errno=-5 IO failure
> [  +0.006071] BTRFS error (device sdg): pending csums is 430080

What kernel version? What drive make/model?

wanted 132477 found 132432 indicates the drive has lost ~45
transactions, that's not good and also weird. There's no crash or any
other errors? A complete dmesg might be more revealing. And also

smartctl -x /dev/sdg
btrfs check --readonly /dev/sdg

After that I suggest
https://btrfs.wiki.kernel.org/index.php/Restore

And try to get any important data out if it's not backed up. You can
try btrfs-find-root to get a listing of roots, most recent to oldest.
Start at the top, and plug that address in as 'btrfs restore -t' and
see if it'll pull anything out. You likely need -i and -v options as
well.

-- 
Chris Murphy

Re: btrfs becomes read-only

[Alexey Isaev]

kernel version:

aleksey@host:~$ sudo uname --all
Linux host 4.15.0-132-generic #136~16.04.1-Ubuntu SMP Tue Jan 12 
18:22:20 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

drive make/model:

Drive is external 5 bay HDD enclosure with raid-5 connected via usb-3 
(made by Orico https://www.orico.cc/us/product/detail/3622.html)
with 5 WD Red 10 Tb. We use this drive for backups.

When i try to run btrfs check i get error message:

aleksey@host:~$ sudo btrfs check --readonly /dev/sdg
Couldn't open file system

aleksey@host:~$ sudo smartctl -x /dev/sdg
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-132-generic] (local 
build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     JMicron H/W RAID5
Serial Number:    1IZ6AZCKMIFZYJ8A7V0W
Firmware Version: 0964
User Capacity:    40,003,191,177,216 bytes [40.0 TB]
Sector Size:      512 bytes logical/physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ATA/ATAPI-7 (minor revision not indicated)
Local Time is:    Wed Jan 27 08:51:02 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Disabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers 
missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                         was never started.
                                         Auto Offline Data Collection: 
Disabled.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x00)         Offline data collection 
not supported.
SMART capabilities:            (0x0000) Automatic saving of SMART 
data                                  is not implemented.
Error logging capability:        (0x00) Error logging NOT supported.
                                         No General Purpose Logging support.

General Purpose Log Directory not supported

SMART Log Directory Version 0
Address    Access  R/W   Size  Description
0x00           SL  R/O      1  Log Directory

SMART Extended Comprehensive Error Log (GP Log 0x03) not supported

SMART Error Log not supported

SMART Extended Self-test Log (GP Log 0x07) not supported

SMART Self-test Log not supported

Selective Self-tests/Logging not supported

SCT Commands not supported

Device Statistics (GP/SMART Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11) not supported

С уважением,
Алексей Исаев,
RQC

27.01.2021 10:38, Chris Murphy пишет:
> On Wed, Jan 27, 2021 at 12:22 AM Alexey Isaev <a.isaev@rqc.ru> wrote:
>> Hello!
>>
>> BTRFS volume becomes read-only with this messages in dmesg.
>> What can i do to repair btrfs partition?
>>
>> [Jan25 08:18] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.007587] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.000132] BTRFS error (device sdg): qgroup scan failed with -5
>>
>> [Jan25 19:52] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.009783] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.000132] BTRFS: error (device sdg) in __btrfs_cow_block:1176:
>> errno=-5 IO failure
>> [  +0.000060] BTRFS info (device sdg): forced readonly
>> [  +0.000004] BTRFS info (device sdg): failed to delete reference to
>> ftrace.h, inode 2986197 parent 2989315
>> [  +0.000002] BTRFS: error (device sdg) in __btrfs_unlink_inode:4220:
>> errno=-5 IO failure
>> [  +0.006071] BTRFS error (device sdg): pending csums is 430080
> What kernel version? What drive make/model?
>
> wanted 132477 found 132432 indicates the drive has lost ~45
> transactions, that's not good and also weird. There's no crash or any
> other errors? A complete dmesg might be more revealing. And also
>
> smartctl -x /dev/sdg
> btrfs check --readonly /dev/sdg
>
> After that I suggest
> https://btrfs.wiki.kernel.org/index.php/Restore
>
> And try to get any important data out if it's not backed up. You can
> try btrfs-find-root to get a listing of roots, most recent to oldest.
> Start at the top, and plug that address in as 'btrfs restore -t' and
> see if it'll pull anything out. You likely need -i and -v options as
> well.
>

Re: btrfs becomes read-only

[Chris Murphy]

On Wed, Jan 27, 2021 at 1:57 AM Alexey Isaev <a.isaev@rqc.ru> wrote:
>
> kernel version:
>
> aleksey@host:~$ sudo uname --all
> Linux host 4.15.0-132-generic #136~16.04.1-Ubuntu SMP Tue Jan 12
> 18:22:20 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

This is an old and EOL kernel. It could be a long fixed Btrfs bug that
caused this problem, I'm not sure. I suggest 5.4.93+ if you need a
longterm kernel, otherwise 5.10.11 is the current stable kernel.


>
> drive make/model:
>
> Drive is external 5 bay HDD enclosure with raid-5 connected via usb-3
> (made by Orico https://www.orico.cc/us/product/detail/3622.html)
> with 5 WD Red 10 Tb. We use this drive for backups.
>
> When i try to run btrfs check i get error message:
>
> aleksey@host:~$ sudo btrfs check --readonly /dev/sdg
> Couldn't open file system

OK is it now on some other dev node? A relatively recent btrfs-progs
is also recommended, 5.10 is current and I probably wouldn't use
anything older than 5.6.1.

> aleksey@host:~$ sudo smartctl -x /dev/sdg

Yeah probably won't work since it's behind a raid5 controller. I think
there's smartctl commands to enable passthrough and get information
for each drive, so that you don't have to put it in JBOD mode. But I'm
not familiar with how to do that. Anyway it's a good idea to find out
if there's SMART reporting any problems about any drive, but not
urgent.

-- 
Chris Murphy

Re: btrfs becomes read-only

[Alexey Isaev]

I managed to run btrs check, but it didn't solve the problem:

aleksey@host:~$ sudo btrfs check --repair /dev/sdg
[sudo] password for aleksey:
enabling repair mode
Checking filesystem on /dev/sdg
UUID: 070ce9af-6511-4b89-a501-0823514320c1
checking extents
parent transid verify failed on 52180048330752 wanted 132477 found 132432
parent transid verify failed on 52180048330752 wanted 132477 found 132432
parent transid verify failed on 52180048330752 wanted 132477 found 132432
parent transid verify failed on 52180048330752 wanted 132477 found 132432
Ignoring transid failure
leaf parent key incorrect 52180048330752
bad block 52180048330752
Errors found in extent allocation tree or chunk allocation
parent transid verify failed on 52180048330752 wanted 132477 found 132432
Ignoring transid failure
Fixed 0 roots.
checking free space cache
parent transid verify failed on 52180048330752 wanted 132477 found 132432
Ignoring transid failure
There is no free space entry for 5100308647936-5101376765952
cache appears valid but isnt 5100303024128
block group 5143252697088 has wrong amount of free space
failed to load free space cache for block group 5143252697088
block group 10289697259520 has wrong amount of free space
failed to load free space cache for block group 10289697259520
block group 10447537307648 has wrong amount of free space
failed to load free space cache for block group 10447537307648
block group 10856632942592 has wrong amount of free space
failed to load free space cache for block group 10856632942592
block group 10877034037248 has wrong amount of free space
failed to load free space cache for block group 10877034037248
block group 10901730099200 has wrong amount of free space
failed to load free space cache for block group 10901730099200
block group 37427448119296 has wrong amount of free space
failed to load free space cache for block group 37427448119296
block group 40151531126784 has wrong amount of free space
failed to load free space cache for block group 40151531126784
block group 40155826094080 has wrong amount of free space
failed to load free space cache for block group 40155826094080
block group 45729619902464 has wrong amount of free space
failed to load free space cache for block group 45729619902464
block group 45751094738944 has wrong amount of free space
failed to load free space cache for block group 45751094738944
block group 45775790800896 has wrong amount of free space
failed to load free space cache for block group 45775790800896
block group 45797265637376 has wrong amount of free space
failed to load free space cache for block group 45797265637376
block group 45839141568512 has wrong amount of free space
failed to load free space cache for block group 45839141568512
found 31266648615205 bytes used err is -22
total csum bytes: 0
total tree bytes: 2949169152
total fs tree bytes: 0
total extent tree bytes: 2927755264
btree space waste bytes: 813795673
file data blocks allocated: 9188147200
  referenced 9188147200

С уважением,
Алексей Исаев,
RQC

27.01.2021 10:38, Chris Murphy пишет:
> On Wed, Jan 27, 2021 at 12:22 AM Alexey Isaev <a.isaev@rqc.ru> wrote:
>> Hello!
>>
>> BTRFS volume becomes read-only with this messages in dmesg.
>> What can i do to repair btrfs partition?
>>
>> [Jan25 08:18] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.007587] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.000132] BTRFS error (device sdg): qgroup scan failed with -5
>>
>> [Jan25 19:52] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.009783] BTRFS error (device sdg): parent transid verify failed on
>> 52180048330752 wanted 132477 found 132432
>> [  +0.000132] BTRFS: error (device sdg) in __btrfs_cow_block:1176:
>> errno=-5 IO failure
>> [  +0.000060] BTRFS info (device sdg): forced readonly
>> [  +0.000004] BTRFS info (device sdg): failed to delete reference to
>> ftrace.h, inode 2986197 parent 2989315
>> [  +0.000002] BTRFS: error (device sdg) in __btrfs_unlink_inode:4220:
>> errno=-5 IO failure
>> [  +0.006071] BTRFS error (device sdg): pending csums is 430080
> What kernel version? What drive make/model?
>
> wanted 132477 found 132432 indicates the drive has lost ~45
> transactions, that's not good and also weird. There's no crash or any
> other errors? A complete dmesg might be more revealing. And also
>
> smartctl -x /dev/sdg
> btrfs check --readonly /dev/sdg
>
> After that I suggest
> https://btrfs.wiki.kernel.org/index.php/Restore
>
> And try to get any important data out if it's not backed up. You can
> try btrfs-find-root to get a listing of roots, most recent to oldest.
> Start at the top, and plug that address in as 'btrfs restore -t' and
> see if it'll pull anything out. You likely need -i and -v options as
> well.
>

Re: btrfs becomes read-only

[Chris Murphy]

On Wed, Jan 27, 2021 at 6:05 AM Alexey Isaev <a.isaev@rqc.ru> wrote:
>
> I managed to run btrs check, but it didn't solve the problem:
>
> aleksey@host:~$ sudo btrfs check --repair /dev/sdg

OK it's risky to run --repair without a developer giving a go ahead,
in particular with older versions of btrfs-progs. There are warnings
in the man page about it.


> [sudo] password for aleksey:
> enabling repair mode
> Checking filesystem on /dev/sdg
> UUID: 070ce9af-6511-4b89-a501-0823514320c1
> checking extents
> parent transid verify failed on 52180048330752 wanted 132477 found 132432
> parent transid verify failed on 52180048330752 wanted 132477 found 132432
> parent transid verify failed on 52180048330752 wanted 132477 found 132432
> parent transid verify failed on 52180048330752 wanted 132477 found 132432
> Ignoring transid failure
> leaf parent key incorrect 52180048330752
> bad block 52180048330752
> Errors found in extent allocation tree or chunk allocation
> parent transid verify failed on 52180048330752 wanted 132477 found 132432

Yeah it's not finding what it's expecting to find there.

Any power fail or crash in the history of the file system?

What do you get for:

btrfs insp dump-s -f /dev/sdg


-- 
Chris Murphy