* Corrupt btrfs filesystem recovery... (Due to *sata* errors)
@ 2013-09-28 19:26 Martin
2013-09-28 20:51 ` Chris Murphy
` (2 more replies)
0 siblings, 3 replies; 21+ messages in thread
From: Martin @ 2013-09-28 19:26 UTC (permalink / raw)
To: linux-btrfs
This may be of interest for the fail cause aswel as how to recover...
I have a known good 2TB (4kByte physical sectors) HDD that supports
sata3 (6Gbit/s). Writing data via rsync at the 6Gbit/s sata rate caused
IO errors for just THREE sectors...
Yet btrfsck bombs out with LOTs of errors...
How best to recover from this?
(This is a 'backup' disk so not 'critical' but it would be nice to avoid
rewriting about 1.5TB of data over the network...)
Is there an obvious sequence/recipe to follow for recovery?
Thanks,
Martin
Further details:
Linux 3.10.7-gentoo-r1 #2 SMP Fri Sep 27 23:38:06 BST 2013 x86_64 AMD
E-450 APU with Radeon(tm) HD Graphics AuthenticAMD GNU/Linux
# btrfs version
Btrfs v0.20-rc1-358-g194aa4a
Single 2TB HDD using default mkbtrfs.
Entire disk (/dev/sdc) is btrfs (no partitions).
The IO errors were:
kernel: end_request: I/O error, dev sdc, sector 3215049328
kernel: end_request: I/O error, dev sdc, sector 3215049328
kernel: end_request: I/O error, dev sdc, sector 3215049328
kernel: end_request: I/O error, dev sdc, sector 3215049328
kernel: end_request: I/O error, dev sdc, sector 3215049328
kernel: end_request: I/O error, dev sdc, sector 3206563752
kernel: end_request: I/O error, dev sdc, sector 3206563752
kernel: end_request: I/O error, dev sdc, sector 3206563752
kernel: end_request: I/O error, dev sdc, sector 3206563752
kernel: end_request: I/O error, dev sdc, sector 3206563752
kernel: end_request: I/O error, dev sdc, sector 3213925248
kernel: end_request: I/O error, dev sdc, sector 3213925248
kernel: end_request: I/O error, dev sdc, sector 3213925248
kernel: end_request: I/O error, dev sdc, sector 3213925248
kernel: end_request: I/O error, dev sdc, sector 3213925248
Lots of sata error noise omitted.
The sata problem was fixed by limiting libata to 3Gbit/s:
libata.force=3.0G
added onto the Grub kernel line.
Running "badblocks" twice in succession (non-destructive data test!)
shows no surface errors and no further errors on the sata interface.
Running btrfsck twice gives the same result, giving a failure with:
Ignoring transid failure
btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino
!= key->objectid || rec->refs > 1)' failed.
An abridged summary is:
checking extents
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
leaf parent key incorrect 907185135616
bad block 907185135616
parent transid verify failed on 915444707328 wanted 16974 found 13021
parent transid verify failed on 915444707328 wanted 16974 found 13021
parent transid verify failed on 915445092352 wanted 16974 found 13021
parent transid verify failed on 915445092352 wanted 16974 found 13021
leaf parent key incorrect 915444883456
bad block 915444883456
leaf parent key incorrect 915445014528
bad block 915445014528
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
leaf parent key incorrect 907183771648
bad block 907183771648
leaf parent key incorrect 907183779840
bad block 907183779840
leaf parent key incorrect 907183783936
bad block 907183783936
[...]
leaf parent key incorrect 907185913856
bad block 907185913856
leaf parent key incorrect 907185917952
bad block 907185917952
parent transid verify failed on 915431579648 wanted 16974 found 16972
parent transid verify failed on 915431579648 wanted 16974 found 16972
parent transid verify failed on 915432382464 wanted 16974 found 16972
parent transid verify failed on 915432382464 wanted 16974 found 16972
parent transid verify failed on 915444707328 wanted 16974 found 13021
parent transid verify failed on 915444707328 wanted 16974 found 13021
parent transid verify failed on 915445092352 wanted 16974 found 13021
parent transid verify failed on 915445092352 wanted 16974 found 13021
parent transid verify failed on 915445100544 wanted 16974 found 13021
parent transid verify failed on 915445100544 wanted 16974 found 13021
parent transid verify failed on 915432734720 wanted 16974 found 16972
parent transid verify failed on 915432734720 wanted 16974 found 16972
parent transid verify failed on 915433144320 wanted 16974 found 16972
parent transid verify failed on 915433144320 wanted 16974 found 16972
parent transid verify failed on 915431862272 wanted 16974 found 16972
parent transid verify failed on 915431862272 wanted 16974 found 16972
parent transid verify failed on 915444715520 wanted 16974 found 13021
parent transid verify failed on 915444715520 wanted 16974 found 13021
parent transid verify failed on 915445166080 wanted 16974 found 13021
parent transid verify failed on 915445166080 wanted 16974 found 13021
parent transid verify failed on 915444740096 wanted 16974 found 13021
parent transid verify failed on 915444740096 wanted 16974 found 13021
bad block 915431026688
leaf parent key incorrect 915431141376
bad block 915431141376
leaf parent key incorrect 915431161856
[...]
leaf parent key incorrect 915445100544
bad block 915445100544
leaf parent key incorrect 915445166080
bad block 915445166080
leaf parent key incorrect 915445268480
bad block 915445268480
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
ref mismatch on [712708972544 4096] extent item 0, found 1
Backref 712708972544 parent 5 root 5 not found in extent tree
backpointer mismatch on [712708972544 4096]
ref mismatch on [712708988928 4096] extent item 0, found 1
Backref 712708988928 parent 5 root 5 not found in extent tree
backpointer mismatch on [712708988928 4096]
ref mismatch on [712708993024 4096] extent item 0, found 1
Backref 712708993024 parent 5 root 5 not found in extent tree
backpointer mismatch on [712708993024 4096]
ref mismatch on [712708997120 4096] extent item 0, found 1
Backref 712708997120 parent 5 root 5 not found in extent tree
backpointer mismatch on [712708997120 4096]
ref mismatch on [712709001216 4096] extent item 0, found 1
Backref 712709001216 parent 5 root 5 not found in extent tree
backpointer mismatch on [712709001216 4096]
[...]
ref mismatch on [712709062656 4096] extent item 0, found 1
Backref 712709062656 parent 5 root 5 not found in extent tree
backpointer mismatch on [712709062656 4096]
ref mismatch on [712709066752 4096] extent item 0, found 1
Backref 712709066752 parent 5 root 5 not found in extent tree
backpointer mismatch on [712709066752 4096]
ref mismatch on [907178082304 4096] extent item 1, found 0
Backref 907178082304 root 5 not referenced back 0x1b96f2a0
Incorrect global backref count on 907178082304 found 1 wanted 0
backpointer mismatch on [907178082304 4096]
owner ref check failed [907178082304 4096]
ref mismatch on [907178090496 4096] extent item 1, found 0
Backref 907178090496 root 5 not referenced back 0x1b98aed0
Incorrect global backref count on 907178090496 found 1 wanted 0
backpointer mismatch on [907178090496 4096]
owner ref check failed [907178090496 4096]
ref mismatch on [907178156032 4096] extent item 1, found 0
Backref 907178156032 root 5 not referenced back 0x3ffe5ce0
Incorrect global backref count on 907178156032 found 1 wanted 0
backpointer mismatch on [907178156032 4096]
owner ref check failed [907178156032 4096]
ref mismatch on [907178160128 4096] extent item 1, found 0
Backref 907178160128 root 5 not referenced back 0x5fbf8b0
Incorrect global backref count on 907178160128 found 1 wanted 0
backpointer mismatch on [907178160128 4096]
owner ref check failed [907178160128 4096]
[...]
ref mismatch on [907180011520 4096] extent item 1, found 0
Backref 907180011520 root 5 not referenced back 0x5980c7e0
Incorrect global backref count on 907180011520 found 1 wanted 0
backpointer mismatch on [907180011520 4096]
owner ref check failed [907180011520 4096]
owner ref check failed [907183771648 4096]
owner ref check failed [907183779840 4096]
owner ref check failed [907183783936 4096]
owner ref check failed [907183792128 4096]
owner ref check failed [907183796224 4096]
owner ref check failed [907183841280 4096]
owner ref check failed [907183874048 4096]
owner ref check failed [907183878144 4096]
owner ref check failed [907183882240 4096]
owner ref check failed [907183886336 4096]
owner ref check failed [907183894528 4096]
owner ref check failed [907183898624 4096]
owner ref check failed [907183902720 4096]
owner ref check failed [907183906816 4096]
owner ref check failed [907183910912 4096]
owner ref check failed [907185057792 4096]
owner ref check failed [907185082368 4096]
owner ref check failed [907185135616 4096]
ref mismatch on [907185139712 4096] extent item 1, found 0
Backref 907185139712 root 5 not referenced back 0x470fa690
Incorrect global backref count on 907185139712 found 1 wanted 0
backpointer mismatch on [907185139712 4096]
owner ref check failed [907185139712 4096]
[...]
ref mismatch on [934316011520 4096] extent item 0, found 1
Backref 934316011520 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316011520 4096]
ref mismatch on [934316019712 4096] extent item 0, found 1
Backref 934316019712 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316019712 4096]
ref mismatch on [934316032000 4096] extent item 0, found 1
Backref 934316032000 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316032000 4096]
ref mismatch on [1128365600768 8192] extent item 1, found 0
Incorrect local backref count on 1128365600768 root 5 owner 889187
offset 0 found 0 wanted 1 back 0x6bb76d90
Backref disk bytenr does not match extent record, bytenr=1128365600768,
ref bytenr=17613768628740554752
backpointer mismatch on [1128365600768 8192]
owner ref check failed [1128365600768 8192]
ref mismatch on [1128365608960 8192] extent item 1, found 0
Incorrect local backref count on 1128365608960 root 5 owner 889188
offset 0 found 0 wanted 1 back 0x6bb76ec0
Backref disk bytenr does not match extent record, bytenr=1128365608960,
ref bytenr=8848955218968205284
backpointer mismatch on [1128365608960 8192]
owner ref check failed [1128365608960 8192]
ref mismatch on [1128365617152 8192] extent item 1, found 0
Incorrect local backref count on 1128365617152 root 5 owner 889189
offset 0 found 0 wanted 1 back 0x6bb76ff0
Backref disk bytenr does not match extent record, bytenr=1128365617152,
ref bytenr=1928784803178016523
backpointer mismatch on [1128365617152 8192]
owner ref check failed [1128365617152 8192]
ref mismatch on [1128365625344 4096] extent item 1, found 0
Incorrect local backref count on 1128365625344 root 5 owner 889190
offset 0 found 0 wanted 1 back 0x6bb77120
Backref disk bytenr does not match extent record, bytenr=1128365625344,
ref bytenr=3735616339648328182
backpointer mismatch on [1128365625344 4096]
owner ref check failed [1128365625344 4096]
[...]
ref mismatch on [1454133166080 12288] extent item 1, found 0
Incorrect local backref count on 1454133166080 root 5 owner 2096965
offset 0 found 0 wanted 1 back 0x50c68ad0
Backref disk bytenr does not match extent record, bytenr=1454133166080,
ref bytenr=64
backpointer mismatch on [1454133166080 12288]
owner ref check failed [1454133166080 12288]
Errors found in extent allocation tree or chunk allocation
checking free space cache
Checking filesystem on /dev/sdc
UUID: 38a60270-f9c6-4ed4-8421-4bf1253ae0b3
free space inode generation (0) did not match free space cache
generation (505)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (484)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (484)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (484)
free space inode generation (0) did not match free space cache
generation (516)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (487)
free space inode generation (0) did not match free space cache
generation (486)
free space inode generation (0) did not match free space cache
generation (501)
free space inode generation (0) did not match free space cache
generation (531)
free space inode generation (0) did not match free space cache
generation (498)
free space inode generation (0) did not match free space cache
generation (498)
free space inode generation (0) did not match free space cache
generation (484)
free space inode generation (0) did not match free space cache
generation (532)
free space inode generation (0) did not match free space cache
generation (502)
free space inode generation (0) did not match free space cache
generation (532)
free space inode generation (0) did not match free space cache
generation (502)
[...]
free space inode generation (0) did not match free space cache
generation (1612)
free space inode generation (0) did not match free space cache
generation (1612)
free space inode generation (0) did not match free space cache
generation (1613)
free space inode generation (0) did not match free space cache
generation (1599)
free space inode generation (0) did not match free space cache
generation (1606)
free space inode generation (0) dparent transid verify failed on
907185127424 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
parent transid verify failed on 915431579648 wanted 16974 found 16972
parent transid verify failed on 915431579648 wanted 16974 found 16972
parent transid verify failed on 915432382464 wanted 16974 found 16972
parent transid verify failed on 915432382464 wanted 16974 found 16972
parent transid verify failed on 915444707328 wanted 16974 found 13021
parent transid verify failed on 915444707328 wanted 16974 found 13021
parent transid verify failed on 915445092352 wanted 16974 found 13021
parent transid verify failed on 915445092352 wanted 16974 found 13021
parent transid verify failed on 915445100544 wanted 16974 found 13021
parent transid verify failed on 915445100544 wanted 16974 found 13021
parent transid verify failed on 915432734720 wanted 16974 found 16972
parent transid verify failed on 915432734720 wanted 16974 found 16972
parent transid verify failed on 915433144320 wanted 16974 found 16972
parent transid verify failed on 915433144320 wanted 16974 found 16972
parent transid verify failed on 915431862272 wanted 16974 found 16972
parent transid verify failed on 915431862272 wanted 16974 found 16972
parent transid verify failed on 915444715520 wanted 16974 found 13021
parent transid verify failed on 915444715520 wanted 16974 found 13021
parent transid verify failed on 915445166080 wanted 16974 found 13021
parent transid verify failed on 915445166080 wanted 16974 found 13021
parent transid verify failed on 915444740096 wanted 16974 found 13021
parent transid verify failed on 915444740096 wanted 16974 found 13021
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
checking fs roots
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185082368 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
parent transid verify failed on 907185127424 wanted 15935 found 12264
[...]
parent transid verify failed on 915444523008 wanted 16974 found 13021
parent transid verify failed on 915444523008 wanted 16974 found 13021
parent transid verify failed on 915444523008 wanted 16974 found 13021
parent transid verify failed on 915444523008 wanted 16974 found 13021
Ignoring transid failure
btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino
!= key->objectid || rec->refs > 1)' failed.
id not match free space cache generation (1625)
free space inode generation (0) did not match free space cache
generation (1607)
free space inode generation (0) did not match free space cache
generation (1604)
free space inode generation (0) did not match free space cache
generation (1606)
free space inode generation (0) did not match free space cache
generation (1620)
free space inode generation (0) did not match free space cache
generation (1626)
free space inode generation (0) did not match free space cache
generation (1609)
free space inode generation (0) did not match free space cache
generation (1653)
free space inode generation (0) did not match free space cache
generation (1628)
free space inode generation (0) did not match free space cache
generation (1628)
free space inode generation (0) did not match free space cache
generation (1649)
End of output
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... (Due to *sata* errors)
2013-09-28 19:26 Corrupt btrfs filesystem recovery... (Due to *sata* errors) Martin
@ 2013-09-28 20:51 ` Chris Murphy
2013-09-28 22:51 ` Martin
2013-09-28 22:54 ` Martin
2013-10-05 12:05 ` ASM1083 rev01 PCIe to PCI Bridge chip (Was: Corrupt btrfs filesystem recovery... (Due to *sata* errors)) Martin
2 siblings, 1 reply; 21+ messages in thread
From: Chris Murphy @ 2013-09-28 20:51 UTC (permalink / raw)
To: Btrfs BTRFS
On Sep 28, 2013, at 1:26 PM, Martin <m_btrfs@ml1.co.uk> wrote:
> Writing data via rsync at the 6Gbit/s sata rate caused
> IO errors for just THREE sectors...
>
> Yet btrfsck bombs out with LOTs of errors…
Any fs will bomb out on write errors.
> How best to recover from this?
Why you're getting I/O errors at SATA 6Gbps link speed needs to be understood. Is it a bad cable? Bad SATA port? Drive or controller firmware bug? Or libata driver bug?
> Lots of sata error noise omitted.
And entire dmesg might still be useful. I don't know if the list will handle the whole dmesg in one email, but it's worth a shot (reply to an email in the thread, don't change the subject).
It's possible software or hardware problems are detected well before writes are even initiated.
> Running "badblocks" twice in succession (non-destructive data test!)
> shows no surface errors and no further errors on the sata interface.
SATA link speed related errors aren't related to bad blocks. If you do a smartctl -x on the drive, chances are it's recording PHY Event errors that might be relevant, and also SMART might record UDMA/CMC errors that would just corroborate that the drive also found link errors.
>
> Running btrfsck twice gives the same result, giving a failure with:
Well honestly at this point I expect file system corruption as it's entirely possible that before the hardware dropped the link speed down to SATA 3Gbps, there was corrupt data already sent to the drive and that's not something Btrfs can know about until trying to read the data back in. So *shrug* - I don't see Btrfs as a way to totally mitigate hardware problems. It's the same problem with bad RAM, and Btrfs doesn't like that either.
Chris Murphy
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... (Due to *sata* errors)
2013-09-28 20:51 ` Chris Murphy
@ 2013-09-28 22:51 ` Martin
2013-09-29 2:06 ` Chris Murphy
0 siblings, 1 reply; 21+ messages in thread
From: Martin @ 2013-09-28 22:51 UTC (permalink / raw)
To: linux-btrfs
Chris,
All agreed. Further comment inlined:
(Should have mentioned more prominently that the hardware problem has
been worked-around by limiting the sata to 3Gbit/s on bootup.)
On 28/09/13 21:51, Chris Murphy wrote:
>
> On Sep 28, 2013, at 1:26 PM, Martin <m_btrfs@ml1.co.uk> wrote:
>
>> Writing data via rsync at the 6Gbit/s sata rate caused IO errors
>> for just THREE sectors...
>>
>> Yet btrfsck bombs out with LOTs of errors…
>
> Any fs will bomb out on write errors.
Indeed. However, are not the sata errors reported back to btrfs so that
it knows whatever parts haven't been updated?
Is there not a mechanism to then go "read-only"?
Also, should not the journal limit the damage?
>> How best to recover from this?
>
> Why you're getting I/O errors at SATA 6Gbps link speed needs to be
> understood. Is it a bad cable? Bad SATA port? Drive or controller
> firmware bug? Or libata driver bug?
I systematically eliminated such as leads, PSU, and NCQ. Limiting libata
to only use 3Gbit/s is the one change that gives a consistent fix. The
HDD and motherboard both support 6Gbit/s, but hey-ho, that's an
experiment I can try again some other time when I have another HDD/SSD
to test in there.
In any case, for the existing HDD - motherboard combination, using sata2
rather than sata3 speeds shouldn't noticeably impact performance. (Other
than sata2 works reliably and so is infinitely better for this case!)
>> Lots of sata error noise omitted.
>
> And entire dmesg might still be useful. I don't know if the list will
> handle the whole dmesg in one email, but it's worth a shot (reply to
> an email in the thread, don't change the subject).
I can email directly if of use/interest. Let me know offlist.
> do a smartctl -x on the drive, chances are it's recording PHY Event
(smartctl -x errors shown further down...)
Nothing untoward noticed:
# smartctl -a /dev/sdc
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD20EARX-00PASB0
Serial Number: WD-...
LU WWN Device Id: ...
Firmware Version: 51.0AB51
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Sep 28 23:35:57 2013 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
[...]
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always
- 9
3 Spin_Up_Time 0x0027 253 159 021 Pre-fail Always
- 1983
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always
- 55
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always
- 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always
- 0
9 Power_On_Hours 0x0032 099 099 000 Old_age Always
- 800
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 53
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always
- 31
193 Load_Cycle_Count 0x0032 199 199 000 Old_age Always
- 3115
194 Temperature_Celsius 0x0022 118 110 000 Old_age Always
- 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
# smartctl -x /dev/sdc
... also shows the errors it saw:
(Just the last 4 copied which look timed for when the HDD was last
exposed to 6Gbit/s sata)
Error 46 [21] occurred at disk power-on lifetime: 755 hours (31 days +
11 hours)
When the command that caused the error occurred, the device was active
or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
01 -- 51 00 08 00 00 6c 1a 4b b0 e0 00 Error: AMNF 8 sectors at LBA =
0x6c1a4bb0 = 1813662640
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time
Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- ---------------
--------------------
25 00 00 00 08 00 00 6c 1a 4b b0 e0 08 10:51:07.192 READ DMA EXT
35 00 00 00 08 00 00 6c 1a 4b a8 e0 08 10:51:07.192 WRITE DMA EXT
25 00 00 00 08 00 00 6c 1a 4b a8 e0 08 10:51:07.192 READ DMA EXT
35 00 00 00 08 00 00 6c 1a 4b a8 e0 08 10:51:07.192 WRITE DMA EXT
25 00 00 00 08 00 00 6c 1a 4b a8 e0 08 10:51:07.157 READ DMA EXT
Error 45 [20] occurred at disk power-on lifetime: 755 hours (31 days +
11 hours)
When the command that caused the error occurred, the device was active
or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
01 -- 51 04 00 00 00 6c 1a 4b b0 e0 00 Error: AMNF 1024 sectors at
LBA = 0x6c1a4bb0 = 1813662640
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time
Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- ---------------
--------------------
25 00 00 04 00 00 00 6c 1a 48 00 e0 08 10:51:03.450 READ DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 10:51:03.449 SET FEATURES
[Enable SATA feature]
27 00 00 00 00 00 00 00 00 00 00 e0 08 10:51:03.449 READ NATIVE
MAX ADDRESS EXT [OBS-ACS-3]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 10:51:03.446 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 10:51:03.446 SET FEATURES
[Set transfer mode]
Error 44 [19] occurred at disk power-on lifetime: 755 hours (31 days +
11 hours)
When the command that caused the error occurred, the device was active
or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
01 -- 51 04 00 00 00 6c 1a 4b b0 e0 00 Error: AMNF 1024 sectors at
LBA = 0x6c1a4bb0 = 1813662640
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time
Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- ---------------
--------------------
25 00 00 04 00 00 00 6c 1a 48 00 e0 08 10:51:00.453 READ DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 10:51:00.452 SET FEATURES
[Enable SATA feature]
27 00 00 00 00 00 00 00 00 00 00 e0 08 10:51:00.452 READ NATIVE
MAX ADDRESS EXT [OBS-ACS-3]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 10:51:00.449 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 10:51:00.449 SET FEATURES
[Set transfer mode]
Error 43 [18] occurred at disk power-on lifetime: 755 hours (31 days +
11 hours)
When the command that caused the error occurred, the device was active
or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
01 -- 51 04 00 00 00 6c 1a 4b b0 e0 00 Error: AMNF 1024 sectors at
LBA = 0x6c1a4bb0 = 1813662640
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time
Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- ---------------
--------------------
25 00 00 04 00 00 00 6c 1a 48 00 e0 08 10:50:57.455 READ DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 10:50:57.455 SET FEATURES
[Enable SATA feature]
27 00 00 00 00 00 00 00 00 00 00 e0 08 10:50:57.455 READ NATIVE
MAX ADDRESS EXT [OBS-ACS-3]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 10:50:57.452 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 10:50:57.452 SET FEATURES
[Set transfer mode]
Error 42 [17] occurred at disk power-on lifetime: 755 hours (31 days +
11 hours)
When the command that caused the error occurred, the device was active
or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
01 -- 51 04 00 00 00 6c 1a 4b b0 e0 00 Error: AMNF 1024 sectors at
LBA = 0x6c1a4bb0 = 1813662640
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time
Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- ---------------
--------------------
25 00 00 04 00 00 00 6c 1a 48 00 e0 08 10:50:54.459 READ DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 10:50:54.458 SET FEATURES
[Enable SATA feature]
27 00 00 00 00 00 00 00 00 00 00 e0 08 10:50:54.458 READ NATIVE
MAX ADDRESS EXT [OBS-ACS-3]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 10:50:54.455 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 10:50:54.455 SET FEATURES
[Set transfer mode]
>> Running btrfsck twice gives the same result, giving a failure
>> with:
>
> Well honestly at this point I expect file system corruption as it's
> entirely possible that before the hardware dropped the link speed
> down to SATA 3Gbps, there was corrupt data already sent to the drive
> and that's not something Btrfs can know about until trying to read
> the data back in. So *shrug* - I don't see Btrfs as a way to totally
> mitigate hardware problems. It's the same problem with bad RAM, and
> Btrfs doesn't like that either.
Indeed. Hence trapping 'unexpectedness' where reasonable to then go
read-only... (I guess a hard compromise though whilst still debugging
bugs 'unexpectedness'! But still good to have in mind. ;-) )
Regards,
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... (Due to *sata* errors)
2013-09-28 19:26 Corrupt btrfs filesystem recovery... (Due to *sata* errors) Martin
2013-09-28 20:51 ` Chris Murphy
@ 2013-09-28 22:54 ` Martin
2013-09-29 2:10 ` Corrupt btrfs filesystem recovery... What best instructions? Martin
2013-10-05 12:05 ` ASM1083 rev01 PCIe to PCI Bridge chip (Was: Corrupt btrfs filesystem recovery... (Due to *sata* errors)) Martin
2 siblings, 1 reply; 21+ messages in thread
From: Martin @ 2013-09-28 22:54 UTC (permalink / raw)
To: linux-btrfs
On 28/09/13 20:26, Martin wrote:
> ... btrfsck bombs out with LOTs of errors...
>
> How best to recover from this?
>
> (This is a 'backup' disk so not 'critical' but it would be nice to avoid
> rewriting about 1.5TB of data over the network...)
>
>
> Is there an obvious sequence/recipe to follow for recovery?
I've got the drive reliably working with the sata limited to 3Gbit/s.
What is the best sequence to try to tidy-up and carry on with the 1.5TB
or so of data on there, rather than working from scratch?
So far, I've only run btrfsck since the corruption errors for the three
sectors...
Suggestions for recovery?
Thanks,
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... (Due to *sata* errors)
2013-09-28 22:51 ` Martin
@ 2013-09-29 2:06 ` Chris Murphy
2013-09-29 2:31 ` Martin
0 siblings, 1 reply; 21+ messages in thread
From: Chris Murphy @ 2013-09-29 2:06 UTC (permalink / raw)
To: Martin; +Cc: linux-btrfs
On Sep 28, 2013, at 4:51 PM, Martin <m_btrfs@ml1.co.uk> wrote:
> Indeed. However, are not the sata errors reported back to btrfs so that
> it knows whatever parts haven't been updated?
It's a good question.
My doubtful speculation of such a mechanism is that it is really not the responsibility of the file system to be prepared for the hardware face planting this spectacularly. The hardware really should do better than this. There are specifications that apply here, and the drive and controller and driver all agreed long before the mounting of a volume and writes started to occur. But then later on, at some point in the middle of the really important part of the conversation (writing your data) something in the hardware chain puked and said "OHHh wait about that prior conversation, I'm really confused, let's talk at a slower speed shall we?" So the before part is just a lost conversation, is my speculation.
The other thing is that SATA and SAS handle these things differently. When there's such a serious error that results in a link speed change, usually the bus is reset and for SATA it means the command queue is lost. And I don't think Btrfs is informed of what commands were completed vs failed in such a case.
But I'd love someone who actually knows what they're talking about to answer that question.
My expectation though, is that unlike perhaps other file systems, Btrfs's design goal is to handle the data that did get written, better. In that it's still accessible where other file systems possibly will have a more difficulty.
> Is there not a mechanism to then go "read-only"?
I don't know. In this case it does seem sorta reasonable. But the dmesg might still be revealing. The PHY Event counters indicate a lot of retries of over 1000 sectors.
>
> Also, should not the journal limit the damage?
Well it's COW so it's not quite like a journaled file system, but yeah it should be in a position to know at the next mount time the most recent state of file system consistency. But that doesn't mean it can fix the parts that are just fundamentally broken. But I think it's a valid question, "now what?" because I don't actually know the state of your file system or how to determine it. So maybe Hugo, or someone else has some thoughts.
But for sure I would move to kernel 3.11.2 or 3.12.rc2 before mounting this file system again.
>
>
>>> How best to recover from this?
>>
>> Why you're getting I/O errors at SATA 6Gbps link speed needs to be
>> understood. Is it a bad cable? Bad SATA port? Drive or controller
>> firmware bug? Or libata driver bug?
>
> I systematically eliminated such as leads, PSU, and NCQ. Limiting libata
> to only use 3Gbit/s is the one change that gives a consistent fix. The
> HDD and motherboard both support 6Gbit/s, but hey-ho, that's an
> experiment I can try again some other time when I have another HDD/SSD
> to test in there.
Stick with forced 3Gbps, but I think it's worth while to find out what the actual problem is. One day you forget about this 3Gbps SATA link, upgrade or regress to another kernel and you don't have the 3Gbps forced speed on the parameter line, and poof - you've got more problems again. The hardware shouldn't negotiate a 6Gbps link and then do a backwards swan dive at 30,000' with your data as if it's an after thought.
> In any case, for the existing HDD - motherboard combination, using sata2
> rather than sata3 speeds shouldn't noticeably impact performance. (Other
> than sata2 works reliably and so is infinitely better for this case!)
It's true.
>
>
>>> Lots of sata error noise omitted.
>>
>> And entire dmesg might still be useful. I don't know if the list will
>> handle the whole dmesg in one email, but it's worth a shot (reply to
>> an email in the thread, don't change the subject).
>
> I can email directly if of use/interest. Let me know offlist.
Use pastebin.com and post the link if it's really huge, but I'd consider setting it to no expiration because if something interesting is learned, people doing searches have a better chance of finding the problem if the link hasn't expired.
I would also separately unmount the file system, note the latest kernel message, then mount the file system and see if there are any kernel messages that might indicate recognition of problems with the fs.
I would not use btrfsck --repair until someone says it's a good idea. That person would not be me.
Chris Murphy
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-09-28 22:54 ` Martin
@ 2013-09-29 2:10 ` Martin
2013-09-29 5:11 ` Duncan
0 siblings, 1 reply; 21+ messages in thread
From: Martin @ 2013-09-29 2:10 UTC (permalink / raw)
To: linux-btrfs
On 28/09/13 23:54, Martin wrote:
> On 28/09/13 20:26, Martin wrote:
>
>> ... btrfsck bombs out with LOTs of errors...
>>
>> How best to recover from this?
>>
>> (This is a 'backup' disk so not 'critical' but it would be nice to avoid
>> rewriting about 1.5TB of data over the network...)
>>
>>
>> Is there an obvious sequence/recipe to follow for recovery?
>
>
> I've got the drive reliably working with the sata limited to 3Gbit/s.
> What is the best sequence to try to tidy-up and carry on with the 1.5TB
> or so of data on there, rather than working from scratch?
>
>
> So far, I've only run btrfsck since the corruption...
So...
Any options for btrfsck to fix things?
Or is anything/everything that is fixable automatically fixed on the
next mount?
Or should:
btrfs scrub /dev/sdX
be run first?
Or?
What does btrfs do (or can do) for recovery?
Advice welcomed,
Thanks,
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... (Due to *sata* errors)
2013-09-29 2:06 ` Chris Murphy
@ 2013-09-29 2:31 ` Martin
0 siblings, 0 replies; 21+ messages in thread
From: Martin @ 2013-09-29 2:31 UTC (permalink / raw)
To: linux-btrfs
Chris,
Thanks for good comment/discussion.
On 29/09/13 03:06, Chris Murphy wrote:
>
> On Sep 28, 2013, at 4:51 PM, Martin <m_btrfs@ml1.co.uk> wrote:
>
> Stick with forced 3Gbps, but I think it's worth while to find out
> what the actual problem is. One day you forget about this 3Gbps SATA
> link, upgrade or regress to another kernel and you don't have the
> 3Gbps forced speed on the parameter line, and poof - you've got more
> problems again. The hardware shouldn't negotiate a 6Gbps link and
> then do a backwards swan dive at 30,000' with your data as if it's an
> after thought.
I've got an engineer's curiosity so that one is very definitely marked
for revisiting at some time... If only to blog that x-y-z combination is
a tar pit for your data...
>> In any case, for the existing HDD - motherboard combination, using
>> sata2 rather than sata3 speeds shouldn't noticeably impact
>> performance. (Other than sata2 works reliably and so is infinitely
>> better for this case!)
>
> It's true.
Well, the IO data rate for badblocks is exactly the same as before,
limited by the speed of the physical rust spinning and data density...
> I would also separately unmount the file system, note the latest
> kernel message, then mount the file system and see if there are any
> kernel messages that might indicate recognition of problems with the
> fs.
>
> I would not use btrfsck --repair until someone says it's a good idea.
> That person would not be me.
It is sat unmounted until some informed opinion is gained...
Thanks again for your notes,
Regards,
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-09-29 2:10 ` Corrupt btrfs filesystem recovery... What best instructions? Martin
@ 2013-09-29 5:11 ` Duncan
2013-09-29 21:29 ` Martin
0 siblings, 1 reply; 21+ messages in thread
From: Duncan @ 2013-09-29 5:11 UTC (permalink / raw)
To: linux-btrfs
Martin posted on Sun, 29 Sep 2013 03:10:37 +0100 as excerpted:
> So...
>
> Any options for btrfsck to fix things?
>
> Or is anything/everything that is fixable automatically fixed on the
> next mount?
>
> Or should:
>
> btrfs scrub /dev/sdX
>
> be run first?
>
> Or?
>
>
> What does btrfs do (or can do) for recovery?
Here's a general-case answer (courtesy gmane) to the order in which to
try recovery question, that Hugo posted a few weeks ago:
http://permalink.gmane.org/gmane.comp.file-systems.btrfs/27999
Note that in specific cases someone who knew what they were doing could
omit some steps and focus on others, but I'm not at that level of "know
what I'm doing", so...
Scrub... would go before this, if it's useful. But scrub depends on a
second, valid copy being available in ordered to fix the bad-checksum
one. On a single device btrfs, btrfs defaults to DUP metadata (unless
it's SSD), so you may have a second copy for that, but you won't have a
second copy of the data. This is a very strong reason to go btrfs raid1
mode (for both data and metadata) if you can, because that gives you a
second copy of everything, thereby actually making use of btrfs' checksum
and scrub ability. (Unfortunately, there is as yet no way to do N-way
mirroring, there's only the second copy not a third, no matter how many
devices you have in that "raid1".)
Finally, if you mentioned your kernel (and btrfs-tools) version(s) I
missed it, but [boilerplate recommendation, stressed repeatedly both in
the wiki and on-list] btrfs being still labeled experimental and under
serious development, there's still lots of bugs fixed every kernel
release. So as Chris Murphy said, if you're not on 3.11-stable or 3.12-
rcX already, get there. Not only can the safety of your data depend on
it, but by choosing to run experimental we're all testers, and our
reports if something does go wrong will be far more usable if we're on a
current kernel. Similarly, btrfs-tools 0.20-rc1 is already somewhat old;
you really should be on a git-snapshot beyond that. (The master branch
is kept stable, work is done in other branches and only merged to master
when it's considered suitably stable, so a recently updated btrfs-tools
master HEAD is at least in theory always the best possible version you
can be running. If that's ever NOT the case, then testers need to be
reporting that ASAP so it can be fixed, too.)
Back to the kernel, it's worth noting that 3.12-rcX includes an option
that turns off most btrfs bugons by default. Unless you're a btrfs
developer (which it doesn't sound like you are), you'll want to activate
that (turning off the bugons), as they're not helpful for ordinary users
and just force unnecessary reboots when something minor and otherwise
immediately recoverable goes wrong. That's just one of the latest fixes.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-09-29 5:11 ` Duncan
@ 2013-09-29 21:29 ` Martin
2013-09-29 21:55 ` Martin
2013-10-03 0:49 ` Martin
0 siblings, 2 replies; 21+ messages in thread
From: Martin @ 2013-09-29 21:29 UTC (permalink / raw)
To: linux-btrfs
On 29/09/13 06:11, Duncan wrote:
> Martin posted on Sun, 29 Sep 2013 03:10:37 +0100 as excerpted:
>
>> So...
>>
>> Any options for btrfsck to fix things?
>>
>> Or is anything/everything that is fixable automatically fixed on the
>> next mount?
>>
>> Or should:
>>
>> btrfs scrub /dev/sdX
>>
>> be run first?
>>
>> Or?
>>
>>
>> What does btrfs do (or can do) for recovery?
>
> Here's a general-case answer (courtesy gmane) to the order in which to
> try recovery question, that Hugo posted a few weeks ago:
>
> http://permalink.gmane.org/gmane.comp.file-systems.btrfs/27999
Thanks for that. Very well found!
The instructions from Hugo are:
####
Let's assume that you don't have a physical device failure (which
is a different set of tools -- mount -odegraded, btrfs dev del
missing).
First thing to do is to take a btrfs-image -c9 -t4 of the
filesystem, and keep a copy of the output to show josef. :)
Then start with -orecovery and -oro,recovery for pretty much
anything.
If those fail, then look in dmesg for errors relating to the log
tree -- if that's corrupt and can't be read (or causes a crash), use
btrfs-zero-log.
If there's problems with the chunk tree -- the only one I've seen
recently was reporting something like "can't map address" -- then
chunk-recover may be of use.
After that, btrfsck is probably the next thing to try. If options
-s1, -s2, -s3 have any success, then btrfs-select-super will help by
replacing the superblock with one that works. If that's not going to
be useful, fall back to btrfsck --repair.
Finally, btrfsck --repair --init-extent-tree may be necessary if
there's a damaged extent tree. Finally, if you've got corruption in
the checksums, there's --init-csum-tree.
Hugo.
####
Those will be tried next...
> Note that in specific cases someone who knew what they were doing could
> omit some steps and focus on others, but I'm not at that level of "know
> what I'm doing", so...
>
> Scrub... would go before this, if it's useful. But scrub depends on a
> second, valid copy being available in ordered to fix the bad-checksum
> one. On a single device btrfs, btrfs defaults to DUP metadata (unless
> it's SSD), so you may have a second copy for that, but you won't have a
> second copy of the data. This is a very strong reason to go btrfs raid1
> mode (for both data and metadata) if you can, because that gives you a
> second copy of everything, thereby actually making use of btrfs' checksum
> and scrub ability. (Unfortunately, there is as yet no way to do N-way
> mirroring, there's only the second copy not a third, no matter how many
> devices you have in that "raid1".)
>
> Finally, if you mentioned your kernel (and btrfs-tools) version(s) I
> missed it, but [boilerplate recommendation, stressed repeatedly both in
> the wiki and on-list] btrfs being still labeled experimental and under
> serious development, there's still lots of bugs fixed every kernel
> release. So as Chris Murphy said, if you're not on 3.11-stable or 3.12-
> rcX already, get there. Not only can the safety of your data depend on
> it, but by choosing to run experimental we're all testers, and our
> reports if something does go wrong will be far more usable if we're on a
> current kernel. Similarly, btrfs-tools 0.20-rc1 is already somewhat old;
> you really should be on a git-snapshot beyond that. (The master branch
> is kept stable, work is done in other branches and only merged to master
> when it's considered suitably stable, so a recently updated btrfs-tools
> master HEAD is at least in theory always the best possible version you
> can be running. If that's ever NOT the case, then testers need to be
> reporting that ASAP so it can be fixed, too.)
>
> Back to the kernel, it's worth noting that 3.12-rcX includes an option
> that turns off most btrfs bugons by default. Unless you're a btrfs
> developer (which it doesn't sound like you are), you'll want to activate
> that (turning off the bugons), as they're not helpful for ordinary users
> and just force unnecessary reboots when something minor and otherwise
> immediately recoverable goes wrong. That's just one of the latest fixes.
Looking up what's available for Gentoo, the maintainers there look to be
nicely sharp with multiple versions available all the way up to kernel
3.11.2...
There's also the latest available from btrfs tools with
sys-fs/btrfs-progs "9999"...
OK, so onto the cutting edge to compile them in...
Thanks all,
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-09-29 21:29 ` Martin
@ 2013-09-29 21:55 ` Martin
2013-09-30 7:51 ` Duncan
2013-10-03 0:49 ` Martin
1 sibling, 1 reply; 21+ messages in thread
From: Martin @ 2013-09-29 21:55 UTC (permalink / raw)
To: linux-btrfs
On 29/09/13 22:29, Martin wrote:
> Looking up what's available for Gentoo, the maintainers there look to be
> nicely sharp with multiple versions available all the way up to kernel
> 3.11.2...
That is being pulled in now as expected:
sys-kernel/gentoo-sources-3.11.2
> There's also the latest available from btrfs tools with
> sys-fs/btrfs-progs "9999"...
Oddly, that caused emerge to report:
[ebuild UD ] sys-fs/btrfs-progs-0.19.11 [0.20_rc1_p358] 0 kB
which is a *downgrade*. Hence, I'm keeping with the 0.20_rc1_p358.
> OK, so onto the cutting edge to compile them in...
Interesting times as is said in a certain part of the world...
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-09-29 21:55 ` Martin
@ 2013-09-30 7:51 ` Duncan
0 siblings, 0 replies; 21+ messages in thread
From: Duncan @ 2013-09-30 7:51 UTC (permalink / raw)
To: linux-btrfs
Martin posted on Sun, 29 Sep 2013 22:55:43 +0100 as excerpted:
> On 29/09/13 22:29, Martin wrote:
>
>> Looking up what's available for Gentoo, the maintainers there look to
>> be nicely sharp with multiple versions available all the way up to
>> kernel 3.11.2...
Cool, another gentooer! =:^)
> That is being pulled in now as expected:
>
> sys-kernel/gentoo-sources-3.11.2
FWIW, I've been doing my own kernels (mainline) since back on mandrake a
decade ago, and I just changed up my scripts a bit when I switched to
gentoo. Then later on I changed them up a bit more to be able to run git
kernels. These days I normally first try (and switch to if no serious
bugs) to the dev kernel around -rc2 or so, by which point I figure
anything likely to eat my system for breakfast should be either worked
thru, or at least news of it available. As a non-dev, it's very cool
being able to spot and report bugs, possibly bisecting to a specific
commit, and watch them get fixed before general kernel release. Just one
way I as a non-dev can contribute back. =:^)
To take care of packages that depend on a kernel package, I used to have
a kernel (gentoo-sources-2.6.9999 or some such, back then, now of course
it'd be 3.9999) in package.provided, but these days I don't even need
that. =:^)
>> There's also the latest available from btrfs tools with
>> sys-fs/btrfs-progs "9999"...
>
> Oddly, that caused emerge to report:
>
> [ebuild UD ] sys-fs/btrfs-progs-0.19.11 [0.20_rc1_p358] 0 kB
>
> which is a *downgrade*. Hence, I'm keeping with the 0.20_rc1_p358.
btrfs-progs-9999 is available, but as a live package, it's masked in
keeping with gentoo policy. So to get it you'd need to unmask it.
But 0.20_rc1_p358 shouldn't be /too/ far back. In fact, I'm guessing the
p-number is the (serialized) patch sequence number indicating the number
of commits forward from the rc1 tag. And on the (locally unmasked) -9999
version here, a git describe --tags gives me
v0.20-rc1-358-g194aa4a
... so 0.20_rc1_p358 is very likely identical to the live version at this
point, and it makes no difference, except that the non-live version is a
stable snapshot instead of a version that might change every time you
merge it, if upstream has done any further commits.
So btrfs-progs-0.20_rc1_p358 should be fine. And you were updating
kernel to 3.11.2, so that should be fine by the time you read this, as
well. =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-09-29 21:29 ` Martin
2013-09-29 21:55 ` Martin
@ 2013-10-03 0:49 ` Martin
2013-10-03 1:31 ` Chris Murphy
2013-10-03 16:56 ` Martin
1 sibling, 2 replies; 21+ messages in thread
From: Martin @ 2013-10-03 0:49 UTC (permalink / raw)
To: linux-btrfs
So... The fix:
(
Summary:
Mounting "-o recovery,noatime" worked well and allowed a diff check to
complete for all but one directory tree. So very nearly all the data is
fine.
Deleting the failed directory tree caused a call stack dump and eventually:
kernel: parent transid verify failed on 915444822016 wanted 16974 found
13021
kernel: BTRFS info (device sdc): failed to delete reference to
eggdrop-1.6.19.ebuild, inode 2096893 parent 5881667
kernel: BTRFS error (device sdc) in __btrfs_unlink_inode:3662: errno=-5
IO failure
kernel: BTRFS info (device sdc): forced readonly
Greater detail listed below.
What next best to try?
Safer to try again but this time with with "no_space_cache,no_inode_cache"?
Thanks,
Martin
)
On 29/09/13 22:29, Martin wrote:
> On 29/09/13 06:11, Duncan wrote:
>>> What does btrfs do (or can do) for recovery?
>>
>> Here's a general-case answer (courtesy gmane) to the order in which to
>> try recovery question, that Hugo posted a few weeks ago:
>>
>> http://permalink.gmane.org/gmane.comp.file-systems.btrfs/27999
>
> Thanks for that. Very well found!
>
> The instructions from Hugo are:
>
> ####
> Let's assume that you don't have a physical device failure (which
> is a different set of tools -- mount -odegraded, btrfs dev del
> missing).
>
> First thing to do is to take a btrfs-image -c9 -t4 of the
> filesystem, and keep a copy of the output to show josef. :)
>
> Then start with -orecovery and -oro,recovery for pretty much
> anything.
For anyone following this, first a health warning:
If your data is in any way critical or important, then you should
already have a backup copy elsewhere. If not, best make a binary image
copy of your disk first!
OK... So with the latest kernel (3.11.2) and btrfs tools
(Btrfs v0.20-rc1-358-g194aa4a) and the sequence went:
mount -v -t btrfs -o recovery LABEL=bu_A /mnt/bu_A
(From syslog:)
kernel: device label bu_A devid 1 transid 17222 /dev/sdc
kernel: btrfs: enabling auto recovery
kernel: btrfs: disk space caching is enabled
kernel: btrfs: bdev /dev/sdc errs: wr 0, rd 27, flush 0, corrupt 0, gen 0
Running through a diff check for part of the backups, syslog reported:
kernel: btrfs read error corrected: ino 1 off 915433144320 (dev /dev/sdc
sector 1813661856)
Also, the HDD was showing quite a few write operations so... Is
"noatime" set?... Ooops... Didn't include a "ro"... So, killed the diff
check and remounted:
mount -v -t btrfs -o remount,recovery,noatime /mnt/bu_A
mount: /dev/sdc mounted on /mnt/bu_A
kernel: btrfs: enabling inode map caching
kernel: btrfs: enabling auto recovery
kernel: btrfs: disk space caching is enabled
And running the diff check again... Now zero writes to the HDD :-)
Various syslog messages were given:
kernel: parent transid verify failed on 907185135616 wanted 15935 found
12264
kernel: btrfs read error corrected: ino 1 off 907185135616 (dev /dev/sdc
sector 1781823824)
kernel: parent transid verify failed on 907185143808 wanted 15935 found
12264
kernel: btrfs read error corrected: ino 1 off 907185143808 (dev /dev/sdc
sector 1781823840)
kernel: parent transid verify failed on 907185139712 wanted 15935 found
12264
kernel: btrfs read error corrected: ino 1 off 907185139712 (dev /dev/sdc
sector 1781823832)
kernel: parent transid verify failed on 907185152000 wanted 15935 found
10903
kernel: btrfs read error corrected: ino 1 off 907185152000 (dev /dev/sdc
sector 1781823856)
kernel: parent transid verify failed on 907183783936 wanted 15935 found
12263
kernel: btrfs read error corrected: ino 1 off 907183783936 (dev /dev/sdc
sector 1781821184)
kernel: parent transid verify failed on 907183792128 wanted 15935 found
10903
kernel: btrfs read error corrected: ino 1 off 907183792128 (dev /dev/sdc
sector 1781821200)
kernel: parent transid verify failed on 907183796224 wanted 15935 found
12263
kernel: btrfs read error corrected: ino 1 off 907183796224 (dev /dev/sdc
sector 1781821208)
kernel: parent transid verify failed on 907183841280 wanted 15935 found
10903
kernel: btrfs read error corrected: ino 1 off 907183841280 (dev /dev/sdc
sector 1781821296)
kernel: parent transid verify failed on 907183878144 wanted 15935 found
12263
kernel: btrfs read error corrected: ino 1 off 907183878144 (dev /dev/sdc
sector 1781821368)
kernel: parent transid verify failed on 907183874048 wanted 15935 found
12263
kernel: btrfs read error corrected: ino 1 off 907183874048 (dev /dev/sdc
sector 1781821360)
kernel: verify_parent_transid: 25 callbacks suppressed
kernel: parent transid verify failed on 915431288832 wanted 16974 found
16972
kernel: repair_io_failure: 25 callbacks suppressed
kernel: btrfs read error corrected: ino 1 off 915431288832 (dev /dev/sdc
sector 1813658232)
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
[...]
One directory tree failed the diff checks so I 'mv'-ed that one tree to
rename it out of the way and then ran an "rm -Rf" to remove it.
That appeared to run fine until:
kernel: parent transid verify failed on 915431862272 wanted 16974 found
16972
kernel: btrfs read error corrected: ino 1 off 915431862272 (dev /dev/sdc
sector 1813659352)
kernel: parent transid verify failed on 907185127424 wanted 15935 found
12264
kernel: btrfs read error corrected: ino 1 off 907185127424 (dev /dev/sdc
sector 1781823808)
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: BTRFS info (device sdc): failed to delete reference to
metadata.xml, inode 1846452 parent 5851502
kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 0 PID: 3236 at fs/btrfs/super.c:253
__btrfs_abort_transaction+0x4a/0xfc()
kernel: btrfs: Transaction aborted (error -5)
kernel: Modules linked in: nfsd auth_rpcgss oid_registry exportfs
nfs_acl lockd sunrpc bridge stp llc snd_hda_codec_realtek
snd_hda_codec_hdmi ppdev evdev serio_raw pcspkr acpi_cpufreq
snd_hda_intel snd_hda_codec mperf snd_pcm freq_table snd_page_alloc
snd_timer parport_pc processor wmi bnx2 snd parport thermal_sys
i2c_piix4 button usbhid firewire_ohci firewire_core xhci_hcd ata_generic
pata_acpi
kernel: CPU: 0 PID: 3236 Comm: nfsd Not tainted 3.11.2-gentoo_muse11_07 #1
kernel: Hardware name: System manufacturer System Product Name/E45M1-M
PRO, BIOS 0502 09/21/2011
kernel: 0000000000000000 ffffffff81700892 ffffffff815261d1 ffff8801f91f1c18
kernel: ffffffff8102ea45 ffff88010b18e5a0 ffffffff811df675 ffff8801f91f1c38
kernel: 00000000fffffffb ffff880233afb000 ffff880230a3b960 0000000000000e4e
kernel: Call Trace:
kernel: [<ffffffff815261d1>] ? dump_stack+0x41/0x51
kernel: [<ffffffff8102ea45>] ? warn_slowpath_common+0x79/0x92
kernel: [<ffffffff811df675>] ? __btrfs_abort_transaction+0x4a/0xfc
kernel: [<ffffffff8102eaf6>] ? warn_slowpath_fmt+0x45/0x4a
kernel: [<ffffffff811df675>] ? __btrfs_abort_transaction+0x4a/0xfc
kernel: [<ffffffff812071e3>] ? __btrfs_unlink_inode+0x19a/0x2c0
kernel: [<ffffffff812093bf>] ? btrfs_unlink_inode+0x12/0x35
kernel: [<ffffffff8120943e>] ? btrfs_unlink+0x5c/0x94
kernel: [<ffffffff810f8e03>] ? vfs_unlink+0x69/0xc8
kernel: [<ffffffffa029f215>] ? nfsd_unlink+0x18e/0x1d1 [nfsd]
kernel: [<ffffffffa02a4e87>] ? nfsd3_proc_remove+0x67/0xab [nfsd]
kernel: [<ffffffffa029a9d2>] ? nfsd_dispatch+0x91/0x148 [nfsd]
kernel: [<ffffffffa0234fc7>] ? svc_process+0x3e1/0x630 [sunrpc]
kernel: [<ffffffffa0235211>] ? svc_process+0x62b/0x630 [sunrpc]
kernel: [<ffffffffa029a574>] ? nfsd+0xc0/0x117 [nfsd]
kernel: [<ffffffffa029a4b4>] ? nfsd_destroy+0x64/0x64 [nfsd]
kernel: [<ffffffff81047287>] ? kthread+0xad/0xb5
kernel: [<ffffffff810471da>] ? kthread_freezable_should_stop+0x41/0x41
kernel: [<ffffffff8152c5ec>] ? ret_from_fork+0x7c/0xb0
kernel: [<ffffffff810471da>] ? kthread_freezable_should_stop+0x41/0x41
kernel: ---[ end trace 53d6fb93a497e75d ]---
kernel: BTRFS warning (device sdc): __btrfs_unlink_inode:3662: Aborting
unused transaction(IO failure).
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: parent transid verify failed on 915444523008 wanted 16974 found
13021
kernel: btrfs read error corrected: ino 1 off 915433652224 (dev /dev/sdc
sector 1813662848)
kernel: btrfs read error corrected: ino 1 off 915433029632 (dev /dev/sdc
sector 1813661632)
kernel: btrfs read error corrected: ino 1 off 915433041920 (dev /dev/sdc
sector 1813661656)
kernel: btrfs read error corrected: ino 1 off 915433955328 (dev /dev/sdc
sector 1813663440)
kernel: btrfs read error corrected: ino 1 off 915433127936 (dev /dev/sdc
sector 1813661824)
kernel: btrfs read error corrected: ino 1 off 915434070016 (dev /dev/sdc
sector 1813663664)
kernel: btrfs read error corrected: ino 1 off 915433132032 (dev /dev/sdc
sector 1813661832)
kernel: btrfs read error corrected: ino 1 off 915433136128 (dev /dev/sdc
sector 1813661840)
kernel: btrfs read error corrected: ino 1 off 915433545728 (dev /dev/sdc
sector 1813662640)
kernel: BTRFS info (device sdc): failed to delete reference to
metadata.xml, inode 1846733 parent 5851559
kernel: BTRFS warning (device sdc): __btrfs_unlink_inode:3662: Aborting
unused transaction(IO failure).
kernel: verify_parent_transid: 96 callbacks suppressed
kernel: parent transid verify failed on 915431579648 wanted 16974 found
16972
kernel: repair_io_failure: 13 callbacks suppressed
kernel: btrfs read error corrected: ino 1 off 915431579648 (dev /dev/sdc
sector 1813658800)
kernel: parent transid verify failed on 915432382464 wanted 16974 found
16972
kernel: btrfs read error corrected: ino 1 off 915432382464 (dev /dev/sdc
sector 1813660368)
kernel: parent transid verify failed on 915444707328 wanted 16974 found
13021
kernel: btrfs read error corrected: ino 1 off 915444707328 (dev /dev/sdc
sector 1813684440)
kernel: parent transid verify failed on 915445092352 wanted 16974 found
13021
kernel: btrfs read error corrected: ino 1 off 915445092352 (dev /dev/sdc
sector 1813685192)
kernel: parent transid verify failed on 915445100544 wanted 16974 found
13021
kernel: btrfs read error corrected: ino 1 off 915445100544 (dev /dev/sdc
sector 1813685208)
kernel: parent transid verify failed on 915431026688 wanted 16974 found
16972
kernel: btrfs read error corrected: ino 1 off 915431026688 (dev /dev/sdc
sector 1813657720)
kernel: parent transid verify failed on 915432538112 wanted 16974 found
16972
kernel: btrfs read error corrected: ino 1 off 915432538112 (dev /dev/sdc
sector 1813660672)
kernel: parent transid verify failed on 915444740096 wanted 16974 found
13021
kernel: btrfs read error corrected: ino 1 off 915444740096 (dev /dev/sdc
sector 1813684504)
kernel: parent transid verify failed on 915444469760 wanted 16974 found
13021
kernel: parent transid verify failed on 915444469760 wanted 16974 found
13021
kernel: parent transid verify failed on 915444469760 wanted 16974 found
13021
kernel: parent transid verify failed on 915444469760 wanted 16974 found
13021
kernel: parent transid verify failed on 915444469760 wanted 16974 found
13021
kernel: parent transid verify failed on 915444469760 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: verify_parent_transid: 45 callbacks suppressed
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: parent transid verify failed on 915444518912 wanted 16974 found
13021
kernel: btrfs read error corrected: ino 1 off 915431141376 (dev /dev/sdc
sector 1813657944)
kernel: btrfs read error corrected: ino 1 off 915431165952 (dev /dev/sdc
sector 1813657992)
kernel: btrfs read error corrected: ino 1 off 915431272448 (dev /dev/sdc
sector 1813658200)
kernel: btrfs read error corrected: ino 1 off 915431161856 (dev /dev/sdc
sector 1813657984)
kernel: btrfs read error corrected: ino 1 off 915445268480 (dev /dev/sdc
sector 1813685536)
kernel: btrfs read error corrected: ino 1 off 915440472064 (dev /dev/sdc
sector 1813676168)
kernel: btrfs read error corrected: ino 1 off 915431170048 (dev /dev/sdc
sector 1813658000)
kernel: btrfs read error corrected: ino 1 off 915431174144 (dev /dev/sdc
sector 1813658008)
kernel: btrfs read error corrected: ino 1 off 915431378944 (dev /dev/sdc
sector 1813658408)
kernel: verify_parent_transid: 147 callbacks suppressed
kernel: parent transid verify failed on 915432869888 wanted 16974 found
16972
kernel: parent transid verify failed on 915444473856 wanted 16974 found
13021
kernel: parent transid verify failed on 915444473856 wanted 16974 found
13021
kernel: parent transid verify failed on 915433119744 wanted 16974 found
16972
kernel: parent transid verify failed on 915433656320 wanted 16974 found
16972
kernel: parent transid verify failed on 915433123840 wanted 16974 found
16972
kernel: parent transid verify failed on 915433050112 wanted 16974 found
16972
kernel: parent transid verify failed on 915444473856 wanted 16974 found
13021
kernel: parent transid verify failed on 915444473856 wanted 16974 found
13021
kernel: parent transid verify failed on 915444822016 wanted 16974 found
13021
kernel: BTRFS info (device sdc): failed to delete reference to
eggdrop-1.6.19.ebuild, inode 2096893 parent 5881667
kernel: BTRFS error (device sdc) in __btrfs_unlink_inode:3662: errno=-5
IO failure
kernel: BTRFS info (device sdc): forced readonly
Next best step to try?
Remount "-o recovery,noatime" again?
Thanks,
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-10-03 0:49 ` Martin
@ 2013-10-03 1:31 ` Chris Murphy
2013-10-03 16:56 ` Martin
1 sibling, 0 replies; 21+ messages in thread
From: Chris Murphy @ 2013-10-03 1:31 UTC (permalink / raw)
To: Martin; +Cc: linux-btrfs
On Oct 2, 2013, at 6:49 PM, Martin <m_btrfs@ml1.co.uk> wrote:
> kernel: btrfs read error corrected: ino 1 off 907183792128 (dev /dev/sdc sector 1781821200)
Can anyone answer if this is what corrupt metadata detection and correction looks like? From the original email this is a single disk, with default mkfs.btrfs. So I guess I'm asking an almost obvious question, but I'm still going to ask it. There is only one copy of data, but two copies of metadata so it can self-correct for metadata corruption.
Next question. Why is -o recovery needed to get this correction behavior? The original post was completely devoid of messages indicating correction.
Chris Murphy
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-10-03 0:49 ` Martin
2013-10-03 1:31 ` Chris Murphy
@ 2013-10-03 16:56 ` Martin
2013-10-04 15:43 ` Martin
1 sibling, 1 reply; 21+ messages in thread
From: Martin @ 2013-10-03 16:56 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2116 bytes --]
On 03/10/13 01:49, Martin wrote:
> Summary:
>
> Mounting "-o recovery,noatime" worked well and allowed a diff check to
> complete for all but one directory tree. So very nearly all the data is
> fine.
>
> Deleting the failed directory tree caused a call stack dump and eventually:
>
> kernel: parent transid verify failed on 915444822016 wanted 16974 found
> 13021
> kernel: BTRFS info (device sdc): failed to delete reference to
> eggdrop-1.6.19.ebuild, inode 2096893 parent 5881667
> kernel: BTRFS error (device sdc) in __btrfs_unlink_inode:3662: errno=-5
> IO failure
> kernel: BTRFS info (device sdc): forced readonly
>
>
> Greater detail listed below.
>
> What next best to try?
>
> Safer to try again but this time with with "no_space_cache,no_inode_cache"?
>
> Thanks,
> Martin
> Next best step to try?
>
> Remount "-o recovery,noatime" again?
In the meantime, trying:
btrfsck /dev/sdc
gave the following output + abort:
parent transid verify failed on 915444523008 wanted 16974 found 13021
Ignoring transid failure
btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino
!= key->objectid || rec->refs > 1)' failed.
id not match free space cache generation (1625)
free space inode generation (0) did not match free space cache
generation (1607)
free space inode generation (0) did not match free space cache
generation (1604)
free space inode generation (0) did not match free space cache
generation (1606)
free space inode generation (0) did not match free space cache
generation (1620)
free space inode generation (0) did not match free space cache
generation (1626)
free space inode generation (0) did not match free space cache
generation (1609)
free space inode generation (0) did not match free space cache
generation (1653)
free space inode generation (0) did not match free space cache
generation (1628)
free space inode generation (0) did not match free space cache
generation (1628)
free space inode generation (0) did not match free space cache
generation (1649)
(There was no syslog output.)
Full btrfsck listing attached.
Suggestions please?
Thanks,
Martin
[-- Attachment #2: btrfsck_sdc.log --]
[-- Type: text/x-log, Size: 59523 bytes --]
checking extents
leaf parent key incorrect 907183771648
bad block 907183771648
leaf parent key incorrect 907183779840
bad block 907183779840
leaf parent key incorrect 907183882240
bad block 907183882240
leaf parent key incorrect 907185160192
bad block 907185160192
leaf parent key incorrect 907185201152
bad block 907185201152
leaf parent key incorrect 915432497152
bad block 915432497152
leaf parent key incorrect 915432509440
bad block 915432509440
leaf parent key incorrect 915432513536
bad block 915432513536
leaf parent key incorrect 915432529920
bad block 915432529920
leaf parent key incorrect 915432701952
bad block 915432701952
leaf parent key incorrect 915433058304
bad block 915433058304
leaf parent key incorrect 915437543424
bad block 915437543424
leaf parent key incorrect 915437563904
bad block 915437563904
leaf parent key incorrect 915444469760
bad block 915444469760
leaf parent key incorrect 915444473856
bad block 915444473856
leaf parent key incorrect 915444506624
bad block 915444506624
leaf parent key incorrect 915444518912
bad block 915444518912
leaf parent key incorrect 915444523008
bad block 915444523008
leaf parent key incorrect 915444527104
bad block 915444527104
leaf parent key incorrect 915444539392
bad block 915444539392
leaf parent key incorrect 915444543488
bad block 915444543488
leaf parent key incorrect 915444547584
bad block 915444547584
leaf parent key incorrect 915444551680
bad block 915444551680
leaf parent key incorrect 915444555776
bad block 915444555776
leaf parent key incorrect 915444559872
bad block 915444559872
leaf parent key incorrect 915444563968
bad block 915444563968
leaf parent key incorrect 915444572160
bad block 915444572160
leaf parent key incorrect 915444576256
bad block 915444576256
leaf parent key incorrect 915444580352
bad block 915444580352
leaf parent key incorrect 915444584448
bad block 915444584448
leaf parent key incorrect 915444588544
bad block 915444588544
leaf parent key incorrect 915444678656
bad block 915444678656
leaf parent key incorrect 915444682752
bad block 915444682752
leaf parent key incorrect 915444793344
bad block 915444793344
leaf parent key incorrect 915444797440
bad block 915444797440
leaf parent key incorrect 915444813824
bad block 915444813824
leaf parent key incorrect 915444817920
bad block 915444817920
leaf parent key incorrect 915444822016
bad block 915444822016
leaf parent key incorrect 915444826112
bad block 915444826112
leaf parent key incorrect 915444830208
bad block 915444830208
leaf parent key incorrect 915444834304
bad block 915444834304
leaf parent key incorrect 915444924416
bad block 915444924416
leaf parent key incorrect 915444973568
bad block 915444973568
leaf parent key incorrect 915444977664
bad block 915444977664
leaf parent key incorrect 915444981760
bad block 915444981760
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
owner ref check failed [907183771648 4096]
owner ref check failed [907183779840 4096]
owner ref check failed [907183882240 4096]
owner ref check failed [907185160192 4096]
owner ref check failed [907185201152 4096]
owner ref check failed [915432497152 4096]
owner ref check failed [915432509440 4096]
owner ref check failed [915432513536 4096]
owner ref check failed [915432529920 4096]
owner ref check failed [915432701952 4096]
owner ref check failed [915433058304 4096]
owner ref check failed [915437543424 4096]
owner ref check failed [915437563904 4096]
owner ref check failed [915444469760 4096]
owner ref check failed [915444473856 4096]
owner ref check failed [915444506624 4096]
owner ref check failed [915444518912 4096]
owner ref check failed [915444523008 4096]
owner ref check failed [915444527104 4096]
owner ref check failed [915444539392 4096]
owner ref check failed [915444543488 4096]
owner ref check failed [915444547584 4096]
owner ref check failed [915444551680 4096]
owner ref check failed [915444555776 4096]
owner ref check failed [915444559872 4096]
owner ref check failed [915444563968 4096]
owner ref check failed [915444572160 4096]
owner ref check failed [915444576256 4096]
owner ref check failed [915444580352 4096]
owner ref check failed [915444584448 4096]
owner ref check failed [915444588544 4096]
owner ref check failed [915444678656 4096]
owner ref check failed [915444682752 4096]
owner ref check failed [915444793344 4096]
owner ref check failed [915444797440 4096]
owner ref check failed [915444813824 4096]
owner ref check failed [915444817920 4096]
owner ref check failed [915444822016 4096]
owner ref check failed [915444826112 4096]
owner ref check failed [915444830208 4096]
owner ref check failed [915444834304 4096]
owner ref check failed [915444924416 4096]
owner ref check failed [915444973568 4096]
owner ref check failed [915444977664 4096]
owner ref check failed [915444981760 4096]
ref mismatch on [918767419392 4096] extent item 0, found 1
Backref 918767419392 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767419392 4096]
ref mismatch on [918767443968 4096] extent item 0, found 1
Backref 918767443968 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767443968 4096]
ref mismatch on [918767448064 4096] extent item 0, found 1
Backref 918767448064 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767448064 4096]
ref mismatch on [918767452160 4096] extent item 0, found 1
Backref 918767452160 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767452160 4096]
ref mismatch on [918767472640 4096] extent item 0, found 1
Backref 918767472640 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767472640 4096]
ref mismatch on [918767489024 4096] extent item 0, found 1
Backref 918767489024 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767489024 4096]
ref mismatch on [918767497216 4096] extent item 0, found 1
Backref 918767497216 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767497216 4096]
ref mismatch on [918767509504 4096] extent item 0, found 1
Backref 918767509504 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767509504 4096]
ref mismatch on [918767525888 4096] extent item 0, found 1
Backref 918767525888 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767525888 4096]
ref mismatch on [918767566848 4096] extent item 0, found 1
Backref 918767566848 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767566848 4096]
ref mismatch on [918767575040 4096] extent item 0, found 1
Backref 918767575040 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767575040 4096]
ref mismatch on [918767579136 4096] extent item 0, found 1
Backref 918767579136 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767579136 4096]
ref mismatch on [918767611904 4096] extent item 0, found 1
Backref 918767611904 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767611904 4096]
ref mismatch on [918767616000 4096] extent item 0, found 1
Backref 918767616000 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767616000 4096]
ref mismatch on [918767620096 4096] extent item 0, found 1
Backref 918767620096 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767620096 4096]
ref mismatch on [918767648768 4096] extent item 0, found 1
Backref 918767648768 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767648768 4096]
ref mismatch on [918767669248 4096] extent item 0, found 1
Backref 918767669248 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767669248 4096]
ref mismatch on [918767677440 4096] extent item 0, found 1
Backref 918767677440 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767677440 4096]
ref mismatch on [918767681536 4096] extent item 0, found 1
Backref 918767681536 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767681536 4096]
ref mismatch on [918767742976 4096] extent item 0, found 1
Backref 918767742976 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767742976 4096]
ref mismatch on [918767771648 4096] extent item 0, found 1
Backref 918767771648 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767771648 4096]
ref mismatch on [918767788032 4096] extent item 0, found 1
Backref 918767788032 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767788032 4096]
ref mismatch on [918767796224 4096] extent item 0, found 1
Backref 918767796224 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767796224 4096]
ref mismatch on [918767845376 4096] extent item 0, found 1
Backref 918767845376 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767845376 4096]
ref mismatch on [918767849472 4096] extent item 0, found 1
Backref 918767849472 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767849472 4096]
ref mismatch on [918767878144 4096] extent item 0, found 1
Backref 918767878144 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767878144 4096]
ref mismatch on [918767902720 4096] extent item 0, found 1
Backref 918767902720 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767902720 4096]
ref mismatch on [918767906816 4096] extent item 0, found 1
Backref 918767906816 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767906816 4096]
ref mismatch on [918767910912 4096] extent item 0, found 1
Backref 918767910912 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767910912 4096]
ref mismatch on [918767915008 4096] extent item 0, found 1
Backref 918767915008 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767915008 4096]
ref mismatch on [918767919104 4096] extent item 0, found 1
Backref 918767919104 parent 5 root 5 not found in extent tree
backpointer mismatch on [918767919104 4096]
ref mismatch on [921135570944 4096] extent item 0, found 1
Backref 921135570944 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135570944 4096]
ref mismatch on [921135575040 4096] extent item 0, found 1
Backref 921135575040 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135575040 4096]
ref mismatch on [921135579136 4096] extent item 0, found 1
Backref 921135579136 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135579136 4096]
ref mismatch on [921135587328 4096] extent item 0, found 1
Backref 921135587328 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135587328 4096]
ref mismatch on [921135591424 4096] extent item 0, found 1
Backref 921135591424 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135591424 4096]
ref mismatch on [921135595520 4096] extent item 0, found 1
Backref 921135595520 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135595520 4096]
ref mismatch on [921135599616 4096] extent item 0, found 1
Backref 921135599616 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135599616 4096]
ref mismatch on [921135603712 4096] extent item 0, found 1
Backref 921135603712 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135603712 4096]
ref mismatch on [921135607808 4096] extent item 0, found 1
Backref 921135607808 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135607808 4096]
ref mismatch on [921135611904 4096] extent item 0, found 1
Backref 921135611904 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135611904 4096]
ref mismatch on [921135616000 4096] extent item 0, found 1
Backref 921135616000 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135616000 4096]
ref mismatch on [921135620096 4096] extent item 0, found 1
Backref 921135620096 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135620096 4096]
ref mismatch on [921135624192 4096] extent item 0, found 1
Backref 921135624192 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135624192 4096]
ref mismatch on [921135628288 4096] extent item 0, found 1
Backref 921135628288 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135628288 4096]
ref mismatch on [921135632384 4096] extent item 0, found 1
Backref 921135632384 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135632384 4096]
ref mismatch on [921135636480 4096] extent item 0, found 1
Backref 921135636480 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135636480 4096]
ref mismatch on [921135640576 4096] extent item 0, found 1
Backref 921135640576 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135640576 4096]
ref mismatch on [921135644672 4096] extent item 0, found 1
Backref 921135644672 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135644672 4096]
ref mismatch on [921135648768 4096] extent item 0, found 1
Backref 921135648768 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135648768 4096]
ref mismatch on [921135652864 4096] extent item 0, found 1
Backref 921135652864 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135652864 4096]
ref mismatch on [921135656960 4096] extent item 0, found 1
Backref 921135656960 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135656960 4096]
ref mismatch on [921135661056 4096] extent item 0, found 1
Backref 921135661056 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135661056 4096]
ref mismatch on [921135665152 4096] extent item 0, found 1
Backref 921135665152 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135665152 4096]
ref mismatch on [921135669248 4096] extent item 0, found 1
Backref 921135669248 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135669248 4096]
ref mismatch on [921135673344 4096] extent item 0, found 1
Backref 921135673344 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135673344 4096]
ref mismatch on [921135677440 4096] extent item 0, found 1
Backref 921135677440 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135677440 4096]
ref mismatch on [921135681536 4096] extent item 0, found 1
Backref 921135681536 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135681536 4096]
ref mismatch on [921135685632 4096] extent item 0, found 1
Backref 921135685632 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135685632 4096]
ref mismatch on [921135689728 4096] extent item 0, found 1
Backref 921135689728 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135689728 4096]
ref mismatch on [921135693824 4096] extent item 0, found 1
Backref 921135693824 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135693824 4096]
ref mismatch on [921135697920 4096] extent item 0, found 1
Backref 921135697920 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135697920 4096]
ref mismatch on [921135702016 4096] extent item 0, found 1
Backref 921135702016 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135702016 4096]
ref mismatch on [921135706112 4096] extent item 0, found 1
Backref 921135706112 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135706112 4096]
ref mismatch on [921135710208 4096] extent item 0, found 1
Backref 921135710208 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135710208 4096]
ref mismatch on [921135714304 4096] extent item 0, found 1
Backref 921135714304 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135714304 4096]
ref mismatch on [921135718400 4096] extent item 0, found 1
Backref 921135718400 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135718400 4096]
ref mismatch on [921135726592 4096] extent item 0, found 1
Backref 921135726592 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135726592 4096]
ref mismatch on [921135837184 4096] extent item 0, found 1
Backref 921135837184 parent 2 root 2 not found in extent tree
backpointer mismatch on [921135837184 4096]
ref mismatch on [921135882240 4096] extent item 0, found 1
Backref 921135882240 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135882240 4096]
ref mismatch on [921135886336 4096] extent item 0, found 1
Backref 921135886336 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135886336 4096]
ref mismatch on [921135890432 4096] extent item 0, found 1
Backref 921135890432 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135890432 4096]
ref mismatch on [921135894528 4096] extent item 0, found 1
Backref 921135894528 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135894528 4096]
ref mismatch on [921135906816 4096] extent item 0, found 1
Backref 921135906816 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135906816 4096]
ref mismatch on [921135910912 4096] extent item 0, found 1
Backref 921135910912 parent 5 root 5 not found in extent tree
backpointer mismatch on [921135910912 4096]
ref mismatch on [921136025600 4096] extent item 0, found 1
Backref 921136025600 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136025600 4096]
ref mismatch on [921136029696 4096] extent item 0, found 1
Backref 921136029696 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136029696 4096]
ref mismatch on [921136037888 4096] extent item 0, found 1
Backref 921136037888 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136037888 4096]
ref mismatch on [921136078848 4096] extent item 0, found 1
Backref 921136078848 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136078848 4096]
ref mismatch on [921136123904 4096] extent item 0, found 1
Backref 921136123904 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136123904 4096]
ref mismatch on [921136160768 4096] extent item 0, found 1
Backref 921136160768 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136160768 4096]
ref mismatch on [921136168960 4096] extent item 0, found 1
Backref 921136168960 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136168960 4096]
ref mismatch on [921136205824 4096] extent item 0, found 1
Backref 921136205824 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136205824 4096]
ref mismatch on [921136214016 4096] extent item 0, found 1
Backref 921136214016 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136214016 4096]
ref mismatch on [921136242688 4096] extent item 0, found 1
Backref 921136242688 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136242688 4096]
ref mismatch on [921136250880 4096] extent item 0, found 1
Backref 921136250880 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136250880 4096]
ref mismatch on [921136259072 4096] extent item 0, found 1
Backref 921136259072 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136259072 4096]
ref mismatch on [921136263168 4096] extent item 0, found 1
Backref 921136263168 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136263168 4096]
ref mismatch on [921136287744 4096] extent item 0, found 1
Backref 921136287744 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136287744 4096]
ref mismatch on [921136300032 4096] extent item 0, found 1
Backref 921136300032 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136300032 4096]
ref mismatch on [921136304128 4096] extent item 0, found 1
Backref 921136304128 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136304128 4096]
ref mismatch on [921136308224 4096] extent item 0, found 1
Backref 921136308224 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136308224 4096]
ref mismatch on [921136312320 4096] extent item 0, found 1
Backref 921136312320 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136312320 4096]
ref mismatch on [921136316416 4096] extent item 0, found 1
Backref 921136316416 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136316416 4096]
ref mismatch on [921136320512 4096] extent item 0, found 1
Backref 921136320512 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136320512 4096]
ref mismatch on [921136332800 4096] extent item 0, found 1
Backref 921136332800 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136332800 4096]
ref mismatch on [921136336896 4096] extent item 0, found 1
Backref 921136336896 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136336896 4096]
ref mismatch on [921136340992 4096] extent item 0, found 1
Backref 921136340992 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136340992 4096]
ref mismatch on [921136345088 4096] extent item 0, found 1
Backref 921136345088 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136345088 4096]
ref mismatch on [921136349184 4096] extent item 0, found 1
Backref 921136349184 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136349184 4096]
ref mismatch on [921136353280 4096] extent item 0, found 1
Backref 921136353280 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136353280 4096]
ref mismatch on [921136357376 4096] extent item 0, found 1
Backref 921136357376 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136357376 4096]
ref mismatch on [921136369664 4096] extent item 0, found 1
Backref 921136369664 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136369664 4096]
ref mismatch on [921136398336 4096] extent item 0, found 1
Backref 921136398336 parent 5 root 5 not found in extent tree
backpointer mismatch on [921136398336 4096]
ref mismatch on [926297280512 4096] extent item 0, found 1
Backref 926297280512 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297280512 4096]
ref mismatch on [926297284608 4096] extent item 0, found 1
Backref 926297284608 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297284608 4096]
ref mismatch on [926297288704 4096] extent item 0, found 1
Backref 926297288704 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297288704 4096]
ref mismatch on [926297292800 4096] extent item 0, found 1
Backref 926297292800 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297292800 4096]
ref mismatch on [926297305088 4096] extent item 0, found 1
Backref 926297305088 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297305088 4096]
ref mismatch on [926297341952 4096] extent item 0, found 1
Backref 926297341952 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297341952 4096]
ref mismatch on [926297382912 4096] extent item 0, found 1
Backref 926297382912 parent 2 root 2 not found in extent tree
backpointer mismatch on [926297382912 4096]
ref mismatch on [926297399296 4096] extent item 0, found 1
Backref 926297399296 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297399296 4096]
ref mismatch on [926297415680 4096] extent item 0, found 1
Backref 926297415680 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297415680 4096]
ref mismatch on [926297423872 4096] extent item 0, found 1
Backref 926297423872 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297423872 4096]
ref mismatch on [926297468928 4096] extent item 0, found 1
Backref 926297468928 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297468928 4096]
ref mismatch on [926297481216 4096] extent item 0, found 1
Backref 926297481216 parent 2 root 2 not found in extent tree
backpointer mismatch on [926297481216 4096]
ref mismatch on [926297526272 4096] extent item 0, found 1
Backref 926297526272 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297526272 4096]
ref mismatch on [926297530368 4096] extent item 0, found 1
Backref 926297530368 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297530368 4096]
ref mismatch on [926297542656 4096] extent item 0, found 1
Backref 926297542656 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297542656 4096]
ref mismatch on [926297575424 4096] extent item 0, found 1
Backref 926297575424 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297575424 4096]
ref mismatch on [926297600000 4096] extent item 0, found 1
Backref 926297600000 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297600000 4096]
ref mismatch on [926297604096 4096] extent item 0, found 1
Backref 926297604096 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297604096 4096]
ref mismatch on [926297608192 4096] extent item 0, found 1
Backref 926297608192 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297608192 4096]
ref mismatch on [926297616384 4096] extent item 0, found 1
Backref 926297616384 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297616384 4096]
ref mismatch on [926297665536 4096] extent item 0, found 1
Backref 926297665536 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297665536 4096]
ref mismatch on [926297686016 4096] extent item 0, found 1
Backref 926297686016 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297686016 4096]
ref mismatch on [926297735168 4096] extent item 0, found 1
Backref 926297735168 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297735168 4096]
ref mismatch on [926297743360 4096] extent item 0, found 1
Backref 926297743360 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297743360 4096]
ref mismatch on [926297800704 4096] extent item 0, found 1
Backref 926297800704 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297800704 4096]
ref mismatch on [926297841664 4096] extent item 0, found 1
Backref 926297841664 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297841664 4096]
ref mismatch on [926297866240 4096] extent item 0, found 1
Backref 926297866240 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297866240 4096]
ref mismatch on [926297915392 4096] extent item 0, found 1
Backref 926297915392 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297915392 4096]
ref mismatch on [926297919488 4096] extent item 0, found 1
Backref 926297919488 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297919488 4096]
ref mismatch on [926297944064 4096] extent item 0, found 1
Backref 926297944064 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297944064 4096]
ref mismatch on [926297976832 4096] extent item 0, found 1
Backref 926297976832 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297976832 4096]
ref mismatch on [926297980928 4096] extent item 0, found 1
Backref 926297980928 parent 5 root 5 not found in extent tree
backpointer mismatch on [926297980928 4096]
ref mismatch on [926298050560 4096] extent item 0, found 1
Backref 926298050560 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298050560 4096]
ref mismatch on [926298087424 4096] extent item 0, found 1
Backref 926298087424 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298087424 4096]
ref mismatch on [926298296320 4096] extent item 0, found 1
Backref 926298296320 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298296320 4096]
ref mismatch on [926298378240 4096] extent item 0, found 1
Backref 926298378240 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298378240 4096]
ref mismatch on [926298411008 4096] extent item 0, found 1
Backref 926298411008 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298411008 4096]
ref mismatch on [926298423296 4096] extent item 0, found 1
Backref 926298423296 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298423296 4096]
ref mismatch on [926298521600 4096] extent item 0, found 1
Backref 926298521600 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298521600 4096]
ref mismatch on [926298562560 4096] extent item 0, found 1
Backref 926298562560 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298562560 4096]
ref mismatch on [926298566656 4096] extent item 0, found 1
Backref 926298566656 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298566656 4096]
ref mismatch on [926298570752 4096] extent item 0, found 1
Backref 926298570752 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298570752 4096]
ref mismatch on [926298632192 4096] extent item 0, found 1
Backref 926298632192 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298632192 4096]
ref mismatch on [926298636288 4096] extent item 0, found 1
Backref 926298636288 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298636288 4096]
ref mismatch on [926298734592 4096] extent item 0, found 1
Backref 926298734592 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298734592 4096]
ref mismatch on [926298738688 4096] extent item 0, found 1
Backref 926298738688 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298738688 4096]
ref mismatch on [926298763264 4096] extent item 0, found 1
Backref 926298763264 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298763264 4096]
ref mismatch on [926298836992 4096] extent item 0, found 1
Backref 926298836992 parent 5 root 5 not found in extent tree
backpointer mismatch on [926298836992 4096]
ref mismatch on [934315782144 4096] extent item 0, found 1
Backref 934315782144 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315782144 4096]
ref mismatch on [934315786240 4096] extent item 0, found 1
Backref 934315786240 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315786240 4096]
ref mismatch on [934315790336 4096] extent item 0, found 1
Backref 934315790336 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315790336 4096]
ref mismatch on [934315794432 4096] extent item 0, found 1
Backref 934315794432 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315794432 4096]
ref mismatch on [934315810816 4096] extent item 0, found 1
Backref 934315810816 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315810816 4096]
ref mismatch on [934315814912 4096] extent item 0, found 1
Backref 934315814912 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315814912 4096]
ref mismatch on [934315823104 4096] extent item 0, found 1
Backref 934315823104 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315823104 4096]
ref mismatch on [934315827200 4096] extent item 0, found 1
Backref 934315827200 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315827200 4096]
ref mismatch on [934315831296 4096] extent item 0, found 1
Backref 934315831296 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315831296 4096]
ref mismatch on [934315835392 4096] extent item 0, found 1
Backref 934315835392 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315835392 4096]
ref mismatch on [934315839488 4096] extent item 0, found 1
Backref 934315839488 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315839488 4096]
ref mismatch on [934315843584 4096] extent item 0, found 1
Backref 934315843584 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315843584 4096]
ref mismatch on [934315855872 4096] extent item 0, found 1
Backref 934315855872 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315855872 4096]
ref mismatch on [934315859968 4096] extent item 0, found 1
Backref 934315859968 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315859968 4096]
ref mismatch on [934315864064 4096] extent item 0, found 1
Backref 934315864064 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315864064 4096]
ref mismatch on [934315868160 4096] extent item 0, found 1
Backref 934315868160 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315868160 4096]
ref mismatch on [934315872256 4096] extent item 0, found 1
Backref 934315872256 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315872256 4096]
ref mismatch on [934315880448 4096] extent item 0, found 1
Backref 934315880448 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315880448 4096]
ref mismatch on [934315884544 4096] extent item 0, found 1
Backref 934315884544 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315884544 4096]
ref mismatch on [934315888640 4096] extent item 0, found 1
Backref 934315888640 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315888640 4096]
ref mismatch on [934315913216 4096] extent item 0, found 1
Backref 934315913216 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315913216 4096]
ref mismatch on [934315917312 4096] extent item 0, found 1
Backref 934315917312 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315917312 4096]
ref mismatch on [934315921408 4096] extent item 0, found 1
Backref 934315921408 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315921408 4096]
ref mismatch on [934315925504 4096] extent item 0, found 1
Backref 934315925504 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315925504 4096]
ref mismatch on [934315937792 4096] extent item 0, found 1
Backref 934315937792 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315937792 4096]
ref mismatch on [934315954176 4096] extent item 0, found 1
Backref 934315954176 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315954176 4096]
ref mismatch on [934315962368 4096] extent item 0, found 1
Backref 934315962368 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315962368 4096]
ref mismatch on [934315974656 4096] extent item 0, found 1
Backref 934315974656 parent 5 root 5 not found in extent tree
backpointer mismatch on [934315974656 4096]
ref mismatch on [934316003328 4096] extent item 0, found 1
Backref 934316003328 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316003328 4096]
ref mismatch on [934316007424 4096] extent item 0, found 1
Backref 934316007424 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316007424 4096]
ref mismatch on [934316011520 4096] extent item 0, found 1
Backref 934316011520 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316011520 4096]
ref mismatch on [934316019712 4096] extent item 0, found 1
Backref 934316019712 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316019712 4096]
ref mismatch on [934316032000 4096] extent item 0, found 1
Backref 934316032000 parent 5 root 5 not found in extent tree
backpointer mismatch on [934316032000 4096]
ref mismatch on [1448539918336 16384] extent item 1, found 0
Incorrect local backref count on 1448539918336 root 5 owner 1846729 offset 0 found 0 wanted 1 back 0x7d8f6aa0
Backref disk bytenr does not match extent record, bytenr=1448539918336, ref bytenr=12105675801913647264
backpointer mismatch on [1448539918336 16384]
owner ref check failed [1448539918336 16384]
ref mismatch on [1454133108736 16384] extent item 1, found 0
Incorrect local backref count on 1454133108736 root 5 owner 2096890 offset 0 found 0 wanted 1 back 0x5ec25860
Backref disk bytenr does not match extent record, bytenr=1454133108736, ref bytenr=272
backpointer mismatch on [1454133108736 16384]
owner ref check failed [1454133108736 16384]
Errors found in extent allocation tree or chunk allocation
checking free space cache
Checking filesystem on /dev/sdc
UUID: 38a60270-f9c6-4ed4-8421-4bf1253ae0b3
free space inode generation (0) did not match free space cache generation (505)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (484)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (484)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (484)
free space inode generation (0) did not match free space cache generation (516)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (487)
free space inode generation (0) did not match free space cache generation (486)
free space inode generation (0) did not match free space cache generation (501)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (498)
free space inode generation (0) did not match free space cache generation (498)
free space inode generation (0) did not match free space cache generation (484)
free space inode generation (0) did not match free space cache generation (532)
free space inode generation (0) did not match free space cache generation (502)
free space inode generation (0) did not match free space cache generation (532)
free space inode generation (0) did not match free space cache generation (502)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (304)
free space inode generation (0) did not match free space cache generation (313)
free space inode generation (0) did not match free space cache generation (312)
free space inode generation (0) did not match free space cache generation (315)
free space inode generation (0) did not match free space cache generation (315)
free space inode generation (0) did not match free space cache generation (320)
free space inode generation (0) did not match free space cache generation (317)
free space inode generation (0) did not match free space cache generation (364)
free space inode generation (0) did not match free space cache generation (370)
free space inode generation (0) did not match free space cache generation (376)
free space inode generation (0) did not match free space cache generation (370)
free space inode generation (0) did not match free space cache generation (342)
free space inode generation (0) did not match free space cache generation (370)
free space inode generation (0) did not match free space cache generation (342)
free space inode generation (0) did not match free space cache generation (342)
free space inode generation (0) did not match free space cache generation (343)
free space inode generation (0) did not match free space cache generation (342)
free space inode generation (0) did not match free space cache generation (370)
free space inode generation (0) did not match free space cache generation (370)
free space inode generation (0) did not match free space cache generation (396)
free space inode generation (0) did not match free space cache generation (342)
free space inode generation (0) did not match free space cache generation (400)
free space inode generation (0) did not match free space cache generation (399)
free space inode generation (0) did not match free space cache generation (351)
free space inode generation (0) did not match free space cache generation (381)
free space inode generation (0) did not match free space cache generation (378)
free space inode generation (0) did not match free space cache generation (402)
free space inode generation (0) did not match free space cache generation (373)
free space inode generation (0) did not match free space cache generation (377)
free space inode generation (0) did not match free space cache generation (380)
free space inode generation (0) did not match free space cache generation (380)
free space inode generation (0) did not match free space cache generation (380)
free space inode generation (0) did not match free space cache generation (379)
free space inode generation (0) did not match free space cache generation (380)
free space inode generation (0) did not match free space cache generation (380)
free space inode generation (0) did not match free space cache generation (409)
free space inode generation (0) did not match free space cache generation (398)
free space inode generation (0) did not match free space cache generation (398)
free space inode generation (0) did not match free space cache generation (409)
free space inode generation (0) did not match free space cache generation (414)
free space inode generation (0) did not match free space cache generation (420)
free space inode generation (0) did not match free space cache generation (428)
free space inode generation (0) did not match free space cache generation (420)
free space inode generation (0) did not match free space cache generation (409)
free space inode generation (0) did not match free space cache generation (432)
free space inode generation (0) did not match free space cache generation (432)
free space inode generation (0) did not match free space cache generation (394)
free space inode generation (0) did not match free space cache generation (447)
free space inode generation (0) did not match free space cache generation (457)
free space inode generation (0) did not match free space cache generation (458)
free space inode generation (0) did not match free space cache generation (417)
free space inode generation (0) did not match free space cache generation (407)
free space inode generation (0) did not match free space cache generation (459)
free space inode generation (0) did not match free space cache generation (468)
free space inode generation (0) did not match free space cache generation (542)
free space inode generation (0) did not match free space cache generation (465)
free space inode generation (0) did not match free space cache generation (514)
free space inode generation (0) did not match free space cache generation (568)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (519)
free space inode generation (0) did not match free space cache generation (532)
free space inode generation (0) did not match free space cache generation (525)
free space inode generation (0) did not match free space cache generation (457)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (569)
free space inode generation (0) did not match free space cache generation (461)
free space inode generation (0) did not match free space cache generation (612)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (505)
free space inode generation (0) did not match free space cache generation (502)
free space inode generation (0) did not match free space cache generation (532)
free space inode generation (0) did not match free space cache generation (569)
free space inode generation (0) did not match free space cache generation (542)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (532)
free space inode generation (0) did not match free space cache generation (539)
free space inode generation (0) did not match free space cache generation (567)
free space inode generation (0) did not match free space cache generation (540)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (484)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (519)
free space inode generation (0) did not match free space cache generation (539)
free space inode generation (0) did not match free space cache generation (613)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (482)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (502)
free space inode generation (0) did not match free space cache generation (505)
free space inode generation (0) did not match free space cache generation (531)
free space inode generation (0) did not match free space cache generation (503)
free space inode generation (0) did not match free space cache generation (502)
free space inode generation (0) did not match free space cache generation (502)
free space inode generation (0) did not match free space cache generation (541)
free space inode generation (0) did not match free space cache generation (539)
free space inode generation (0) did not match free space cache generation (532)
free space inode generation (0) did not match free space cache generation (539)
free space inode generation (0) did not match free space cache generation (615)
free space inode generation (0) did not match free space cache generation (624)
free space inode generation (0) did not match free space cache generation (630)
free space inode generation (0) did not match free space cache generation (568)
free space inode generation (0) did not match free space cache generation (567)
free space inode generation (0) did not match free space cache generation (567)
free space inode generation (0) did not match free space cache generation (613)
free space inode generation (0) did not match free space cache generation (614)
free space inode generation (0) did not match free space cache generation (614)
free space inode generation (0) did not match free space cache generation (606)
free space inode generation (0) did not match free space cache generation (605)
free space inode generation (0) did not match free space cache generation (605)
free space inode generation (0) did not match free space cache generation (606)
free space inode generation (0) did not match free space cache generation (606)
free space inode generation (0) did not match free space cache generation (607)
free space inode generation (0) did not match free space cache generation (621)
free space inode generation (0) did not match free space cache generation (631)
free space inode generation (0) did not match free space cache generation (626)
free space inode generation (0) did not match free space cache generation (647)
free space inode generation (0) did not match free space cache generation (659)
free space inode generation (0) did not match free space cache generation (661)
free space inode generation (0) did not match free space cache generation (660)
free space inode generation (0) did not match free space cache generation (665)
free space inode generation (0) did not match free space cache generation (673)
free space inode generation (0) did not match free space cache generation (680)
free space inode generation (0) did not match free space cache generation (689)
free space inode generation (0) did not match free space cache generation (693)
free space inode generation (0) did not match free space cache generation (702)
free space inode generation (0) did not match free space cache generation (716)
free space inode generation (0) did not match free space cache generation (717)
free space inode generation (0) did not match free space cache generation (720)
free space inode generation (0) did not match free space cache generation (727)
free space inode generation (0) did not match free space cache generation (739)
free space inode generation (0) did not match free space cache generation (744)
free space inode generation (0) did not match free space cache generation (759)
free space inode generation (0) did not match free space cache generation (762)
free space inode generation (0) did not match free space cache generation (770)
free space inode generation (0) did not match free space cache generation (780)
free space inode generation (0) did not match free space cache generation (786)
free space inode generation (0) did not match free space cache generation (787)
free space inode generation (0) did not match free space cache generation (807)
free space inode generation (0) did not match free space cache generation (808)
free space inode generation (0) did not match free space cache generation (816)
free space inode generation (0) did not match free space cache generation (821)
free space inode generation (0) did not match free space cache generation (819)
free space inode generation (0) did not match free space cache generation (817)
free space inode generation (0) did not match free space cache generation (824)
free space inode generation (0) did not match free space cache generation (854)
free space inode generation (0) did not match free space cache generation (843)
free space inode generation (0) did not match free space cache generation (852)
free space inode generation (0) did not match free space cache generation (846)
free space inode generation (0) did not match free space cache generation (852)
free space inode generation (0) did not match free space cache generation (861)
free space inode generation (0) did not match free space cache generation (883)
free space inode generation (0) did not match free space cache generation (873)
free space inode generation (0) did not match free space cache generation (879)
free space inode generation (0) did not match free space cache generation (884)
free space inode generation (0) did not match free space cache generation (890)
free space inode generation (0) did not match free space cache generation (928)
free space inode generation (0) did not match free space cache generation (916)
free space inode generation (0) did not match free space cache generation (929)
free space inode generation (0) did not match free space cache generation (931)
free space inode generation (0) did not match free space cache generation (934)
free space inode generation (0) did not match free space cache generation (937)
free space inode generation (0) did not match free space cache generation (949)
free space inode generation (0) did not match free space cache generation (945)
free space inode generation (0) did not match free space cache generation (956)
free space inode generation (0) did not match free space cache generation (960)
free space inode generation (0) did not match free space cache generation (971)
free space inode generation (0) did not match free space cache generation (983)
free space inode generation (0) did not match free space cache generation (996)
free space inode generation (0) did not match free space cache generation (1014)
free space inode generation (0) did not match free space cache generation (1150)
free space inode generation (0) did not match free space cache generation (1146)
free space inode generation (0) did not match free space cache generation (1149)
free space inode generation (0) did not match free space cache generation (1077)
free space inode generation (0) did not match free space cache generation (1093)
free space inode generation (0) did not match free space cache generation (1106)
free space inode generation (0) did not match free space cache generation (1118)
free space inode generation (0) did not match free space cache generation (1126)
free space inode generation (0) did not match free space cache generation (1137)
free space inode generation (0) did not match free space cache generation (1151)
free space inode generation (0) did not match free space cache generation (1152)
free space inode generation (0) did not match free space cache generation (1163)
free space inode generation (0) did not match free space cache generation (1181)
free space inode generation (0) did not match free space cache generation (1193)
free space inode generation (0) did not match free space cache generation (1221)
free space inode generation (0) did not match free space cache generation (1240)
free space inode generation (0) did not match free space cache generation (1238)
free space inode generation (0) did not match free space cache generation (1295)
free space inode generation (0) did not match free space cache generation (1298)
free space inode generation (0) did not match free space cache generation (1296)
free space inode generation (0) did not match free space cache generation (1337)
free space inode generation (0) did not match free space cache generation (1335)
free space inode generation (0) did not match free space cache generation (1341)
free space inode generation (0) did not match free space cache generation (1350)
free space inode generation (0) did not match free space cache generation (1354)
free space inode generation (0) did not match free space cache generation (1376)
free space inode generation (0) did not match free space cache generation (1396)
free space inode generation (0) did not match free space cache generation (1397)
free space inode generation (0) did not match free space cache generation (1403)
free space inode generation (0) did not match free space cache generation (1413)
free space inode generation (0) did not match free space cache generation (1426)
free space inode generation (0) did not match free space cache generation (1425)
free space inode generation (0) did not match free space cache generation (1445)
free space inode generation (0) did not match free space cache generation (1454)
free space inode generation (0) did not match free space cache generation (1474)
free space inode generation (0) did not match free space cache generation (1479)
free space inode generation (0) did not match free space cache generation (1479)
free space inode generation (0) did not match free space cache generation (1542)
free space inode generation (0) did not match free space cache generation (1588)
free space inode generation (0) did not match free space cache generation (1593)
free space inode generation (0) did not match free space cache generation (1594)
free space inode generation (0) did not match free space cache generation (1601)
free space inode generation (0) did not match free space cache generation (1598)
free space inode generation (0) did not match free space cache generation (1607)
free space inode generation (0) did not match free space cache generation (1599)
free space inode generation (0) did not match free space cache generation (1612)
free space inode generation (0) did not match free space cache generation (1612)
free space inode generation (0) did not match free space cache generation (1612)
free space inode generation (0) did not match free space cache generation (1613)
free space inode generation (0) did not match free space cache generation (1599)
free space inode generation (0) did not match free space cache generation (1606)
free space inode generation (0) dparent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
checking fs roots
parent transid verify failed on 915444523008 wanted 16974 found 13021
parent transid verify failed on 915444523008 wanted 16974 found 13021
parent transid verify failed on 915444523008 wanted 16974 found 13021
parent transid verify failed on 915444523008 wanted 16974 found 13021
Ignoring transid failure
btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino != key->objectid || rec->refs > 1)' failed.
id not match free space cache generation (1625)
free space inode generation (0) did not match free space cache generation (1607)
free space inode generation (0) did not match free space cache generation (1604)
free space inode generation (0) did not match free space cache generation (1606)
free space inode generation (0) did not match free space cache generation (1620)
free space inode generation (0) did not match free space cache generation (1626)
free space inode generation (0) did not match free space cache generation (1609)
free space inode generation (0) did not match free space cache generation (1653)
free space inode generation (0) did not match free space cache generation (1628)
free space inode generation (0) did not match free space cache generation (1628)
free space inode generation (0) did not match free space cache generation (1649)
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-10-03 16:56 ` Martin
@ 2013-10-04 15:43 ` Martin
2013-10-05 11:32 ` Martin
0 siblings, 1 reply; 21+ messages in thread
From: Martin @ 2013-10-04 15:43 UTC (permalink / raw)
To: linux-btrfs
What best to try next?
mount "-o recovery,noatime"
btrfsck:
--repair try to repair the filesystem
--init-csum-tree create a new CRC tree
--init-extent-tree create a new extent tree
or is a "scrub" worthwhile?
The fail and switch to read-only occured whilst trying to delete a known
bad directory tree. No worries for losing the data in that.
But how best to clean up the filesystem errors?
Thanks,
Martin
On 03/10/13 17:56, Martin wrote:
> On 03/10/13 01:49, Martin wrote:
>
>> Summary:
>>
>> Mounting "-o recovery,noatime" worked well and allowed a diff check to
>> complete for all but one directory tree. So very nearly all the data is
>> fine.
>>
>> Deleting the failed directory tree caused a call stack dump and eventually:
>>
>> kernel: parent transid verify failed on 915444822016 wanted 16974 found
>> 13021
>> kernel: BTRFS info (device sdc): failed to delete reference to
>> eggdrop-1.6.19.ebuild, inode 2096893 parent 5881667
>> kernel: BTRFS error (device sdc) in __btrfs_unlink_inode:3662: errno=-5
>> IO failure
>> kernel: BTRFS info (device sdc): forced readonly
>>
>>
>> Greater detail listed below.
>>
>> What next best to try?
>>
>> Safer to try again but this time with with "no_space_cache,no_inode_cache"?
>>
>> Thanks,
>> Martin
>
>
>> Next best step to try?
>>
>> Remount "-o recovery,noatime" again?
>
>
> In the meantime, trying:
>
> btrfsck /dev/sdc
>
> gave the following output + abort:
>
> parent transid verify failed on 915444523008 wanted 16974 found 13021
> Ignoring transid failure
> btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino
> != key->objectid || rec->refs > 1)' failed.
> id not match free space cache generation (1625)
> free space inode generation (0) did not match free space cache
> generation (1607)
> free space inode generation (0) did not match free space cache
> generation (1604)
> free space inode generation (0) did not match free space cache
> generation (1606)
> free space inode generation (0) did not match free space cache
> generation (1620)
> free space inode generation (0) did not match free space cache
> generation (1626)
> free space inode generation (0) did not match free space cache
> generation (1609)
> free space inode generation (0) did not match free space cache
> generation (1653)
> free space inode generation (0) did not match free space cache
> generation (1628)
> free space inode generation (0) did not match free space cache
> generation (1628)
> free space inode generation (0) did not match free space cache
> generation (1649)
>
>
> (There was no syslog output.)
>
> Full btrfsck listing attached.
>
>
> Suggestions please?
>
> Thanks,
> Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-10-04 15:43 ` Martin
@ 2013-10-05 11:32 ` Martin
2013-10-05 13:18 ` Martin
0 siblings, 1 reply; 21+ messages in thread
From: Martin @ 2013-10-05 11:32 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]
No comment so blindly trying:
btrfsck --repair /dev/sdc
gave the following abort:
btrfsck: extent-tree.c:2736: alloc_reserved_tree_block: Assertion
`!(ret)' failed.
Full output attached.
All on:
3.11.2-gentoo
Btrfs v0.20-rc1-358-g194aa4a
For a 2TB single HDD formatted with defaults.
What next?
Thanks,
Martin
>> In the meantime, trying:
>>
>> btrfsck /dev/sdc
>>
>> gave the following output + abort:
>>
>> parent transid verify failed on 915444523008 wanted 16974 found 13021
>> Ignoring transid failure
>> btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino
>> != key->objectid || rec->refs > 1)' failed.
>> id not match free space cache generation (1625)
>> free space inode generation (0) did not match free space cache
>> generation (1607)
>> free space inode generation (0) did not match free space cache
>> generation (1604)
>> free space inode generation (0) did not match free space cache
>> generation (1606)
>> free space inode generation (0) did not match free space cache
>> generation (1620)
>> free space inode generation (0) did not match free space cache
>> generation (1626)
>> free space inode generation (0) did not match free space cache
>> generation (1609)
>> free space inode generation (0) did not match free space cache
>> generation (1653)
>> free space inode generation (0) did not match free space cache
>> generation (1628)
>> free space inode generation (0) did not match free space cache
>> generation (1628)
>> free space inode generation (0) did not match free space cache
>> generation (1649)
>>
>>
>> (There was no syslog output.)
>>
>> Full btrfsck listing attached.
>>
>>
>> Suggestions please?
>>
>> Thanks,
>> Martin
[-- Attachment #2: btrfsck_repair_sdc.log --]
[-- Type: text/x-log, Size: 3965 bytes --]
checking extents
leaf parent key incorrect 907183771648
bad block 907183771648
leaf parent key incorrect 907183779840
bad block 907183779840
leaf parent key incorrect 907183882240
bad block 907183882240
leaf parent key incorrect 907185160192
bad block 907185160192
leaf parent key incorrect 907185201152
bad block 907185201152
leaf parent key incorrect 915432497152
bad block 915432497152
leaf parent key incorrect 915432509440
bad block 915432509440
leaf parent key incorrect 915432513536
bad block 915432513536
leaf parent key incorrect 915432529920
bad block 915432529920
leaf parent key incorrect 915432701952
bad block 915432701952
leaf parent key incorrect 915433058304
bad block 915433058304
leaf parent key incorrect 915437543424
bad block 915437543424
leaf parent key incorrect 915437563904
bad block 915437563904
leaf parent key incorrect 915444469760
bad block 915444469760
leaf parent key incorrect 915444473856
bad block 915444473856
leaf parent key incorrect 915444506624
bad block 915444506624
leaf parent key incorrect 915444518912
bad block 915444518912
leaf parent key incorrect 915444523008
bad block 915444523008
leaf parent key incorrect 915444527104
bad block 915444527104
leaf parent key incorrect 915444539392
bad block 915444539392
leaf parent key incorrect 915444543488
bad block 915444543488
leaf parent key incorrect 915444547584
bad block 915444547584
leaf parent key incorrect 915444551680
bad block 915444551680
leaf parent key incorrect 915444555776
bad block 915444555776
leaf parent key incorrect 915444559872
bad block 915444559872
leaf parent key incorrect 915444563968
bad block 915444563968
leaf parent key incorrect 915444572160
bad block 915444572160
leaf parent key incorrect 915444576256
bad block 915444576256
leaf parent key incorrect 915444580352
bad block 915444580352
leaf parent key incorrect 915444584448
bad block 915444584448
leaf parent key incorrect 915444588544
bad block 915444588544
leaf parent key incorrect 915444678656
bad block 915444678656
leaf parent key incorrect 915444682752
bad block 915444682752
leaf parent key incorrect 915444793344
bad block 915444793344
leaf parent key incorrect 915444797440
bad block 915444797440
leaf parent key incorrect 915444813824
bad block 915444813824
leaf parent key incorrect 915444817920
bad block 915444817920
leaf parent key incorrect 915444822016
bad block 915444822016
leaf parent key incorrect 915444826112
bad block 915444826112
leaf parent key incorrect 915444830208
bad block 915444830208
leaf parent key incorrect 915444834304
bad block 915444834304
leaf parent key incorrect 915444924416
bad block 915444924416
leaf parent key incorrect 915444973568
bad block 915444973568
leaf parent key incorrect 915444977664
bad block 915444977664
leaf parent key incorrect 915444981760
bad block 915444981760
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444973568 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444977664 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915444981760 wanted 16974 found 13021
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915432701952 wanted 16974 found 16972
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444678656 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
parent transid verify failed on 915444682752 wanted 16974 found 13021
deleting pointer to block 915432701952
owner ref check failed [907183771648 4096]
repair deleting extent record: key 907183771648 168 4096
btrfsck: extent-tree.c:2736: alloc_reserved_tree_block: Assertion `!(ret)' failed.
enabling repair mode
Checking filesystem on /dev/sdc
UUID: 38a60270-f9c6-4ed4-8421-4bf1253ae0b3
^ permalink raw reply [flat|nested] 21+ messages in thread
* ASM1083 rev01 PCIe to PCI Bridge chip (Was: Corrupt btrfs filesystem recovery... (Due to *sata* errors))
2013-09-28 19:26 Corrupt btrfs filesystem recovery... (Due to *sata* errors) Martin
2013-09-28 20:51 ` Chris Murphy
2013-09-28 22:54 ` Martin
@ 2013-10-05 12:05 ` Martin
2 siblings, 0 replies; 21+ messages in thread
From: Martin @ 2013-10-05 12:05 UTC (permalink / raw)
To: linux-btrfs
On 28/09/13 20:26, Martin wrote:
> AMD
> E-450 APU with Radeon(tm) HD Graphics AuthenticAMD GNU/Linux
Just in case someone else stumbles across this thread due to a related
problem for my particular motherboard...
There appears to be a fatal hardware bug for the interrupt line deassert
for a PCIe to PCI Bridge chip:
ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
See the thread on https://lkml.org/lkml/2012/1/30/216
For that chip, the interrupt line is not always deasserted for PCI
interrupts. The hardware fault appears to be fixed in ASM1083 rev 03.
Unfortunately, there is no useful OS workaround possible for rev 01.
Hence, the PCI interrupts are unusable for ASM1083 rev01 ? :-(
In brief, this means that the PCI card slots on the motherboard cannot
be used for any hardware that might generate an interrupt. That means
pretty much all normal PCI cards. (The PCIe card slots are fine.)
For my own example, there does not appear to be any other devices using
that bridge chip. The only concern is for the sound chip but I happen to
never use sound on that system and so that is disabled.
The problem is listed in syslog/dmesg by lines such as:
kernel: irq 16: nobody cared (try booting with the "irqpoll" option)
kernel: Disabling IRQ #16
Unfortunately, the HDDs and network interfaces also use that irq or "irg
17" (which can also be affected). Losing the irq will badly slow down
your system and can cause data corruption for heavy use of the HDD.
Use:
lspci | grep -i ASM1083
to see if you have that chip and if so, what revision.
To see if you have any irqpoll messages, use:
grep -ia irqpoll /var/log/messages
To list what devices use what interrupts, use either of:
grep -ia ' irq ' /var/log/messages
cat /proc/interrupts
Note that there should no longer be any ASM1083 rev01 chips being
supplied by now. (ASM1083 rev03 chips have been seen in products.)
Hope that helps for that bit of obscurity!
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Corrupt btrfs filesystem recovery... What best instructions?
2013-10-05 11:32 ` Martin
@ 2013-10-05 13:18 ` Martin
2013-10-07 14:56 ` btrfsck --repair --init-extent-tree: segfault error 4 Martin
0 siblings, 1 reply; 21+ messages in thread
From: Martin @ 2013-10-05 13:18 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2177 bytes --]
So...
The hint there is "btrfsck: extent-tree.c:2736", so trying:
btrfsck --repair --init-extent-tree /dev/sdc
That ran for a while until:
kernel: btrfsck[16610]: segfault at cc ip 000000000041d2a7 sp
00007fffd2c2d710 error 4 in btrfsck[400000+4d000]
There's no other messages in the syslog.
The output attached.
What next?
Thanks,
Martin
On 05/10/13 12:32, Martin wrote:
> No comment so blindly trying:
>
> btrfsck --repair /dev/sdc
>
> gave the following abort:
>
> btrfsck: extent-tree.c:2736: alloc_reserved_tree_block: Assertion
> `!(ret)' failed.
>
> Full output attached.
>
>
> All on:
>
> 3.11.2-gentoo
> Btrfs v0.20-rc1-358-g194aa4a
>
> For a 2TB single HDD formatted with defaults.
>
>
> What next?
>
> Thanks,
> Martin
>
>
>
>
>>> In the meantime, trying:
>>>
>>> btrfsck /dev/sdc
>>>
>>> gave the following output + abort:
>>>
>>> parent transid verify failed on 915444523008 wanted 16974 found 13021
>>> Ignoring transid failure
>>> btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino
>>> != key->objectid || rec->refs > 1)' failed.
>>> id not match free space cache generation (1625)
>>> free space inode generation (0) did not match free space cache
>>> generation (1607)
>>> free space inode generation (0) did not match free space cache
>>> generation (1604)
>>> free space inode generation (0) did not match free space cache
>>> generation (1606)
>>> free space inode generation (0) did not match free space cache
>>> generation (1620)
>>> free space inode generation (0) did not match free space cache
>>> generation (1626)
>>> free space inode generation (0) did not match free space cache
>>> generation (1609)
>>> free space inode generation (0) did not match free space cache
>>> generation (1653)
>>> free space inode generation (0) did not match free space cache
>>> generation (1628)
>>> free space inode generation (0) did not match free space cache
>>> generation (1628)
>>> free space inode generation (0) did not match free space cache
>>> generation (1649)
>>>
>>>
>>> (There was no syslog output.)
>>>
>>> Full btrfsck listing attached.
>>>
>>>
>>> Suggestions please?
>>>
>>> Thanks,
>>> Martin
[-- Attachment #2: btrfsck_repair_extent-tree_sdc.log --]
[-- Type: text/x-log, Size: 10239 bytes --]
btrfs unable to find ref byte nr 912043257856 parent 0 root 1 owner 2 offset 0
btrfs unable to find ref byte nr 912043343872 parent 0 root 1 owner 1 offset 1
btrfs unable to find ref byte nr 912044331008 parent 0 root 1 owner 0 offset 1
btrfs unable to find ref byte nr 912043261952 parent 0 root 1 owner 1 offset 1
btrfs unable to find ref byte nr 912043266048 parent 0 root 1 owner 0 offset 1
checking extents
leaf parent key incorrect 907183771648
bad block 907183771648
leaf parent key incorrect 907183779840
bad block 907183779840
leaf parent key incorrect 907183882240
bad block 907183882240
leaf parent key incorrect 907185160192
bad block 907185160192
leaf parent key incorrect 907185201152
bad block 907185201152
leaf parent key incorrect 915432497152
bad block 915432497152
leaf parent key incorrect 915432509440
bad block 915432509440
leaf parent key incorrect 915432513536
bad block 915432513536
leaf parent key incorrect 915432529920
bad block 915432529920
leaf parent key incorrect 915433058304
bad block 915433058304
leaf parent key incorrect 915437543424
bad block 915437543424
leaf parent key incorrect 915437563904
bad block 915437563904
leaf parent key incorrect 915444469760
bad block 915444469760
leaf parent key incorrect 915444473856
bad block 915444473856
leaf parent key incorrect 915444506624
bad block 915444506624
leaf parent key incorrect 915444518912
bad block 915444518912
leaf parent key incorrect 915444523008
bad block 915444523008
leaf parent key incorrect 915444527104
bad block 915444527104
leaf parent key incorrect 915444539392
bad block 915444539392
leaf parent key incorrect 915444543488
bad block 915444543488
leaf parent key incorrect 915444547584
bad block 915444547584
leaf parent key incorrect 915444551680
bad block 915444551680
leaf parent key incorrect 915444555776
bad block 915444555776
leaf parent key incorrect 915444559872
bad block 915444559872
leaf parent key incorrect 915444563968
bad block 915444563968
leaf parent key incorrect 915444572160
bad block 915444572160
leaf parent key incorrect 915444576256
bad block 915444576256
leaf parent key incorrect 915444580352
bad block 915444580352
leaf parent key incorrect 915444584448
bad block 915444584448
leaf parent key incorrect 915444588544
bad block 915444588544
leaf parent key incorrect 915444793344
bad block 915444793344
leaf parent key incorrect 915444797440
bad block 915444797440
leaf parent key incorrect 915444813824
bad block 915444813824
leaf parent key incorrect 915444817920
bad block 915444817920
leaf parent key incorrect 915444822016
bad block 915444822016
leaf parent key incorrect 915444826112
bad block 915444826112
leaf parent key incorrect 915444830208
bad block 915444830208
leaf parent key incorrect 915444834304
bad block 915444834304
leaf parent key incorrect 915444924416
bad block 915444924416
ref mismatch on [12582912 8065024] extent item 0, found 1
btrfs unable to find ref byte nr 912014393344 parent 0 root 2 owner 0 offset 0
adding new data backref on 12582912 root 5 owner 1903 offset 0 found 1
Backref 12582912 root 5 owner 1903 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 12582912 root 5 owner 1903 offset 0 found 1 wanted 0 back 0x1abd230
backpointer mismatch on [12582912 8065024]
ref mismatch on [20647936 12288] extent item 0, found 1
adding new data backref on 20647936 root 5 owner 1911 offset 0 found 1
Backref 20647936 root 5 owner 1911 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20647936 root 5 owner 1911 offset 0 found 1 wanted 0 back 0x1a83e60
backpointer mismatch on [20647936 12288]
ref mismatch on [20660224 90112] extent item 0, found 1
adding new data backref on 20660224 root 5 owner 1919 offset 0 found 1
Backref 20660224 root 5 owner 1919 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20660224 root 5 owner 1919 offset 0 found 1 wanted 0 back 0x620844c0
backpointer mismatch on [20660224 90112]
ref mismatch on [20750336 102400] extent item 0, found 1
adding new data backref on 20750336 root 5 owner 1925 offset 0 found 1
Backref 20750336 root 5 owner 1925 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20750336 root 5 owner 1925 offset 0 found 1 wanted 0 back 0x62084be0
backpointer mismatch on [20750336 102400]
ref mismatch on [20852736 28672] extent item 0, found 1
adding new data backref on 20852736 root 5 owner 1936 offset 0 found 1
Backref 20852736 root 5 owner 1936 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20852736 root 5 owner 1936 offset 0 found 1 wanted 0 back 0x10e6f480
backpointer mismatch on [20852736 28672]
ref mismatch on [20881408 28672] extent item 0, found 1
adding new data backref on 20881408 root 5 owner 1959 offset 0 found 1
Backref 20881408 root 5 owner 1959 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20881408 root 5 owner 1959 offset 0 found 1 wanted 0 back 0x10143d60
backpointer mismatch on [20881408 28672]
ref mismatch on [20910080 8192] extent item 0, found 1
adding new data backref on 20910080 root 5 owner 1977 offset 0 found 1
Backref 20910080 root 5 owner 1977 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20910080 root 5 owner 1977 offset 0 found 1 wanted 0 back 0xec22ec0
backpointer mismatch on [20910080 8192]
ref mismatch on [20918272 8192] extent item 0, found 1
adding new data backref on 20918272 root 5 owner 1985 offset 0 found 1
Backref 20918272 root 5 owner 1985 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20918272 root 5 owner 1985 offset 0 found 1 wanted 0 back 0x62fc1d00
backpointer mismatch on [20918272 8192]
ref mismatch on [20926464 8192] extent item 0, found 1
adding new data backref on 20926464 root 5 owner 1994 offset 0 found 1
Backref 20926464 root 5 owner 1994 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20926464 root 5 owner 1994 offset 0 found 1 wanted 0 back 0x877ba00
backpointer mismatch on [20926464 8192]
ref mismatch on [20934656 8192] extent item 0, found 1
adding new data backref on 20934656 root 5 owner 2002 offset 0 found 1
Backref 20934656 root 5 owner 2002 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20934656 root 5 owner 2002 offset 0 found 1 wanted 0 back 0x877b7a0
backpointer mismatch on [20934656 8192]
ref mismatch on [20942848 24576] extent item 0, found 1
adding new data backref on 20942848 root 5 owner 2064 offset 0 found 1
Backref 20942848 root 5 owner 2064 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20942848 root 5 owner 2064 offset 0 found 1 wanted 0 back 0xa79e490
backpointer mismatch on [20942848 24576]
ref mismatch on [20967424 4096] extent item 0, found 1
adding new data backref on 20967424 root 5 owner 2241 offset 0 found 1
Backref 20967424 root 5 owner 2241 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 20967424 root 5 owner 2241 offset 0 found 1 wanted 0 back 0xc0cba20
backpointer mismatch on [20967424 4096]
ref mismatch on [20983808 4096] extent item 0, found 1
adding new tree backref on start 20983808 len 4096 parent 3 root 3
Backref 20983808 parent 3 root 3 not found in extent tree
backpointer mismatch on [20983808 4096]
ref mismatch on [20987904 4096] extent item 0, found 1
adding new tree backref on start 20987904 len 4096 parent 3 root 3
Backref 20987904 parent 3 root 3 not found in extent tree
backpointer mismatch on [20987904 4096]
ref mismatch on [20992000 4096] extent item 0, found 1
adding new tree backref on start 20992000 len 4096 parent 3 root 3
Backref 20992000 parent 3 root 3 not found in extent tree
backpointer mismatch on [20992000 4096]
ref mismatch on [20996096 4096] extent item 0, found 1
adding new tree backref on start 20996096 len 4096 parent 3 root 3
Backref 20996096 parent 3 root 3 not found in extent tree
backpointer mismatch on [20996096 4096]
ref mismatch on [21000192 4096] extent item 0, found 1
adding new tree backref on start 21000192 len 4096 parent 3 root 3
Backref 21000192 parent 3 root 3 not found in extent tree
backpointer mismatch on [21000192 4096]
ref mismatch on [21004288 4096] extent item 0, found 1
adding new tree backref on start 21004288 len 4096 parent 3 root 3
Backref 21004288 parent 3 root 3 not found in extent tree
backpointer mismatch on [21004288 4096]
ref mismatch on [21008384 4096] extent item 0, found 1
adding new tree backref on start 21008384 len 4096 parent 3 root 3
Backref 21008384 parent 3 root 3 not found in extent tree
backpointer mismatch on [21008384 4096]
ref mismatch on [21012480 4096] extent item 0, found 1
adding new tree backref on start 21012480 len 4096 parent 3 root 3
Backref 21012480 parent 3 root 3 not found in extent tree
backpointer mismatch on [21012480 4096]
ref mismatch on [21016576 4096] extent item 0, found 1
adding new tree backref on start 21016576 len 4096 parent 3 root 3
Backref 21016576 parent 3 root 3 not found in extent tree
backpointer mismatch on [21016576 4096]
ref mismatch on [21020672 4096] extent item 0, found 1
adding new tree backref on start 21020672 len 4096 parent 3 root 3
Backref 21020672 parent 3 root 3 not found in extent tree
backpointer mismatch on [21020672 4096]
ref mismatch on [21024768 4096] extent item 0, found 1
adding new tree backref on start 21024768 len 4096 parent 3 root 3
Backref 21024768 parent 3 root 3 not found in extent tree
backpointer mismatch on [21024768 4096]
ref mismatch on [21028864 4096] extent item 0, found 1
adding new tree backref on start 21028864 len 4096 parent 3 root 3
Backref 21028864 parent 3 root 3 not found in extent tree
backpointer mismatch on [21028864 4096]
ref mismatch on [21032960 4096] extent item 0, found 1
adding new tree backref on start 21032960 len 4096 parent 3 root 3
Backref 21032960 parent 3 root 3 not found in extent tree
backpointer mismatch on [21032960 4096]
ref mismatch on [21037056 4096] extent item 0, found 1
adding new tree backref on start 21037056 len 4096 parent 3 root 3
Backref 21037056 parent 3 root 3 not found in extent tree
backpointer mismatch on [21037056 4096]
ref mismatch on [21041152 4096] extent item 0, found 1
^ permalink raw reply [flat|nested] 21+ messages in thread
* btrfsck --repair --init-extent-tree: segfault error 4
2013-10-05 13:18 ` Martin
@ 2013-10-07 14:56 ` Martin
2013-10-07 19:03 ` Chris Murphy
0 siblings, 1 reply; 21+ messages in thread
From: Martin @ 2013-10-07 14:56 UTC (permalink / raw)
To: linux-btrfs
Any clues or educated comment please?
Can the corrupt directory tree safely be ignored and left in place? Or
might that cause everything to fall over in a big heap as soon as I try
to write data again?
Could these other tricks work-around or fix the corrupt tree:
Run a scrub?
Make a snapshot and work from the snapshot?
Or try "mount -o recovery,noatime" again?
Or is it dead?
(The 1.5TB of backup data is replicated elsewhere but it would be good
to rescue this version rather than completely redo from scratch.
Especially so for the sake of just a few MBytes of one corrupt directory
tree.)
Thanks,
Martin
On 05/10/13 14:18, Martin wrote:
> So...
>
> The hint there is "btrfsck: extent-tree.c:2736", so trying:
>
> btrfsck --repair --init-extent-tree /dev/sdc
>
> That ran for a while until:
>
> kernel: btrfsck[16610]: segfault at cc ip 000000000041d2a7 sp
> 00007fffd2c2d710 error 4 in btrfsck[400000+4d000]
>
> There's no other messages in the syslog.
>
> The output attached.
>
>
> What next?
>
>
> Thanks,
> Martin
>
>
>
> On 05/10/13 12:32, Martin wrote:
>> No comment so blindly trying:
>>
>> btrfsck --repair /dev/sdc
>>
>> gave the following abort:
>>
>> btrfsck: extent-tree.c:2736: alloc_reserved_tree_block: Assertion
>> `!(ret)' failed.
>>
>> Full output attached.
>>
>>
>> All on:
>>
>> 3.11.2-gentoo
>> Btrfs v0.20-rc1-358-g194aa4a
>>
>> For a 2TB single HDD formatted with defaults.
>>
>>
>> What next?
>>
>> Thanks,
>> Martin
>>
>>
>>
>>
>>>> In the meantime, trying:
>>>>
>>>> btrfsck /dev/sdc
>>>>
>>>> gave the following output + abort:
>>>>
>>>> parent transid verify failed on 915444523008 wanted 16974 found 13021
>>>> Ignoring transid failure
>>>> btrfsck: cmds-check.c:1066: process_file_extent: Assertion `!(rec->ino
>>>> != key->objectid || rec->refs > 1)' failed.
>>>> id not match free space cache generation (1625)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1607)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1604)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1606)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1620)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1626)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1609)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1653)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1628)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1628)
>>>> free space inode generation (0) did not match free space cache
>>>> generation (1649)
>>>>
>>>>
>>>> (There was no syslog output.)
>>>>
>>>> Full btrfsck listing attached.
>>>>
>>>>
>>>> Suggestions please?
>>>>
>>>> Thanks,
>>>> Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: btrfsck --repair --init-extent-tree: segfault error 4
2013-10-07 14:56 ` btrfsck --repair --init-extent-tree: segfault error 4 Martin
@ 2013-10-07 19:03 ` Chris Murphy
2013-10-09 16:03 ` Martin
0 siblings, 1 reply; 21+ messages in thread
From: Chris Murphy @ 2013-10-07 19:03 UTC (permalink / raw)
To: Martin; +Cc: linux-btrfs
On Oct 7, 2013, at 8:56 AM, Martin <m_btrfs@ml1.co.uk> wrote:
>
> Or try "mount -o recovery,noatime" again?
Because of this: free space inode generation (0) did not match free space cache generation (1607)
Try mount option clear_cache. You could then use iotop to make sure the btrfs-freespace process becomes inactive before unmounting the file system; I don't think you need to wait in order to use the file system, nor do you need to unmount then remount without the option. But if it works, it should only be needed once, not as a persistent mount option.
> Or is it dead?
>
> (The 1.5TB of backup data is replicated elsewhere but it would be good
> to rescue this version rather than completely redo from scratch.
> Especially so for the sake of just a few MBytes of one corrupt directory
> tree.)
Right. If you snapshot the subvolume containing the corrupt portion of the file system, the snapshot probably inherits that corruption. But if you write to only one of them, if those writes make the problem worse, should be isolated only to the one you write to. I might avoid writing to it, honestly. To save time, get increasingly aggressive to get data out of this directory and once you succeed, blow away the file system and start from scratch.
You could also then try kernel 3.12 rc4, as there are some btrfs bug fixes I'm seeing in there also, but I don't know if any of them will help your case. If you try it, mount normally, then try to get your data. If that doesn't work, try the recovery option. Maybe you'll get different results.
Chris Murphy
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: btrfsck --repair --init-extent-tree: segfault error 4
2013-10-07 19:03 ` Chris Murphy
@ 2013-10-09 16:03 ` Martin
0 siblings, 0 replies; 21+ messages in thread
From: Martin @ 2013-10-09 16:03 UTC (permalink / raw)
To: linux-btrfs
In summary:
Looks like minimal damage remains and yet I'm still suffering
"Input/output error" from btrfs and btrfsck appears to have looped...
A diff check suggests the damage to be in one (heavily linked to) tree
of a few MBytes.
Would a scrub clear out the damaged trees?
Worth debugging?
Thanks,
Martin
Further detail:
On 07/10/13 20:03, Chris Murphy wrote:
>
> On Oct 7, 2013, at 8:56 AM, Martin <m_btrfs@ml1.co.uk> wrote:
>
>>
>> Or try "mount -o recovery,noatime" again?
>
> Because of this: free space inode generation (0) did not match free
> space cache generation (1607)
>
> Try mount option clear_cache. You could then use iotop to make sure
> the btrfs-freespace process becomes inactive before unmounting the
> file system; I don't think you need to wait in order to use the file
> system, nor do you need to unmount then remount without the option.
> But if it works, it should only be needed once, not as a persistent
> mount option.
Thanks for that.
So, trying:
mount -v -t btrfs -o recovery,noatime,clear_cache /dev/sdc
gave:
kernel: device label bu_A devid 1 transid 17448 /dev/sdc
kernel: btrfs: enabling inode map caching
kernel: btrfs: enabling auto recovery
kernel: btrfs: force clearing of disk cache
kernel: btrfs: disk space caching is enabled
kernel: btrfs: bdev /dev/sdc errs: wr 0, rd 27, flush 0, corrupt 0, gen 0
btrfs-freespace appeared occasionally briefly in atop but there's no
noticeable disk activity. All very rapidly done?
Running a diff check to see if all ok and what might be missing gave the
syslog output:
kernel: verify_parent_transid: 165 callbacks suppressed
kernel: parent transid verify failed on 915444506624 wanted 16974 found
13021
kernel: parent transid verify failed on 915444506624 wanted 16974 found
13021
kernel: parent transid verify failed on 915444506624 wanted 16974 found
13021
kernel: parent transid verify failed on 915444506624 wanted 16974 found
13021
kernel: parent transid verify failed on 915444506624 wanted 16974 found
13021
kernel: parent transid verify failed on 915444506624 wanted 16974 found
13021
The diff eventually failed with "Input/output error".
'mv' to move this failed directory tree out of the way worked.
Attempting to use 'ln -s' gave the attached syslog output and the
filesystem was made "Read-only".
Remounting:
mount -v -o remount,recovery,noatime,clear_cache,rw /dev/sdc
and the mv looks fine. Trying the 'ln -s' again gives:
ln: creating symbolic link `./portage': Read-only file system
unmounting gave the syslog message:
kernel: btrfs: commit super ret -30
Mounting again:
mount -v -t btrfs -o recovery,noatime,clear_cache /dev/sdc
showed that the symbolic link was put in place ok.
Rerunning the diff check eventually found another "Input/output error".
So unmounted and tried again:
btrfsck --repair --init-extent-tree /dev/sdc
Failed with:
btrfs unable to find ref byte nr 911367733248 parent 0 root 1 owner 2
offset 0
btrfs unable to find ref byte nr 911367737344 parent 0 root 1 owner 1
offset 1
btrfs unable to find ref byte nr 911367741440 parent 0 root 1 owner 0
offset 1
leaf free space ret -297791851, leaf data size 3995, used 297795846
nritems 2
checking extents
btrfsck: extent_io.c:606: free_extent_buffer: Assertion `!(eb->refs <
0)' failed.
enabling repair mode
Checking filesystem on /dev/sdc
UUID: 38a60270-f9c6-4ed4-8421-4bf1253ae0b3
Creating a new extent tree
Failed to find [911367733248, 168, 4096]
Failed to find [911367737344, 168, 4096]
Failed to find [911367741440, 168, 4096]
Rerunning again and this time btrfsck is sat there at 100% CPU for the
last 24 hours. Full output so far is:
parent transid verify failed on 911904604160 wanted 17448 found 17449
parent transid verify failed on 911904604160 wanted 17448 found 17449
parent transid verify failed on 911904604160 wanted 17448 found 17449
parent transid verify failed on 911904604160 wanted 17448 found 17449
Ignoring transid failure
Nothing syslog and no disk activity.
Looped?...
>> Or is it dead?
>>
>> (The 1.5TB of backup data is replicated elsewhere but it would be
>> good to rescue this version rather than completely redo from
>> scratch. Especially so for the sake of just a few MBytes of one
>> corrupt directory tree.)
>
> Right. If you snapshot the subvolume containing the corrupt portion
> of the file system, the snapshot probably inherits that corruption.
> But if you write to only one of them, if those writes make the
> problem worse, should be isolated only to the one you write to. I
> might avoid writing to it, honestly. To save time, get increasingly
> aggressive to get data out of this directory and once you succeed,
> blow away the file system and start from scratch.
>
> You could also then try kernel 3.12 rc4, as there are some btrfs bug
> fixes I'm seeing in there also, but I don't know if any of them will
> help your case. If you try it, mount normally, then try to get your
> data. If that doesn't work, try the recovery option. Maybe you'll get
> different results.
As suspected, thanks.
Would a scrub clear out the damaged trees?
Anything useful to try? Any debug value in looking at the fail cases?
Is there a btrfsck mode of making good everything that is certain and
dumping any remaining fragments into "lost + found"? (Or is that way
down the developments yet?)
Aside: btrfs looks to be usable enough, especially so with the disk
format now stable, to at least offer the well established features as
'stable'...?
(This is the first fail I've had, and considering the sata failed, is
no surprise... Too severe a test! But can the limited damage be
recovered...?)
Thanks,
Martin
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2013-10-09 16:04 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-09-28 19:26 Corrupt btrfs filesystem recovery... (Due to *sata* errors) Martin
2013-09-28 20:51 ` Chris Murphy
2013-09-28 22:51 ` Martin
2013-09-29 2:06 ` Chris Murphy
2013-09-29 2:31 ` Martin
2013-09-28 22:54 ` Martin
2013-09-29 2:10 ` Corrupt btrfs filesystem recovery... What best instructions? Martin
2013-09-29 5:11 ` Duncan
2013-09-29 21:29 ` Martin
2013-09-29 21:55 ` Martin
2013-09-30 7:51 ` Duncan
2013-10-03 0:49 ` Martin
2013-10-03 1:31 ` Chris Murphy
2013-10-03 16:56 ` Martin
2013-10-04 15:43 ` Martin
2013-10-05 11:32 ` Martin
2013-10-05 13:18 ` Martin
2013-10-07 14:56 ` btrfsck --repair --init-extent-tree: segfault error 4 Martin
2013-10-07 19:03 ` Chris Murphy
2013-10-09 16:03 ` Martin
2013-10-05 12:05 ` ASM1083 rev01 PCIe to PCI Bridge chip (Was: Corrupt btrfs filesystem recovery... (Due to *sata* errors)) Martin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.