* raid6 file system in a bad state
@ 2016-10-10 16:04 Jason D. Michaelson
2016-10-10 20:59 ` Chris Murphy
0 siblings, 1 reply; 12+ messages in thread
From: Jason D. Michaelson @ 2016-10-10 16:04 UTC (permalink / raw)
To: linux-btrfs
At some point in the last week, I had a 6-disk raid6 pool go south on me.
One of the disks had a write problem, unbeknownst to me, which caused the
entire pool and its subvolumes to remount read only.
When this problem occurred I was on debian jessie kernel 3.16.something.
Following list advice I upgraded to the latest in jessie-backports, 4.7.5.
My git clone of btrfs-progs is at commit
81f4d96f3d6368dc4e5edf7e3cb9d19bb4d00c4f
Not knowing the cause of the problem, I unmounted and attempted to remount,
which failed, with the following coming from dmesg:
[308063.610960] BTRFS info (device sda): allowing degraded mounts
[308063.610972] BTRFS info (device sda): disk space caching is enabled
[308063.723461] BTRFS error (device sda): parent transid verify failed on
5752357961728 wanted 161562 found 159746
[308063.815224] BTRFS info (device sda): bdev /dev/sdh errs: wr 261, rd 1,
flush 87, corrupt 0, gen 0
[308063.849613] BTRFS error (device sda): parent transid verify failed on
5752642420736 wanted 161562 found 159786
[308063.881024] BTRFS error (device sda): parent transid verify failed on
5752472338432 wanted 161562 found 159751
[308063.940225] BTRFS error (device sda): parent transid verify failed on
5752478842880 wanted 161562 found 159752
[308063.979517] BTRFS error (device sda): parent transid verify failed on
5752543526912 wanted 161562 found 159764
[308064.012479] BTRFS error (device sda): parent transid verify failed on
5752513036288 wanted 161562 found 159764
[308064.049169] BTRFS error (device sda): parent transid verify failed on
5752642617344 wanted 161562 found 159786
[308064.080507] BTRFS error (device sda): parent transid verify failed on
5752642650112 wanted 161562 found 159786
[308064.138951] BTRFS error (device sda): parent transid verify failed on
5752610603008 wanted 161562 found 159783
[308064.164326] BTRFS error (device sda): bad tree block start
5918360357649457268 5752610603008
[308064.173752] BTRFS error (device sda): bad tree block start
5567295971165396096 5752610603008
[308064.182026] BTRFS error (device sda): failed to read block groups: -5
[308064.234174] BTRFS: open_ctree failed
/dev/sdh is the disc that had the write error
btrfs filesystem show produces this:
root@castor:~/btrfs-progs# btrfs filesystem show
Label: none uuid: 73ed01df-fb2a-4b27-b6fc-12a57da934bd
Total devices 6 FS bytes used 6.46TiB
devid 1 size 2.73TiB used 1.64TiB path /dev/sda
devid 2 size 2.73TiB used 1.64TiB path /dev/sdh
devid 3 size 2.73TiB used 1.64TiB path /dev/sdd
devid 4 size 2.73TiB used 1.64TiB path /dev/sdg
devid 5 size 2.73TiB used 1.64TiB path /dev/sdf
devid 6 size 2.73TiB used 1.64TiB path /dev/sde
I just now discovered the raid5/6 checksum bug and am hoping I haven't
somehow hit that, since I haven't actually written much of anything to the
discs in quite a long time (save for a few recently-ripped ISOs that must
have been going there when the sdh write error happened).
While there's a lot of stuff I don't care about on the pool, I've got a lot
of Blu Ray ISOs on it that I'd rather not have to re-rip if I can avoid it
(the backups for those are the original discs in my movie cabinet), plus a
local Debian mirror that I'd rather not have to re-sync.
btrfs restore gives this:
parent transid verify failed on 5752357961728 wanted 161562 found 159746
parent transid verify failed on 5752357961728 wanted 161562 found 159746
checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
bytenr mismatch, want=5752357961728, have=56504706479104
Couldn't setup extent tree
This is a dry-run, no files are going to be restored
Restoring /dev/null/BluRay-StarWars
Restoring /dev/null/BluRay-StarWars/Star Wars 4 A New Hope.iso
Restoring /dev/null/BluRay-StarWars/Star Wars 6 Return of the Jedi.iso
Restoring /dev/null/BluRay-StarWars/Star Wars Bonus Episodes 1 2 3.iso
Restoring /dev/null/BluRay-StarWars/Star Wars 5 The Empire Strikes Back.iso
Restoring /dev/null/BluRay-StarWars/Star Wars Spoofs and Documentaries.iso
Restoring /dev/null/BluRay-StarWars/Star Wars Bonus Episodes 4 5 6.iso
Restoring /dev/null/BluRay-StarWars/Star Wars 1 The Phantom Menace.iso
Restoring /dev/null/BluRay-StarWars/Star Wars 2 Attack of the Clones.iso
Restoring /dev/null/BluRay-StarWars/Star Wars 3 Revenge of the Sith.iso
Found objectid=257, key=256
Done searching /BluRay-StarWars
Restoring /dev/null/BluRay-HarryPotter
Restoring /dev/null/BluRay-HarryPotter/Year 1 Harry Potter and the Sorcerers
Stone.iso
Restoring /dev/null/BluRay-HarryPotter/Year 2 Harry Potter and the Chamber
of Secrets.iso
Restoring /dev/null/BluRay-HarryPotter/Year 3 Harry Potter and the Prizoner
of Azkaban.iso
Restoring /dev/null/BluRay-HarryPotter/Year 4 Harry Potter and the Goblet of
Fire.iso
Restoring /dev/null/BluRay-HarryPotter/Year 5 Harry Potter and the Order of
the Phoenix.iso
Found objectid=257, key=256
Done searching /BluRay-HarryPotter
Restoring /dev/null/BluRay-DowntonAbbey
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 1 Disc 1.iso
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 1 Disc 2.iso
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 2 Disc 1.iso
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 2 Disc 2.iso
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 2 Disc 3.iso
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 3 Disc 1.iso
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 3 Disc 2.iso
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 3 Disc 3.iso
Found objectid=257, key=256
Done searching /BluRay-DowntonAbbey
And there's a lot of stuff in here in the output that I don't really care
about so moving on to the end:
Restoring /dev/null/root/Software
Found objectid=304888, key=257402
Done searching /root/Software
Found objectid=257, key=256
Done searching /root
parent transid verify failed on 5752616845312 wanted 161562 found 159784
parent transid verify failed on 5752616845312 wanted 161562 found 159784
checksum verify failed on 5752616845312 found 0EB38D74 wanted 1BB07DCA
checksum verify failed on 5752616845312 found B4E0DBD6 wanted DD91E4E9
checksum verify failed on 5752616845312 found B4E0DBD6 wanted DD91E4E9
bytenr mismatch, want=5752616845312, have=857463135251540499
Error reading subvolume /dev/null/BluRay-Disney: 18446744073709551611
After reading some of the suggestions, I attempted a btrfs rescue
chunk-recover, which results in a SIGBUS error ~40% through the process:
root@castor:~/btrfs-progs# gdb --args ./btrfs rescue chunk-recover -vvvv
/dev/sda
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
(gdb) r
Starting program: /root/btrfs-progs/btrfs rescue chunk-recover -vvvv
/dev/sda
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
All Devices:
Device: id = 2, name = /dev/sdh
Device: id = 3, name = /dev/sdd
Device: id = 5, name = /dev/sdf
Device: id = 6, name = /dev/sde
Device: id = 4, name = /dev/sdg
Device: id = 1, name = /dev/sda
[New Thread 0x7ffff6f91700 (LWP 14913)]
[New Thread 0x7ffff6790700 (LWP 14914)]
[New Thread 0x7ffff5f8f700 (LWP 14915)]
[New Thread 0x7ffff578e700 (LWP 14916)]
[New Thread 0x7ffff4f8d700 (LWP 14917)]
[New Thread 0x7fffe7fff700 (LWP 14918)]
Scanning: 449541562368 in dev0, 689038929920 in dev1, 681330704384 in dev2,
669722726400 in dev3, 681526239232 in dev4, 675649212416 in dev5[Thread
0x7ffff6f91700 (LWP 14913) exited]
Scanning: DONE in dev0, 1203854462976 in dev1, 1209906450432 in dev2,
1194740371456 in dev3, 1211076476928 in dev4, 1212511375360 in dev5
Program received signal SIGBUS, Bus error.
[Switching to Thread 0x7ffff4f8d700 (LWP 14917)]
btrfs_new_block_group_record (leaf=leaf@entry=0x7fffdc0008c0,
key=key@entry=0x7ffff4f8ccb0, slot=slot@entry=30) at cmds-check.c:5258
5258 rec->flags = btrfs_disk_block_group_flags(leaf, ptr);
(gdb) p leaf
p $1 = (struct extent_buffer *) 0x7fffdc0008c0
(gdb) p ptr
$2 = (struct btrfs_block_group_item *) 0x68eb1bad
(gdb) p *leaf
$3 = {cache_node = {rb_node = {__rb_parent_color = 0, rb_right = 0x0,
rb_left = 0x0}, objectid = 0, start = 0, size = 0}, start = 0, dev_bytenr =
0,
len = 16384, tree = 0x0, lru = {next = 0x0, prev = 0x0}, recow = {next =
0x0, prev = 0x0}, refs = 0, flags = 0, fd = 0, data = 0x7fffdc000940
"5\f\004\n"}
(gdb) p *ptr
Cannot access memory at address 0x68eb1bad
(gdb) bt
#0 btrfs_new_block_group_record (leaf=leaf@entry=0x7fffdc0008c0,
key=key@entry=0x7ffff4f8ccb0, slot=slot@entry=30) at cmds-check.c:5258
#1 0x0000000000434c2f in process_block_group_item (slot=30,
key=0x7ffff4f8ccb0, leaf=0x7fffdc0008c0, bg_cache=0x7fffffffe998) at
chunk-recover.c:232
#2 extract_metadata_record (rc=rc@entry=0x7fffffffe960,
leaf=leaf@entry=0x7fffdc0008c0) at chunk-recover.c:717
#3 0x000000000043538c in scan_one_device (dev_scan_struct=0x6a6450) at
chunk-recover.c:807
#4 0x00007ffff73450a4 in start_thread (arg=0x7ffff4f8d700) at
pthread_create.c:309
#5 0x00007ffff707a62d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
I was disappointed to see dev0 (which corresponds to /dev/sdh) come out as
DONE because of these dmesg entries:
[232572.871164] mpt2sas_cm0: log_info(0x31080000): originator(PL),
code(0x08), sub_code(0x0000)
[232572.871185] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 c0 00 00 01 00 00 00
[232572.871190] mpt2sas_cm0: sas_address(0x50030480002e5946), phy(6)
[232572.871193] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232572.871197] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000),
smid(36)
[232572.871200] mpt2sas_cm0: request_len(131072), underflow(131072),
resid(131072)
[232572.871202] mpt2sas_cm0: tag(1), transfer_count(0),
sc->result(0x00000000)
[232572.871205] mpt2sas_cm0: scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232572.871208] mpt2sas_cm0: [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232572.871239] sd 0:0:8:0: [sdh] tag#4 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232572.871243] sd 0:0:8:0: [sdh] tag#4 Sense Key : Medium Error [current]
[232572.871248] sd 0:0:8:0: [sdh] tag#4 Add. Sense: Unrecovered read error
[232572.871252] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 c0 00 00 01 00 00 00
[232572.871256] blk_update_request: critical medium error, dev sdh, sector
878010816
[232578.796809] mpt2sas_cm0: log_info(0x31080000): originator(PL),
code(0x08), sub_code(0x0000)
[232578.796838] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 f0 00 00 00 40 00 00
[232578.796845] mpt2sas_cm0: sas_address(0x50030480002e5946), phy(6)
[232578.796850] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232578.796856] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000),
smid(36)
[232578.796860] mpt2sas_cm0: request_len(32768), underflow(32768),
resid(0)
[232578.796864] mpt2sas_cm0: tag(0), transfer_count(32768),
sc->result(0x00000000)
[232578.796869] mpt2sas_cm0: scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232578.796874] mpt2sas_cm0: [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232578.797129] sd 0:0:8:0: [sdh] tag#4 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232578.797138] sd 0:0:8:0: [sdh] tag#4 Sense Key : Medium Error [current]
[232578.797146] sd 0:0:8:0: [sdh] tag#4 Add. Sense: Unrecovered read error
[232578.797154] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 f0 00 00 00 40 00 00
[232578.797160] blk_update_request: critical medium error, dev sdh, sector
878010888
[232581.663794] mpt2sas_cm0: log_info(0x31080000): originator(PL),
code(0x08), sub_code(0x0000)
[232581.663823] sd 0:0:8:0: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 34
55 62 30 00 00 00 80 00 00
[232581.663830] mpt2sas_cm0: sas_address(0x50030480002e5946), phy(6)
[232581.663835] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232581.663841] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000),
smid(62)
[232581.663845] mpt2sas_cm0: request_len(65536), underflow(65536),
resid(65536)
[232581.663849] mpt2sas_cm0: tag(0), transfer_count(0),
sc->result(0x00000000)
[232581.663854] mpt2sas_cm0: scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232581.663859] mpt2sas_cm0: [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232581.663918] sd 0:0:8:0: [sdh] tag#1 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232581.663937] sd 0:0:8:0: [sdh] tag#1 Sense Key : Medium Error [current]
[232581.663951] sd 0:0:8:0: [sdh] tag#1 Add. Sense: Unrecovered read error
[232581.663960] sd 0:0:8:0: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 34
55 62 30 00 00 00 80 00 00
[232581.663967] blk_update_request: critical medium error, dev sdh, sector
878010928
[232584.622191] sd 0:0:8:0: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 34
55 62 08 00 00 00 08 00 00
[232584.622207] mpt2sas_cm0: sas_address(0x50030480002e5946), phy(6)
[232584.622213] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232584.622219] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000),
smid(55)
[232584.622224] mpt2sas_cm0: request_len(4096), underflow(4096),
resid(4096)
[232584.622228] mpt2sas_cm0: tag(0), transfer_count(0),
sc->result(0x00000000)
[232584.622233] mpt2sas_cm0: scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232584.622237] mpt2sas_cm0: [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232584.622272] sd 0:0:8:0: [sdh] tag#0 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232584.622282] sd 0:0:8:0: [sdh] tag#0 Sense Key : Medium Error [current]
[232584.622295] sd 0:0:8:0: [sdh] tag#0 Add. Sense: Unrecovered read error
[232584.622304] sd 0:0:8:0: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 34
55 62 08 00 00 00 08 00 00
[232584.622311] blk_update_request: critical medium error, dev sdh, sector
878010888
[232584.630625] Buffer I/O error on dev sdh, logical block 109751361, async
page read
rather than moving on but that's neither here nor there, since the disc
really can't be trusted as it is.
btrfs check produces this output:
root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sda
parent transid verify failed on 5752357961728 wanted 161562 found 159746
parent transid verify failed on 5752357961728 wanted 161562 found 159746
checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
bytenr mismatch, want=5752357961728, have=56504706479104
Couldn't setup extent tree
ERROR: cannot open file system
Like I said, the vast majority of what's on this disc is either BluRay ISOs
that I can re-rip, stuff I don't care about recovering, or stuff that I can
always re-mirror if I have to. Given that I'm well versed in C programming,
I'd much rather devote my time to working with the code to resolve whatever
problem may be happening here than re-acquiring or re-ripping what's on that
pool.
Since it took somewhere near an hour and a half to get to that SIGBUS in
gdb, I've left my session open for anyone who may have ideas to chime in.
Just let me know what information you need!
Thanks
Jason Michaelson
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: raid6 file system in a bad state 2016-10-10 16:04 raid6 file system in a bad state Jason D. Michaelson @ 2016-10-10 20:59 ` Chris Murphy [not found] ` <5ce201d22364$96702780$c3507680$@com> 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-10-10 20:59 UTC (permalink / raw) To: Jason D. Michaelson; +Cc: Btrfs BTRFS On Mon, Oct 10, 2016 at 10:04 AM, Jason D. Michaelson <jasondmichaelson@gmail.com> wrote: > One of the disks had a write problem, unbeknownst to me, which caused the > entire pool and its subvolumes to remount read only. Can you be more specific about the write problem? What are the messages from the logs about these write problems and is that problem now fixed? > > When this problem occurred I was on debian jessie kernel 3.16.something. > Following list advice I upgraded to the latest in jessie-backports, 4.7.5. > My git clone of btrfs-progs is at commit > 81f4d96f3d6368dc4e5edf7e3cb9d19bb4d00c4f > > Not knowing the cause of the problem, I unmounted and attempted to remount, > which failed, with the following coming from dmesg: > > [308063.610960] BTRFS info (device sda): allowing degraded mounts > [308063.610972] BTRFS info (device sda): disk space caching is enabled > [308063.723461] BTRFS error (device sda): parent transid verify failed on > 5752357961728 wanted 161562 found 159746 > [308063.815224] BTRFS info (device sda): bdev /dev/sdh errs: wr 261, rd 1, > flush 87, corrupt 0, gen 0 > [308063.849613] BTRFS error (device sda): parent transid verify failed on > 5752642420736 wanted 161562 found 159786 > [308063.881024] BTRFS error (device sda): parent transid verify failed on > 5752472338432 wanted 161562 found 159751 > [308063.940225] BTRFS error (device sda): parent transid verify failed on > 5752478842880 wanted 161562 found 159752 > [308063.979517] BTRFS error (device sda): parent transid verify failed on > 5752543526912 wanted 161562 found 159764 > [308064.012479] BTRFS error (device sda): parent transid verify failed on > 5752513036288 wanted 161562 found 159764 > [308064.049169] BTRFS error (device sda): parent transid verify failed on > 5752642617344 wanted 161562 found 159786 > [308064.080507] BTRFS error (device sda): parent transid verify failed on > 5752642650112 wanted 161562 found 159786 > [308064.138951] BTRFS error (device sda): parent transid verify failed on > 5752610603008 wanted 161562 found 159783 > [308064.164326] BTRFS error (device sda): bad tree block start > 5918360357649457268 5752610603008 > [308064.173752] BTRFS error (device sda): bad tree block start > 5567295971165396096 5752610603008 > [308064.182026] BTRFS error (device sda): failed to read block groups: -5 > [308064.234174] BTRFS: open_ctree failed Sometimes it will be more tolerant with mount -o degraded,ro. > /dev/sdh is the disc that had the write error > > [232578.796809] mpt2sas_cm0: log_info(0x31080000): originator(PL), > code(0x08), sub_code(0x0000) > [232578.796838] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34 > 55 61 f0 00 00 00 40 00 00 > [232578.796845] mpt2sas_cm0: sas_address(0x50030480002e5946), phy(6) > [232578.796850] mpt2sas_cm0: > enclosure_logical_id(0x50030442523a2033),slot(2) > [232578.796856] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000), > smid(36) > [232578.796860] mpt2sas_cm0: request_len(32768), underflow(32768), > resid(0) > [232578.796864] mpt2sas_cm0: tag(0), transfer_count(32768), > sc->result(0x00000000) > [232578.796869] mpt2sas_cm0: scsi_status(check condition)(0x02), > scsi_state(autosense valid )(0x01) > [232578.796874] mpt2sas_cm0: [sense_key,asc,ascq]: [0x03,0x11,0x00], > count(18) > [232578.797129] sd 0:0:8:0: [sdh] tag#4 FAILED Result: hostbyte=DID_OK > driverbyte=DRIVER_SENSE > [232578.797138] sd 0:0:8:0: [sdh] tag#4 Sense Key : Medium Error [current] > [232578.797146] sd 0:0:8:0: [sdh] tag#4 Add. Sense: Unrecovered read error > [232578.797154] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34 > 55 61 f0 00 00 00 40 00 00 > [232578.797160] blk_update_request: critical medium error, dev sdh, sector > 878010888 Each one of these complains about a read error and a different LBA is reported. These should get fixed automatically by Btrfs since kernel 3.19. The problem is that you were using 3.16 so they were left to accumulate. 3.16 kernel needed a full balance to fix these. > [232581.663794] mpt2sas_cm0: log_info(0x31080000): originator(PL), > code(0x08), sub_code(0x0000) > [232581.663823] sd 0:0:8:0: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 34 > 55 62 30 00 00 00 80 00 00 > [232581.663830] mpt2sas_cm0: sas_address(0x50030480002e5946), phy(6) > [232581.663835] mpt2sas_cm0: > enclosure_logical_id(0x50030442523a2033),slot(2) > [232581.663841] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000), > smid(62) > [232581.663845] mpt2sas_cm0: request_len(65536), underflow(65536), > resid(65536) > [232581.663849] mpt2sas_cm0: tag(0), transfer_count(0), > sc->result(0x00000000) > [232581.663854] mpt2sas_cm0: scsi_status(check condition)(0x02), > scsi_state(autosense valid )(0x01) > [232581.663859] mpt2sas_cm0: [sense_key,asc,ascq]: [0x03,0x11,0x00], > count(18) > [232581.663918] sd 0:0:8:0: [sdh] tag#1 FAILED Result: hostbyte=DID_OK > driverbyte=DRIVER_SENSE > [232581.663937] sd 0:0:8:0: [sdh] tag#1 Sense Key : Medium Error [current] > [232581.663951] sd 0:0:8:0: [sdh] tag#1 Add. Sense: Unrecovered read error > [232581.663960] sd 0:0:8:0: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 34 > 55 62 30 00 00 00 80 00 00 > [232581.663967] blk_update_request: critical medium error, dev sdh, sector > 878010928 > [232584.622191] sd 0:0:8:0: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 34 > 55 62 08 00 00 00 08 00 00 > [232584.622207] mpt2sas_cm0: sas_address(0x50030480002e5946), phy(6) > [232584.622213] mpt2sas_cm0: > enclosure_logical_id(0x50030442523a2033),slot(2) > [232584.622219] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000), > smid(55) > [232584.622224] mpt2sas_cm0: request_len(4096), underflow(4096), > resid(4096) > [232584.622228] mpt2sas_cm0: tag(0), transfer_count(0), > sc->result(0x00000000) > [232584.622233] mpt2sas_cm0: scsi_status(check condition)(0x02), > scsi_state(autosense valid )(0x01) > [232584.622237] mpt2sas_cm0: [sense_key,asc,ascq]: [0x03,0x11,0x00], > count(18) > [232584.622272] sd 0:0:8:0: [sdh] tag#0 FAILED Result: hostbyte=DID_OK > driverbyte=DRIVER_SENSE > [232584.622282] sd 0:0:8:0: [sdh] tag#0 Sense Key : Medium Error [current] > [232584.622295] sd 0:0:8:0: [sdh] tag#0 Add. Sense: Unrecovered read error > [232584.622304] sd 0:0:8:0: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 34 > 55 62 08 00 00 00 08 00 00 > [232584.622311] blk_update_request: critical medium error, dev sdh, sector > 878010888 > [232584.630625] Buffer I/O error on dev sdh, logical block 109751361, async > page read > > > rather than moving on but that's neither here nor there, since the disc > really can't be trusted as it is. You don't have to, that's Btrfs problem. What's confusing to me is that interspersed with these read errors, there are no Btrfs messages. I'm expecting Btrfs to do something about the read error if those sectors are in a Btrfs volume. Either Btrfs should say whether it's fixing the problem, in either case. But I see nothing. > > btrfs check produces this output: > > root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sda > parent transid verify failed on 5752357961728 wanted 161562 found 159746 > parent transid verify failed on 5752357961728 wanted 161562 found 159746 > checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76 > checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 > checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 > bytenr mismatch, want=5752357961728, have=56504706479104 > Couldn't setup extent tree > ERROR: cannot open file system Seems like there should be more to the story than just one bad drive or it'd be possible to mount minus the bad drive. Maybe sdh is spitting out spurious data. Have you tried physically removing /dev/sdh and then doing a degraded mount? An entire syslog, journalctl, or dmesg might be helpful rather than excerpts. > Like I said, the vast majority of what's on this disc is either BluRay ISOs > that I can re-rip, stuff I don't care about recovering, or stuff that I can > always re-mirror if I have to. Given that I'm well versed in C programming, > I'd much rather devote my time to working with the code to resolve whatever > problem may be happening here than re-acquiring or re-ripping what's on that > pool. > > Since it took somewhere near an hour and a half to get to that SIGBUS in > gdb, I've left my session open for anyone who may have ideas to chime in. > Just let me know what information you need! That part I can't help out with. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <5ce201d22364$96702780$c3507680$@com>]
* Re: raid6 file system in a bad state [not found] ` <5ce201d22364$96702780$c3507680$@com> @ 2016-10-11 4:23 ` Chris Murphy 2016-10-11 15:52 ` Jason D. Michaelson 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-10-11 4:23 UTC (permalink / raw) To: Jason D. Michaelson; +Cc: Chris Murphy, Btrfs BTRFS What do you get for btrfs-find-root <dev> btrfs rescue super-recover -v <dev> It shouldn't matter which dev you pick, unless it face plants, then try another. ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: raid6 file system in a bad state 2016-10-11 4:23 ` Chris Murphy @ 2016-10-11 15:52 ` Jason D. Michaelson 2016-10-11 16:06 ` Chris Murphy 0 siblings, 1 reply; 12+ messages in thread From: Jason D. Michaelson @ 2016-10-11 15:52 UTC (permalink / raw) To: 'Chris Murphy'; +Cc: 'Btrfs BTRFS' > -----Original Message----- > From: chris@colorremedies.com [mailto:chris@colorremedies.com] On > Behalf Of Chris Murphy > Sent: Monday, October 10, 2016 11:23 PM > To: Jason D. Michaelson > Cc: Chris Murphy; Btrfs BTRFS > Subject: Re: raid6 file system in a bad state > > What do you get for > > btrfs-find-root <dev> root@castor:~/logs# btrfs-find-root /dev/sda parent transid verify failed on 5752357961728 wanted 161562 found 159746 parent transid verify failed on 5752357961728 wanted 161562 found 159746 Couldn't setup extent tree Superblock thinks the generation is 161562 Superblock thinks the level is 1 There's no further output, and btrfs-find-root is pegged at 100%. At the moment, the perceived bad disc is connected. I received the same results without as well. > btrfs rescue super-recover -v <dev> root@castor:~/logs# btrfs rescue super-recover -v /dev/sda All Devices: Device: id = 2, name = /dev/sdh Device: id = 3, name = /dev/sdd Device: id = 5, name = /dev/sdf Device: id = 6, name = /dev/sde Device: id = 4, name = /dev/sdg Device: id = 1, name = /dev/sda Before Recovering: [All good supers]: device name = /dev/sdd superblock bytenr = 65536 device name = /dev/sdd superblock bytenr = 67108864 device name = /dev/sdd superblock bytenr = 274877906944 device name = /dev/sdf superblock bytenr = 65536 device name = /dev/sdf superblock bytenr = 67108864 device name = /dev/sdf superblock bytenr = 274877906944 device name = /dev/sde superblock bytenr = 65536 device name = /dev/sde superblock bytenr = 67108864 device name = /dev/sde superblock bytenr = 274877906944 device name = /dev/sdg superblock bytenr = 65536 device name = /dev/sdg superblock bytenr = 67108864 device name = /dev/sdg superblock bytenr = 274877906944 device name = /dev/sda superblock bytenr = 65536 device name = /dev/sda superblock bytenr = 67108864 device name = /dev/sda superblock bytenr = 274877906944 [All bad supers]: device name = /dev/sdh superblock bytenr = 65536 device name = /dev/sdh superblock bytenr = 67108864 device name = /dev/sdh superblock bytenr = 274877906944 Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: n Aborted to recover bad superblocks I aborted this waiting for instructions on whether to proceed from the list. > > > > It shouldn't matter which dev you pick, unless it face plants, then try > another. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: raid6 file system in a bad state 2016-10-11 15:52 ` Jason D. Michaelson @ 2016-10-11 16:06 ` Chris Murphy 2016-10-11 16:10 ` Jason D. Michaelson 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-10-11 16:06 UTC (permalink / raw) To: Jason D. Michaelson; +Cc: Chris Murphy, Btrfs BTRFS On Tue, Oct 11, 2016 at 9:52 AM, Jason D. Michaelson <jasondmichaelson@gmail.com> wrote: >> btrfs rescue super-recover -v <dev> > > root@castor:~/logs# btrfs rescue super-recover -v /dev/sda > All Devices: > Device: id = 2, name = /dev/sdh > Device: id = 3, name = /dev/sdd > Device: id = 5, name = /dev/sdf > Device: id = 6, name = /dev/sde > Device: id = 4, name = /dev/sdg > Device: id = 1, name = /dev/sda > > Before Recovering: > [All good supers]: > device name = /dev/sdd > superblock bytenr = 65536 > > device name = /dev/sdd > superblock bytenr = 67108864 > > device name = /dev/sdd > superblock bytenr = 274877906944 > > device name = /dev/sdf > superblock bytenr = 65536 > > device name = /dev/sdf > superblock bytenr = 67108864 > > device name = /dev/sdf > superblock bytenr = 274877906944 > > device name = /dev/sde > superblock bytenr = 65536 > > device name = /dev/sde > superblock bytenr = 67108864 > > device name = /dev/sde > superblock bytenr = 274877906944 > > device name = /dev/sdg > superblock bytenr = 65536 > > device name = /dev/sdg > superblock bytenr = 67108864 > > device name = /dev/sdg > superblock bytenr = 274877906944 > > device name = /dev/sda > superblock bytenr = 65536 > > device name = /dev/sda > superblock bytenr = 67108864 > > device name = /dev/sda > superblock bytenr = 274877906944 > > [All bad supers]: > device name = /dev/sdh > superblock bytenr = 65536 > > device name = /dev/sdh > superblock bytenr = 67108864 > > device name = /dev/sdh > superblock bytenr = 274877906944 > > > Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: n > Aborted to recover bad superblocks > > I aborted this waiting for instructions on whether to proceed from the list. Bad superblocks can't be a good thing and would only cause confusion. I'd think that a known bad superblock would be ignored at mount time and even by btrfs-find-root, or maybe even replaced like any other kind of known bad metadata where good copies are available. btrfs-show-super -f /dev/sda btrfs-show-super -f /dev/sdh Find out what the difference is between good and bad supers. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: raid6 file system in a bad state 2016-10-11 16:06 ` Chris Murphy @ 2016-10-11 16:10 ` Jason D. Michaelson 2016-10-11 17:41 ` Chris Murphy 0 siblings, 1 reply; 12+ messages in thread From: Jason D. Michaelson @ 2016-10-11 16:10 UTC (permalink / raw) To: 'Chris Murphy'; +Cc: 'Btrfs BTRFS' > > > Bad superblocks can't be a good thing and would only cause confusion. > I'd think that a known bad superblock would be ignored at mount time > and even by btrfs-find-root, or maybe even replaced like any other kind > of known bad metadata where good copies are available. > > btrfs-show-super -f /dev/sda > btrfs-show-super -f /dev/sdh > > > Find out what the difference is between good and bad supers. > root@castor:~# btrfs-show-super -f /dev/sda superblock: bytenr=65536, device=/dev/sda --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x45278835 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd label generation 161562 root 5752616386560 sys_array_size 354 chunk_root_generation 156893 root_level 1 chunk_root 20971520 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 18003557892096 bytes_used 7107627130880 sectorsize 4096 nodesize 16384 leafsize 16384 stripesize 4096 root_dir 6 num_devices 6 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0xe1 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | RAID56 ) cache_generation 161562 uuid_tree_generation 161562 dev_item.uuid 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 dev_item.fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd [match] dev_item.type 0 dev_item.total_bytes 3000592982016 dev_item.bytes_used 1800957198336 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0) chunk length 4194304 owner 2 stripe_len 65536 type SYSTEM num_stripes 1 stripe 0 devid 1 offset 0 dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) chunk length 11010048 owner 2 stripe_len 65536 type SYSTEM|RAID6 num_stripes 6 stripe 0 devid 6 offset 1048576 dev uuid: 390a1fd8-cc6c-40e7-b0b5-88ca7dcbcc32 stripe 1 devid 5 offset 1048576 dev uuid: 2df974c5-9dde-4062-81e9-c6eeee13db62 stripe 2 devid 4 offset 1048576 dev uuid: dce3d159-721d-4859-9955-37a03769bb0d stripe 3 devid 3 offset 1048576 dev uuid: 6f7142db-824c-4791-a5b2-d6ce11c81c8f stripe 4 devid 2 offset 1048576 dev uuid: dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c stripe 5 devid 1 offset 20971520 dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 backup_roots[4]: backup 0: backup_tree_root: 5752437456896 gen: 161561 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 5752385224704 gen: 161561 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 5752437587968 gen: 161561 level: 1 backup_csum_root: 5752389615616 gen: 161561 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7112579833856 backup_num_devices: 6 backup 1: backup_tree_root: 5752616386560 gen: 161562 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 5752649416704 gen: 161563 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 5752616501248 gen: 161562 level: 1 backup_csum_root: 5752650203136 gen: 161563 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7107602407424 backup_num_devices: 6 backup 2: backup_tree_root: 5752112103424 gen: 161559 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 5752207409152 gen: 161560 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 5752113463296 gen: 161559 level: 1 backup_csum_root: 5752205492224 gen: 161560 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7112514002944 backup_num_devices: 6 backup 3: backup_tree_root: 5752298307584 gen: 161560 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 5752385224704 gen: 161561 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 5752299978752 gen: 161560 level: 1 backup_csum_root: 5752389615616 gen: 161561 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7112542425088 backup_num_devices: 6 root@castor:~# btrfs-show-super -f /dev/sdh superblock: bytenr=65536, device=/dev/sdh --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x0f7dfe09 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd label generation 161474 root 4844272943104 sys_array_size 354 chunk_root_generation 156893 root_level 1 chunk_root 20971520 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 18003557892096 bytes_used 7110395990016 sectorsize 4096 nodesize 16384 leafsize 16384 stripesize 4096 root_dir 6 num_devices 6 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0xe1 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | RAID56 ) cache_generation 161474 uuid_tree_generation 161474 dev_item.uuid dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c dev_item.fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd [match] dev_item.type 0 dev_item.total_bytes 3000592982016 dev_item.bytes_used 1800936226816 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0) chunk length 4194304 owner 2 stripe_len 65536 type SYSTEM num_stripes 1 stripe 0 devid 1 offset 0 dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) chunk length 11010048 owner 2 stripe_len 65536 type SYSTEM|RAID6 num_stripes 6 stripe 0 devid 6 offset 1048576 dev uuid: 390a1fd8-cc6c-40e7-b0b5-88ca7dcbcc32 stripe 1 devid 5 offset 1048576 dev uuid: 2df974c5-9dde-4062-81e9-c6eeee13db62 stripe 2 devid 4 offset 1048576 dev uuid: dce3d159-721d-4859-9955-37a03769bb0d stripe 3 devid 3 offset 1048576 dev uuid: 6f7142db-824c-4791-a5b2-d6ce11c81c8f stripe 4 devid 2 offset 1048576 dev uuid: dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c stripe 5 devid 1 offset 20971520 dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 backup_roots[4]: backup 0: backup_tree_root: 4844253364224 gen: 161473 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 4844248121344 gen: 161473 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 1411186688 gen: 156893 level: 1 backup_csum_root: 4844247793664 gen: 161473 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7110380077056 backup_num_devices: 6 backup 1: backup_tree_root: 4844272943104 gen: 161474 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 4844268240896 gen: 161474 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 1411186688 gen: 156893 level: 1 backup_csum_root: 4844254216192 gen: 161474 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7110395990016 backup_num_devices: 6 backup 2: backup_tree_root: 4844252168192 gen: 161471 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 4844242698240 gen: 161471 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 1411186688 gen: 156893 level: 1 backup_csum_root: 4844241764352 gen: 161471 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7110343888896 backup_num_devices: 6 backup 3: backup_tree_root: 4844263358464 gen: 161472 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 backup_extent_root: 4844261965824 gen: 161472 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 backup_dev_root: 1411186688 gen: 156893 level: 1 backup_csum_root: 4844261801984 gen: 161472 level: 3 backup_total_bytes: 18003557892096 backup_bytes_used: 7110370037760 backup_num_devices: 6 root@castor:~# btrfs-show-super -f /dev/sda > sda root@castor:~# btrfs-show-super -f /dev/sdh > sdh root@castor:~# diff -u sda sdh --- sda 2016-10-11 11:09:42.853170807 -0500 +++ sdh 2016-10-11 11:09:46.469082028 -0500 @@ -1,16 +1,16 @@ -superblock: bytenr=65536, device=/dev/sda +superblock: bytenr=65536, device=/dev/sdh --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 -csum 0x45278835 [match] +csum 0x0f7dfe09 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd label -generation 161562 -root 5752616386560 +generation 161474 +root 4844272943104 sys_array_size 354 chunk_root_generation 156893 root_level 1 @@ -20,7 +20,7 @@ log_root_transid 0 log_root_level 0 total_bytes 18003557892096 -bytes_used 7107627130880 +bytes_used 7110395990016 sectorsize 4096 nodesize 16384 leafsize 16384 @@ -34,17 +34,17 @@ BIG_METADATA | EXTENDED_IREF | RAID56 ) -cache_generation 161562 -uuid_tree_generation 161562 -dev_item.uuid 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 +cache_generation 161474 +uuid_tree_generation 161474 +dev_item.uuid dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c dev_item.fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd [match] dev_item.type 0 dev_item.total_bytes 3000592982016 -dev_item.bytes_used 1800957198336 +dev_item.bytes_used 1800936226816 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 -dev_item.devid 1 +dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 @@ -72,47 +72,47 @@ dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 backup_roots[4]: backup 0: - backup_tree_root: 5752437456896 gen: 161561 level: 1 + backup_tree_root: 4844253364224 gen: 161473 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 - backup_extent_root: 5752385224704 gen: 161561 level: 2 + backup_extent_root: 4844248121344 gen: 161473 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 - backup_dev_root: 5752437587968 gen: 161561 level: 1 - backup_csum_root: 5752389615616 gen: 161561 level: 3 + backup_dev_root: 1411186688 gen: 156893 level: 1 + backup_csum_root: 4844247793664 gen: 161473 level: 3 backup_total_bytes: 18003557892096 - backup_bytes_used: 7112579833856 + backup_bytes_used: 7110380077056 backup_num_devices: 6 backup 1: - backup_tree_root: 5752616386560 gen: 161562 level: 1 + backup_tree_root: 4844272943104 gen: 161474 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 - backup_extent_root: 5752649416704 gen: 161563 level: 2 + backup_extent_root: 4844268240896 gen: 161474 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 - backup_dev_root: 5752616501248 gen: 161562 level: 1 - backup_csum_root: 5752650203136 gen: 161563 level: 3 + backup_dev_root: 1411186688 gen: 156893 level: 1 + backup_csum_root: 4844254216192 gen: 161474 level: 3 backup_total_bytes: 18003557892096 - backup_bytes_used: 7107602407424 + backup_bytes_used: 7110395990016 backup_num_devices: 6 backup 2: - backup_tree_root: 5752112103424 gen: 161559 level: 1 + backup_tree_root: 4844252168192 gen: 161471 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 - backup_extent_root: 5752207409152 gen: 161560 level: 2 + backup_extent_root: 4844242698240 gen: 161471 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 - backup_dev_root: 5752113463296 gen: 161559 level: 1 - backup_csum_root: 5752205492224 gen: 161560 level: 3 + backup_dev_root: 1411186688 gen: 156893 level: 1 + backup_csum_root: 4844241764352 gen: 161471 level: 3 backup_total_bytes: 18003557892096 - backup_bytes_used: 7112514002944 + backup_bytes_used: 7110343888896 backup_num_devices: 6 backup 3: - backup_tree_root: 5752298307584 gen: 161560 level: 1 + backup_tree_root: 4844263358464 gen: 161472 level: 1 backup_chunk_root: 20971520 gen: 156893 level: 1 - backup_extent_root: 5752385224704 gen: 161561 level: 2 + backup_extent_root: 4844261965824 gen: 161472 level: 2 backup_fs_root: 124387328 gen: 74008 level: 0 - backup_dev_root: 5752299978752 gen: 161560 level: 1 - backup_csum_root: 5752389615616 gen: 161561 level: 3 + backup_dev_root: 1411186688 gen: 156893 level: 1 + backup_csum_root: 4844261801984 gen: 161472 level: 3 backup_total_bytes: 18003557892096 - backup_bytes_used: 7112542425088 + backup_bytes_used: 7110370037760 backup_num_devices: 6 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: raid6 file system in a bad state 2016-10-11 16:10 ` Jason D. Michaelson @ 2016-10-11 17:41 ` Chris Murphy [not found] ` <5e8701d223f1$c7ea0960$57be1c20$@com> 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-10-11 17:41 UTC (permalink / raw) To: Jason D. Michaelson; +Cc: Chris Murphy, Btrfs BTRFS On Tue, Oct 11, 2016 at 10:10 AM, Jason D. Michaelson <jasondmichaelson@gmail.com> wrote: > superblock: bytenr=65536, device=/dev/sda > --------------------------------------------------------- > generation 161562 > root 5752616386560 > superblock: bytenr=65536, device=/dev/sdh > --------------------------------------------------------- > generation 161474 > root 4844272943104 OK so most obvious is that the bad super is many generations back than the good super. That's expected given all the write errors. >root@castor:~/logs# btrfs-find-root /dev/sda >parent transid verify failed on 5752357961728 wanted 161562 found 159746 >parent transid verify failed on 5752357961728 wanted 161562 found 159746 >Couldn't setup extent tree >Superblock thinks the generation is 161562 >Superblock thinks the level is 1 This squares with the good super. So btrfs-find-root is using a good super. I don't know what 5752357961728 is for, maybe it's possible to read that with btrfs-debug-tree -b 5752357961728 <anydev> and see what comes back. This is not the tree root according to the super though. So what do you get for btrfs-debug-tree -b 5752616386560 <anydev> Going back to your logs.... [ 38.810575] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 38.810595] NFSD: starting 90-second grace period (net ffffffffb12e5b80) [ 241.292816] INFO: task bfad_worker:234 blocked for more than 120 seconds. [ 241.299135] Not tainted 4.7.0-0.bpo.1-amd64 #1 [ 241.305645] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. I don't know what this kernel is. I think you'd be better off with stable 4.7.7 or 4.8.1 for this work, so you're not running into a bunch of weird blocked task problems in addition to whatever is going on with the fs. [ 38.810575] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 38.810595] NFSD: starting 90-second grace period (net ffffffffb12e5b80) [ 241.292816] INFO: task bfad_worker:234 blocked for more than 120 seconds. [ 241.299135] Not tainted 4.7.0-0.bpo.1-amd64 #1 [ 241.305645] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. I don't know what this kernel is. I think you'd be better off with stable 4.7.7 or 4.8.1 for this work, so you're not running into a bunch of weird blocked task problems in addition to whatever is going on with the fs. [ 20.552205] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd devid 3 transid 161562 /dev/sdd [ 20.552372] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd devid 5 transid 161562 /dev/sdf [ 20.552524] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd devid 6 transid 161562 /dev/sde [ 20.552689] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd devid 4 transid 161562 /dev/sdg [ 20.552858] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd devid 1 transid 161562 /dev/sda [ 669.843166] BTRFS warning (device sda): devid 2 uuid dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c is missing [232572.871243] sd 0:0:8:0: [sdh] tag#4 Sense Key : Medium Error [current] Two items missing, in effect, for this failed read. One literally missing, and the other one missing due to unrecoverable read error. The fact it's not trying to fix anything suggests it hasn't really finished mounting, there must be something wrong where it either just gets confused and won't fix (because it might make things worse) or there isn't reduncancy. [52799.495999] mce: [Hardware Error]: Machine check events logged [53249.491975] mce: [Hardware Error]: Machine check events logged [231298.005594] mce: [Hardware Error]: Machine check events logged Bunch of other hardware issues... I *really* think you need to get the hardware issues sorted out before working on this file system unless you just don't care that much about it. There are already enough unknowns without contributing who knows what effect the hardware issues are having while trying to repair things. Or even understand what's going on. > sys_chunk_array[2048]: > item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0) > chunk length 4194304 owner 2 stripe_len 65536 > type SYSTEM num_stripes 1 > stripe 0 devid 1 offset 0 > dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 > item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) > chunk length 11010048 owner 2 stripe_len 65536 > type SYSTEM|RAID6 num_stripes 6 > stripe 0 devid 6 offset 1048576 > dev uuid: 390a1fd8-cc6c-40e7-b0b5-88ca7dcbcc32 > stripe 1 devid 5 offset 1048576 > dev uuid: 2df974c5-9dde-4062-81e9-c6eeee13db62 > stripe 2 devid 4 offset 1048576 > dev uuid: dce3d159-721d-4859-9955-37a03769bb0d > stripe 3 devid 3 offset 1048576 > dev uuid: 6f7142db-824c-4791-a5b2-d6ce11c81c8f > stripe 4 devid 2 offset 1048576 > dev uuid: dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c > stripe 5 devid 1 offset 20971520 > dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4 Huh, well item 0 is damn strange. I wonder how that happened. The dev uuid of that single system chunk matches devid 1. This is a single source of failure. This could be an artifact of creating the raid6 with an old btrfs-progs. I just created a volume with btrfs-progs 4.7.3: # mkfs.btrfs -draid6 -mraid6 /dev/mapper/VG-1 /dev/mapper/VG-2 /dev/mapper/VG-3 /dev/mapper/VG-4 /dev/mapper/VG-5 /dev/mapper/VG-6 And the super block sys_chunk_array has only a SYSTEM|RAID6 chunk. There is no single SYSTEM chunk. After mounting and copying some data over, then umounting, same thing. One system chunk, raid6. So *IF* there is anything wrong with this single system chunk, it's all bets are off, no way to even attempt to fix the problem. That might explain why it's not getting past the very earliest stage of mounting. But it's inconclusive. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <5e8701d223f1$c7ea0960$57be1c20$@com>]
* Re: raid6 file system in a bad state [not found] ` <5e8701d223f1$c7ea0960$57be1c20$@com> @ 2016-10-11 20:38 ` Chris Murphy 2016-10-12 17:59 ` Jason D. Michaelson 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-10-11 20:38 UTC (permalink / raw) To: Jason D. Michaelson, Btrfs BTRFS; +Cc: Chris Murphy readding btrfs On Tue, Oct 11, 2016 at 1:00 PM, Jason D. Michaelson <jasondmichaelson@gmail.com> wrote: > > >> -----Original Message----- >> From: chris@colorremedies.com [mailto:chris@colorremedies.com] On >> Behalf Of Chris Murphy >> Sent: Tuesday, October 11, 2016 12:41 PM >> To: Jason D. Michaelson >> Cc: Chris Murphy; Btrfs BTRFS >> Subject: Re: raid6 file system in a bad state >> >> On Tue, Oct 11, 2016 at 10:10 AM, Jason D. Michaelson >> <jasondmichaelson@gmail.com> wrote: >> > superblock: bytenr=65536, device=/dev/sda >> > --------------------------------------------------------- >> > generation 161562 >> > root 5752616386560 >> >> >> >> > superblock: bytenr=65536, device=/dev/sdh >> > --------------------------------------------------------- >> > generation 161474 >> > root 4844272943104 >> >> OK so most obvious is that the bad super is many generations back than >> the good super. That's expected given all the write errors. >> >> > > Is there any chance/way of going back to use this generation/root as a source for btrfs restore? Yes with -t option and that root bytenr for the generation you want to restore. Thing is, that's so far back the metadata may be gone (overwritten) already. But worth a shot. I've recovered recently deleted files this way. OK at this point I'm thinking that fixing the super blocks won't change anything because it sounds like it's using the new ones anyway and maybe the thing to try is going back to a tree root that isn't in any of the new supers. That means losing anything that was being written when the lost writes happened. However, for all we know some overwrites happened so this won't work. And also it does nothing to deal with the fragile state of having at least two flaky devices, and one of the system chunks with no redundancy. Try 'btrfs check' without repair. And then also try it with -r flag using the various tree roots we've seen so far. Try explicitly using 5752616386560, which is what it ought to use first anyway. And then also 4844272943104. That might go far enough back before the bad sectors were a factor. Normally what you'd want is for it to use one of the backup roots, but it's consistently running into a problem with all of them when using recovery mount option. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: raid6 file system in a bad state 2016-10-11 20:38 ` Chris Murphy @ 2016-10-12 17:59 ` Jason D. Michaelson 2016-10-12 19:36 ` Chris Murphy 0 siblings, 1 reply; 12+ messages in thread From: Jason D. Michaelson @ 2016-10-12 17:59 UTC (permalink / raw) To: 'Chris Murphy', 'Btrfs BTRFS' > -----Original Message----- > From: chris@colorremedies.com [mailto:chris@colorremedies.com] On > Behalf Of Chris Murphy > Sent: Tuesday, October 11, 2016 3:38 PM > To: Jason D. Michaelson; Btrfs BTRFS > Cc: Chris Murphy > Subject: Re: raid6 file system in a bad state > > readding btrfs > > On Tue, Oct 11, 2016 at 1:00 PM, Jason D. Michaelson > <jasondmichaelson@gmail.com> wrote: > > > > > >> -----Original Message----- > >> From: chris@colorremedies.com [mailto:chris@colorremedies.com] On > >> Behalf Of Chris Murphy > >> Sent: Tuesday, October 11, 2016 12:41 PM > >> To: Jason D. Michaelson > >> Cc: Chris Murphy; Btrfs BTRFS > >> Subject: Re: raid6 file system in a bad state > >> > >> On Tue, Oct 11, 2016 at 10:10 AM, Jason D. Michaelson > >> <jasondmichaelson@gmail.com> wrote: > >> > superblock: bytenr=65536, device=/dev/sda > >> > --------------------------------------------------------- > >> > generation 161562 > >> > root 5752616386560 > >> > >> > >> > >> > superblock: bytenr=65536, device=/dev/sdh > >> > --------------------------------------------------------- > >> > generation 161474 > >> > root 4844272943104 > >> > >> OK so most obvious is that the bad super is many generations back > >> than the good super. That's expected given all the write errors. > >> > >> > > > > Is there any chance/way of going back to use this generation/root as > a source for btrfs restore? > > Yes with -t option and that root bytenr for the generation you want to > restore. Thing is, that's so far back the metadata may be gone > (overwritten) already. But worth a shot. I've recovered recently > deleted files this way. With the bad disc in place: root@castor:~/btrfs-progs# ./btrfs restore -t 4844272943104 -D /dev/sda /dev/null parent transid verify failed on 4844272943104 wanted 161562 found 161476 parent transid verify failed on 4844272943104 wanted 161562 found 161476 checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 bytenr mismatch, want=4844272943104, have=66211125067776 Couldn't read tree root Could not open root, trying backup super warning, device 6 is missing warning, device 5 is missing warning, device 4 is missing warning, device 3 is missing warning, device 2 is missing checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB bytenr mismatch, want=20971520, have=267714560 ERROR: cannot read chunk root Could not open root, trying backup super warning, device 6 is missing warning, device 5 is missing warning, device 4 is missing warning, device 3 is missing warning, device 2 is missing checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB bytenr mismatch, want=20971520, have=267714560 ERROR: cannot read chunk root Could not open root, trying backup super And what's interesting is that when I move the /dev/sdd (the current bad disc) out of /dev, rescan, and run btrfs restore with the main root I get similar output: root@castor:~/btrfs-progs# ./btrfs restore -D /dev/sda /dev/null warning, device 2 is missing checksum verify failed on 21430272 found 71001E6E wanted 95E3A3D8 checksum verify failed on 21430272 found 992E0C37 wanted 36992D8B checksum verify failed on 21430272 found 992E0C37 wanted 36992D8B bytenr mismatch, want=21430272, have=264830976 Couldn't read chunk tree Could not open root, trying backup super warning, device 6 is missing warning, device 5 is missing warning, device 4 is missing warning, device 3 is missing warning, device 2 is missing checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB bytenr mismatch, want=20971520, have=267714560 ERROR: cannot read chunk root Could not open root, trying backup super warning, device 6 is missing warning, device 5 is missing warning, device 4 is missing warning, device 3 is missing warning, device 2 is missing checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB bytenr mismatch, want=20971520, have=267714560 ERROR: cannot read chunk root Could not open root, trying backup super So it doesn't seem to work, but the difference in output between the two, at least to my untrained eyes, is intriguing, to say the least. > > > OK at this point I'm thinking that fixing the super blocks won't change > anything because it sounds like it's using the new ones anyway and > maybe the thing to try is going back to a tree root that isn't in any > of the new supers. That means losing anything that was being written > when the lost writes happened. However, for all we know some overwrites > happened so this won't work. And also it does nothing to deal with the > fragile state of having at least two flaky devices, and one of the > system chunks with no redundancy. > This is the one thing I'm not following you on. I know there's one device that's flaky. Originally sdi, switched to sdh, and today (after reboot to 4.7.7), sdd. You'll have to forgive my ignorance, but I'm missing how you determined that a second was flaky (or was that from the ITEM 0 not being replicated you mentioned yesterday?) > > Try 'btrfs check' without repair. And then also try it with -r flag > using the various tree roots we've seen so far. Try explicitly using > 5752616386560, which is what it ought to use first anyway. And then > also 4844272943104. > root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sda parent transid verify failed on 5752357961728 wanted 161562 found 159746 parent transid verify failed on 5752357961728 wanted 161562 found 159746 checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76 checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 bytenr mismatch, want=5752357961728, have=56504706479104 Couldn't setup extent tree ERROR: cannot open file system root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sdd parent transid verify failed on 4844272943104 wanted 161474 found 161476 parent transid verify failed on 4844272943104 wanted 161474 found 161476 checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 bytenr mismatch, want=4844272943104, have=66211125067776 Couldn't read tree root ERROR: cannot open file system root@castor:~/btrfs-progs# ./btrfs check --readonly -r 5752616386560 /dev/sda parent transid verify failed on 5752357961728 wanted 161562 found 159746 parent transid verify failed on 5752357961728 wanted 161562 found 159746 checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76 checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 bytenr mismatch, want=5752357961728, have=56504706479104 Couldn't setup extent tree ERROR: cannot open file system root@castor:~/btrfs-progs# ./btrfs check --readonly -r 5752616386560 /dev/sdd parent transid verify failed on 5752616386560 wanted 161474 found 161562 parent transid verify failed on 5752616386560 wanted 161474 found 161562 checksum verify failed on 5752616386560 found 2A134884 wanted CEF0F532 checksum verify failed on 5752616386560 found B7FE62DB wanted 3786D60F checksum verify failed on 5752616386560 found B7FE62DB wanted 3786D60F bytenr mismatch, want=5752616386560, have=56504661311488 Couldn't read tree root ERROR: cannot open file system root@castor:~/btrfs-progs# ./btrfs check --readonly -r 4844272943104 /dev/sda parent transid verify failed on 4844272943104 wanted 161562 found 161476 parent transid verify failed on 4844272943104 wanted 161562 found 161476 checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 bytenr mismatch, want=4844272943104, have=66211125067776 Couldn't read tree root ERROR: cannot open file system root@castor:~/btrfs-progs# ./btrfs check --readonly -r 4844272943104 /dev/sdd parent transid verify failed on 4844272943104 wanted 161474 found 161476 parent transid verify failed on 4844272943104 wanted 161474 found 161476 checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 bytenr mismatch, want=4844272943104, have=66211125067776 Couldn't read tree root ERROR: cannot open file system > That might go far enough back before the bad sectors were a factor. > Normally what you'd want is for it to use one of the backup roots, but > it's consistently running into a problem with all of them when using > recovery mount option. > Is that a result of all of them being identical, save for the bad disc? Again, Chris, Thank you so much for your time looking at this! Btrfs on the whole is something that, as a developer, I'd love to become involved with. Alas, there are only 24 hours in the day. > > > > > -- > Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: raid6 file system in a bad state 2016-10-12 17:59 ` Jason D. Michaelson @ 2016-10-12 19:36 ` Chris Murphy 2016-10-14 21:54 ` Chris Murphy 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-10-12 19:36 UTC (permalink / raw) To: Jason D. Michaelson; +Cc: Chris Murphy, Btrfs BTRFS On Wed, Oct 12, 2016 at 11:59 AM, Jason D. Michaelson <jasondmichaelson@gmail.com> wrote: > With the bad disc in place: > > root@castor:~/btrfs-progs# ./btrfs restore -t 4844272943104 -D /dev/sda /dev/null > parent transid verify failed on 4844272943104 wanted 161562 found 161476 > parent transid verify failed on 4844272943104 wanted 161562 found 161476 > checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > bytenr mismatch, want=4844272943104, have=66211125067776 > Couldn't read tree root > Could not open root, trying backup super > warning, device 6 is missing > warning, device 5 is missing > warning, device 4 is missing > warning, device 3 is missing > warning, device 2 is missing > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > bytenr mismatch, want=20971520, have=267714560 > ERROR: cannot read chunk root > Could not open root, trying backup super > warning, device 6 is missing > warning, device 5 is missing > warning, device 4 is missing > warning, device 3 is missing > warning, device 2 is missing > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > bytenr mismatch, want=20971520, have=267714560 > ERROR: cannot read chunk root > Could not open root, trying backup super Don't all of these device missing messages seem bogus? I don't know how to find out what's going on here. If it were me, I'd try to reproduce this with a couple of distros's live images (Fedora Rawhide and openSUSE Tumbleweed), and if they're both reproducing this "missing" output, I'd file a bugzilla.kernel.org bug with a strace. I mean, this stuff is hard enough as it is without bugs like this getting in the way. Fedora 25 nightly: https://kojipkgs.fedoraproject.org/compose/branched/Fedora-25-20161008.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-25-20161008.n.0.iso That'll have some version of kernel 4.8, not sure which one. And it will have btrfs-progs 4.6.1 which is safe but showing its age in Btrfs years. You can do this from inside the live environment: sudo dnf update https://kojipkgs.fedoraproject.org//packages/btrfs-progs/4.7.3/1.fc26/x86_64/btrfs-progs-4.7.3-1.fc26.x86_64.rpm or sudo dnf update https://kojipkgs.fedoraproject.org//packages/btrfs-progs/4.8.1/1.fc26/x86_64/btrfs-progs-4.8.1-1.fc26.x86_64.rpm It's probably just as valid to do this with whatever you have now, strace that and file a bug. But it doesn't really for sure isolate whether it's a local problem or not. > > And what's interesting is that when I move the /dev/sdd (the current bad disc) out of /dev, rescan, and run btrfs restore with the main root I get similar output: > > root@castor:~/btrfs-progs# ./btrfs restore -D /dev/sda /dev/null > warning, device 2 is missing > checksum verify failed on 21430272 found 71001E6E wanted 95E3A3D8 > checksum verify failed on 21430272 found 992E0C37 wanted 36992D8B > checksum verify failed on 21430272 found 992E0C37 wanted 36992D8B > bytenr mismatch, want=21430272, have=264830976 > Couldn't read chunk tree > Could not open root, trying backup super > warning, device 6 is missing > warning, device 5 is missing > warning, device 4 is missing > warning, device 3 is missing > warning, device 2 is missing > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > bytenr mismatch, want=20971520, have=267714560 > ERROR: cannot read chunk root > Could not open root, trying backup super > warning, device 6 is missing > warning, device 5 is missing > warning, device 4 is missing > warning, device 3 is missing > warning, device 2 is missing > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB > bytenr mismatch, want=20971520, have=267714560 > ERROR: cannot read chunk root > Could not open root, trying backup super > > So it doesn't seem to work, but the difference in output between the two, at least to my untrained eyes, is intriguing, to say the least. Yeah I'm not sure what to recommend now. > >> >> >> OK at this point I'm thinking that fixing the super blocks won't change >> anything because it sounds like it's using the new ones anyway and >> maybe the thing to try is going back to a tree root that isn't in any >> of the new supers. That means losing anything that was being written >> when the lost writes happened. However, for all we know some overwrites >> happened so this won't work. And also it does nothing to deal with the >> fragile state of having at least two flaky devices, and one of the >> system chunks with no redundancy. >> > > This is the one thing I'm not following you on. I know there's one device that's flaky. Originally sdi, switched to sdh, and today (after reboot to 4.7.7), sdd. You'll have to forgive my ignorance, but I'm missing how you determined that a second was flaky (or was that from the ITEM 0 not being replicated you mentioned yesterday?) In your dmesg there was one device reported missing entirely, and then a separate device had a sector read failure. > >> >> Try 'btrfs check' without repair. And then also try it with -r flag >> using the various tree roots we've seen so far. Try explicitly using >> 5752616386560, which is what it ought to use first anyway. And then >> also 4844272943104. >> > > root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sda > parent transid verify failed on 5752357961728 wanted 161562 found 159746 > parent transid verify failed on 5752357961728 wanted 161562 found 159746 > checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76 > checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 > checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 > bytenr mismatch, want=5752357961728, have=56504706479104 > Couldn't setup extent tree > ERROR: cannot open file system > root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sdd > parent transid verify failed on 4844272943104 wanted 161474 found 161476 > parent transid verify failed on 4844272943104 wanted 161474 found 161476 > checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > bytenr mismatch, want=4844272943104, have=66211125067776 > Couldn't read tree root > ERROR: cannot open file system > > root@castor:~/btrfs-progs# ./btrfs check --readonly -r 5752616386560 /dev/sda > parent transid verify failed on 5752357961728 wanted 161562 found 159746 > parent transid verify failed on 5752357961728 wanted 161562 found 159746 > checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76 > checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 > checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 > bytenr mismatch, want=5752357961728, have=56504706479104 > Couldn't setup extent tree > ERROR: cannot open file system > root@castor:~/btrfs-progs# ./btrfs check --readonly -r 5752616386560 /dev/sdd > parent transid verify failed on 5752616386560 wanted 161474 found 161562 > parent transid verify failed on 5752616386560 wanted 161474 found 161562 > checksum verify failed on 5752616386560 found 2A134884 wanted CEF0F532 > checksum verify failed on 5752616386560 found B7FE62DB wanted 3786D60F > checksum verify failed on 5752616386560 found B7FE62DB wanted 3786D60F > bytenr mismatch, want=5752616386560, have=56504661311488 > Couldn't read tree root > ERROR: cannot open file system > > root@castor:~/btrfs-progs# ./btrfs check --readonly -r 4844272943104 /dev/sda > parent transid verify failed on 4844272943104 wanted 161562 found 161476 > parent transid verify failed on 4844272943104 wanted 161562 found 161476 > checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > bytenr mismatch, want=4844272943104, have=66211125067776 > Couldn't read tree root > ERROR: cannot open file system > root@castor:~/btrfs-progs# ./btrfs check --readonly -r 4844272943104 /dev/sdd > parent transid verify failed on 4844272943104 wanted 161474 found 161476 > parent transid verify failed on 4844272943104 wanted 161474 found 161476 > checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640 > bytenr mismatch, want=4844272943104, have=66211125067776 > Couldn't read tree root > ERROR: cannot open file system Someone else who knows more will have to speak up. This is one of the more annoying things about Btrfs's state right now, is it's not at all clear to a regular user what sequence to attempt repairs in. It's a shot in the dark. Other file systems it's much easier. It fails to mount, you run fsck with default options, and it either can fix it or it can't. Btrfs, it's many options, many orders, very developer oriented messages, and no hints what the next step is to take. At this point you could set up some kind of overlay on each drive, maybe also using blockdev to set each block device read only to ensure the original is not modified. Something like this: https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file But that will avoid this: "Block-level copies of devices" https://btrfs.wiki.kernel.org/index.php/Gotchas I haven't tried this, so I'm not really certain how to hide the original and overlay from the kernel since they both have to be present at the same time. Maybe an LVM snapshot LV can be presented to libvirt/virt-manager and you can use one a recent distro image to boot the VM and try some repairs. I just can't tell you what order to do them in. Cannot read chunk root is a problem, maybe it can be repaired with btrfs rescue chunk-recover. Cannot read tree root is also a problem, once the chunk is repaired, maybe it's possible to repair it. The extent tree can't be used until the chunk tree is readable so that ought to just take care of itself. You might be looking at chunk recover, super recover, check --repair, and maybe even check --repair --init-extent-tree. And as a last resort --init-csum-tree which really is just papering over real problems in a way that now the file system won't know what's bad and makes things worse but it might survive long enough to get more data off. And actually, before any of the above, you could see if you can take a btrfs-image -t4 -c9 -s, and also btrfs-debug-tree and output to a file somewhere. Maybe then it's a useful donation image for making the tools better. > > >> That might go far enough back before the bad sectors were a factor. >> Normally what you'd want is for it to use one of the backup roots, but >> it's consistently running into a problem with all of them when using >> recovery mount option. >> > > Is that a result of all of them being identical, save for the bad disc? I don't understand the question. The bad disk is the one that has the bad super, but all the tools are clearly ignoring the bad super when looking for the tree root. So I don't think the bad disk is a factor. I can't prove it but I think the problems were happening before the bad disk, it's just that the bad disk added to the confusion and may also be preventing repairs. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: raid6 file system in a bad state 2016-10-12 19:36 ` Chris Murphy @ 2016-10-14 21:54 ` Chris Murphy 2016-10-17 18:52 ` Jason D. Michaelson 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-10-14 21:54 UTC (permalink / raw) To: Chris Murphy; +Cc: Jason D. Michaelson, Btrfs BTRFS This may be relevant and is pretty terrible. http://www.spinics.net/lists/linux-btrfs/msg59741.html Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: raid6 file system in a bad state 2016-10-14 21:54 ` Chris Murphy @ 2016-10-17 18:52 ` Jason D. Michaelson 0 siblings, 0 replies; 12+ messages in thread From: Jason D. Michaelson @ 2016-10-17 18:52 UTC (permalink / raw) To: 'Chris Murphy'; +Cc: 'Btrfs BTRFS' I've been following that thread. It's been my fear. I'm in the process of doing a restore of what I can get off of it so that i can re-create the file system with raid1 which, if i'm reading that thread correctly doesn't suffer at all from the rmw problems extant in the raid5/6 code at the moment. Again, thanks for your help. > -----Original Message----- > From: chris@colorremedies.com [mailto:chris@colorremedies.com] On > Behalf Of Chris Murphy > Sent: Friday, October 14, 2016 4:55 PM > To: Chris Murphy > Cc: Jason D. Michaelson; Btrfs BTRFS > Subject: Re: raid6 file system in a bad state > > This may be relevant and is pretty terrible. > > http://www.spinics.net/lists/linux-btrfs/msg59741.html > > > Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-10-17 18:52 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-10 16:04 raid6 file system in a bad state Jason D. Michaelson
2016-10-10 20:59 ` Chris Murphy
[not found] ` <5ce201d22364$96702780$c3507680$@com>
2016-10-11 4:23 ` Chris Murphy
2016-10-11 15:52 ` Jason D. Michaelson
2016-10-11 16:06 ` Chris Murphy
2016-10-11 16:10 ` Jason D. Michaelson
2016-10-11 17:41 ` Chris Murphy
[not found] ` <5e8701d223f1$c7ea0960$57be1c20$@com>
2016-10-11 20:38 ` Chris Murphy
2016-10-12 17:59 ` Jason D. Michaelson
2016-10-12 19:36 ` Chris Murphy
2016-10-14 21:54 ` Chris Murphy
2016-10-17 18:52 ` Jason D. Michaelson
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.