* btrfs raid1 degraded does not mount or fsck
@ 2016-04-09 22:44 Vladi Gergov
2016-04-10 6:55 ` Duncan
0 siblings, 1 reply; 6+ messages in thread
From: Vladi Gergov @ 2016-04-09 22:44 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-btrfs
Anyone know if there is currently with updated kernel and tools a way to
recover the data on this? I have tried btrfs chunk-recovery with not
luck. Anything else I can do to try and get the data off at least?
Thanks in advance!
On Tuesday, 16.11.10 at 16:50, Chris Mason wrote:
> Excerpts from Vladi Gergov's message of 2010-10-29 16:53:42 -0400:
> > >>> gypsyops @ /mnt > sudo mount -o degraded /dev/sdc das3/
> > Password:
> > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > [ 684.577540] device label das4 devid 2 transid 107954 /dev/sdc
> > [ 684.595150] btrfs: allowing degraded mounts
> > [ 684.595594] btrfs: failed to read chunk root on sdb
> > [ 684.604110] btrfs: open_ctree failed
> >
> > >>> gypsyops @ /mnt > sudo btrfsck /dev/sdc
> > btrfsck: volumes.c:1367: btrfs_read_sys_array: Assertion `!(ret)' failed.
>
> Ok, I dug through this and found the bug responsible for your
> unmountable FS. When we're mounted in degraded mode, and we don't have
> enough drives available to do raid1,10, we're can use the wrong raid
> level for new allocations.
>
> I'm fixing the kernel side so this doesn't happen anymore, but I'll need
> to rebuild the chunk tree (and probably a few others) off your good disk to
> fix things.
>
> I've got it reproduced here though, so I'll make an fsck that can scan
> for the correct trees and fix it for you.
>
> Since you're basically going to be my first external fsck customer, is
> there anyway you can do a raw device based backup of the blocks? This
> way if I do mess things up we can repeat the experiment.
>
> -chris
>
> !DSPAM:4ce2fd55191821234852255!
>
>
--
,-| Vladi
`-| Gergov
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: btrfs raid1 degraded does not mount or fsck 2016-04-09 22:44 btrfs raid1 degraded does not mount or fsck Vladi Gergov @ 2016-04-10 6:55 ` Duncan 0 siblings, 0 replies; 6+ messages in thread From: Duncan @ 2016-04-10 6:55 UTC (permalink / raw) To: linux-btrfs [Please keep your replies in standard quoted-context, then reply-in- context, order. I'm reordering here, but often either don't include complete context or simply don't bother replying, when it's too much work to get the reply in the proper context.] Vladi Gergov posted on Sat, 09 Apr 2016 15:44:41 -0700 as excerpted: > On Tuesday, 16.11.10 at 16:50, Chris Mason wrote: >> Excerpts from Vladi Gergov's message of 2010-10-29 16:53:42 -0400: [snip] Wow, followups to old message threads! A lot has changed since 2010! >> Ok, I dug through this and found the bug responsible for your >> unmountable FS. When we're mounted in degraded mode, and we don't have >> enough drives available to do raid1,10, we're can use the wrong raid >> level for new allocations. >> >> I'm fixing the kernel side so this doesn't happen anymore [...] Interestingly that's still an issue, tho AFAIK now deliberate. [1] When mounted degraded-writable and there's not enough devices with free space to create new chunks of the appropriate raid type, btrfs will "degrade" to writing single-mode chunks. While this does allow writing to continue, including either balances to change the raid level or btrfs replace or btrfs device add to again provide enough devices to properly handle the configured raid type, it usually results in a one-shot-writable bug, because once those single- mode chunks are there, current kernels see single-mode on a filesystem missing devices and refuse to mount it degraded-writable again. That means you get a one-time writable mount shot at correcting the problem, after which you won't be able to mount writable again (using current kernels) to correct it later, if the repair of the filesystem wasn't completed in that single degraded-writable mount. You *can*, however, still mount degraded,read-only, which should give you access to the files to copy them elsewhere. There are patches available that will change the chunks available check from per-filesystem to per-chunk. With these patches applied, the kernel will allow degraded writable mounting of a nominally raidN filesystem with devices missing, even if there's single-mode chunks on the filesystem as well, *AS*LONG*AS* all chunks have at least one copy available. What that means, in effect, is that as long as the degraded, writable filesystem doesn't lose ANOTHER device, this one with some of the single chunks already written to it, which would thus leave those chunks unavailable, it can mount a degraded filesystem writable more than once. Unfortunately, those patches got added to a more complex patch set that wasn't considered ready for kernel 4.5 and didn't get into it. I'm not sure whether they're in the current 4.6 development kernel or not. But the patches ARE available. > Anyone know if there is currently with updated kernel and tools a way to > recover the data on this? I have tried btrfs chunk-recovery with not > luck. Anything else I can do to try and get the data off at least? > Thanks in advance! As suggested above, if the only problem was a missing device, with anything approaching a current kernel (I'd suggest LTS 4.1 or 4.4 if you're conservative, 4.5 if you want the latest stable kernel series, and preferably a similarly versioned btrfs-progs userspace) you should at least be able to mount the filesystem degraded,ro. A degraded,ro mount should be possible even if there are chunks entirely missing, as long as the superblocks and various other structures remain intact. Or find and apply those patches and you *may* be able to mount the filesystem degraded,rw again, in ordered to replace the missing device and rebalance to normal operation. However, this is a bit more iffy as it depends on no chunks being entirely missing, among other things, some of which may have changed since 2010, if you're still trying to recover that 2010 filesystem. If the filesystem won't mount in degraded,ro or degraded,ro,recovery mode, then there's something wrong with it beyond the missing device. In that case you may not be able to actually mount the filesystem (at least not without intensive surgery which the devs might be able to help you with but which is beyond my level as a simple sysadmin-level btrfs user and list regular)... BUT THERE'S STILL HOPE to at least recover the files. The tool you're looking for in that case is btrfs restore. With luck you can simply point it at the remaining device and give it a place to restore the files to (and fill in options such as whether you want it to try to restore ownership/perms and other metadata, whether you want it to try to restore symlinks as well, etc), and it can do its thing. There's a -D dry-run option you can use to see if it looks promising, before attempting to run it for real. If restore on its own can't help, there's a more advanced mode that works in conjunction with btrfs-find-root, but fair warning, this gets rather technical and most people require some help to do it the first time, if they can manage it at all. I won't go into detail here as there's a page on the wiki that describes the process, and there's another very recent on-list thread where I (and others) was working with someone else trying to do an advanced-mode btrfs-find-root and btrfs restore, and besides, with luck you won't need it anyway. But here's the link to the wiki page and the thread in question, as found on gmane's web interface. https://btrfs.wiki.kernel.org/index.php/Restore http://thread.gmane.org/gmane.comp.file-systems.btrfs/55022 --- [1] C. Mason was working on patches to change it back then, but obviously it didn't work out as he then anticipated it would. He'd have to fill in the specific details, but as I explained above, btrfs now requires a particular number of devices with writable unallocated space in ordered to allocate new chunks of the corresponding raid level, two devices for raid1, for instance. If there's not enough devices with unallocated free space to create a new chunk in that raid mode, as explained, it falls back to single mode, which only requires a single device. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 6+ messages in thread
* btrfs raid1 degraded does not mount or fsck @ 2010-10-29 20:53 Vladi Gergov 2010-10-30 6:55 ` Goffredo Baroncelli 2010-11-16 21:50 ` Chris Mason 0 siblings, 2 replies; 6+ messages in thread From: Vladi Gergov @ 2010-10-29 20:53 UTC (permalink / raw) To: linux-btrfs kernel: scratch git repo from today 10.29.10 @ 14:30 PST Btrfs v0.19-35-g1b444cd-dirty >>> gypsyops @ /mnt > sudo btrfs filesystem show Label: 'das4' uuid: d0e5137f-e5e7-49da-91f6-a9c4e4e72c6f Total devices 3 FS bytes used 1.38TB devid 3 size 1.82TB used 0.00 path /dev/sdb devid 2 size 1.82TB used 1.38TB path /dev/sdc *** Some devices missing Btrfs v0.19-35-g1b444cd-dirty >>> gypsyops @ /mnt > sudo mount -o degraded /dev/sdc das3/ Password: mount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so [ 684.577540] device label das4 devid 2 transid 107954 /dev/sdc [ 684.595150] btrfs: allowing degraded mounts [ 684.595594] btrfs: failed to read chunk root on sdb [ 684.604110] btrfs: open_ctree failed >>> gypsyops @ /mnt > sudo btrfsck /dev/sdc btrfsck: volumes.c:1367: btrfs_read_sys_array: Assertion `!(ret)' failed. any help please? -- ,-| Vladi `-| Gergov !DSPAM:4ccb39a9191821603519226! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: btrfs raid1 degraded does not mount or fsck 2010-10-29 20:53 Vladi Gergov @ 2010-10-30 6:55 ` Goffredo Baroncelli 2010-11-16 21:50 ` Chris Mason 1 sibling, 0 replies; 6+ messages in thread From: Goffredo Baroncelli @ 2010-10-30 6:55 UTC (permalink / raw) To: linux-btrfs On Friday, 29 October, 2010, Vladi Gergov wrote: > kernel: scratch git repo from today 10.29.10 @ 14:30 PST > Btrfs v0.19-35-g1b444cd-dirty > > >>> gypsyops @ /mnt > sudo btrfs filesystem show > Label: 'das4' uuid: d0e5137f-e5e7-49da-91f6-a9c4e4e72c6f > Total devices 3 FS bytes used 1.38TB > devid 3 size 1.82TB used 0.00 path /dev/sdb > devid 2 size 1.82TB used 1.38TB path /dev/sdc > *** Some devices missing > > Btrfs v0.19-35-g1b444cd-dirty > > >>> gypsyops @ /mnt > sudo mount -o degraded /dev/sdc das3/ > Password: > mount: wrong fs type, bad option, bad superblock on /dev/sdc, > missing codepage or helper program, or other error > In some cases useful info is found in syslog - try > dmesg | tail or so > > [ 684.577540] device label das4 devid 2 transid 107954 /dev/sdc > [ 684.595150] btrfs: allowing degraded mounts > [ 684.595594] btrfs: failed to read chunk root on sdb > [ 684.604110] btrfs: open_ctree failed > > >>> gypsyops @ /mnt > sudo btrfsck /dev/sdc > btrfsck: volumes.c:1367: btrfs_read_sys_array: Assertion `!(ret)' failed. > > any help please? did you execute btrfs dev scan before trying to mount the filesystem ? > -- > > ,-| Vladi > `-| Gergov > > !DSPAM:4ccb39a9191821603519226! > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- gpg key@ keyserver.linux.it: Goffredo Baroncelli (ghigo) <kreijack@inwind.it> Key fingerprint = 4769 7E51 5293 D36C 814E C054 BF04 F161 3DC5 0512 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: btrfs raid1 degraded does not mount or fsck 2010-10-29 20:53 Vladi Gergov 2010-10-30 6:55 ` Goffredo Baroncelli @ 2010-11-16 21:50 ` Chris Mason 2011-10-10 4:42 ` Larry Reaves 1 sibling, 1 reply; 6+ messages in thread From: Chris Mason @ 2010-11-16 21:50 UTC (permalink / raw) To: Vladi Gergov; +Cc: linux-btrfs Excerpts from Vladi Gergov's message of 2010-10-29 16:53:42 -0400: > >>> gypsyops @ /mnt > sudo mount -o degraded /dev/sdc das3/ > Password: > mount: wrong fs type, bad option, bad superblock on /dev/sdc, > missing codepage or helper program, or other error > In some cases useful info is found in syslog - try > dmesg | tail or so > > [ 684.577540] device label das4 devid 2 transid 107954 /dev/sdc > [ 684.595150] btrfs: allowing degraded mounts > [ 684.595594] btrfs: failed to read chunk root on sdb > [ 684.604110] btrfs: open_ctree failed > > >>> gypsyops @ /mnt > sudo btrfsck /dev/sdc > btrfsck: volumes.c:1367: btrfs_read_sys_array: Assertion `!(ret)' failed. Ok, I dug through this and found the bug responsible for your unmountable FS. When we're mounted in degraded mode, and we don't have enough drives available to do raid1,10, we're can use the wrong raid level for new allocations. I'm fixing the kernel side so this doesn't happen anymore, but I'll need to rebuild the chunk tree (and probably a few others) off your good disk to fix things. I've got it reproduced here though, so I'll make an fsck that can scan for the correct trees and fix it for you. Since you're basically going to be my first external fsck customer, is there anyway you can do a raw device based backup of the blocks? This way if I do mess things up we can repeat the experiment. -chris ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: btrfs raid1 degraded does not mount or fsck 2010-11-16 21:50 ` Chris Mason @ 2011-10-10 4:42 ` Larry Reaves 0 siblings, 0 replies; 6+ messages in thread From: Larry Reaves @ 2011-10-10 4:42 UTC (permalink / raw) To: linux-btrfs Chris Mason <chris.mason <at> oracle.com> writes: > > Excerpts from Vladi Gergov's message of 2010-10-29 16:53:42 -0400: > > >>> gypsyops @ /mnt > sudo mount -o degraded /dev/sdc das3/ > > Password: > > mount: wrong fs type, bad option, bad superblock on /dev/sdc, > > missing codepage or helper program, or other error > > In some cases useful info is found in syslog - try > > dmesg | tail or so > > > > [ 684.577540] device label das4 devid 2 transid 107954 /dev/sdc > > [ 684.595150] btrfs: allowing degraded mounts > > [ 684.595594] btrfs: failed to read chunk root on sdb > > [ 684.604110] btrfs: open_ctree failed > > > > >>> gypsyops @ /mnt > sudo btrfsck /dev/sdc > > btrfsck: volumes.c:1367: btrfs_read_sys_array: Assertion `!(ret)' failed. > > Ok, I dug through this and found the bug responsible for your > unmountable FS. When we're mounted in degraded mode, and we don't have > enough drives available to do raid1,10, we're can use the wrong raid > level for new allocations. > > I'm fixing the kernel side so this doesn't happen anymore, but I'll need > to rebuild the chunk tree (and probably a few others) off your good disk to > fix things. > > I've got it reproduced here though, so I'll make an fsck that can scan > for the correct trees and fix it for you. > > Since you're basically going to be my first external fsck customer, is > there anyway you can do a raw device based backup of the blocks? This > way if I do mess things up we can repeat the experiment. > > -chris I'm seeing this same error, uname -a shows: 2.6.38-11-generic although, my fs went down a month or so ago so it could have been a slightly earlier Ubuntu kernel. Is this the same issue? Is that very targeted fsck available somewhere? I've got my replacement disk ready to add, I just need to get the other 3 mounted. Data and metadata are both raid10. I really need to start tracking the kernel again :(. Thanks for any help. -Larry ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-04-10 6:55 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-04-09 22:44 btrfs raid1 degraded does not mount or fsck Vladi Gergov 2016-04-10 6:55 ` Duncan -- strict thread matches above, loose matches on Subject: below -- 2010-10-29 20:53 Vladi Gergov 2010-10-30 6:55 ` Goffredo Baroncelli 2010-11-16 21:50 ` Chris Mason 2011-10-10 4:42 ` Larry Reaves
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).