Problem with unmountable filesystem.

Problem with unmountable filesystem.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

So, I just recently had to hard reset a system running  root on BTRFS,
and when it tried to come back up, it chocked on the root filesystem.
Based on the kernel messages, the primary issue is log corruption, and
in theory btrfs-zero-log should fix it.  The actual issue however, is
that the primary superblock appears to be pointing at a corrupted root
tree, which causes pretty much everything that does anything other than
just read the sb to fail.  The first backup sb does point to a good
tree, but only btrfs check and btrfs restore have any option to ignore
the first sb and use one of the backups instead.  To make matters more
complicated, the first sb still has a valid checksum and passes the
tests done by btrfs rescue super-recover, and therefore that can't be
used to recover either.  I was wondering if anyone here might have any
advice.  I'm fine using dd to replace the primary sb with one of the
backups, but don't know the exact parameters that would be needed.
Also, we should consider adding a mount option to select a specific sb
mirror to use; I know that ext* have such an option, and that has
actually saved me a couple of times.  I'm using btrfs-progs 3.16 and
kernel 3.16.1.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: Problem with unmountable filesystem.

[Chris Murphy]

On Sep 16, 2014, at 8:40 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:

> Based on the kernel messages, the primary issue is log corruption, and
> in theory btrfs-zero-log should fix it.

Can you provide a complete dmesg somewhere for this initial failure, just for reference? I'm curious what this indication looks like compared to other problems.

>  The actual issue however, is
> that the primary superblock appears to be pointing at a corrupted root
> tree, which causes pretty much everything that does anything other than
> just read the sb to fail.  The first backup sb does point to a good
> tree, but only btrfs check and btrfs restore have any option to ignore
> the first sb and use one of the backups instead.

Maybe use wipefs -a on this volume, which removes the magic from only the first superblock by default (you can specify another location). And then try btrfs-show-super -F which "dumps" supers with bad magic.

I just tried this:
# wipefs -a /dev/sdb
/dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
# btrfs-show-super -F /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum			0x5c1196d7 [DON'T MATCH]
bytenr			65536
flags			0x1
magic			........ [DON'T MATCH]
[…]
# btrfs-show-super -i1 /dev/sdb
superblock: bytenr=67108864, device=/dev/sdb
---------------------------------------------------------
csum			0xfc70be19 [match]
bytenr			67108864
flags			0x1
magic			_BHRfS_M [match]

So the mirror is definitely there and valid.
# btrfs rescue super-recover -yv /dev/sdb
No valid Btrfs found on /dev/sdb
Usage or syntax errors

Not expected at all, man page says "Recover bad superblocks from good copies." There's a good copy, it's not being found by btrfs rescue super-recover. Seems like a bug.


# btrfs check /dev/sdb
No valid Btrfs found on /dev/sdb
Couldn't open file system

# btrfs check -s1 /dev/sdb
using SB copy 1, bytenr 67108864
Checking filesystem on /dev/sdb
UUID: 9acf13de-5b98-4f28-9992-533e4a99d348
[snip]
OK it finds it, maybe a --repair will fix the bad first one?
# btrfs check -s1 /dev/sdb
using SB copy 1, bytenr 67108864
enabling repair mode
Checking filesystem on /dev/sdb
UUID: 9acf13de-5b98-4f28-9992-533e4a99d348
[snip]
No indication of repair
# btrfs check /dev/sdb
No valid Btrfs found on /dev/sdb
Couldn't open file system
# btrfs check /dev/sdb
No valid Btrfs found on /dev/sdb
Couldn't open file system
[root@f21v ~]# btrfs-show-super -F /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum			0x5c1196d7 [DON'T MATCH]
bytenr			65536
flags			0x1
magic			........ [DON'T MATCH]


Still not fixed. Maybe I needed to corrupt something else in the superblock other than the magic and this behavior is intentional, otherwise wipefs -a, followed by btrfsck would resurrect an intentionally wiped btrfs fs, potentially wiping out some newer file system in the process.



> I'm fine using dd to replace the primary sb with one of the
> backups, but don't know the exact parameters that would be needed.

Here's an idea:

# btrfs-show-super /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum			0x92aa51ab [match]
[snip]
So I know what I'm looking for starts at LBA 65536/512

# dd if=/dev/sdb skip=128 count=4 2>/dev/null | hexdump -C
00000000  92 aa 51 ab 00 00 00 00  00 00 00 00 00 00 00 00  |..Q…..........|
[snip]

And as it turns out the csum is right at the beginning, 4 bytes. So use bs of 4 bytes, seek 65536/4, count of 1. This should zero just 4 bytes starting at 65536 bytes in.

# dd if=/dev/zero of=/dev/sdb bs=4 seek=16384 count=1

Checked it with the earlier skip=128 command and it looks like everything else is intact.

# btrfs-show-super -F /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum			0x00000000 [DON'T MATCH]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
[snip]
OK so the csum is bad, the magic is good. Now see if btrfs rescue super-recover does anything
# btrfs rescue super-recover /dev/sdb
Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: Y
Recovered bad superblocks successful
*** Error in `btrfs': corrupted double-linked list: 0x0000000002289e40 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7a77e)[0x7f388663977e]
/lib64/libc.so.6(+0x80b03)[0x7f388663fb03]
/lib64/libc.so.6(+0x81c88)[0x7f3886640c88]
/lib64/libc.so.6(cfree+0x4c)[0x7f38866456ec]
btrfs[0x425ec6]
btrfs[0x406902]
/lib64/libc.so.6(__libc_start_main+0xf0)[0x7f38865df0e0]
btrfs[0x406a04]
======= Memory m
[snip]

kaboom!

But was it really successful?
# btrfs-show-super -F /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum			0x92aa51ab [match]
[skip]
Looks fixed. And it mounts.

NOW, I didn't actually have my first superblock pointing to a corrupt root tree. So it's possible that while the csum was fixed in my case, that the subsequent crash has not properly copied all good parts of superblock1 to superblock0. *shrug*

And since it crashes, looks like I found a bug.

> I'm using btrfs-progs 3.16 and
> kernel 3.16.1.

So did I for all of the above.


Chris Murphy

Re: Problem with unmountable filesystem.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 7616 bytes --]

On 2014-09-16 16:57, Chris Murphy wrote:
> 
> On Sep 16, 2014, at 8:40 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
> 
>> Based on the kernel messages, the primary issue is log corruption, and
>> in theory btrfs-zero-log should fix it.
> 
> Can you provide a complete dmesg somewhere for this initial failure, just for reference? I'm curious what this indication looks like compared to other problems.
> 
Okay, I can't really get a 'complete' dmesg, because the system panics 
on the mount failure (the filesystem in question is the system's root 
filesystem), the system has no serial ports, and I didn't think to 
build in support for console on ttyUSB0.  I can however get what the 
recovery environment (locally compiled based on buildroot) shows when I 
try to mount the filesystem:
[   30.871036] BTRFS: device label gentoo devid 1 transid 160615 /dev/sda3
[   30.875225] BTRFS info (device sda3): disk space caching is enabled
[   30.917091] BTRFS: detected SSD devices, enabling SSD mode
[   30.920536] BTRFS: bad tree block start 0 130402254848
[   30.924018] BTRFS: bad tree block start 0 130402254848
[   30.926234] BTRFS: failed to read log tree
[   30.953055] BTRFS: open_ctree failed
>>  The actual issue however, is
>> that the primary superblock appears to be pointing at a corrupted root
>> tree, which causes pretty much everything that does anything other than
>> just read the sb to fail.  The first backup sb does point to a good
>> tree, but only btrfs check and btrfs restore have any option to ignore
>> the first sb and use one of the backups instead.
> 
> Maybe use wipefs -a on this volume, which removes the magic from only the first superblock by default (you can specify another location). And then try btrfs-show-super -F which "dumps" supers with bad magic.
> 
Thanks for the suggestion, I hadn't thought of that...
> I just tried this:
> # wipefs -a /dev/sdb
> /dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
> # btrfs-show-super -F /dev/sdb
> superblock: bytenr=65536, device=/dev/sdb
> ---------------------------------------------------------
> csum			0x5c1196d7 [DON'T MATCH]
> bytenr			65536
> flags			0x1
> magic			........ [DON'T MATCH]
> […]
> # btrfs-show-super -i1 /dev/sdb
> superblock: bytenr=67108864, device=/dev/sdb
> ---------------------------------------------------------
> csum			0xfc70be19 [match]
> bytenr			67108864
> flags			0x1
> magic			_BHRfS_M [match]
> 
> So the mirror is definitely there and valid.
> # btrfs rescue super-recover -yv /dev/sdb
> No valid Btrfs found on /dev/sdb
> Usage or syntax errors
> 
> Not expected at all, man page says "Recover bad superblocks from good copies." There's a good copy, it's not being found by btrfs rescue super-recover. Seems like a bug.
> 
> 
> # btrfs check /dev/sdb
> No valid Btrfs found on /dev/sdb
> Couldn't open file system
> 
> # btrfs check -s1 /dev/sdb
> using SB copy 1, bytenr 67108864
> Checking filesystem on /dev/sdb
> UUID: 9acf13de-5b98-4f28-9992-533e4a99d348
> [snip]
> OK it finds it, maybe a --repair will fix the bad first one?
> # btrfs check -s1 /dev/sdb
> using SB copy 1, bytenr 67108864
> enabling repair mode
> Checking filesystem on /dev/sdb
> UUID: 9acf13de-5b98-4f28-9992-533e4a99d348
> [snip]
> No indication of repair
> # btrfs check /dev/sdb
> No valid Btrfs found on /dev/sdb
> Couldn't open file system
> # btrfs check /dev/sdb
> No valid Btrfs found on /dev/sdb
> Couldn't open file system
> [root@f21v ~]# btrfs-show-super -F /dev/sdb
> superblock: bytenr=65536, device=/dev/sdb
> ---------------------------------------------------------
> csum			0x5c1196d7 [DON'T MATCH]
> bytenr			65536
> flags			0x1
> magic			........ [DON'T MATCH]
> 
> 
> Still not fixed. Maybe I needed to corrupt something else in the superblock other than the magic and this behavior is intentional, otherwise wipefs -a, followed by btrfsck would resurrect an intentionally wiped btrfs fs, potentially wiping out some newer file system in the process.
> 
...though maybe it's a good thing I didn't.
> 
> 
>> I'm fine using dd to replace the primary sb with one of the
>> backups, but don't know the exact parameters that would be needed.
> 
> Here's an idea:
> 
> # btrfs-show-super /dev/sdb
> superblock: bytenr=65536, device=/dev/sdb
> ---------------------------------------------------------
> csum			0x92aa51ab [match]
> [snip]
> So I know what I'm looking for starts at LBA 65536/512
> 
> # dd if=/dev/sdb skip=128 count=4 2>/dev/null | hexdump -C
> 00000000  92 aa 51 ab 00 00 00 00  00 00 00 00 00 00 00 00  |..Q…..........|
> [snip]
> 
> And as it turns out the csum is right at the beginning, 4 bytes. So use bs of 4 bytes, seek 65536/4, count of 1. This should zero just 4 bytes starting at 65536 bytes in.
> 
> # dd if=/dev/zero of=/dev/sdb bs=4 seek=16384 count=1
> 
> Checked it with the earlier skip=128 command and it looks like everything else is intact.
> 
> # btrfs-show-super -F /dev/sdb
> superblock: bytenr=65536, device=/dev/sdb
> ---------------------------------------------------------
> csum			0x00000000 [DON'T MATCH]
> bytenr			65536
> flags			0x1
> magic			_BHRfS_M [match]
> [snip]
> OK so the csum is bad, the magic is good. Now see if btrfs rescue super-recover does anything
> # btrfs rescue super-recover /dev/sdb
> Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: Y
> Recovered bad superblocks successful
> *** Error in `btrfs': corrupted double-linked list: 0x0000000002289e40 ***
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x7a77e)[0x7f388663977e]
> /lib64/libc.so.6(+0x80b03)[0x7f388663fb03]
> /lib64/libc.so.6(+0x81c88)[0x7f3886640c88]
> /lib64/libc.so.6(cfree+0x4c)[0x7f38866456ec]
> btrfs[0x425ec6]
> btrfs[0x406902]
> /lib64/libc.so.6(__libc_start_main+0xf0)[0x7f38865df0e0]
> btrfs[0x406a04]
> ======= Memory m
> [snip]
> 
> kaboom!
> 
> But was it really successful?
> # btrfs-show-super -F /dev/sdb
> superblock: bytenr=65536, device=/dev/sdb
> ---------------------------------------------------------
> csum			0x92aa51ab [match]
> [skip]
> Looks fixed. And it mounts.
> 
> NOW, I didn't actually have my first superblock pointing to a corrupt root tree. So it's possible that while the csum was fixed in my case, that the subsequent crash has not properly copied all good parts of superblock1 to superblock0. *shrug*
> 
> And since it crashes, looks like I found a bug.
> 
>> I'm using btrfs-progs 3.16 and
>> kernel 3.16.1.
> 
> So did I for all of the above.
> 
> 
Since posting this, I realized that the recovery environment I'm working from is actually btrfs-progs 3.14.1 and kernel 3.14.5, I need to make a point to update that once I get the system working again.

I've also discovered, when trying to use btrfs restore to copy out the data to a different system, that 3.14.1 restore apparently chokes on filesystem that have lzo compression turned on.  It's reporting errors trying to inflate compressed files, and I know for a fact that none of those files were even open, let alone being written to, when the system crashed.  I don't know if this is a known bug or even if it is still the case with btrfs-progs 3.16, but I figured I'd comment about it because I haven't seen anything about it anywhere.

Also, I interestingly didn't get the crash you saw above with btrfs rescue super-recover, so that might be a regression in 3.16 btrfs-progs.

Thanks for all the help.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: Problem with unmountable filesystem.

[Chris Murphy]

On Sep 17, 2014, at 5:23 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
> 
> Thanks for all the help.

Well, it's not much help. It seems possible to "corrupt" a primary superblock that points to a corrupt tree root, and use btrfs rescure super-recover to replace it, and then mount should work. One thing I didn't try was corrupting the primary superblock and just mounting normally or with recovery, to see if it'll automatically ignore the primary superblock and use the backup.

But I think you're onto something, that a good superblock can point to a corrupt tree root, and then not have a straight forward way to mount the good tree root. If I understand this correctly.


Chris Murphy

Re: Problem with unmountable filesystem.

[Duncan]

Chris Murphy posted on Wed, 17 Sep 2014 12:57:59 -0600 as excerpted:

> But I think you're onto something, that a good superblock can point to a
> corrupt tree root, and then not have a straight forward way to mount the
> good tree root. If I understand this correctly.

This is what I ran into myself a couple months ago.

I ended up doing a restore as the simplest recovery I could do, then 
recreated the affected filesystems.  It happened on two  filesystems 
(separate, I don't trust subvolumes precisely because it's the same 
filesystem underneath and that's too many eggs in one basket for my 
comfort), /var/log and /home.  I was able to restore nearly all of /home 
(enough that I wasn't aware of anything missing except symlinks that 
restore doesn't work with), but lost about half of /var/log, which I did 
first and made a couple mistakes on that I didn't repeat on /home.

I figured there was a way to fix the filesystems as they were, but these 
threads about deliberately corrupting the first superblock in ordered to 
use the others hadn't appeared yet, and between restore and backups I got 
enough back not to be worried about the rest, so restore was "good 
enough".

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Problem with unmountable filesystem.

[Duncan]

Austin S Hemmelgarn posted on Wed, 17 Sep 2014 07:23:46 -0400 as
excerpted:

> I've also discovered, when trying to use btrfs restore to copy out the
> data to a different system, that 3.14.1 restore apparently chokes on
> filesystem that have lzo compression turned on.  It's reporting errors
> trying to inflate compressed files, and I know for a fact that none of
> those files were even open, let alone being written to, when the system
> crashed.  I don't know if this is a known bug or even if it is still the
> case with btrfs-progs 3.16, but I figured I'd comment about it because I
> haven't seen anything about it anywhere.

FWIW that's a known and recently patched issue.  If you're still seeing 
issues with it with btrfs-progs 3.16, report it, but 3.14.1 almost 
certainly wouldn't have had the fix.  (This is one related patch turned 
up by a quick search; there may be others.)

* commit 93ebec96f2ae1d3276ebe89e2d6188f9b46692fb
| Author: Vincent Stehlé <vincent.stehle@laposte.net>
| Date:   Wed Jun 18 18:51:19 2014 +0200
|
|     btrfs-progs: restore: check lzo compress length
|
|     When things go wrong for lzo-compressed btrfs, feeding
|     lzo1x_decompress_safe() with corrupt data during restore
|     can lead to crashes. Reduce the risk by adding
|     a check on the input length.
|
|     Signed-off-by: Vincent Stehlé <vincent.stehle@laposte.net>
|     Signed-off-by: David Sterba <dsterba@suse.cz>
|
|  cmds-restore.c | 6 ++++++
|  1 file changed, 6 insertions(+)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Problem with unmountable filesystem.

[Austin S Hemmelgarn]

On 09/17/2014 02:57 PM, Chris Murphy wrote:
> 
> On Sep 17, 2014, at 5:23 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
>>
>> Thanks for all the help.
> 
> Well, it's not much help. It seems possible to "corrupt" a primary superblock that points to a corrupt tree root, and use btrfs rescure super-recover to replace it, and then mount should work. One thing I didn't try was corrupting the primary superblock and just mounting normally or with recovery, to see if it'll automatically ignore the primary superblock and use the backup.
> 
> But I think you're onto something, that a good superblock can point to a corrupt tree root, and then not have a straight forward way to mount the good tree root. If I understand this correctly.
> 
Corrupting the primary superblock did in fact work, and I decided to try
mounting immediately, which failed.  I didn't try with -o recovery, but
I think that would probably fail as well.  Things worked perfectly
however after using btrfs rescue super-recover.  As far as avoiding
future problems, I think the best solution would be to have the mount
operation try the tree root pointed to by the backup superblock if the
one pointed to by the primary seems corrupted.

Secondarily, this almost makes me want to set the ssd option on all
BTRFS filesystems, just to get the rotating superblock updates, because
if it weren't for that behavior, I probably wouldn't have been able to
recovery anything in this particular case.

Re: Problem with unmountable filesystem.

[Austin S Hemmelgarn]

On 09/17/2014 04:22 PM, Duncan wrote:
> Austin S Hemmelgarn posted on Wed, 17 Sep 2014 07:23:46 -0400 as
> excerpted:
> 
>> I've also discovered, when trying to use btrfs restore to copy out the
>> data to a different system, that 3.14.1 restore apparently chokes on
>> filesystem that have lzo compression turned on.  It's reporting errors
>> trying to inflate compressed files, and I know for a fact that none of
>> those files were even open, let alone being written to, when the system
>> crashed.  I don't know if this is a known bug or even if it is still the
>> case with btrfs-progs 3.16, but I figured I'd comment about it because I
>> haven't seen anything about it anywhere.
> 
> FWIW that's a known and recently patched issue.  If you're still seeing 
> issues with it with btrfs-progs 3.16, report it, but 3.14.1 almost 
> certainly wouldn't have had the fix.  (This is one related patch turned 
> up by a quick search; there may be others.)
> 
> * commit 93ebec96f2ae1d3276ebe89e2d6188f9b46692fb
> | Author: Vincent Stehlé <vincent.stehle@laposte.net>
> | Date:   Wed Jun 18 18:51:19 2014 +0200
> |
> |     btrfs-progs: restore: check lzo compress length
> |
> |     When things go wrong for lzo-compressed btrfs, feeding
> |     lzo1x_decompress_safe() with corrupt data during restore
> |     can lead to crashes. Reduce the risk by adding
> |     a check on the input length.
> |
> |     Signed-off-by: Vincent Stehlé <vincent.stehle@laposte.net>
> |     Signed-off-by: David Sterba <dsterba@suse.cz>
> |
> |  cmds-restore.c | 6 ++++++
> |  1 file changed, 6 insertions(+)
> 
Yeah, 3.16 seems fine, I just hadn't updated my recovery environment
yet.  Ironically, I did some performance testing afterwards, and
realized that using any compression was actually slowing down my system
(my disk appears to be faster than my RAM, which is really sad, even for
a laptop).

Re: Problem with unmountable filesystem.

[Chris Murphy]

On Sep 18, 2014, at 11:12 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:

> On 09/17/2014 02:57 PM, Chris Murphy wrote:
>> 
>> On Sep 17, 2014, at 5:23 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
>>> 
>>> Thanks for all the help.
>> 
>> Well, it's not much help. It seems possible to "corrupt" a primary superblock that points to a corrupt tree root, and use btrfs rescure super-recover to replace it, and then mount should work. One thing I didn't try was corrupting the primary superblock and just mounting normally or with recovery, to see if it'll automatically ignore the primary superblock and use the backup.
>> 
>> But I think you're onto something, that a good superblock can point to a corrupt tree root, and then not have a straight forward way to mount the good tree root. If I understand this correctly.
>> 
> Corrupting the primary superblock did in fact work, and I decided to try
> mounting immediately, which failed.

I just tried corrupting primary superblock by overwriting the 4 bytes csum with zeros.
Normal mount = fail
[16576.124611] BTRFS: superblock checksum mismatch
[16576.141673] BTRFS: open_ctree failed

-o recovery = fail
[16595.532801] BTRFS: superblock checksum mismatch
[16595.547868] BTRFS: open_ctree failed

btrfs check = reports no problems
btrfs check --repair = reports no problems or fixes but does fix the problem

[root@f21v ~]# btrfs check --repair /dev/sdb
enabling repair mode
Checking filesystem on /dev/sdb
UUID: 9acf13de-5b98-4f28-9992-533e4a99d348
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 510612493 bytes used err is 0
total csum bytes: 864464
total tree bytes: 1576960
total fs tree bytes: 389120
total extent tree bytes: 163840
btree space waste bytes: 196561
file data blocks allocated: 885604352
 referenced 895991808
Btrfs v3.16


>  I didn't try with -o recovery, but
> I think that would probably fail as well.  Things worked perfectly
> however after using btrfs rescue super-recover.  As far as avoiding
> future problems, I think the best solution would be to have the mount
> operation try the tree root pointed to by the backup superblock if the
> one pointed to by the primary seems corrupted.

Right. I'm not understanding how you determined a tree root is corrupt. The mount code clearly knows it can't mount a corrupt tree root, but it's neither fixing this corruption nor falling back to another superblock to see if it points to a good tree root. It's beyond my expertise what comes after detecting the problem, which the code already does. Should it fix the problem? And if not, or if it can't, should it automatically fallback to a different tree root specified by a backup superblock? And if so how far back in generation should it go before asking for user intervention? Because it's possible, on say very large file systems with ssd mount option, that there could be 3 or 4 superblocks with different tree roots pointed to with different generations. We probably don't want to always go back up to 1.5 or 2 minutes to a different tree, rather than try to fix it if possible.



> 
> Secondarily, this almost makes me want to set the ssd option on all
> BTRFS filesystems, just to get the rotating superblock updates, because
> if it weren't for that behavior, I probably wouldn't have been able to
> recovery anything in this particular case.

It's a catch 22 in that what if you get a torn write on a superblock update but it pointed to a good tree root? A mirrored superblock would allow this to be fixed pretty easily. But rotating ones means you have to go to an older tree root, unless you somehow fix up the bad superblock with the pointer to the newest good tree root - could be tedious.

One day these multiple repair methods should merge, I'd think. There are a lot of different ways to fix a Btrfs right now.


Chris Murphy

Re: Problem with unmountable filesystem.

[Duncan]

Austin S Hemmelgarn posted on Thu, 18 Sep 2014 13:12:03 -0400 as
excerpted:

> Secondarily, this almost makes me want to set the ssd option on all
> BTRFS filesystems, just to get the rotating superblock updates, because
> if it weren't for that behavior, I probably wouldn't have been able to
> recovery anything in this particular case.

That's an interesting point.  All my btrfs are on ssd ATM and I'm still 
using reiserfs on the spinning rust until btrfs further stabilizes, but 
at some point I imagine I'll switch everything to btrfs, and I hadn't 
thought about this.

Tho I could do with a good explanation of how both ssd and rotational 
superblock update handling works and the differences between the two, as 
I'm not entirely clear on it myself, at this point.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Problem with unmountable filesystem.

[Chris Murphy]

Possibly btrfs-select-super can do some of the things I was doing the hard way. It's possible to select a super to overwrite other supers, even if they're "good" ones. Whereas btrfs rescue super-recover won't do that, and neither will btrfsck, hence why I corrupted the one I didn't want first. This command isn't built by default (at least not on Fedora).

Chris

Re: Problem with unmountable filesystem.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On 2014-09-19 13:07, Chris Murphy wrote:
> Possibly btrfs-select-super can do some of the things I was doing the hard way. It's possible to select a super to overwrite other supers, even if they're "good" ones. Whereas btrfs rescue super-recover won't do that, and neither will btrfsck, hence why I corrupted the one I didn't want first. This command isn't built by default (at least not on Fedora).
I don't think it's built by default on any of the major distributions. 
On Gentoo you need to set package specific configure options.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: Problem with unmountable filesystem.

[Chris Murphy]

On Sep 17, 2014, at 5:23 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:

[   30.920536] BTRFS: bad tree block start 0 130402254848
[   30.924018] BTRFS: bad tree block start 0 130402254848
[   30.926234] BTRFS: failed to read log tree
[   30.953055] BTRFS: open_ctree failed

I'm still confused. Btrfs knows this tree root is bad, but it has backup roots. So why wasn't one of those used by -o recovery? I thought that's the whole point of that mount option. Backup tree roots are per superblock, so conceivably you'd have up to 8 of these with two superblocks, they're shown with
btrfs-show-super -af  ## and -F even if a super is bad

But skipping that, to fix this you need to know which super is pointing to the wrong tree root, since you're using ssd mount option with rotating supers. I assume mount uses the super with the highest generation number. So you'd need to:
btrfs-show-super -a
to find out the super with the most recent generation. You'd assume that one was wrong. And then use btrfs-select-super to pick the right one, and replace the wrong one. Then you could mount.

I also wonder if btrfs check -sX would show different results in your case. I'd think it would because it ought to know one of those tree roots is bad, seeing as mount knows it. And then it seems (I'm speculating a ton) that --repair might try to fix the bad tree root, and then if it fails I'd like to think it can just find the most recent good tree root, ideally one listed as a backup_tree_root by any good superblock, and then have the next mount use that.

I'm not sure why this persistently fails, and I wonder if there are cases of users giving up and blowing away file systems that could actually be mountable. But it's just really a manual process figuring out what things to do in what order to get them to mount. 


Chris Murphy

Re: Problem with unmountable filesystem.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2876 bytes --]

On 2014-09-19 13:54, Chris Murphy wrote:
>
> On Sep 17, 2014, at 5:23 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
>
> [   30.920536] BTRFS: bad tree block start 0 130402254848
> [   30.924018] BTRFS: bad tree block start 0 130402254848
> [   30.926234] BTRFS: failed to read log tree
> [   30.953055] BTRFS: open_ctree failed
>
> I'm still confused. Btrfs knows this tree root is bad, but it has backup roots. So why wasn't one of those used by -o recovery? I thought that's the whole point of that mount option. Backup tree roots are per superblock, so conceivably you'd have up to 8 of these with two superblocks, they're shown with
> btrfs-show-super -af  ## and -F even if a super is bad
>
> But skipping that, to fix this you need to know which super is pointing to the wrong tree root, since you're using ssd mount option with rotating supers. I assume mount uses the super with the highest generation number. So you'd need to:
> btrfs-show-super -a
> to find out the super with the most recent generation. You'd assume that one was wrong. And then use btrfs-select-super to pick the right one, and replace the wrong one. Then you could mount.
>
> I also wonder if btrfs check -sX would show different results in your case. I'd think it would because it ought to know one of those tree roots is bad, seeing as mount knows it. And then it seems (I'm speculating a ton) that --repair might try to fix the bad tree root, and then if it fails I'd like to think it can just find the most recent good tree root, ideally one listed as a backup_tree_root by any good superblock, and then have the next mount use that.
>
> I'm not sure why this persistently fails, and I wonder if there are cases of users giving up and blowing away file systems that could actually be mountable. But it's just really a manual process figuring out what things to do in what order to get them to mount.
>
 From what I can tell, btrfs check doesn't do anything about backup 
superblocks unless you specifically tell it to.  In this case, running 
btrfs check without specifying a superblock mirror, and with explicitly 
specifying the primary superblock produced identical results (namely it 
choked, hard, with an error message similar to that from the kernel. 
However, running it with -s1 to select the first backup superblock 
returned no errors at all other than the space_cache being invalid and 
the count of used blocks being wrong.

Based on my (limited) understanding of the mount code, it does try to 
use the superblock with the highest generation (regardless of whether we 
are on an ssd or not), but doesn't properly fall back to a secondary 
superblock after trying to mount using the primary.

As far as btrfs check repair trying to fix this, I don't think that it 
does so currently, probably for the same reason that mount fails.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]