Re: Mount issue, mount /dev/sdc2: can't read superblock

From: Chris Murphy <lists@colorremedies.com>
To: Peter Chant <pete@petezilla.co.uk>, Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: Re: Mount issue, mount /dev/sdc2: can't read superblock
Date: Sun, 23 Dec 2018 17:58:08 -0700	[thread overview]
Message-ID: <CAJCQCtRv2NOabzMB0A9A7hoZcu_-6Fy2+umc8+qYcWnv=wn7bQ@mail.gmail.com> (raw)
In-Reply-To: <3b4d0ed3-4151-50b9-b1da-6be240bb58b3@petezilla.co.uk>

On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk> wrote:

> btrfs rescue super -v /dev/sdb2
...
> All supers are valid, no need to recover
>
>
> btrfs insp dump-s -f <dev>
...
> generation              7937947
...
>         backup 0:
>                 backup_tree_root:       1113909100544   gen: 7937935    level: 1
...
>         backup 1:
>                 backup_tree_root:       1113907347456   gen: 7937936    level: 1
...
>         backup 2:
>                 backup_tree_root:       1113911951360   gen: 7937937    level: 1
...
>         backup 3:
>                 backup_tree_root:       1113907494912   gen: 7937934    level: 1
...

The kernel wrote out three valid checksummed supers, with what seems
to be a rather significant sanity violation. The super generation and
tree root address do not match any of the backup tree roots. The
*current* tree root is supposed to be in one of the backups as well.

Qu, any idea how this is even theoretically possible? Bit flip right
before the super is computed and checksummed? Seems like some kind of
corruption before checksum is computed.

> I'm getting suspicious of the drive as when I was trying the various
> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
> I also have a separate basic install on ext4 on the same disk.  Though
> e2fsck shows no errors and mounts fine I cannot log into that install.
> Maybe a coincidence, but too many bad things thrown up make me
> suspicious.  Whatever is happening this seems to be really fighting me.

I'm not sure how even a bad device accounts for the super generation
and backup mismatches. That's damn strange.

If you get bored with the back and forth and just want to give up,
that's fine. I suggest that if you have the time and space, to take a
btrfs-image in case Qu or some other developer wants to look at this
file system at some point. The btrfs-image is a read only process, can
be set to scrub filenames, and only contains metadata. Size of the
resulting file is around 1/2 of the size of metadata, when doing
'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
much free space to direct the command to.

btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile

It might fail, if so you can try adding -w and see if that helps.

There is no log listed in the super so zero-log isn't indicated, and
also tells me there were no fsync's still flushing at the time of the
crash. The loss should be at most a minute of data, not an
inconsistent file system that can't be mounted anymore. Pretty weird.

What were your mount options? Defaults? Anything custom like discard,
commit=, notreelog? Any non-default mount options themselves would not
be the cause of the problem, but might suggest partial ideas for what
might have happened.

-- 
Chris Murphy