Re: corrupted root, doesnt check, repair or mount

From: Chris Murphy <lists@colorremedies.com>
To: Daniel Brunner <daniel@brunner.ninja>
Cc: Chris Murphy <lists@colorremedies.com>,
	Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: Re: corrupted root, doesnt check, repair or mount
Date: Tue, 8 Dec 2020 22:52:22 -0700	[thread overview]
Message-ID: <CAJCQCtSTgxdjwXs+hKbka1s8YUJxuGxqHVTdCkV0bAasp8Zz4g@mail.gmail.com> (raw)
In-Reply-To: <CAD7Y51hdeJVNBAYQXxvf=kUOKh_KYX286Hzoy-qJ8izn+crVtQ@mail.gmail.com>

On Tue, Dec 8, 2020 at 5:30 PM Daniel Brunner <daniel@brunner.ninja> wrote:
>
> Hi again,
>
> do the outputs of said commands help in finding the issue?

Not really.

The btrfs super blocks are valid and are the same. So at least those
three locations aren't affected by this problem.

btrfs superblock reports:
total_bytes
29202801745920 is 27197.2 GiB
bytes_used
17674898116608 us 16461.0 GiB

mdadm superblock reports:
  Array Size : 39065219072 (37255.50 GiB 40002.78 GB)  this is
40002784329728 bytes
Used Dev Size : 9766304768 (9313.87 GiB 10000.70 GB)

 # blockdev --getsize64 /dev/mapper/bcache0-open
40002767544320 is 37255.4804611206055

Array size minus bcache size
40002784329728 - 40002767544320 = 16785408 is ~16.0078125 MiB

Is it normal for the mdadm array to be bigger than the bcache device?
I have no idea. I don't know enough about bcache operation.

But the btrfs file system is at least smaller than both the bcache
device and array. So maybe it's safe? Ideally they are the same size
but smaller Btrfs than array size isn't bad in that it won't break it.

However, I don't know why Btrfs report ~16T used, and yet mdadm thinks
~9T is used. That seems like a big deal.

Based on the description, this isn't a Btrfs problem. The file system
shrink happened first? And it succeeded? And then you used mdadm to
shrink the array and now btrfs won't mount anymore. So it seems to me
the problem is either with the mdadm array or bcache, and I don't know
enough about either one of them to help figure out what is wrong. And
you'd have to know what's wrong first, to know how to fix it.

You could ask on both linux-raid@ and bcache lists.

>
> This corrupted fs blocks my homeserver usage and if the data is not
> recoverable, i would start all over again.
> I do not have enough disks to backup the current corrupted state.
>
> Losing the data would mean about 10 years of hamstering/collecting
> stuff would be lost (only some parts are backed up externally),
> so if there is any chance of recovering I would keep the machine
> offline until any new ideas pop up.

The data is there. The questions remain: what exactly did happen? and
what exactly should have happened? If you can infer the first
question, you can reverse it and get back to the data. But I don't
understand the relationship between bcache and mdadm, it's a bit
complicated and that makes recovery more difficult.

-- 
Chris Murphy