Re: corruption with multi-device btrfs + single bcache, won't mount

[Thiago Ramon <thiagoramon@gmail.com>]

On Sun, Feb 10, 2019 at 5:07 AM STEVE LEUNG <sjleung@shaw.ca> wrote:
>
> Hi all,
>
> I decided to try something a bit crazy, and try multi-device raid1 btrfs on
> top of dm-crypt and bcache.  That is:
>
>   btrfs -> dm-crypt -> bcache -> physical disks
>
> I have a single cache device in front of 4 disks.  Maybe this wasn't
> that good of an idea, because the filesystem went read-only a few
> days after setting it up, and now it won't mount.  I'd been running
> btrfs on top of 4 dm-crypt-ed disks for some time without any
> problems, and only added bcache (taking one device out at a time,
> converting it over, adding it back) recently.
>
> This was on Arch Linux x86-64, kernel 4.20.1.
>
> dmesg from a mount attempt (using -o usebackuproot,nospace_cache,clear_cache):
>
>   [  267.355024] BTRFS info (device dm-5): trying to use backup root at mount time
>   [  267.355027] BTRFS info (device dm-5): force clearing of disk cache
>   [  267.355030] BTRFS info (device dm-5): disabling disk space caching
>   [  267.355032] BTRFS info (device dm-5): has skinny extents
>   [  271.446808] BTRFS error (device dm-5): parent transid verify failed on 13069706166272 wanted 4196588 found 4196585
>   [  271.447485] BTRFS error (device dm-5): parent transid verify failed on 13069706166272 wanted 4196588 found 4196585
>   [  271.447491] BTRFS error (device dm-5): failed to read block groups: -5
>   [  271.455868] BTRFS error (device dm-5): open_ctree failed
>
> btrfs check:
>
>   parent transid verify failed on 13069706166272 wanted 4196588 found 4196585
>   parent transid verify failed on 13069706166272 wanted 4196588 found 4196585
>   parent transid verify failed on 13069706166272 wanted 4196588 found 4196585
>   parent transid verify failed on 13069706166272 wanted 4196588 found 4196585
>   Ignoring transid failure
>   ERROR: child eb corrupted: parent bytenr=13069708722176 item=7 parent level=2 child level=0
>   ERROR: cannot open file system
>
> Any simple fix for the filesystem?  It'd be nice to recover the data
> that's hopefully still intact.  I have some backups that I can dust
> off if it really comes down to it, but it's more convenient to
> recover the data in-place.
>
> This is complete speculation, but I do wonder if having the single
> cache device for multiple btrfs disks triggered the problem.
No, having a single cache device with multiple backing devices is the
most common way to use bcache. I've used a setup similar to yours for
a couple of years without problems (until it broke down recently due
to other issues).

Your current filesystem is probably too damaged to properly repair
right now (some other people here might be able to help with that),
but you probably haven't lost much of what's in there. You can dump
the files out with "btrfs restore", or you can use a patch to allow
you to mount the damaged filesystem read-only
(https://patchwork.kernel.org/patch/10738583/).

But before you try to restore anything, can you go back in your kernel
logs and check for errors? Either one of your devices is failing, you
might have physical link issues or bad memory. Even with a complex
setup like this you shouldn't be getting random corruption like this.

>
> Thanks for any assistance.
>
> Steve