From: Erik Jensen <erikjensen@rkjnsn.net>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: Hugo Mills <hugo@carfax.org.uk>,
linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: "bad tree block start" when trying to mount on ARM
Date: Mon, 18 Jan 2021 04:12:31 -0800 [thread overview]
Message-ID: <CAMj6ewNDQFzXsvF5c1=raJc11iMvMKcHH=AbkUkrNeV2e3XGVg@mail.gmail.com> (raw)
In-Reply-To: <3b2fe3d7-1919-d236-e6bb-483593287cc5@gmx.com>
The offending system is indeed ARMv7 (specifically a Marvell ARMADA®
388), but I believe the Broadcom BCM2835 in my Raspberry Pi is
actually ARMv6 (with hardware float support).
On Mon, Jan 18, 2021 at 4:01 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/1/18 下午7:55, Erik Jensen wrote:
> > On Mon, Jan 18, 2021 at 3:07 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >> On 2021/1/18 下午6:33, Erik Jensen wrote:
> >>> I ended up having other priorities occupying my time since 2019, and the
> >>> "solution" of exporting the individual drives on my NAS using NBD and
> >>> mounting them on my desktop worked, even if it wasn't pretty.
> >>>
> >>> However, I am currently looking into Syncthing, which I would like to
> >>> run on the NAS directly. That would, of course, require accessing the
> >>> filesystem directly on the NAS rather than just exporting the raw
> >>> devices, which means circling back to this issue.
> >>>
> >>> After updating my NAS, I have determined that the issue still occurs
> >>> with Linux 5.8.
> >>>
> >>> What's the next best step for debugging the issue? Ideally, I'd like to
> >>> help track down the issue to find a proper fix, rather than just trying
> >>> to bypass the issue. I wasn't sure if the suggestion to comment out
> >>> btrfs_verify_dev_extents() was more geared toward the former or the latter.
> >>
> >> After rewinding my memory on this case, the problem is really that the
> >> ARM btrfs kernel is reading garbage, while X86 or ARM user space tool
> >> works as expected.
> >>
> >> Can you recompile your kernel on the ARM board to add extra debugging
> >> messages?
> >> If possible, we can try to add some extra debug points to bombarding
> >> your dmesg.
> >>
> >> Or do you have other ARM boards to test the same fs?
> >>
> >>
> >> Thanks,
> >> Qu
> >
> > It's pretty easy to build a kernel with custom patches applied, though
> > the actual building takes a while, so I'd be happy to add whatever
> > debug messages would be useful. I also have an old Raspberry Pi
> > (original model B) I can dig out and try to get going, tomorrow. I
> > can't hook it up to the drives directly, but I should be able to
> > access them via NBD like I was doing from my desktop.
>
> RPI 1B would be a little slow but should be enough to expose the
> problem, if the problem is for all arm builds (as long as you're also
> using armv7 for the offending system).
>
> Thanks,
> Qu
>
> > If I can't get
> > that going for whatever reason, I could also try running an emulated
> > ARM system with QEMU.
> >
> >>>
> >>> On Fri, Jun 28, 2019 at 1:15 AM Qu Wenruo <quwenruo.btrfs@gmx.com
> >>> <mailto:quwenruo.btrfs@gmx.com>> wrote:
> >>>
> >>>
> >>>
> >>> On 2019/6/28 下午4:00, Erik Jensen wrote:
> >>> >> So it's either the block layer reading some wrong from the disk
> >>> or btrfs
> >>> >> layer doesn't do correct endian convert.
> >>> >
> >>> > My ARM board is running in little endian mode, so it doesn't seem
> >>> like
> >>> > endianness should be an issue. (It is 32-bits versus my desktop's 64,
> >>> > though.) I've also tried exporting the drives via NBD to my x86_64
> >>> > system, and that worked fine, so if the problem is under btrfs, it
> >>> > would have to be in the encryption layer, but fsck succeeding on the
> >>> > ARM board would seem to rule that out, as well.
> >>> >
> >>> >> Would you dump the following data (X86 and ARM should output the
> >>> same
> >>> >> content, thus one output is enough).
> >>> >> # btrfs ins dump-tree -b 17628726968320 /dev/dm-3
> >>> >> # btrfs ins dump-tree -b 17628727001088 /dev/dm-3
> >>> >
> >>> > Attached, and also 17628705964032, since that's the block
> >>> mentioned in
> >>> > my most recent mount attempt (see below).
> >>>
> >>> The trees are completely fine.
> >>>
> >>> So it should be something else causing the problem.
> >>>
> >>> >
> >>> >> And then, for the ARM system, please apply the following diff,
> >>> and try
> >>> >> mount again.
> >>> >> The diff adds extra debug info, to exam the vital members of a
> >>> tree block.
> >>> >>
> >>> >> Correct fs should output something like:
> >>> >> BTRFS error (device dm-4): bad tree block start, want 30408704
> >>> have 0
> >>> >> tree block gen=4 owner=5 nritems=2 level=0
> >>> >> csum:
> >>> >>
> >>> a304e483-0000-0000-0000-00000000000000000000-0000-0000-0000-000000000000
> >>> >>
> >>> >> The csum one is the most important one, if there aren't so many
> >>> zeros,
> >>> >> it means at that timing, btrfs just got a bunch of garbage, thus we
> >>> >> could do further debug.
> >>> >
> >>> > [ 131.725573] BTRFS info (device dm-1): disk space caching is
> >>> enabled
> >>> > [ 131.731884] BTRFS info (device dm-1): has skinny extents
> >>> > [ 133.046145] BTRFS error (device dm-1): bad tree block start, want
> >>> > 17628705964032 have 2807793151171243621
> >>> > [ 133.055775] tree block gen=7888986126946982446
> >>> > owner=11331573954727661546 nritems=4191910623 level=112
> >>> > [ 133.065661] csum:
> >>> >
> >>> 416a456c-1e68-dbc3-185d-aaad410beaef5493ab3f-3cb9-4ba1-2214-b41cba9656fc
> >>>
> >>> Completely garbage here, so I'd say the data we got isn't what we want.
> >>>
> >>> > [ 133.108383] BTRFS error (device dm-1): bad tree block start, want
> >>> > 17628705964032 have 2807793151171243621
> >>> > [ 133.117999] tree block gen=7888986126946982446
> >>> > owner=11331573954727661546 nritems=4191910623 level=112
> >>> > [ 133.127756] csum:
> >>> >
> >>> 416a456c-1e68-dbc3-185d-aaad410beaef5493ab3f-3cb9-4ba1-2214-b41cba9656fc
> >>>
> >>> But strangely, the 2nd try still gives us the same result, if it's
> >>> really some garbage, we should get some different result.
> >>>
> >>> > [ 133.136241] BTRFS error (device dm-1): failed to verify dev
> >>> extents
> >>> > against chunks: -5
> >>>
> >>> You can try to skip the dev extents verification by commenting out the
> >>> btrfs_verify_dev_extents() call in disk-io.c::open_ctree().
> >>>
> >>> It may fail at another location though.
> >>>
> >>> The more strange part is, we have the device tree root node read out
> >>> without problem.
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>> > [ 133.166165] BTRFS error (device dm-1): open_ctree failed
> >>> >
> >>> > I copied some files over last time I had it mounted on my desktop,
> >>> > which may be why it's now failing at a different block.
> >>> >
> >>> > Thanks!
> >>> >
> >>>
next prev parent reply other threads:[~2021-01-18 12:14 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-21 8:34 "bad tree block start" when trying to mount on ARM Erik Jensen
2019-05-21 8:56 ` Patrik Lundquist
2019-05-21 9:01 ` Erik Jensen
2019-05-21 9:18 ` Hugo Mills
2019-05-22 16:02 ` Erik Jensen
2019-06-26 7:04 ` Erik Jensen
2019-06-26 8:10 ` Qu Wenruo
[not found] ` <CAMj6ewO229vq6=s+T7GhUegwDADv4dzhqPiM0jo10QiKujvytA@mail.gmail.com>
2019-06-28 8:15 ` Qu Wenruo
2021-01-18 10:50 ` Erik Jensen
[not found] ` <CAMj6ewMqXLtrBQgTJuz04v3MBZ0W95fU4pT0jP6kFhuP830TuA@mail.gmail.com>
2021-01-18 11:07 ` Qu Wenruo
2021-01-18 11:55 ` Erik Jensen
2021-01-18 12:01 ` Qu Wenruo
2021-01-18 12:12 ` Erik Jensen [this message]
2021-01-19 5:22 ` Erik Jensen
2021-01-19 9:28 ` Erik Jensen
2021-01-20 8:21 ` Qu Wenruo
2021-01-20 8:30 ` Qu Wenruo
[not found] ` <CAMj6ewOqCJTGjykDijun9_LWYELA=92HrE+KjGo-ehJTutR_+w@mail.gmail.com>
2021-01-26 4:54 ` Erik Jensen
2021-01-29 6:39 ` Erik Jensen
2021-02-01 2:35 ` Qu Wenruo
2021-02-01 5:49 ` Su Yue
2021-02-04 6:16 ` Erik Jensen
2021-02-06 1:57 ` Erik Jensen
2021-02-10 5:47 ` Qu Wenruo
2021-02-10 22:17 ` Erik Jensen
2021-02-10 23:47 ` Qu Wenruo
2021-02-18 1:24 ` Qu Wenruo
2021-02-18 4:03 ` Erik Jensen
2021-02-18 5:24 ` Qu Wenruo
2021-02-18 5:49 ` Erik Jensen
2021-02-18 6:09 ` Qu Wenruo
2021-02-18 6:59 ` Erik Jensen
2021-02-18 7:24 ` Qu Wenruo
2021-02-18 7:59 ` Erik Jensen
2021-02-18 8:38 ` Qu Wenruo
2021-02-18 8:52 ` Erik Jensen
2021-02-18 8:59 ` Qu Wenruo
2021-02-20 2:47 ` Erik Jensen
2021-02-20 3:16 ` Qu Wenruo
2021-02-20 4:28 ` Erik Jensen
2021-02-20 6:01 ` Qu Wenruo
2021-02-21 5:36 ` Erik Jensen
2021-02-18 7:25 ` Erik Jensen
2019-05-21 10:17 ` Qu Wenruo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAMj6ewNDQFzXsvF5c1=raJc11iMvMKcHH=AbkUkrNeV2e3XGVg@mail.gmail.com' \
--to=erikjensen@rkjnsn.net \
--cc=hugo@carfax.org.uk \
--cc=linux-btrfs@vger.kernel.org \
--cc=quwenruo.btrfs@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).