Re: "bad tree block start" when trying to mount on ARM

[Erik Jensen <erikjensen@rkjnsn.net>]

On Tue, May 21, 2019 at 2:18 AM Hugo Mills <hugo@carfax.org.uk> wrote:
>
> On Tue, May 21, 2019 at 01:34:42AM -0700, Erik Jensen wrote:
> > I have a 5-drive btrfs filesystem. (raid-5 data, dup metadata). I can
> > mount it fine on my x86_64 system, and running `btrfs check` there
> > reveals no errors. However, I am not able to mount the filesystem on
> > my 32-bit ARM board, which I am hoping to use for lower-power file
> > serving. dmesg shows the following:
> >
> > [   83.066301] BTRFS info (device dm-3): disk space caching is enabled
> > [   83.072817] BTRFS info (device dm-3): has skinny extents
> > [   83.553973] BTRFS error (device dm-3): bad tree block start, want
> > 17628726968320 have 396461950000496896
> > [   83.554089] BTRFS error (device dm-3): bad tree block start, want
> > 17628727001088 have 5606876608493751477
> > [   83.601176] BTRFS error (device dm-3): bad tree block start, want
> > 17628727001088 have 5606876608493751477
> > [   83.610811] BTRFS error (device dm-3): failed to verify dev extents
> > against chunks: -5
> > [   83.639058] BTRFS error (device dm-3): open_ctree failed
> >
> > Is this expected to work? I did notice that there are gotchas on the
> > wiki related to filesystems over 8TiB on 32-bit systems, but it
> > sounded like they were mostly related to running the tools, as opposed
> > to the filesystem driver itself. (Each of the five drives is
> > 8TB/7.28TiB)
>
>    Yes, it should work. We had problems with ARM several years ago,
> because of its unusual behaviour with unaligned word accesses, but
> those were in userspace, and, as far as I know, fixed now. Looking at
> the want/have numbers, it doesn't look like an endianness problem or
> an ARM-unaligned-access problem.
>
> > If this isn't expected, what should I do to help track down the issue?
>
>    Can you show us the output of "btrfs check --readonly", on both the
> x86_64 machine and the ARM machine? It might give some more insight
> into the nature of the breakage.

On x86_64:
Opening filesystem to check...
Checking filesystem on /dev/mapper/storage1
UUID: aafd9149-9cfe-4970-ae21-f1065c36ed63
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 17647861833728 bytes used, no error found
total csum bytes: 17211131512
total tree bytes: 19333480448
total fs tree bytes: 202801152
total extent tree bytes: 183894016
btree space waste bytes: 1474174626
file data blocks allocated: 17628822319104
 referenced 17625817141248

On ARM:
Opening filesystem to check...
Checking filesystem on /dev/mapper/storage1
UUID: aafd9149-9cfe-4970-ae21-f1065c36ed63
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 17647861833728 bytes used, no error found
total csum bytes: 17211131512
total tree bytes: 19333480448
total fs tree bytes: 202801152
total extent tree bytes: 183894016
btree space waste bytes: 1474174626
file data blocks allocated: 17628822319104
 referenced 17625817141248

>    Possibly also "btrfs inspect dump-super" on both machines.

On x86_64:
superblock: bytenr=65536, device=/dev/dm-5
---------------------------------------------------------
csum_type        0 (crc32c)
csum_size        4
csum            0x737fcf72 [match]
bytenr            65536
flags            0x1
            ( WRITTEN )
magic            _BHRfS_M [match]
fsid            aafd9149-9cfe-4970-ae21-f1065c36ed63
label            Storage
generation        97532
root            30687232
sys_array_size        129
chunk_root_generation    97526
root_level        1
chunk_root        20971520
chunk_root_level    1
log_root        0
log_root_transid    0
log_root_level        0
total_bytes        40007732224000
bytes_used        17647861833728
sectorsize        4096
nodesize        16384
leafsize (deprecated)        16384
stripesize        4096
root_dir        6
num_devices        5
compat_flags        0x0
compat_ro_flags        0x0
incompat_flags        0x1e1
            ( MIXED_BACKREF |
              BIG_METADATA |
              EXTENDED_IREF |
              RAID56 |
              SKINNY_METADATA )
cache_generation    97532
uuid_tree_generation    97532
dev_item.uuid        39a9463d-65f5-499b-bca8-dae6b52eb729
dev_item.fsid        aafd9149-9cfe-4970-ae21-f1065c36ed63 [match]
dev_item.type        0
dev_item.total_bytes    8001546444800
dev_item.bytes_used    4436709605376
dev_item.io_align    4096
dev_item.io_width    4096
dev_item.sector_size    4096
dev_item.devid        5
dev_item.dev_group    0
dev_item.seek_speed    0
dev_item.bandwidth    0
dev_item.generation    0

On ARM:
superblock: bytenr=65536, device=/dev/dm-2
---------------------------------------------------------
csum_type        0 (crc32c)
csum_size        4
csum            0x737fcf72 [match]
bytenr            65536
flags            0x1
            ( WRITTEN )
magic            _BHRfS_M [match]
fsid            aafd9149-9cfe-4970-ae21-f1065c36ed63
metadata_uuid        aafd9149-9cfe-4970-ae21-f1065c36ed63
label            Storage
generation        97532
root            30687232
sys_array_size        129
chunk_root_generation    97526
root_level        1
chunk_root        20971520
chunk_root_level    1
log_root        0
log_root_transid    0
log_root_level        0
total_bytes        40007732224000
bytes_used        17647861833728
sectorsize        4096
nodesize        16384
leafsize (deprecated)    16384
stripesize        4096
root_dir        6
num_devices        5
compat_flags        0x0
compat_ro_flags        0x0
incompat_flags        0x1e1
            ( MIXED_BACKREF |
              BIG_METADATA |
              EXTENDED_IREF |
              RAID56 |
              SKINNY_METADATA )
cache_generation    97532
uuid_tree_generation    97532
dev_item.uuid        39a9463d-65f5-499b-bca8-dae6b52eb729
dev_item.fsid        aafd9149-9cfe-4970-ae21-f1065c36ed63 [match]
dev_item.type        0
dev_item.total_bytes    8001546444800
dev_item.bytes_used    4436709605376
dev_item.io_align    4096
dev_item.io_width    4096
dev_item.sector_size    4096
dev_item.devid        5
dev_item.dev_group    0
dev_item.seek_speed    0
dev_item.bandwidth    0
dev_item.generation    0

Only difference appears to the extra metadata_uuid line on ARM. I
assume that's because the ARM system is running btrfs-progs v4.20.2 vs
v4.19 on the x86_64 system.

> > Also potentially relevant: The x86_64 system is currently running
> > 4.19.27, while the ARM system is running 5.1.3.
>
>    Shouldn't make a difference.
>
> > Finally, just in case it's relevant, I just finished reencrypting the
> > array, which involved doing a `btrfs replace` on each device in the
> > array.
>
>    If you can still mount on x86_64, then the FS is at least
> reasonably complete and undamaged. I don't think this will make a
> difference.  However, it's worth checking whether there are any
> funnies about your encryption layer on ARM (I wouldn't expect any,
> since it's recognising the decrypted device as btrfs, rather than
> random crud).

I took the sha256 hash of the first GiB of plaintext on each drive,
and got the same result on both systems, so I think things should be
okay, there.

>    Hugo.
>
> --
> Hugo Mills             | Prisoner unknown: Return to Zenda.
> hugo@... carfax.org.uk |
> http://carfax.org.uk/  |
> PGP: E2AB1DE4          |