Re: "bad tree block start" when trying to mount on ARM

[Erik Jensen <erikjensen@rkjnsn.net>]

I'm still seeing this. Anything else I can try?

On Wed, May 22, 2019 at 9:02 AM Erik Jensen <erikjensen@rkjnsn.net> wrote:
>
> On Tue, May 21, 2019 at 2:18 AM Hugo Mills <hugo@carfax.org.uk> wrote:
> >
> > On Tue, May 21, 2019 at 01:34:42AM -0700, Erik Jensen wrote:
> > > I have a 5-drive btrfs filesystem. (raid-5 data, dup metadata). I can
> > > mount it fine on my x86_64 system, and running `btrfs check` there
> > > reveals no errors. However, I am not able to mount the filesystem on
> > > my 32-bit ARM board, which I am hoping to use for lower-power file
> > > serving. dmesg shows the following:
> > >
> > > [   83.066301] BTRFS info (device dm-3): disk space caching is enabled
> > > [   83.072817] BTRFS info (device dm-3): has skinny extents
> > > [   83.553973] BTRFS error (device dm-3): bad tree block start, want
> > > 17628726968320 have 396461950000496896
> > > [   83.554089] BTRFS error (device dm-3): bad tree block start, want
> > > 17628727001088 have 5606876608493751477
> > > [   83.601176] BTRFS error (device dm-3): bad tree block start, want
> > > 17628727001088 have 5606876608493751477
> > > [   83.610811] BTRFS error (device dm-3): failed to verify dev extents
> > > against chunks: -5
> > > [   83.639058] BTRFS error (device dm-3): open_ctree failed
> > >
> > > Is this expected to work? I did notice that there are gotchas on the
> > > wiki related to filesystems over 8TiB on 32-bit systems, but it
> > > sounded like they were mostly related to running the tools, as opposed
> > > to the filesystem driver itself. (Each of the five drives is
> > > 8TB/7.28TiB)
> >
> >    Yes, it should work. We had problems with ARM several years ago,
> > because of its unusual behaviour with unaligned word accesses, but
> > those were in userspace, and, as far as I know, fixed now. Looking at
> > the want/have numbers, it doesn't look like an endianness problem or
> > an ARM-unaligned-access problem.
> >
> > > If this isn't expected, what should I do to help track down the issue?
> >
> >    Can you show us the output of "btrfs check --readonly", on both the
> > x86_64 machine and the ARM machine? It might give some more insight
> > into the nature of the breakage.
>
> On x86_64:
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/storage1
> UUID: aafd9149-9cfe-4970-ae21-f1065c36ed63
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 17647861833728 bytes used, no error found
> total csum bytes: 17211131512
> total tree bytes: 19333480448
> total fs tree bytes: 202801152
> total extent tree bytes: 183894016
> btree space waste bytes: 1474174626
> file data blocks allocated: 17628822319104
>  referenced 17625817141248
>
> On ARM:
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/storage1
> UUID: aafd9149-9cfe-4970-ae21-f1065c36ed63
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 17647861833728 bytes used, no error found
> total csum bytes: 17211131512
> total tree bytes: 19333480448
> total fs tree bytes: 202801152
> total extent tree bytes: 183894016
> btree space waste bytes: 1474174626
> file data blocks allocated: 17628822319104
>  referenced 17625817141248
>
> >    Possibly also "btrfs inspect dump-super" on both machines.
>
> On x86_64:
> superblock: bytenr=65536, device=/dev/dm-5
> ---------------------------------------------------------
> csum_type        0 (crc32c)
> csum_size        4
> csum            0x737fcf72 [match]
> bytenr            65536
> flags            0x1
>             ( WRITTEN )
> magic            _BHRfS_M [match]
> fsid            aafd9149-9cfe-4970-ae21-f1065c36ed63
> label            Storage
> generation        97532
> root            30687232
> sys_array_size        129
> chunk_root_generation    97526
> root_level        1
> chunk_root        20971520
> chunk_root_level    1
> log_root        0
> log_root_transid    0
> log_root_level        0
> total_bytes        40007732224000
> bytes_used        17647861833728
> sectorsize        4096
> nodesize        16384
> leafsize (deprecated)        16384
> stripesize        4096
> root_dir        6
> num_devices        5
> compat_flags        0x0
> compat_ro_flags        0x0
> incompat_flags        0x1e1
>             ( MIXED_BACKREF |
>               BIG_METADATA |
>               EXTENDED_IREF |
>               RAID56 |
>               SKINNY_METADATA )
> cache_generation    97532
> uuid_tree_generation    97532
> dev_item.uuid        39a9463d-65f5-499b-bca8-dae6b52eb729
> dev_item.fsid        aafd9149-9cfe-4970-ae21-f1065c36ed63 [match]
> dev_item.type        0
> dev_item.total_bytes    8001546444800
> dev_item.bytes_used    4436709605376
> dev_item.io_align    4096
> dev_item.io_width    4096
> dev_item.sector_size    4096
> dev_item.devid        5
> dev_item.dev_group    0
> dev_item.seek_speed    0
> dev_item.bandwidth    0
> dev_item.generation    0
>
> On ARM:
> superblock: bytenr=65536, device=/dev/dm-2
> ---------------------------------------------------------
> csum_type        0 (crc32c)
> csum_size        4
> csum            0x737fcf72 [match]
> bytenr            65536
> flags            0x1
>             ( WRITTEN )
> magic            _BHRfS_M [match]
> fsid            aafd9149-9cfe-4970-ae21-f1065c36ed63
> metadata_uuid        aafd9149-9cfe-4970-ae21-f1065c36ed63
> label            Storage
> generation        97532
> root            30687232
> sys_array_size        129
> chunk_root_generation    97526
> root_level        1
> chunk_root        20971520
> chunk_root_level    1
> log_root        0
> log_root_transid    0
> log_root_level        0
> total_bytes        40007732224000
> bytes_used        17647861833728
> sectorsize        4096
> nodesize        16384
> leafsize (deprecated)    16384
> stripesize        4096
> root_dir        6
> num_devices        5
> compat_flags        0x0
> compat_ro_flags        0x0
> incompat_flags        0x1e1
>             ( MIXED_BACKREF |
>               BIG_METADATA |
>               EXTENDED_IREF |
>               RAID56 |
>               SKINNY_METADATA )
> cache_generation    97532
> uuid_tree_generation    97532
> dev_item.uuid        39a9463d-65f5-499b-bca8-dae6b52eb729
> dev_item.fsid        aafd9149-9cfe-4970-ae21-f1065c36ed63 [match]
> dev_item.type        0
> dev_item.total_bytes    8001546444800
> dev_item.bytes_used    4436709605376
> dev_item.io_align    4096
> dev_item.io_width    4096
> dev_item.sector_size    4096
> dev_item.devid        5
> dev_item.dev_group    0
> dev_item.seek_speed    0
> dev_item.bandwidth    0
> dev_item.generation    0
>
> Only difference appears to the extra metadata_uuid line on ARM. I
> assume that's because the ARM system is running btrfs-progs v4.20.2 vs
> v4.19 on the x86_64 system.
>
> > > Also potentially relevant: The x86_64 system is currently running
> > > 4.19.27, while the ARM system is running 5.1.3.
> >
> >    Shouldn't make a difference.
> >
> > > Finally, just in case it's relevant, I just finished reencrypting the
> > > array, which involved doing a `btrfs replace` on each device in the
> > > array.
> >
> >    If you can still mount on x86_64, then the FS is at least
> > reasonably complete and undamaged. I don't think this will make a
> > difference.  However, it's worth checking whether there are any
> > funnies about your encryption layer on ARM (I wouldn't expect any,
> > since it's recognising the decrypted device as btrfs, rather than
> > random crud).
>
> I took the sha256 hash of the first GiB of plaintext on each drive,
> and got the same result on both systems, so I think things should be
> okay, there.
>
> >    Hugo.
> >
> > --
> > Hugo Mills             | Prisoner unknown: Return to Zenda.
> > hugo@... carfax.org.uk |
> > http://carfax.org.uk/  |
> > PGP: E2AB1DE4          |