Re: Bug when mounting XFS with external SATA drives in USB enclosures

[Eric Sandeen <sandeen@sandeen.net>]

On 11/22/19 9:21 PM, Pedro Ribeiro wrote:
> Hi,
> 
> I have been trying to find out the cause of a bug that's affecting all
> my external hard drive backups.
> 
> I have three external drives, in different USB enclosures, with the same
> configuration and the same problem.
> 
> Drive A: 2TB HDD, USB3 Seagate self enclosed drive
> Drive B: 4TB HDD, USB3 Toshiba self enclosed drive
> Drive C: 512MB SSD, Crucial MX500 with USB-C third party enclosure
> 
> All of the drives have a dm-crypt / LUKS on top, with a XFS partition
> inside. Drive A is a few months old, Drive B is about 3 years old, drive
> C about 1.5 years old. They are seldomly used (they're backup drives) so
> they are all fine mechanically.
> 
> The problem is when I attach any of the drives, enter the LUKS password
> and then try to mount, this happens:
> [   66.039772] XFS (dm-0): Mounting V5 Filesystem
> [   66.060934] XFS (dm-0): log recovery read I/O error at daddr 0x0 len
> 8 error -5
> [   66.060939] XFS (dm-0): empty log check failed
> [   66.060940] XFS (dm-0): log mount/recovery failed: error -5
> [   66.061064] XFS (dm-0): log mount failed

I assume that it used to work, right?  When did it stop working?
<reads further, sees 5.3>

> No matter what I do, using all the recovery tools, etc, it's impossible
> to mount...
> 
> The thing is that is there is NOTHING wrong with these drives. The above
> happens when running my specific, stripped and locked down kernel config.
> 
> If I take Debian's 4.19 kernel config, put it on a 5.3.11 tree, run make
> oldconfig and just answer the defaults on all prompts, all of the drives
> above mount fine:
> [   46.184068] XFS (dm-0): Mounting V5 Filesystem
> [   46.412566] XFS (dm-0): Ending clean mount

> I hit this problem recently when I moved from kernel 4.18.20, which I
> was using for a long time, to 5.3.X. In kernel 4.18.20, I did not have
> any problems with my specific stripped down config.

Could be related to memory alignment.  Commit:

commit f8f9ee479439c1be9e33c4404912a2a112c46200
Author: Dave Chinner <dchinner@redhat.com>
Date:   Mon Aug 26 12:08:39 2019 -0700

    xfs: add kmem_alloc_io()
    
    Memory we use to submit for IO needs strict alignment to the
    underlying driver contraints. Worst case, this is 512 bytes. Given
    that all allocations for IO are always a power of 2 multiple of 512
    bytes, the kernel heap provides natural alignment for objects of
    these sizes and that suffices.

went into kernel 5.4, and doesn't look like it's in the 5.3.x stable
stream.

I haven't looked very closely at your config deltas for what might change
alignment but it'd be worth giving:

f8f9ee479439 xfs: add kmem_alloc_io()
d916275aa4dd xfs: get allocation alignment from the buftarg
0ad95687c3ad xfs: add kmem allocation trace points

a try.

-Eric

> I have asked for help in IRC at #xfs, and one of the guys there (ailiop)
> was very helpful in trying to track down the problem, but we ultimately
> failed, hence why I'm asking for help here.
> 
> I'm attaching the kernel configs and the dmesg outputs. There is nothing
> obvious in the kernel config diff that should make this happen... it's a
> very weird bug.
> 
> Regards,
> Pedro
>