Re: xfs_repair: superblock read failed, fatal error -- Input/output error

From: Carlos Maiolino <cmaiolino@redhat.com>
To: Daniel Storey <daniel.storey202@gmail.com>
Cc: "linux-xfs@vger.kernel.org" <linux-xfs@vger.kernel.org>
Subject: Re: xfs_repair: superblock read failed, fatal error -- Input/output error
Date: Fri, 3 Jan 2020 08:04:38 +0100	[thread overview]
Message-ID: <20200103070438.wnyo3a6ubdccptz7@orion.redhat.com> (raw)
In-Reply-To: <MN2PR07MB59347B8FC6B9E92D0107D9A1A4200@MN2PR07MB5934.namprd07.prod.outlook.com>

Hi Daniel.

>     Also on a vmware machine? On the same hypervisor? For sure not on the same host,
>     since UFS explorer (AFAIK) does not have a Linux version.
> 
> It does, actually (have a Linux version).  I'm running it on the same host.

Oh, I didn't know that, thanks for the information.

>     
>     And btw, UFS Explorer is built so that you can scan/recover data on very damaged
>     filesystems and disks, while filesystems won't let you mount a corrupted
>     filesystem to avoid doing even more damage. So, yeah, you might still see
>     filesystem data/metadata using UFS explorer with damaged filesystems or block
>     devices.
>     
> Okay.

Just adding to the information above, UFS (and basically most of the disaster
recovery tools), will ignore IO errors as much as it can, as an attempt to read
as much data as possible from the failing devices.

>     
>     So, again, I'd try to open these devices on a bare-metal machine and check the
>     device for errors. If the errors are still present, replace the devices.
>     
> Ok - I'll try opening these devices on a bare-metal (not a VMware host) and check them for errors. What do I do if there are no errors present?  As the SMART check revealed no problems with the disks.
> 

I did another look into the dmesg output you provided, and:

>     > [52819.637179] Buffer I/O error on dev dm-4, logical block 1610612731, async page read

This is an I/O error even below the vdo driver, so as much as XFS is only the
victim here, I am inclined to say again VDO is also one more victim here of a
failing device, even though I don't know much details about VDO driver.

So you really need to check your storage stack to know where the errors might
be.

The HDD itself, the storage enclosure, usb cable, VMWare hypervisor, etc. I
really can't say, I couldn't spot any errors pointing to anything specific other
than generic I/O errors, which essentially means kernel failed to issue I/O
commands to your device. It will require some investigation to determine where
the error lies, that's why I suggested plugging the usb HDD into a bare-metal
machine, so you can start by better isolating the problem.

But XFS there is just the messenger there of some problem with your device or
storage stack.

Cheers.

-- 
Carlos