I got file system corruption with XFS

I got file system corruption with XFS

[Luciano ES]

I needed to restore something from my backups, an external hard disk 
that is kept separately, always disconnected until I really need it. 

But the file system refused to be mounted: "structure needs cleaning," 
it said.

I googled and didn't find much hope about it. I followed what little 
advice I found: I ran xfs_repair and it didn't work. So I ran it 
again with -L and it worked, but the software itself warned me that 
some files could not be recovered. I'll never know which ones.

I always liked XFS and thought those dreaded days of file system 
corruption and lost files were far behind. So my only question is: 
Why does that happen? The disk is not even used 99% of the time.
How does an XFS file system go belly up just like that?

TIA

-- 
Luciano ES
>>

Re: I got file system corruption with XFS

[Eric Sandeen]

On 12/11/18 2:32 PM, Luciano ES wrote:
> I needed to restore something from my backups, an external hard disk 
> that is kept separately, always disconnected until I really need it. 
> 
> But the file system refused to be mounted: "structure needs cleaning," 
> it said.
> 
> I googled and didn't find much hope about it. I followed what little 
> advice I found: I ran xfs_repair and it didn't work. So I ran it 
> again with -L and it worked, but the software itself warned me that 
> some files could not be recovered. I'll never know which ones.
> 
> I always liked XFS and thought those dreaded days of file system 
> corruption and lost files were far behind. So my only question is: 
> Why does that happen? The disk is not even used 99% of the time.
> How does an XFS file system go belly up just like that?

There is no way for us to know.  You didn't provide nearly enough
information to even hazard a guess.

But ok fine, I'll hazard a wild guess anyway: your external drive had
a corrupt log because the enclosure didn't honor a cache flush issued
by the filesystem after some previous mount.

-Eric

Re: I got file system corruption with XFS

[Dave Chinner]

On Tue, Dec 11, 2018 at 06:32:03PM -0200, Luciano ES wrote:
> I needed to restore something from my backups, an external hard disk 
> that is kept separately, always disconnected until I really need it. 
> 
> But the file system refused to be mounted: "structure needs cleaning," 
> it said.
> 
> I googled and didn't find much hope about it. I followed what little 
> advice I found: I ran xfs_repair and it didn't work. So I ran it 
> again with -L and it worked, but the software itself warned me that 
> some files could not be recovered. I'll never know which ones.
> 
> I always liked XFS and thought those dreaded days of file system 
> corruption and lost files were far behind. So my only question is: 
> Why does that happen? The disk is not even used 99% of the time.
> How does an XFS file system go belly up just like that?

What was the actual error message?

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: I got file system corruption with XFS

[Luciano ES]

On Tue, 11 Dec 2018 14:57:45 -0600, Eric Sandeen wrote:

> On 12/11/18 2:32 PM, Luciano ES wrote:
> > I needed to restore something from my backups, an external hard
> > disk that is kept separately, always disconnected until I really
> > need it. 
> > 
> > But the file system refused to be mounted: "structure needs
> > cleaning," it said.
> > 
> > I googled and didn't find much hope about it. I followed what
> > little advice I found: I ran xfs_repair and it didn't work. So I
> > ran it again with -L and it worked, but the software itself warned
> > me that some files could not be recovered. I'll never know which
> > ones.
> > 
> > I always liked XFS and thought those dreaded days of file system 
> > corruption and lost files were far behind. So my only question is: 
> > Why does that happen? The disk is not even used 99% of the time.
> > How does an XFS file system go belly up just like that?  
> 
> There is no way for us to know.  You didn't provide nearly enough
> information to even hazard a guess.
> 
> But ok fine, I'll hazard a wild guess anyway: your external drive had
> a corrupt log because the enclosure didn't honor a cache flush issued
> by the filesystem after some previous mount.
> 
> -Eric


**************************

I understand you don't have much to work with, but I can't tell you
more than I have. It happened several days ago and I didn't write 
anything down. As far as I can remember, there wasn't really much 
to note. "Structure needs cleaning" was pretty much all I was ever 
told. There must have been more when I ran xfs_repair, but it looked 
incomprehensible to me and I thought I shouldn't bother anyone else 
about it.

Your theory about the enclosure sounds good. Do you think it is so 
flawed that issuing a 'sync' command manually before umounting 
wouldn't have made any difference?

-- 
Luciano ES
>>

Re: I got file system corruption with XFS

[Eric Sandeen]

On 12/11/18 4:54 PM, Luciano ES wrote:
> On Tue, 11 Dec 2018 14:57:45 -0600, Eric Sandeen wrote:
> 
>> On 12/11/18 2:32 PM, Luciano ES wrote:
>>> I needed to restore something from my backups, an external hard
>>> disk that is kept separately, always disconnected until I really
>>> need it. 
>>>
>>> But the file system refused to be mounted: "structure needs
>>> cleaning," it said.
>>>
>>> I googled and didn't find much hope about it. I followed what
>>> little advice I found: I ran xfs_repair and it didn't work. So I
>>> ran it again with -L and it worked, but the software itself warned
>>> me that some files could not be recovered. I'll never know which
>>> ones.
>>>
>>> I always liked XFS and thought those dreaded days of file system 
>>> corruption and lost files were far behind. So my only question is: 
>>> Why does that happen? The disk is not even used 99% of the time.
>>> How does an XFS file system go belly up just like that?  
>>
>> There is no way for us to know.  You didn't provide nearly enough
>> information to even hazard a guess.
>>
>> But ok fine, I'll hazard a wild guess anyway: your external drive had
>> a corrupt log because the enclosure didn't honor a cache flush issued
>> by the filesystem after some previous mount.
>>
>> -Eric
> 
> 
> **************************
> 
> I understand you don't have much to work with, but I can't tell you
> more than I have.

That's ok.  It just means I can't help you more than I have.  ;)

> It happened several days ago and I didn't write 
> anything down. As far as I can remember, there wasn't really much 
> to note. "Structure needs cleaning" was pretty much all I was ever 
> told. There must have been more when I ran xfs_repair, but it looked 
> incomprehensible to me and I thought I shouldn't bother anyone else 
> about it.
> 
> Your theory about the enclosure sounds good. Do you think it is so 
> flawed that issuing a 'sync' command manually before umounting 
> wouldn't have made any difference?

If the enclosure ignores cache flush commands, then sync wouldn't have
helped.  Sync /shoudn't/ be necessary before unmount tho.  You may have
been in more jeopardy if you pulled out the USB plug, or lost power.
But that's all just a guess.

-Eric

Re: File system corruption in two hard disks

[Dave Chinner]

On Sun, Mar 31, 2019 at 04:41:47PM -0300, Luciano ES wrote:
> I have two hard disks that have been kept in storage for a little
> more than one year. I can read neither of them.
> 
> DISK 1: I tried to open it with cryptsetup. It wasn't recognized 
> as a LUKS partition, which is weird because I always use LUKS.
> 
> But it pops up as a mountable file system on the pcmanfm file manager.
> Clicking it to mount it as is gives me this error message:
> 
> Error mounting /dev/sdc1 at /media/ext/sm640: Command-line 
> `mount -t "xfs" -o "uhelper=udisks2,nodev,nosuid" "/dev/sdc1" "/media/ext/sm640"' 
> exited with non-zero exit status 32: mount: 
> mount /dev/sdc1 on /media/ext/sm640 failed: Structure needs cleaning

And the errors that XFS emitted in dmesg are... ?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: File system corruption in two hard disks

[Luciano ES]

On Mon, 1 Apr 2019 09:49:18 +1100, Dave Chinner wrote:

> On Sun, Mar 31, 2019 at 04:41:47PM -0300, Luciano ES wrote:
> > I have two hard disks that have been kept in storage for a little
> > more than one year. I can read neither of them.
> > 
> > DISK 1: I tried to open it with cryptsetup. It wasn't recognized 
> > as a LUKS partition, which is weird because I always use LUKS.
> > 
> > But it pops up as a mountable file system on the pcmanfm file
> > manager. Clicking it to mount it as is gives me this error message:
> > 
> > Error mounting /dev/sdc1 at /media/ext/sm640: Command-line 
> > `mount -t "xfs" -o "uhelper=udisks2,nodev,nosuid" "/dev/sdc1"
> > "/media/ext/sm640"' exited with non-zero exit status 32: mount: 
> > mount /dev/sdc1 on /media/ext/sm640 failed: Structure needs
> > cleaning  
> 
> And the errors that XFS emitted in dmesg are... ?
> 
> Cheers,
> 
> Dave.


**************************
Thank you for your reply. Sorry I couldn't get back to you earlier.

Here is the dmesg output. I tried mounting them then running 'xfs_repair -n' 
on each one of them.

DISK 1:
[58267.353026] usb 2-1.2.4: new high-speed USB device number 17 using ehci-pci
[58267.470031] usb 2-1.2.4: New USB device found, idVendor=13fd, idProduct=0840
[58267.470033] usb 2-1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[58267.470033] usb 2-1.2.4: Product: External
[58267.470034] usb 2-1.2.4: Manufacturer: Generic
[58267.470035] usb 2-1.2.4: SerialNumber: [elided]
[58267.470472] usb-storage 2-1.2.4:1.0: USB Mass Storage device detected
[58267.470676] scsi host4: usb-storage 2-1.2.4:1.0
[58268.481677] scsi 4:0:0:0: Direct-Access     Generic  External         1.14 PQ: 0 ANSI: 4
[58268.482369] sd 4:0:0:0: Attached scsi generic sg2 type 0
[58268.482918] sd 4:0:0:0: [sdc] 1250263727 512-byte logical blocks: (640 GB/596 GiB)
[58268.483543] sd 4:0:0:0: [sdc] Write Protect is off
[58268.483544] sd 4:0:0:0: [sdc] Mode Sense: 23 00 00 00
[58268.484183] sd 4:0:0:0: [sdc] No Caching mode page found
[58268.484186] sd 4:0:0:0: [sdc] Assuming drive cache: write through
[58268.515922]  sdc: sdc1
[58268.518417] sd 4:0:0:0: [sdc] Attached SCSI disk
[58268.723169] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[58268.723171] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
[58268.723173] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
[58268.723175] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[58269.531556] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[58269.531559] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
[58269.531561] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
[58269.531563] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(12)/Blank a1 06 20 da 00 00 4f c2 00 b0 00 00
[58271.587320] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58271.587323] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58271.587324] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58271.587326] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 00 00 00 f0 00
[58271.587327] blk_update_request: critical medium error, dev sdc, sector 2560
[58273.860713] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58273.860716] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58273.860717] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58273.860719] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 50 00 00 02 00
[58273.860720] blk_update_request: critical medium error, dev sdc, sector 2640
[58273.860724] Buffer I/O error on dev sdc1, logical block 296, async page read
[58276.091606] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58276.091609] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58276.091610] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58276.091612] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 54 00 00 04 00
[58276.091613] blk_update_request: critical medium error, dev sdc, sector 2644
[58276.091616] Buffer I/O error on dev sdc1, logical block 298, async page read
[58276.091618] Buffer I/O error on dev sdc1, logical block 299, async page read
[58278.368375] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58278.368377] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58278.368378] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58278.368380] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 00 00 00 f0 00
[58278.368381] blk_update_request: critical medium error, dev sdc, sector 2560
[58280.608664] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58280.608667] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58280.608669] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58280.608671] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 50 00 00 02 00
[58280.608672] blk_update_request: critical medium error, dev sdc, sector 2640
[58280.608676] Buffer I/O error on dev sdc1, logical block 296, async page read
[58282.839557] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58282.839560] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58282.839561] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58282.839563] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 54 00 00 04 00
[58282.839565] blk_update_request: critical medium error, dev sdc, sector 2644
[58282.839568] Buffer I/O error on dev sdc1, logical block 298, async page read
[58282.839570] Buffer I/O error on dev sdc1, logical block 299, async page read
[58285.094179] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58285.094181] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58285.094183] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58285.094185] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 00 00 00 f0 00
[58285.094186] blk_update_request: critical medium error, dev sdc, sector 2560
[58287.323337] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58287.323340] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58287.323341] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58287.323344] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 50 00 00 02 00
[58287.323345] blk_update_request: critical medium error, dev sdc, sector 2640
[58287.323349] Buffer I/O error on dev sdc1, logical block 296, async page read
[58289.587476] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58289.587479] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58289.587480] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58289.587482] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 54 00 00 04 00
[58289.587483] blk_update_request: critical medium error, dev sdc, sector 2644
[58289.587487] Buffer I/O error on dev sdc1, logical block 298, async page read
[58289.587489] Buffer I/O error on dev sdc1, logical block 299, async page read
[58289.788582] XFS (sdc1): Offline file system operation in progress!
[58289.788614] XFS (sdc1): Metadata corruption detected at xfs_sb_read_verify+0x111/0x140 [xfs], xfs_sb block 0xffffffffffffffff
[58289.788617] XFS (sdc1): Unmount and run xfs_repair
[58289.788618] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
[58289.788620] ffff9b34d9cd3000: 58 46 53 42 00 00 10 00 00 00 00 00 09 50 af 55  XFSB.........P.U
[58289.788621] ffff9b34d9cd3010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[58289.788622] ffff9b34d9cd3020: 53 df 87 aa a0 e0 42 30 ba 5f 36 4c c5 42 be e7  S.....B0._6L.B..
[58289.788623] ffff9b34d9cd3030: 00 00 00 00 08 00 00 05 ff ff ff ff ff ff ff ff  ................
[58289.788628] XFS (sdc1): SB validate failed with error -117.
[58290.764466] XFS (sdc1): Offline file system operation in progress!
[58290.764499] XFS (sdc1): Metadata corruption detected at xfs_sb_read_verify+0x111/0x140 [xfs], xfs_sb block 0xffffffffffffffff
[58290.764502] XFS (sdc1): Unmount and run xfs_repair
[58290.764502] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
[58290.764504] ffff9b348d3fa000: 58 46 53 42 00 00 10 00 00 00 00 00 09 50 af 55  XFSB.........P.U
[58290.764505] ffff9b348d3fa010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[58290.764506] ffff9b348d3fa020: 53 df 87 aa a0 e0 42 30 ba 5f 36 4c c5 42 be e7  S.....B0._6L.B..
[58290.764507] ffff9b348d3fa030: 00 00 00 00 08 00 00 05 ff ff ff ff ff ff ff ff  ................
[58290.764511] XFS (sdc1): SB validate failed with error -117.
[58316.492438] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58316.492440] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58316.492441] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58316.492443] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 00 00 00 f0 00
[58316.492445] blk_update_request: critical medium error, dev sdc, sector 2560
[58318.699428] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58318.699431] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58318.699432] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58318.699434] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 50 00 00 02 00
[58318.699435] blk_update_request: critical medium error, dev sdc, sector 2640
[58318.699438] Buffer I/O error on dev sdc1, logical block 296, async page read
[58320.952446] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58320.952449] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58320.952450] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58320.952452] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 54 00 00 04 00
[58320.952453] blk_update_request: critical medium error, dev sdc, sector 2644
[58320.952457] Buffer I/O error on dev sdc1, logical block 298, async page read
[58320.952459] Buffer I/O error on dev sdc1, logical block 299, async page read
[58321.152815] XFS (sdc1): Offline file system operation in progress!
[58321.152849] XFS (sdc1): Metadata corruption detected at xfs_sb_read_verify+0x111/0x140 [xfs], xfs_sb block 0xffffffffffffffff
[58321.152852] XFS (sdc1): Unmount and run xfs_repair
[58321.152853] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
[58321.152855] ffff9b34acf07000: 58 46 53 42 00 00 10 00 00 00 00 00 09 50 af 55  XFSB.........P.U
[58321.152856] ffff9b34acf07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[58321.152857] ffff9b34acf07020: 53 df 87 aa a0 e0 42 30 ba 5f 36 4c c5 42 be e7  S.....B0._6L.B..
[58321.152859] ffff9b34acf07030: 00 00 00 00 08 00 00 05 ff ff ff ff ff ff ff ff  ................
[58321.152864] XFS (sdc1): SB validate failed with error -117.
[58332.196671] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[58332.196674] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[58332.196675] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[58332.196677] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 09 e0 00 00 f0 00
[58332.196678] blk_update_request: critical medium error, dev sdc, sector 2528
[58341.573221] XFS (sdc1): Offline file system operation in progress!
[58341.573252] XFS (sdc1): Metadata corruption detected at xfs_sb_read_verify+0x111/0x140 [xfs], xfs_sb block 0xffffffffffffffff
[58341.573255] XFS (sdc1): Unmount and run xfs_repair
[58341.573256] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
[58341.573257] ffff9b34c7b05000: 58 46 53 42 00 00 10 00 00 00 00 00 09 50 af 55  XFSB.........P.U
[58341.573258] ffff9b34c7b05010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[58341.573259] ffff9b34c7b05020: 53 df 87 aa a0 e0 42 30 ba 5f 36 4c c5 42 be e7  S.....B0._6L.B..
[58341.573260] ffff9b34c7b05030: 00 00 00 00 08 00 00 05 ff ff ff ff ff ff ff ff  ................
[58341.573286] XFS (sdc1): SB validate failed with error -117.
[58345.707124] usb 2-1.2.4: USB disconnect, device number 17


DISK 2:
[57949.398186] usb 2-1.2.4: new high-speed USB device number 16 using ehci-pci
[57949.519158] usb 2-1.2.4: New USB device found, idVendor=13fd, idProduct=0840
[57949.519160] usb 2-1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[57949.519161] usb 2-1.2.4: Product: External
[57949.519161] usb 2-1.2.4: Manufacturer: Generic
[57949.519162] usb 2-1.2.4: SerialNumber: [elided]
[57949.519441] usb-storage 2-1.2.4:1.0: USB Mass Storage device detected
[57949.519883] scsi host4: usb-storage 2-1.2.4:1.0
[57950.522787] scsi 4:0:0:0: Direct-Access     Generic  External         1.14 PQ: 0 ANSI: 4
[57950.523080] sd 4:0:0:0: Attached scsi generic sg2 type 0
[57950.523790] sd 4:0:0:0: [sdc] 234441647 512-byte logical blocks: (120 GB/112 GiB)
[57950.524526] sd 4:0:0:0: [sdc] Write Protect is off
[57950.524528] sd 4:0:0:0: [sdc] Mode Sense: 23 00 00 00
[57950.525248] sd 4:0:0:0: [sdc] No Caching mode page found
[57950.525251] sd 4:0:0:0: [sdc] Assuming drive cache: write through
[57950.573656]  sdc: sdc1
[57950.573659] sdc: p1 size 234439600 extends beyond EOD, enabling native capacity
[57950.578529]  sdc: sdc1
[57950.578530] sdc: p1 size 234439600 extends beyond EOD, truncated
[57950.580963] sd 4:0:0:0: [sdc] Attached SCSI disk
[57950.913655] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[57950.913658] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
[57950.913659] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
[57950.913661] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[57951.045034] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[57951.045037] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
[57951.045038] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
[57951.045040] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(12)/Blank a1 06 20 da 00 00 4f c2 00 b0 00 00
[57997.645037] attempt to access beyond end of device
[57997.645039] dm-4: rw=16, want=234435504, limit=234435503
[57997.645041] XFS (dm-4): last sector read failed
[58040.866246] attempt to access beyond end of device
[58040.866249] dm-4: rw=16, want=234435504, limit=234435503
[58040.866251] XFS (dm-4): last sector read failed
[58046.339021] attempt to access beyond end of device
[58046.339023] dm-4: rw=16, want=234435504, limit=234435503
[58046.339025] XFS (dm-4): last sector read failed
[58190.058091] usb 2-1.2.4: USB disconnect, device number 16


(Note: I ran disk2 before disk 1 and inverted the order of the 
outputs here just to keep it consistent with my initial report. 
It probably doesn't matter.)


-- 
Luciano ES
>>

Re: File system corruption in two hard disks

[Dave Chinner]

On Mon, Apr 01, 2019 at 06:13:11PM -0300, Luciano ES wrote:
> On Mon, 1 Apr 2019 09:49:18 +1100, Dave Chinner wrote:
> 
> > On Sun, Mar 31, 2019 at 04:41:47PM -0300, Luciano ES wrote:
> > > I have two hard disks that have been kept in storage for a little
> > > more than one year. I can read neither of them.
> > > 
> > > DISK 1: I tried to open it with cryptsetup. It wasn't recognized 
> > > as a LUKS partition, which is weird because I always use LUKS.
> > > 
> > > But it pops up as a mountable file system on the pcmanfm file
> > > manager. Clicking it to mount it as is gives me this error message:
> > > 
> > > Error mounting /dev/sdc1 at /media/ext/sm640: Command-line 
> > > `mount -t "xfs" -o "uhelper=udisks2,nodev,nosuid" "/dev/sdc1"
> > > "/media/ext/sm640"' exited with non-zero exit status 32: mount: 
> > > mount /dev/sdc1 on /media/ext/sm640 failed: Structure needs
> > > cleaning  
> > 
> > And the errors that XFS emitted in dmesg are... ?
> > 
> > Cheers,
> > 
> > Dave.
> 
> 
> **************************
> Thank you for your reply. Sorry I couldn't get back to you earlier.
> 
> Here is the dmesg output. I tried mounting them then running 'xfs_repair -n' 
> on each one of them.
> 
> DISK 1:
> [58267.353026] usb 2-1.2.4: new high-speed USB device number 17 using ehci-pci
> [58267.470031] usb 2-1.2.4: New USB device found, idVendor=13fd, idProduct=0840
> [58267.470033] usb 2-1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> [58267.470033] usb 2-1.2.4: Product: External
> [58267.470034] usb 2-1.2.4: Manufacturer: Generic
> [58267.470035] usb 2-1.2.4: SerialNumber: [elided]
> [58267.470472] usb-storage 2-1.2.4:1.0: USB Mass Storage device detected
> [58267.470676] scsi host4: usb-storage 2-1.2.4:1.0
> [58268.481677] scsi 4:0:0:0: Direct-Access     Generic  External         1.14 PQ: 0 ANSI: 4
> [58268.482369] sd 4:0:0:0: Attached scsi generic sg2 type 0
> [58268.482918] sd 4:0:0:0: [sdc] 1250263727 512-byte logical blocks: (640 GB/596 GiB)
> [58268.483543] sd 4:0:0:0: [sdc] Write Protect is off
> [58268.483544] sd 4:0:0:0: [sdc] Mode Sense: 23 00 00 00
> [58268.484183] sd 4:0:0:0: [sdc] No Caching mode page found
> [58268.484186] sd 4:0:0:0: [sdc] Assuming drive cache: write through
> [58268.515922]  sdc: sdc1
> [58268.518417] sd 4:0:0:0: [sdc] Attached SCSI disk
> [58268.723169] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
> [58268.723171] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
> [58268.723173] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
> [58268.723175] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
> [58269.531556] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
> [58269.531559] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
> [58269.531561] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
> [58269.531563] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(12)/Blank a1 06 20 da 00 00 4f c2 00 b0 00 00
> [58271.587320] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [58271.587323] sd 4:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current] 
> [58271.587324] sd 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
> [58271.587326] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a 00 00 00 f0 00
> [58271.587327] blk_update_request: critical medium error, dev sdc, sector 2560

You're getting read errors from the disk itself i.e. the data
on the storage medium has degraded and no longer contains the same
information that was written to it.

> [58289.587483] blk_update_request: critical medium error, dev sdc, sector 2644
> [58289.587487] Buffer I/O error on dev sdc1, logical block 298, async page read
> [58289.587489] Buffer I/O error on dev sdc1, logical block 299, async page read
> [58289.788582] XFS (sdc1): Offline file system operation in progress!

Indicative of single bit errors in the superblock i.e. the "in
progress flag is set, which is only set during mkfs and is zero at
all other times. You could probably clear that field using xfs_db,
but it's likely there are lots of other problems with the disk...

> DISK 2:
> [57949.398186] usb 2-1.2.4: new high-speed USB device number 16 using ehci-pci
> [57949.519158] usb 2-1.2.4: New USB device found, idVendor=13fd, idProduct=0840
> [57949.519160] usb 2-1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> [57949.519161] usb 2-1.2.4: Product: External
> [57949.519161] usb 2-1.2.4: Manufacturer: Generic
> [57949.519162] usb 2-1.2.4: SerialNumber: [elided]
> [57949.519441] usb-storage 2-1.2.4:1.0: USB Mass Storage device detected
> [57949.519883] scsi host4: usb-storage 2-1.2.4:1.0
> [57950.522787] scsi 4:0:0:0: Direct-Access     Generic  External         1.14 PQ: 0 ANSI: 4
> [57950.523080] sd 4:0:0:0: Attached scsi generic sg2 type 0
> [57950.523790] sd 4:0:0:0: [sdc] 234441647 512-byte logical blocks: (120 GB/112 GiB)
> [57950.524526] sd 4:0:0:0: [sdc] Write Protect is off
> [57950.524528] sd 4:0:0:0: [sdc] Mode Sense: 23 00 00 00
> [57950.525248] sd 4:0:0:0: [sdc] No Caching mode page found
> [57950.525251] sd 4:0:0:0: [sdc] Assuming drive cache: write through
> [57950.573656]  sdc: sdc1
> [57950.573659] sdc: p1 size 234439600 extends beyond EOD, enabling native capacity
> [57950.578529]  sdc: sdc1
> [57950.578530] sdc: p1 size 234439600 extends beyond EOD, truncated

The partition table indicates that partition 1 extends past the end
of the disk.

> [57950.580963] sd 4:0:0:0: [sdc] Attached SCSI disk
> [57950.913655] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
> [57950.913658] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
> [57950.913659] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
> [57950.913661] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
> [57951.045034] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
> [57951.045037] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] [descriptor] 
> [57951.045038] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
> [57951.045040] sd 4:0:0:0: [sdc] tag#0 CDB: ATA command pass through(12)/Blank a1 06 20 da 00 00 4f c2 00 b0 00 00
> [57997.645037] attempt to access beyond end of device
> [57997.645039] dm-4: rw=16, want=234435504, limit=234435503
> [57997.645041] XFS (dm-4): last sector read failed
> [58040.866246] attempt to access beyond end of device

And XFs is trying to validate that it can read the last block of the
filesystem, which it can't because the partition is beyond the end
of the device.

At this point, I suspect your problems are the USB enclosure your
are using. Plug the disks directly into a SATA port and see if that
makes the IO errors go away. This looks like hardware problems, not
XFS issues...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: File system corruption in two hard disks

[Luciano ES]

On Tue, 2 Apr 2019 08:32:26 +1100, Dave Chinner wrote:

> On Mon, Apr 01, 2019 at 06:13:11PM -0300, Luciano ES wrote:
> > On Mon, 1 Apr 2019 09:49:18 +1100, Dave Chinner wrote:
> >   
> > > On Sun, Mar 31, 2019 at 04:41:47PM -0300, Luciano ES wrote:  
> > > > I have two hard disks that have been kept in storage for a
> > > > little more than one year. I can read neither of them.
> > > > 
> > > > DISK 1: I tried to open it with cryptsetup. It wasn't
> > > > recognized as a LUKS partition, which is weird because I always
> > > > use LUKS.
> > > > 
> > > > But it pops up as a mountable file system on the pcmanfm file
> > > > manager. Clicking it to mount it as is gives me this error
> > > > message:
> > > > 
> > > > Error mounting /dev/sdc1 at /media/ext/sm640: Command-line 
> > > > `mount -t "xfs" -o "uhelper=udisks2,nodev,nosuid" "/dev/sdc1"
> > > > "/media/ext/sm640"' exited with non-zero exit status 32: mount: 
> > > > mount /dev/sdc1 on /media/ext/sm640 failed: Structure needs
> > > > cleaning    
> > > 
> > > And the errors that XFS emitted in dmesg are... ?
> > > 
> > > Cheers,
> > > 
> > > Dave.  
> > 
> > 
> > **************************
> > Thank you for your reply. Sorry I couldn't get back to you earlier.
> > 
> > Here is the dmesg output. I tried mounting them then running
> > 'xfs_repair -n' on each one of them.
> > 
> > DISK 1:
> > [58267.353026] usb 2-1.2.4: new high-speed USB device number 17
> > using ehci-pci [58267.470031] usb 2-1.2.4: New USB device found,
> > idVendor=13fd, idProduct=0840 [58267.470033] usb 2-1.2.4: New USB
> > device strings: Mfr=1, Product=2, SerialNumber=3 [58267.470033] usb
> > 2-1.2.4: Product: External [58267.470034] usb 2-1.2.4:
> > Manufacturer: Generic [58267.470035] usb 2-1.2.4: SerialNumber:
> > [elided] [58267.470472] usb-storage 2-1.2.4:1.0: USB Mass Storage
> > device detected [58267.470676] scsi host4: usb-storage 2-1.2.4:1.0
> > [58268.481677] scsi 4:0:0:0: Direct-Access     Generic
> > External         1.14 PQ: 0 ANSI: 4 [58268.482369] sd 4:0:0:0:
> > Attached scsi generic sg2 type 0 [58268.482918] sd 4:0:0:0: [sdc]
> > 1250263727 512-byte logical blocks: (640 GB/596 GiB) [58268.483543]
> > sd 4:0:0:0: [sdc] Write Protect is off [58268.483544] sd 4:0:0:0:
> > [sdc] Mode Sense: 23 00 00 00 [58268.484183] sd 4:0:0:0: [sdc] No
> > Caching mode page found [58268.484186] sd 4:0:0:0: [sdc] Assuming
> > drive cache: write through [58268.515922]  sdc: sdc1
> > [58268.518417] sd 4:0:0:0: [sdc] Attached SCSI disk
> > [58268.723169] sd 4:0:0:0: [sdc] tag#0 FAILED Result:
> > hostbyte=DID_ERROR driverbyte=DRIVER_SENSE [58268.723171] sd
> > 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current]
> > [descriptor] [58268.723173] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No
> > additional sense information [58268.723175] sd 4:0:0:0: [sdc] tag#0
> > CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00
> > 00 00 00 e5 00 [58269.531556] sd 4:0:0:0: [sdc] tag#0 FAILED
> > Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE [58269.531559]
> > sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current]
> > [descriptor] [58269.531561] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No
> > additional sense information [58269.531563] sd 4:0:0:0: [sdc] tag#0
> > CDB: ATA command pass through(12)/Blank a1 06 20 da 00 00 4f c2 00
> > b0 00 00 [58271.587320] sd 4:0:0:0: [sdc] tag#0 FAILED Result:
> > hostbyte=DID_OK driverbyte=DRIVER_SENSE [58271.587323] sd 4:0:0:0:
> > [sdc] tag#0 Sense Key : Medium Error [current] [58271.587324] sd
> > 4:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
> > [58271.587326] sd 4:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 0a
> > 00 00 00 f0 00 [58271.587327] blk_update_request: critical medium
> > error, dev sdc, sector 2560  
> 
> You're getting read errors from the disk itself i.e. the data
> on the storage medium has degraded and no longer contains the same
> information that was written to it.
> 
> > [58289.587483] blk_update_request: critical medium error, dev sdc,
> > sector 2644 [58289.587487] Buffer I/O error on dev sdc1, logical
> > block 298, async page read [58289.587489] Buffer I/O error on dev
> > sdc1, logical block 299, async page read [58289.788582] XFS (sdc1):
> > Offline file system operation in progress!  
> 
> Indicative of single bit errors in the superblock i.e. the "in
> progress flag is set, which is only set during mkfs and is zero at
> all other times. You could probably clear that field using xfs_db,
> but it's likely there are lots of other problems with the disk...
> 
> > DISK 2:
> > [57949.398186] usb 2-1.2.4: new high-speed USB device number 16
> > using ehci-pci [57949.519158] usb 2-1.2.4: New USB device found,
> > idVendor=13fd, idProduct=0840 [57949.519160] usb 2-1.2.4: New USB
> > device strings: Mfr=1, Product=2, SerialNumber=3 [57949.519161] usb
> > 2-1.2.4: Product: External [57949.519161] usb 2-1.2.4:
> > Manufacturer: Generic [57949.519162] usb 2-1.2.4: SerialNumber:
> > [elided] [57949.519441] usb-storage 2-1.2.4:1.0: USB Mass Storage
> > device detected [57949.519883] scsi host4: usb-storage 2-1.2.4:1.0
> > [57950.522787] scsi 4:0:0:0: Direct-Access     Generic
> > External         1.14 PQ: 0 ANSI: 4 [57950.523080] sd 4:0:0:0:
> > Attached scsi generic sg2 type 0 [57950.523790] sd 4:0:0:0: [sdc]
> > 234441647 512-byte logical blocks: (120 GB/112 GiB) [57950.524526]
> > sd 4:0:0:0: [sdc] Write Protect is off [57950.524528] sd 4:0:0:0:
> > [sdc] Mode Sense: 23 00 00 00 [57950.525248] sd 4:0:0:0: [sdc] No
> > Caching mode page found [57950.525251] sd 4:0:0:0: [sdc] Assuming
> > drive cache: write through [57950.573656]  sdc: sdc1
> > [57950.573659] sdc: p1 size 234439600 extends beyond EOD, enabling
> > native capacity [57950.578529]  sdc: sdc1
> > [57950.578530] sdc: p1 size 234439600 extends beyond EOD,
> > truncated  
> 
> The partition table indicates that partition 1 extends past the end
> of the disk.
> 
> > [57950.580963] sd 4:0:0:0: [sdc] Attached SCSI disk
> > [57950.913655] sd 4:0:0:0: [sdc] tag#0 FAILED Result:
> > hostbyte=DID_ERROR driverbyte=DRIVER_SENSE [57950.913658] sd
> > 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current]
> > [descriptor] [57950.913659] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No
> > additional sense information [57950.913661] sd 4:0:0:0: [sdc] tag#0
> > CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00
> > 00 00 00 e5 00 [57951.045034] sd 4:0:0:0: [sdc] tag#0 FAILED
> > Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE [57951.045037]
> > sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current]
> > [descriptor] [57951.045038] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No
> > additional sense information [57951.045040] sd 4:0:0:0: [sdc] tag#0
> > CDB: ATA command pass through(12)/Blank a1 06 20 da 00 00 4f c2 00
> > b0 00 00 [57997.645037] attempt to access beyond end of device
> > [57997.645039] dm-4: rw=16, want=234435504, limit=234435503
> > [57997.645041] XFS (dm-4): last sector read failed [58040.866246]
> > attempt to access beyond end of device  
> 
> And XFs is trying to validate that it can read the last block of the
> filesystem, which it can't because the partition is beyond the end
> of the device.
> 
> At this point, I suspect your problems are the USB enclosure your
> are using. Plug the disks directly into a SATA port and see if that
> makes the IO errors go away. This looks like hardware problems, not
> XFS issues...
> 
> Cheers,
> 
> Dave.

**************************

Hi. Thank you for your kind attention again.

I tested the disks directly connected to the SATA ports this time. 
One of them worked flawlessly. 

Damn! It's the third USB enclosure that fails on me in one year!

The other disk failed again. Here is some dmesg:

[    1.243231] sd 1:0:0:0: [sdb] 1250263728 512-byte logical blocks: (640 GB/596 GiB)
[    1.243274] sd 1:0:0:0: [sdb] Write Protect is off
[    1.243276] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    1.243299] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.270253]  sdb: sdb1
[    1.271133] sd 1:0:0:0: [sdb] Attached SCSI disk
[    1.272908]  sda: sda1 sda2 sda3 sda4 sda5 sda6
[    1.273680] sd 0:0:0:0: [sda] Attached SCSI disk
elided: lots of stuff about probing USB ports...
[    3.777831] ata2.00: exception Emask 0x0 SAct 0x1000 SErr 0x0 action 0x0
[    3.777882] ata2.00: irq_stat 0x40000008
[    3.777926] ata2.00: failed command: READ FPDMA QUEUED
[    3.777974] ata2.00: cmd 60/00:60:00:0a:00/01:00:00:00:00/40 tag 12 ncq dma 131072 in
                        res 41/40:00:50:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[    3.778038] ata2.00: status: { DRDY ERR }
[    3.778080] ata2.00: error: { UNC }
[    3.790298] ata2.00: configured for UDMA/133
[    3.790312] sd 1:0:0:0: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[    3.790315] sd 1:0:0:0: [sdb] tag#12 Sense Key : Medium Error [current] 
[    3.790318] sd 1:0:0:0: [sdb] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed
[    3.790321] sd 1:0:0:0: [sdb] tag#12 CDB: Read(10) 28 00 00 00 0a 00 00 01 00 00
[    3.790323] blk_update_request: I/O error, dev sdb, sector 2640
[    3.790401] ata2: EH complete
[    6.033996] ata2.00: exception Emask 0x0 SAct 0x3c000000 SErr 0x0 action 0x0
[    6.034060] ata2.00: irq_stat 0x40000008
[    6.034115] ata2.00: failed command: READ FPDMA QUEUED
[    6.034175] ata2.00: cmd 60/02:d0:50:0a:00/00:00:00:00:00/40 tag 26 ncq dma 1024 in
                        res 41/40:00:50:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[    6.034287] ata2.00: status: { DRDY ERR }
[    6.034341] ata2.00: error: { UNC }
[    6.046572] ata2.00: configured for UDMA/133
[    6.046584] sd 1:0:0:0: [sdb] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[    6.046586] sd 1:0:0:0: [sdb] tag#26 Sense Key : Medium Error [current] 
[    6.046588] sd 1:0:0:0: [sdb] tag#26 Add. Sense: Unrecovered read error - auto reallocate failed
[    6.046591] sd 1:0:0:0: [sdb] tag#26 CDB: Read(10) 28 00 00 00 0a 50 00 00 02 00
[    6.046593] blk_update_request: I/O error, dev sdb, sector 2640
[    6.046653] Buffer I/O error on dev sdb1, logical block 296, async page read
[    6.046723] ata2: EH complete
[    8.286163] ata2.00: exception Emask 0x0 SAct 0x1c000 SErr 0x0 action 0x0
[    8.286226] ata2.00: irq_stat 0x40000008
[    8.286282] ata2.00: failed command: READ FPDMA QUEUED
[    8.286342] ata2.00: cmd 60/02:70:56:0a:00/00:00:00:00:00/40 tag 14 ncq dma 1024 in
                        res 41/40:00:56:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[    8.286454] ata2.00: status: { DRDY ERR }
[    8.286508] ata2.00: error: { UNC }
[    8.298738] ata2.00: configured for UDMA/133
[    8.298749] sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[    8.298752] sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium Error [current] 
[    8.298754] sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed
[    8.298757] sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 00 00 0a 56 00 00 02 00
[    8.298758] blk_update_request: I/O error, dev sdb, sector 2646
[    8.298818] Buffer I/O error on dev sdb1, logical block 299, async page read
[    8.298889] ata2: EH complete
elided: lots of initialization/ACPI stuff...
[   15.630752] ata2.00: exception Emask 0x0 SAct 0x8000000 SErr 0x0 action 0x0
[   15.630803] ata2.00: irq_stat 0x40000008
[   15.630830] ata2.00: failed command: READ FPDMA QUEUED
[   15.630865] ata2.00: cmd 60/00:d8:00:0a:00/01:00:00:00:00/40 tag 27 ncq dma 131072 in
                        res 41/40:00:50:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[   15.630951] ata2.00: status: { DRDY ERR }
[   15.630976] ata2.00: error: { UNC }
[   15.643146] ata2.00: configured for UDMA/133
[   15.643162] sd 1:0:0:0: [sdb] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   15.643165] sd 1:0:0:0: [sdb] tag#27 Sense Key : Medium Error [current] 
[   15.643167] sd 1:0:0:0: [sdb] tag#27 Add. Sense: Unrecovered read error - auto reallocate failed
[   15.643170] sd 1:0:0:0: [sdb] tag#27 CDB: Read(10) 28 00 00 00 0a 00 00 01 00 00
[   15.643172] blk_update_request: I/O error, dev sdb, sector 2640
[   15.643229] ata2: EH complete
[   17.882935] ata2.00: exception Emask 0x0 SAct 0x3c00 SErr 0x0 action 0x0
[   17.882984] ata2.00: irq_stat 0x40000008
[   17.883011] ata2.00: failed command: READ FPDMA QUEUED
[   17.883045] ata2.00: cmd 60/02:50:50:0a:00/00:00:00:00:00/40 tag 10 ncq dma 1024 in
                        res 41/40:00:50:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[   17.883130] ata2.00: status: { DRDY ERR }
[   17.883155] ata2.00: error: { UNC }
[   17.895264] ata2.00: configured for UDMA/133
[   17.895276] sd 1:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   17.895279] sd 1:0:0:0: [sdb] tag#10 Sense Key : Medium Error [current] 
[   17.895281] sd 1:0:0:0: [sdb] tag#10 Add. Sense: Unrecovered read error - auto reallocate failed
[   17.895284] sd 1:0:0:0: [sdb] tag#10 CDB: Read(10) 28 00 00 00 0a 50 00 00 02 00
[   17.895286] blk_update_request: I/O error, dev sdb, sector 2640
[   17.895328] Buffer I/O error on dev sdb1, logical block 296, async page read
[   17.895377] ata2: EH complete
[   20.127110] ata2.00: exception Emask 0x0 SAct 0x60000001 SErr 0x0 action 0x0
[   20.127161] ata2.00: irq_stat 0x40000008
[   20.127188] ata2.00: failed command: READ FPDMA QUEUED
[   20.127223] ata2.00: cmd 60/02:e8:56:0a:00/00:00:00:00:00/40 tag 29 ncq dma 1024 in
                        res 41/40:00:56:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[   20.127307] ata2.00: status: { DRDY ERR }
[   20.127332] ata2.00: error: { UNC }
[   20.139450] ata2.00: configured for UDMA/133
[   20.139468] sd 1:0:0:0: [sdb] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   20.139470] sd 1:0:0:0: [sdb] tag#29 Sense Key : Medium Error [current] 
[   20.139473] sd 1:0:0:0: [sdb] tag#29 Add. Sense: Unrecovered read error - auto reallocate failed
[   20.139475] sd 1:0:0:0: [sdb] tag#29 CDB: Read(10) 28 00 00 00 0a 56 00 00 02 00
[   20.139477] blk_update_request: I/O error, dev sdb, sector 2646
[   20.139517] Buffer I/O error on dev sdb1, logical block 299, async page read
[   20.139564] ata2: EH complete
elided: more initialization, sound, other partitions being mounted cleanly etc...
[  149.808922] XFS (sdb1): Offline file system operation in progress!
[  149.808973] XFS (sdb1): Metadata corruption detected at xfs_sb_read_verify+0x111/0x140 [xfs], xfs_sb block 0xffffffffffffffff
[  149.808978] XFS (sdb1): Unmount and run xfs_repair
[  149.808980] XFS (sdb1): First 64 bytes of corrupted metadata buffer:
[  149.808983] ffff9b459bf9a000: 58 46 53 42 00 00 10 00 00 00 00 00 09 50 af 55  XFSB.........P.U
[  149.808985] ffff9b459bf9a010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  149.808987] ffff9b459bf9a020: 53 df 87 aa a0 e0 42 30 ba 5f 36 4c c5 42 be e7  S.....B0._6L.B..
[  149.808989] ffff9b459bf9a030: 00 00 00 00 08 00 00 05 ff ff ff ff ff ff ff ff  ................
[  149.809036] XFS (sdb1): SB validate failed with error -117.
[  234.394294] ata2.00: exception Emask 0x0 SAct 0x400 SErr 0x0 action 0x0
[  234.394299] ata2.00: irq_stat 0x40000008
[  234.394303] ata2.00: failed command: READ FPDMA QUEUED
[  234.394308] ata2.00: cmd 60/00:50:00:08:00/04:00:00:00:00/40 tag 10 ncq dma 524288 in
                        res 41/40:00:50:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[  234.394311] ata2.00: status: { DRDY ERR }
[  234.394313] ata2.00: error: { UNC }
[  234.406484] ata2.00: configured for UDMA/133
[  234.406502] sd 1:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  234.406505] sd 1:0:0:0: [sdb] tag#10 Sense Key : Medium Error [current] 
[  234.406507] sd 1:0:0:0: [sdb] tag#10 Add. Sense: Unrecovered read error - auto reallocate failed
[  234.406510] sd 1:0:0:0: [sdb] tag#10 CDB: Read(10) 28 00 00 00 08 00 00 04 00 00
[  234.406512] blk_update_request: I/O error, dev sdb, sector 2640
[  234.406538] ata2: EH complete


Also:
# xfs_repair -n /dev/sdb1
Phase 1 - find and verify superblock...
superblock read failed, offset 0, size 524288, ag 0, rval -1
fatal error -- Input/output error


I didn't have time to investigate more so I didn't even try smartctl on it.
But looks like that disk is dead, doesn't it?
:-(


-- 
Luciano ES
>>

Re: File system corruption in two hard disks

[Eric Sandeen]

On 4/2/19 11:23 AM, Luciano ES wrote:

> The other disk failed again. Here is some dmesg:
> 
> [    1.243231] sd 1:0:0:0: [sdb] 1250263728 512-byte logical blocks: (640 GB/596 GiB)
> [    1.243274] sd 1:0:0:0: [sdb] Write Protect is off
> [    1.243276] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
> [    1.243299] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [    1.270253]  sdb: sdb1
> [    1.271133] sd 1:0:0:0: [sdb] Attached SCSI disk
> [    1.272908]  sda: sda1 sda2 sda3 sda4 sda5 sda6
> [    1.273680] sd 0:0:0:0: [sda] Attached SCSI disk
> elided: lots of stuff about probing USB ports...
> [    3.777831] ata2.00: exception Emask 0x0 SAct 0x1000 SErr 0x0 action 0x0
> [    3.777882] ata2.00: irq_stat 0x40000008
> [    3.777926] ata2.00: failed command: READ FPDMA QUEUED
> [    3.777974] ata2.00: cmd 60/00:60:00:0a:00/01:00:00:00:00/40 tag 12 ncq dma 131072 in
>                         res 41/40:00:50:0a:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
> [    3.778038] ata2.00: status: { DRDY ERR }
> [    3.778080] ata2.00: error: { UNC }
> [    3.790298] ata2.00: configured for UDMA/133
> [    3.790312] sd 1:0:0:0: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

This continues to look like a hardware/drive failure, not an xfs problem.

...

> Also:
> # xfs_repair -n /dev/sdb1
> Phase 1 - find and verify superblock...
> superblock read failed, offset 0, size 524288, ag 0, rval -1
> fatal error -- Input/output error

as does this.

> 
> I didn't have time to investigate more so I didn't even try smartctl on it.
> But looks like that disk is dead, doesn't it?

yep!

> :-(
> 
> 

Re: File system corruption in two hard disks

[Chris Murphy]

On Tue, Apr 2, 2019 at 10:24 AM Luciano ES <lucmove@gmail.com> wrote:

> [    3.790321] sd 1:0:0:0: [sdb] tag#12 CDB: Read(10) 28 00 00 00 0a 00 00 01 00 00
> [    3.790323] blk_update_request: I/O error, dev sdb, sector 2640

Common bad sector error, includes the LBA for the sector.

There's a scant chance it's recoverable if the drive supports
configurable SCT ERC, and just happens to have a low timeout value
(common on NAS and enterprise drives). You can check it with:

# smartctl -l scterc /dev/sdb
# cat /sys/block/sdb/device/timeout

These are two different things. The first is internal to the drive
(firmware). The second is the kernel's command queue timer for that
block device. If the SCT ERC value is something short like 70
deciseconds, you can try disabling it.

# smartctl -l scterc,0,0 /dev/sdb

And then increase the kernel command timer to something ridiculous
like 180 seconds.

# echo 180 > /sys/block/sdb/device/timeout

Try your repair again. xfs_repair might appear to hang. My guess is it
fails again right away. But there's some chance giving the drive more
time to recover that sector, and it might just do it. Thing is, if
there's no problem with the contents on that bad sector, it won't
likely be overwritten, and it only gets "repaired" by an overwrite.
Once the xfs_repair completes and if successful, you'll want to mount
the file system rw, make some trivial change like touching a file,
then unmount.

A reboot will reset all of these values, and you'll quickly learn if
this is fixed. If not...well cross that bridge later depending on what
results you get.


> [    8.298754] sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed
> [    8.298757] sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 00 00 0a 56 00 00 02 00
> [    8.298758] blk_update_request: I/O error, dev sdb, sector 2646

2640 and 2646 are likely the same 4096 physical sector; they get
different values because of 512 byte sector emulation. What do you get
for

# blockdev --getss --getpbsz


> I didn't have time to investigate more so I didn't even try smartctl on it.
> But looks like that disk is dead, doesn't it?
> :-(

Uncertain. Some number of bad sectors are considered acceptable by the
manufacturer if they remap. Well, yours went bad before the remap so
I'd complain if the drive is under warranty. But that's separate from
recovery...

# smartctl -x /dev/sdb



-- 
Chris Murphy

Re: File system corruption in two hard disks

[Dave Chinner]

On Tue, Apr 02, 2019 at 01:23:57PM -0300, Luciano ES wrote:
> On Tue, 2 Apr 2019 08:32:26 +1100, Dave Chinner wrote:
> > At this point, I suspect your problems are the USB enclosure your
> > are using. Plug the disks directly into a SATA port and see if that
> > makes the IO errors go away. This looks like hardware problems, not
> > XFS issues...
> 
> **************************
> 
> Hi. Thank you for your kind attention again.
> 
> I tested the disks directly connected to the SATA ports this time. 
> One of them worked flawlessly. 

That's good!

> Damn! It's the third USB enclosure that fails on me in one year!

That's about par for the course, I think. I simply don't trust USB
storage enclosures at all...

> The other disk failed again. Here is some dmesg:

....

> [    3.790315] sd 1:0:0:0: [sdb] tag#12 Sense Key : Medium Error [current] 
> [    3.790318] sd 1:0:0:0: [sdb] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed

Basically, this is saying the sector on the disk is bad, and there
is no spare sectors on the disk for recovery of that sector. i.e
the sector is now permanently dead. The drive is essentially
unusable at this point - it will only continue to get worse and
lose more data.

AFAICT, the only way you'll get anything off this drive is via
ddrescue to image it and then perform data recovery on the
decayed corpse it extracts....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

XFS file system corruption, refuses to mount

[Luciano ES]

I've had this internal disk running for a long time. I had to 
disconnect it from the SATA and power plugs for two days. 
Now it won't mount. 

mount: wrong fs type, bad option, bad superblock on /dev/mapper/cab3,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.

I get this in dmesg:

[   30.301450] XFS (dm-1): Mounting V5 Filesystem
[   30.426206] XFS (dm-1): Corruption warning: Metadata has LSN (16:367696) ahead of current LSN (16:367520). Please unmount and run xfs_repair (>= v4.3) to resolve.
[   30.426209] XFS (dm-1): log mount/recovery failed: error -22
[   30.426310] XFS (dm-1): log mount failed

Note that the entire disk is encrypted with cryptsetup/LUKS, 
which is working fine. Wrong passwords fail. The right password 
opens it. But then it refuses to mount.

This has been happening a lot to me with XFS file systems. 
Why is this happening?

Is there something I can do to recover the data?

-- 
Luciano ES
>>

Re: XFS file system corruption, refuses to mount

[Darrick J. Wong]

On Fri, Aug 02, 2019 at 09:53:56PM -0300, Luciano ES wrote:
> I've had this internal disk running for a long time. I had to 
> disconnect it from the SATA and power plugs for two days. 
> Now it won't mount. 
> 
> mount: wrong fs type, bad option, bad superblock on /dev/mapper/cab3,
>        missing codepage or helper program, or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
> 
> I get this in dmesg:
> 
> [   30.301450] XFS (dm-1): Mounting V5 Filesystem
> [   30.426206] XFS (dm-1): Corruption warning: Metadata has LSN
> (16:367696) ahead of current LSN (16:367520). Please unmount and run
> xfs_repair (>= v4.3) to resolve.

Hm, I think this means the superblock LSN is behind the log LSN, which
could mean that... software is buggy?  The disk didn't flush its cache
before it was unplugged?  Something else?

What kernel & xfsprogs?

And how did you disconnect it from the power plugs?

> [   30.426209] XFS (dm-1): log mount/recovery failed: error -22
> [   30.426310] XFS (dm-1): log mount failed
> 
> Note that the entire disk is encrypted with cryptsetup/LUKS, 
> which is working fine. Wrong passwords fail. The right password 
> opens it. But then it refuses to mount.
> 
> This has been happening a lot to me with XFS file systems. 
> Why is this happening?
> 
> Is there something I can do to recover the data?

Try xfs_repair -n to see what it would do if you ran repair?

--D

> -- 
> Luciano ES
> >>

Re: XFS file system corruption, refuses to mount

[Luciano ES]

On Fri, 2 Aug 2019 18:11:06 -0700, Darrick J. Wong wrote:

> On Fri, Aug 02, 2019 at 09:53:56PM -0300, Luciano ES wrote:
> > I've had this internal disk running for a long time. I had to 
> > disconnect it from the SATA and power plugs for two days. 
> > Now it won't mount. 
> > 
> > mount: wrong fs type, bad option, bad superblock
> > on /dev/mapper/cab3, missing codepage or helper program, or other
> > error In some cases useful info is found in syslog - try
> >        dmesg | tail or so.
> > 
> > I get this in dmesg:
> > 
> > [   30.301450] XFS (dm-1): Mounting V5 Filesystem
> > [   30.426206] XFS (dm-1): Corruption warning: Metadata has LSN
> > (16:367696) ahead of current LSN (16:367520). Please unmount and run
> > xfs_repair (>= v4.3) to resolve.  
> 
> Hm, I think this means the superblock LSN is behind the log LSN, which
> could mean that... software is buggy?  The disk didn't flush its cache
> before it was unplugged?  Something else?
> 
> What kernel & xfsprogs?

Debian 4.9.0-3-amd64, xfsprogs 4.9.0.


> And how did you disconnect it from the power plugs?

I shut down the machine, opened the box's cover and disconnected the 
data and power cables. I used them on the CD/DVD drive, which I never 
use but this time I had to. The hard disk drive remained quiet in its 
bay. Then I shut down the machine and reconnected the cables to the 
hard disk and this problem came up. I also tried another cable and 
another SATA port, to no avail.


> > [   30.426209] XFS (dm-1): log mount/recovery failed: error -22
> > [   30.426310] XFS (dm-1): log mount failed
> > 
> > Note that the entire disk is encrypted with cryptsetup/LUKS, 
> > which is working fine. Wrong passwords fail. The right password 
> > opens it. But then it refuses to mount.
> > 
> > This has been happening a lot to me with XFS file systems. 
> > Why is this happening?
> > 
> > Is there something I can do to recover the data?  
> 
> Try xfs_repair -n to see what it would do if you ran repair?

I tried and got this output:


Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...


and it's been printing an endless stream of dots for a very long 
time. I'm about to go to bed and let this running overnight. 
It looks like it has a long way to go.

-- 
Luciano ES
>>

Re: XFS file system corruption, refuses to mount

[Dave Chinner]

On Fri, Aug 02, 2019 at 10:53:20PM -0300, Luciano ES wrote:
> On Fri, 2 Aug 2019 18:11:06 -0700, Darrick J. Wong wrote:
> 
> > On Fri, Aug 02, 2019 at 09:53:56PM -0300, Luciano ES wrote:
> > > I've had this internal disk running for a long time. I had to 
> > > disconnect it from the SATA and power plugs for two days. 
> > > Now it won't mount. 
> > > 
> > > mount: wrong fs type, bad option, bad superblock
> > > on /dev/mapper/cab3, missing codepage or helper program, or other
> > > error In some cases useful info is found in syslog - try
> > >        dmesg | tail or so.
> > > 
> > > I get this in dmesg:
> > > 
> > > [   30.301450] XFS (dm-1): Mounting V5 Filesystem
> > > [   30.426206] XFS (dm-1): Corruption warning: Metadata has LSN
> > > (16:367696) ahead of current LSN (16:367520). Please unmount and run
> > > xfs_repair (>= v4.3) to resolve.  
> > 
> > Hm, I think this means the superblock LSN is behind the log LSN, which
> > could mean that... software is buggy?  The disk didn't flush its cache
> > before it was unplugged?  Something else?

Given the difference in LSNs is only 176 sectors, it seems very
likely that the drive isn't honoring device flushes and so the log
writes that moved the tail haven't hit the disk before the metadata
which was issued after the log write completed...

What is the drive you are using (brand, model number age, etc)? What
is the output from demsg when the device is first discovered on
boot?

> > What kernel & xfsprogs?
> 
> Debian 4.9.0-3-amd64, xfsprogs 4.9.0.
> 
> 
> > And how did you disconnect it from the power plugs?
> 
> I shut down the machine, opened the box's cover and disconnected the 
> data and power cables. I used them on the CD/DVD drive, which I never 
> use but this time I had to. The hard disk drive remained quiet in its 
> bay. Then I shut down the machine and reconnected the cables to the 
> hard disk and this problem came up. I also tried another cable and 
> another SATA port, to no avail.
> 
> 
> > > [   30.426209] XFS (dm-1): log mount/recovery failed: error -22
> > > [   30.426310] XFS (dm-1): log mount failed
> > > 
> > > Note that the entire disk is encrypted with cryptsetup/LUKS, 
> > > which is working fine. Wrong passwords fail. The right password 
> > > opens it. But then it refuses to mount.
> > > 
> > > This has been happening a lot to me with XFS file systems. 
> > > Why is this happening?
> > > 
> > > Is there something I can do to recover the data?  
> > 
> > Try xfs_repair -n to see what it would do if you ran repair?
> 
> I tried and got this output:
> 
> 
> Phase 1 - find and verify superblock...
> bad primary superblock - bad magic number !!!
> 
> attempting to find secondary superblock...

Unlock the encrypted device first? What does blkid tell you about
that device?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS file system corruption, refuses to mount

[Luciano ES]

On Fri, 2 Aug 2019 18:11:06 -0700, Darrick J. Wong wrote:

> > Is there something I can do to recover the data?  
> 
> Try xfs_repair -n to see what it would do if you ran repair?

I tried it. It took a very long time.

Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
...................................... (a million dots)
Sorry, could not find valid secondary superblock
Exiting now.

I tested the file system (unlocked) with the fsck, lsblk, blkid and 
file commands, all of which confirm that it is an XFS file system. 
It just won't mount. Is there any recovery procedure?

-- 
Luciano ES
>>

XFS file system refuses to mount

[Luciano ES]

If this sounds familiar, it's because I've been here before about 
this same problem. New events are at the end of this message.

I've had this internal disk running for a long time. I had to 
disconnect it from the SATA and power plugs for two days. 
Now it won't mount. 

mount: wrong fs type, bad option, bad superblock on /dev/mapper/cab3,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.

I get this in dmesg:

[   30.301450] XFS (dm-1): Mounting V5 Filesystem
[   30.426206] XFS (dm-1): Corruption warning: Metadata has LSN (16:367696) ahead of current LSN (16:367520). Please unmount and run xfs_repair (>= v4.3) to resolve.
[   30.426209] XFS (dm-1): log mount/recovery failed: error -22
[   30.426310] XFS (dm-1): log mount failed

Note that the entire disk is encrypted with cryptsetup/LUKS, 
which is working fine. Wrong passwords fail. The right password 
opens it. But then the partition refuses to mount.

I've tried xfs_repair -n on it and got this:

Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...

It prints an endless stream of dots for a very long time then it 
says no file system has been found. 

Is there something I can do to recover the data?

The new event is that I am cloning the partition to another one 
with dd right now as I type, and I would like to know how I 
can use that to fix the partition so it's mountable or if I can 
at least scan it and rescue any files.

Thank you in advance.

-- 
Luciano ES
>>