Input/Output error reading from a clean raid

Input/Output error reading from a clean raid

[Salatiel Filho]

I am trying to recover a few  files from my backup. The backup is on a
raid 5 + ext4.
There are several files where i get I/O error. The raid appears to be
clean and fsck shows no errors. Any ideas what could it be ?

md1 : active raid5 sdd1[0] sdg1[4] sdf1[2] sde1[1]
      3220829184 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/8 pages [4KB], 65536KB chunk



[]'s
Salatiel

Re: Input/Output error reading from a clean raid

[John Stoffel]

Salatiel> I am trying to recover a few files from my backup. The
Salatiel> backup is on a raid 5 + ext4.  There are several files where
Salatiel> i get I/O error. The raid appears to be clean and fsck shows
Salatiel> no errors. Any ideas what could it be ?

Salatiel> md1 : active raid5 sdd1[0] sdg1[4] sdf1[2] sde1[1]
Salatiel>       3220829184 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
Salatiel>       bitmap: 1/8 pages [4KB], 65536KB chunk

It would help if you could post the error(s) you're getting, along
with any output from dmesg during that time.  Have you done a full
scan of the disk looking for errors?  You might just have silent
read errors in your array.  So as root do:

   # echo check >>/sys/block/md??/md/sync_action

where md?? is the name of your md array you want to check.  You can
get the name from:

   cat /proc/mdstat

and of course it would help to post that info as well if you want more
help.

John

Re: Input/Output error reading from a clean raid

[Salatiel Filho]

Ok, i have run echo check >>/sys/block/md1/md/sync_action, and now the output of
mdadm mdadm --examine-badblocks /dev/sdd1 /dev/sdg1 /dev/sdf1  /dev/sde1

Bad-blocks on /dev/sdd1:
          1515723072 for 512 sectors
          1515723584 for 512 sectors
          1515724096 for 512 sectors
          1515724608 for 512 sectors
          1515725120 for 512 sectors
          1515725632 for 512 sectors
          1515726144 for 512 sectors
          1515726656 for 512 sectors
          1515727168 for 512 sectors
          1515727680 for 512 sectors
          1515728192 for 512 sectors
          1515728704 for 512 sectors
          1515729216 for 512 sectors
          1515729728 for 512 sectors
          1515730240 for 512 sectors
          1515730752 for 512 sectors
          1515731264 for 512 sectors
          1515731776 for 512 sectors
          1515732288 for 512 sectors
          1515732800 for 512 sectors
          1515733312 for 512 sectors
          1515733824 for 512 sectors
          1515734336 for 512 sectors
          1515734848 for 512 sectors
          1515735360 for 512 sectors
          1515735872 for 512 sectors
          1515736384 for 512 sectors
          1515736896 for 512 sectors
          1515737408 for 512 sectors
          1515737920 for 512 sectors
          1515738432 for 512 sectors
          1515738944 for 512 sectors
          1515739456 for 512 sectors
          1515739968 for 512 sectors
          1515740480 for 512 sectors
          1515740992 for 512 sectors
          1515741504 for 512 sectors
          1515742016 for 192 sectors
          1515743712 for 512 sectors
          1515744224 for 512 sectors
          1515744736 for 512 sectors
          1515745248 for 512 sectors
          1515745760 for 512 sectors
          1515746272 for 512 sectors
          1515746784 for 512 sectors
          1515747296 for 512 sectors
          1515747808 for 512 sectors
          1515748320 for 512 sectors
          1515749072 for 304 sectors
          1515750400 for 512 sectors
          1515750912 for 512 sectors
          1515751424 for 512 sectors
          1515751936 for 512 sectors
          1515752448 for 512 sectors
          1515752960 for 512 sectors
          1515753472 for 512 sectors
          1515753984 for 512 sectors
          1515754496 for 232 sectors
Bad-blocks list is empty in /dev/sdg1
Bad-blocks list is empty in /dev/sdf1
Bad-blocks on /dev/sde1:
          1515723072 for 512 sectors
          1515723584 for 512 sectors
          1515724096 for 512 sectors
          1515724608 for 512 sectors
          1515725120 for 512 sectors
          1515725632 for 512 sectors
          1515726144 for 512 sectors
          1515726656 for 512 sectors
          1515727168 for 512 sectors
          1515727680 for 512 sectors
          1515728192 for 512 sectors
          1515728704 for 512 sectors
          1515729216 for 512 sectors
          1515729728 for 512 sectors
          1515730240 for 512 sectors
          1515730752 for 512 sectors
          1515731264 for 512 sectors
          1515731776 for 512 sectors
          1515732288 for 512 sectors
          1515732800 for 512 sectors
          1515733312 for 512 sectors
          1515733824 for 512 sectors
          1515734336 for 512 sectors
          1515734848 for 512 sectors
          1515735360 for 512 sectors
          1515735872 for 512 sectors
          1515736384 for 512 sectors
          1515736896 for 512 sectors
          1515737408 for 512 sectors
          1515737920 for 512 sectors
          1515738432 for 512 sectors
          1515738944 for 512 sectors
          1515739456 for 512 sectors
          1515739968 for 512 sectors
          1515740480 for 512 sectors
          1515740992 for 512 sectors
          1515741504 for 512 sectors
          1515742016 for 192 sectors
          1515743712 for 512 sectors
          1515744224 for 512 sectors
          1515744736 for 512 sectors
          1515745248 for 512 sectors
          1515745760 for 512 sectors
          1515746272 for 512 sectors
          1515746784 for 512 sectors
          1515747296 for 512 sectors
          1515747808 for 512 sectors
          1515748320 for 512 sectors
          1515749072 for 304 sectors
          1515750400 for 512 sectors
          1515750912 for 512 sectors
          1515751424 for 512 sectors
          1515751936 for 512 sectors
          1515752448 for 512 sectors
          1515752960 for 512 sectors
          1515753472 for 512 sectors
          1515753984 for 512 sectors
          1515754496 for 232 sectors
[]'s
Salatiel


On Sun, Jan 22, 2017 at 10:03 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Sun, Jan 22, 2017 at 11:08:40AM -0300, Salatiel Filho wrote:
>> Any ideas what could it be ?
>
> mdadm --examine-badblocks
>
> Regards
> Andreas Klauer

Re: Input/Output error reading from a clean raid

[Salatiel Filho]

The output of the command is:

# dd if=Fotos.zip of=/dev/null
dd: error reading ‘Fotos.zip’: Input/output error
328704+0 records in
328704+0 records out
168296448 bytes (168 MB) copied, 0.127723 s, 1.3 GB/s

or

# cp Fotos.zip /tmp/
cp: error reading ‘Fotos.zip’: Input/output error
cp: failed to extend ‘/tmp/Fotos.zip’: Input/output error


There is nothing on dmesg after running those commands;

[]'s
Salatiel


On Sun, Jan 22, 2017 at 9:18 PM, John Stoffel <john@stoffel.org> wrote:
>
> Salatiel> I am trying to recover a few files from my backup. The
> Salatiel> backup is on a raid 5 + ext4.  There are several files where
> Salatiel> i get I/O error. The raid appears to be clean and fsck shows
> Salatiel> no errors. Any ideas what could it be ?
>
> Salatiel> md1 : active raid5 sdd1[0] sdg1[4] sdf1[2] sde1[1]
> Salatiel>       3220829184 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
> Salatiel>       bitmap: 1/8 pages [4KB], 65536KB chunk
>
> It would help if you could post the error(s) you're getting, along
> with any output from dmesg during that time.  Have you done a full
> scan of the disk looking for errors?  You might just have silent
> read errors in your array.  So as root do:
>
>    # echo check >>/sys/block/md??/md/sync_action
>
> where md?? is the name of your md array you want to check.  You can
> get the name from:
>
>    cat /proc/mdstat
>
> and of course it would help to post that info as well if you want more
> help.
>
> John

Re: Input/Output error reading from a clean raid

[John Stoffel]

Salatiel> The output of the command is:
Salatiel> # dd if=Fotos.zip of=/dev/null
Salatiel> dd: error reading ‘Fotos.zip’: Input/output error
Salatiel> 328704+0 records in
Salatiel> 328704+0 records out
Salatiel> 168296448 bytes (168 MB) copied, 0.127723 s, 1.3 GB/s

Salatiel> or

Salatiel> # cp Fotos.zip /tmp/
Salatiel> cp: error reading ‘Fotos.zip’: Input/output error
Salatiel> cp: failed to extend ‘/tmp/Fotos.zip’: Input/output error

Can you do a 'zip -l Fotos.zip' and get anything back?  It looks like
the first 168mb might be ok... so you might get something back.

You might also want to try and start doing a dd from 328705 records
(or even a couple more records farther) to see if you can get anything
else from there.

In this case, the tool 'ddrescue' might be your answer, since it is
designed to handle errors like this and continue reading past errors.
It might, or might not, let you get more of your data back.  On debian based
systems you should be able to just do:

	apt-get install gddrescue

or just do:

   apt-cache search ddrescue

For RedHat fedora you could do:

   dnf search ddrescue

too.  

Did you run the "echo check > ..." command at all?  What did it say in
the output of: cat /proc/mdstat  when you did this?  

Salatiel> There is nothing on dmesg after running those commands;

You might be out of luck.  This is one reason why I like A) mirroring
my data and B) saving multiple copies to multiple locations.  Storage
is cheap these days.

Though I admit I'm not perfect either.

Please get us more information so we can try to help more.

Also, have you unmounted the filesystem and done an 'fsck -y /dev/...'
on it as well?  You might want to do a more in-depth check of the
filesystem to see if there's any corruption somewhere.

Also, going to the end of the file, and seeking backwards and reading
off blocks might help you recover more of the zip file.



Salatiel> On Sun, Jan 22, 2017 at 9:18 PM, John Stoffel <john@stoffel.org> wrote:
>> 
Salatiel> I am trying to recover a few files from my backup. The
Salatiel> backup is on a raid 5 + ext4.  There are several files where
Salatiel> i get I/O error. The raid appears to be clean and fsck shows
Salatiel> no errors. Any ideas what could it be ?
>> 
Salatiel> md1 : active raid5 sdd1[0] sdg1[4] sdf1[2] sde1[1]
Salatiel> 3220829184 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
Salatiel> bitmap: 1/8 pages [4KB], 65536KB chunk
>> 
>> It would help if you could post the error(s) you're getting, along
>> with any output from dmesg during that time.  Have you done a full
>> scan of the disk looking for errors?  You might just have silent
>> read errors in your array.  So as root do:
>> 
>> # echo check >>/sys/block/md??/md/sync_action
>> 
>> where md?? is the name of your md array you want to check.  You can
>> get the name from:
>> 
>> cat /proc/mdstat
>> 
>> and of course it would help to post that info as well if you want more
>> help.
>> 
>> John

Re: Input/Output error reading from a clean raid

[John Stoffel]

>>>>> "Salatiel" == Salatiel Filho <salatiel.filho@gmail.com> writes:

Salatiel> Ok, i have run echo check >>/sys/block/md1/md/sync_action,
Salatiel> and now the output of mdadm mdadm --examine-badblocks
Salatiel> /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sde1


Salatiel> Bad-blocks on /dev/sdd1:
Salatiel>           1515723072 for 512 sectors
Salatiel>           1515723584 for 512 sectors
Salatiel>           1515724096 for 512 sectors
Salatiel>           1515724608 for 512 sectors

You have bad disks in your array.  First thing off is that I would go
buy replacements and then use 'ddrescue' to copy the data from the old
disks to new disks.  Then I'd try to assemble the NEW disks only into
an array, and then I'd fsck the filesystem(s).

You're going to lose data, no doubt about it.  You're now in the mode
where you're trying to save as much as you can as quickly as possible.

Personally, I'd be setting up a RAID6 array for your new setup.  Then
I would also be setting up weekly checks of the raid array as well.

You're going to lose data no matter what.  So get new disks and start
copying what you can.

John

Re: Input/Output error reading from a clean raid

[Wols Lists]

On 23/01/17 17:07, John Stoffel wrote:
> You have bad disks in your array.  First thing off is that I would go
> buy replacements and then use 'ddrescue' to copy the data from the old
> disks to new disks.  Then I'd try to assemble the NEW disks only into
> an array, and then I'd fsck the filesystem(s).
> 
> You're going to lose data, no doubt about it.  You're now in the mode
> where you're trying to save as much as you can as quickly as possible.
> 
> Personally, I'd be setting up a RAID6 array for your new setup.  Then
> I would also be setting up weekly checks of the raid array as well.
> 
> You're going to lose data no matter what.  So get new disks and start
> copying what you can.

Go read the raid wiki. https://raid.wiki.kernel.org/index.php/Linux_Raid

Especially replacing a failed drive
https://raid.wiki.kernel.org/index.php/Replacing_a_failed_drive

And please - can you get ddrescue's error log that it mentions and email
me a copy. If you've got some Perl or Python or shell skills, maybe you
could even write that script it mentions (which is described in a bit
more detail in programming projects
https://raid.wiki.kernel.org/index.php/Programming_projects) Otherwise
I'll try and write it - might be a good way of learning Python :-) but
at the moment I think I'm learning by jumping in out of my depth, so
we'll see how far I get :-)

Cheers,
Wol

Re: Input/Output error reading from a clean raid

[Andreas Klauer]

On Mon, Jan 23, 2017 at 11:02:24AM -0300, Salatiel Filho wrote:
> mdadm mdadm --examine-badblocks /dev/sdd1 /dev/sdg1 /dev/sdf1  /dev/sde1
> 
> Bad-blocks on /dev/sdd1:
>           1515723072 for 512 sectors
> Bad-blocks on /dev/sde1:
>           1515723072 for 512 sectors

md believes you have bad blocks in identical places so it won't return 
whatever data is in these blocks. Thus you get read errors even if there 
is no bad block on the disk itself. Those bad block entries can be caused 
by cable or controller flukes, making temporary problems permanent...

Personally I disable the bad block list everywhere.

You can search this list for old messages regarding --examine-badblocks, 
this problem came up several times. Clearing the mdadm bad block list is 
worth a try. There's an undocumented option, update=force-no-bbl or such.

Regards
Andreas Klauer

Re: Input/Output error reading from a clean raid

[Salatiel Filho]

On Mon, Jan 23, 2017 at 2:34 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Mon, Jan 23, 2017 at 11:02:24AM -0300, Salatiel Filho wrote:
>> mdadm mdadm --examine-badblocks /dev/sdd1 /dev/sdg1 /dev/sdf1  /dev/sde1
>>
>> Bad-blocks on /dev/sdd1:
>>           1515723072 for 512 sectors
>> Bad-blocks on /dev/sde1:
>>           1515723072 for 512 sectors
>
> md believes you have bad blocks in identical places so it won't return
> whatever data is in these blocks. Thus you get read errors even if there
> is no bad block on the disk itself. Those bad block entries can be caused
> by cable or controller flukes, making temporary problems permanent...
>
> Personally I disable the bad block list everywhere.
>
> You can search this list for old messages regarding --examine-badblocks,
> this problem came up several times. Clearing the mdadm bad block list is
> worth a try. There's an undocumented option, update=force-no-bbl or such.
>
> Regards
> Andreas Klauer

Thanks all of you for the help.
Andreas, the force-no-bbl from mdadm 3.4 did the trick. I was able to
retrieve all files and their md5 matches, so it is great =)
I really think it is very unlikely that two different disks from two
different brands would have problems at exactly the same block.
I have a question, who populates the badblock list ? Is the check
action send to the /sys/block/md??/md/sync_action OR each read error
updates it ?
I think it was maybe some problem with the cable ( it is a 4 disks usb3 bay ).
Anyway, thank you very much !

Re: Input/Output error reading from a clean raid

[Wols Lists]

On 24/01/17 21:15, Salatiel Filho wrote:
> I really think it is very unlikely that two different disks from two
> different brands would have problems at exactly the same block.
> I have a question, who populates the badblock list ? Is the check
> action send to the /sys/block/md??/md/sync_action OR each read error
> updates it ?

I think it's a known problem - nobody seems to know quite why it happens
but when a block is added to the badblocks list it seems to get added to
every device. Given that modern hard-drives are supposed to relocate bad
blocks and not need a badblock list, I think that's why it's not been
found, most people especially those in the know just tend to disable
os-level badblocks.

Cheers,
Wol

Re: Input/Output error reading from a clean raid

[John Stoffel]

>>>>> "Salatiel" == Salatiel Filho <salatiel.filho@gmail.com> writes:

Salatiel> On Mon, Jan 23, 2017 at 2:34 PM, Andreas Klauer
Salatiel> <Andreas.Klauer@metamorpher.de> wrote:
>> On Mon, Jan 23, 2017 at 11:02:24AM -0300, Salatiel Filho wrote:
>>> mdadm mdadm --examine-badblocks /dev/sdd1 /dev/sdg1 /dev/sdf1  /dev/sde1
>>> 
>>> Bad-blocks on /dev/sdd1:
>>> 1515723072 for 512 sectors
>>> Bad-blocks on /dev/sde1:
>>> 1515723072 for 512 sectors
>> 
>> md believes you have bad blocks in identical places so it won't return
>> whatever data is in these blocks. Thus you get read errors even if there
>> is no bad block on the disk itself. Those bad block entries can be caused
>> by cable or controller flukes, making temporary problems permanent...
>> 
>> Personally I disable the bad block list everywhere.
>> 
>> You can search this list for old messages regarding --examine-badblocks,
>> this problem came up several times. Clearing the mdadm bad block list is
>> worth a try. There's an undocumented option, update=force-no-bbl or such.
>> 
>> Regards
>> Andreas Klauer

Salatiel> Thanks all of you for the help.
Salatiel> Andreas, the force-no-bbl from mdadm 3.4 did the trick. I was able to
Salatiel> retrieve all files and their md5 matches, so it is great =)

Great news, glad I could help, wish I had pin-pointed the root cause
better.

john