getting rid of "csum failed" on a hw raid

getting rid of "csum failed" on a hw raid

[Stefan G. Weichinger]

greets

I have a customer server that was set up with a hardware raid and on top
of that /dev/sda exists a btrfs filesystem.

Yes, I know, that is stupid (can't change that easily).

In dmesg I get flooding:

[2329426.792480] BTRFS warning (device sda): csum failed ino 7454384 off
81708052480 csum 3746940842 expected csum 3305376500
[2329427.844757] BTRFS warning (device sda): csum failed ino 7454384 off
81708052480 csum 3746940842 expected csum 3305376500
[2329428.929354] BTRFS warning (device sda): csum failed ino 7454384 off
81708052480 csum 3746940842 expected csum 3305376500

...

I googled how to spot the file containing that problematic block(?):

find / -type f -exec cp {} /dev/null \;

I have that running within tmux now, are there any other ways to find
that file and somehow fix it?

That btrfs contains the root-fs of the server ... so I would maybe have
to driver there, boot from stick or so and btrfsck the unmounted fs?

thanks, Stefan

Re: getting rid of "csum failed" on a hw raid

[Timofey Titovets]

2017-06-07 11:48 GMT+03:00 Stefan G. Weichinger <lists@xunil.at>:
>
> greets
>
> I have a customer server that was set up with a hardware raid and on top
> of that /dev/sda exists a btrfs filesystem.
>
> Yes, I know, that is stupid (can't change that easily).
>
> In dmesg I get flooding:
>
> [2329426.792480] BTRFS warning (device sda): csum failed ino 7454384 off
> 81708052480 csum 3746940842 expected csum 3305376500
> [2329427.844757] BTRFS warning (device sda): csum failed ino 7454384 off
> 81708052480 csum 3746940842 expected csum 3305376500
> [2329428.929354] BTRFS warning (device sda): csum failed ino 7454384 off
> 81708052480 csum 3746940842 expected csum 3305376500
>
> ...
>
> I googled how to spot the file containing that problematic block(?):
>
> find / -type f -exec cp {} /dev/null \;
>
> I have that running within tmux now, are there any other ways to find
> that file and somehow fix it?
>
> That btrfs contains the root-fs of the server ... so I would maybe have
> to driver there, boot from stick or so and btrfsck the unmounted fs?
>
> thanks, Stefan

btrfs scrub start /mnt_path do this trick

After, you can find info with paths in dmesg

-- 
Have a nice day,
Timofey.

Re: getting rid of "csum failed" on a hw raid

[Stefan G. Weichinger]

Am 2017-06-07 um 11:37 schrieb Timofey Titovets:

> btrfs scrub start /mnt_path do this trick
> 
> After, you can find info with paths in dmesg

thank you, I think I have the file, it's a qemu-img-file.
I try cp-ing it to another fs first, but assume this will fail, right?

Re: getting rid of "csum failed" on a hw raid

[Timofey Titovets]

2017-06-07 13:05 GMT+03:00 Stefan G. Weichinger <lists@xunil.at>:
> Am 2017-06-07 um 11:37 schrieb Timofey Titovets:
>
>> btrfs scrub start /mnt_path do this trick
>>
>> After, you can find info with paths in dmesg
>
> thank you, I think I have the file, it's a qemu-img-file.
> I try cp-ing it to another fs first, but assume this will fail, right?

Yes, because btrfs will return -EIO
So try dd_rescue

-- 
Have a nice day,
Timofey.

Re: getting rid of "csum failed" on a hw raid

[Adam Borowski]

On Wed, Jun 07, 2017 at 01:10:26PM +0300, Timofey Titovets wrote:
> 2017-06-07 13:05 GMT+03:00 Stefan G. Weichinger <lists@xunil.at>:
> > Am 2017-06-07 um 11:37 schrieb Timofey Titovets:
> >
> >> btrfs scrub start /mnt_path do this trick
> >>
> >> After, you can find info with paths in dmesg
> >
> > thank you, I think I have the file, it's a qemu-img-file.
> > I try cp-ing it to another fs first, but assume this will fail, right?
> 
> Yes, because btrfs will return -EIO
> So try dd_rescue

Or even plain dd conv=noerror.  Both will do a faithful analogue of a
physical disk with a silent data corruption on the affected sectors.

It might be a better idea to try running the VM in question: it will avoid
silencing the errors, thus let you find out what you lost -- if you run any
filesystem other than btrfs or zfs in the guest, it has no means of spotting
silent corruption.

Obviously, do make the dd copy first.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ A tit a day keeps the vet away.
⣾⠁⢰⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ (Rejoice as my small-animal-murder-machine got unbroken after
⠈⠳⣄⠀⠀⠀⠀ nearly two years of no catch!)

Re: getting rid of "csum failed" on a hw raid

[Roman Mamedov]

On Wed, 7 Jun 2017 15:09:02 +0200
Adam Borowski <kilobyte@angband.pl> wrote:

> On Wed, Jun 07, 2017 at 01:10:26PM +0300, Timofey Titovets wrote:
> > 2017-06-07 13:05 GMT+03:00 Stefan G. Weichinger <lists@xunil.at>:
> > > Am 2017-06-07 um 11:37 schrieb Timofey Titovets:
> > >
> > >> btrfs scrub start /mnt_path do this trick
> > >>
> > >> After, you can find info with paths in dmesg
> > >
> > > thank you, I think I have the file, it's a qemu-img-file.
> > > I try cp-ing it to another fs first, but assume this will fail, right?
> > 
> > Yes, because btrfs will return -EIO
> > So try dd_rescue
> 
> Or even plain dd conv=noerror.  Both will do a faithful analogue of a
> physical disk with a silent data corruption on the affected sectors.

Yeah, except "plain dd conv=noerror" will produce a useless corrupted image,
because it will be shifted forward by the number of unreadable bytes after the
first error.

You also need the "sync" flag in there.
https://superuser.com/questions/622541/what-does-dd-conv-sync-noerror-do
http://www.debianadmin.com/recover-data-from-a-dead-hard-drive-using-dd.html
https://wiki.archlinux.org/index.php/disk_cloning
Or just stick with dd_rescue and not try to correct people's perfectly good
suggestions with completely wrong and harmful ones.

-- 
With respect,
Roman

Re: getting rid of "csum failed" on a hw raid

[Adam Borowski]

On Wed, Jun 07, 2017 at 06:48:48PM +0500, Roman Mamedov wrote:
> On Wed, 7 Jun 2017 15:09:02 +0200
> Adam Borowski <kilobyte@angband.pl> wrote:
> > > Yes, because btrfs will return -EIO
> > > So try dd_rescue
> > 
> > Or even plain dd conv=noerror.  Both will do a faithful analogue of a
> > physical disk with a silent data corruption on the affected sectors.
> 
> Yeah, except "plain dd conv=noerror" will produce a useless corrupted image,
> because it will be shifted forward by the number of unreadable bytes after the
> first error.
> 
> You also need the "sync" flag in there.

Doh, yeah.

> Or just stick with dd_rescue and not try to correct people's perfectly good
> suggestions with completely wrong and harmful ones.

On the other hand, you have dd everywhere, while dd_rescue is not available
on most bootable media not specifically made for rescue purposes.  And
installing extra software when your disk is already unsound but mostly
working might be risky.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ A tit a day keeps the vet away.
⣾⠁⢰⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ (Rejoice as my small-animal-murder-machine got unbroken after
⠈⠳⣄⠀⠀⠀⠀ nearly two years of no catch!)

Re: getting rid of "csum failed" on a hw raid

[Stefan G. Weichinger]

Am 2017-06-07 um 15:52 schrieb Adam Borowski:
> On Wed, Jun 07, 2017 at 06:48:48PM +0500, Roman Mamedov wrote:
>> On Wed, 7 Jun 2017 15:09:02 +0200
>> Adam Borowski <kilobyte@angband.pl> wrote:
>>>> Yes, because btrfs will return -EIO
>>>> So try dd_rescue
>>>
>>> Or even plain dd conv=noerror.  Both will do a faithful analogue of a
>>> physical disk with a silent data corruption on the affected sectors.
>>
>> Yeah, except "plain dd conv=noerror" will produce a useless corrupted image,
>> because it will be shifted forward by the number of unreadable bytes after the
>> first error.
>>
>> You also need the "sync" flag in there.
> 
> Doh, yeah.
> 
>> Or just stick with dd_rescue and not try to correct people's perfectly good
>> suggestions with completely wrong and harmful ones.
> 
> On the other hand, you have dd everywhere, while dd_rescue is not available
> on most bootable media not specifically made for rescue purposes.  And
> installing extra software when your disk is already unsound but mostly
> working might be risky.

I had ddrescue (without "_") on the given server (gentoo linux) and was
able to copy the 2 files via

ddrescue -n -e1 $filename /other_fs/$filename

I then cp-ied the file back via plain "cp" and the VM started again.
Looks good, now also btrfs scr is happy!

Now it's interesting if there is an underlying corruption happening in
the hardware (on disks).

thanks so far, Stefan

Re: getting rid of "csum failed" on a hw raid

[Chris Murphy]

I'm confused about a few items in this thread:

1. On normal read, if there's a csum mismatch, there should be a path
to file and EIO, because Btrfs won't propagate bad data up to user
space. I'd expect if there's metadata corruption, there'd be no path
to file, and no EIO. The original email does include an inode, but
that's ambiguous since each fs tree has its own set of inodes, so we
can't always easily infer a file from an inode.

2. Why does ddrescue work around this problem on a mounted file
system? It's just reading the file, through the file system, the exact
same errors should have happened.

3. My take on this would have been to use btrfs restore and go after
the file path if I absolutely needed a copy of this file (no backup),
and then copied that back to the file system.


Chris Murphy

Re: getting rid of "csum failed" on a hw raid

[Goffredo Baroncelli]

On 2017-06-07 17:58, Chris Murphy wrote:
> I'm confused about a few items in this thread:
> 
> 1. On normal read, if there's a csum mismatch, there should be a path
> to file and EIO, because Btrfs won't propagate bad data up to user
> space. I'd expect if there's metadata corruption, there'd be no path
> to file, and no EIO. The original email does include an inode, but
> that's ambiguous since each fs tree has its own set of inodes, so we
> can't always easily infer a file from an inode.

Usually the number of fs-trees is a lot smaller than the number of files. So even if the fs-root is not displayed, it could be easily guessed. Anyway I understood that the file was found doing "cp -r / /dev/null" (which means try all the files until you find a problem)
> 
> 2. Why does ddrescue work around this problem on a mounted file
> system? It's just reading the file, through the file system, the exact
> same errors should have happened.

Which problem are you discussing ? Finding the path or getting the data ? Anyway ddrescue is not able to get the data with a wrong checksum. When the data is not good, it is skipped leaving an hole in the destination file.

> 
> 3. My take on this would have been to use btrfs restore and go after
> the file path if I absolutely needed a copy of this file (no backup),
> and then copied that back to the file system.


It might be useful to have a command to handle these situations: read all the good data, read even the wrong data logging the range were the checksum is incorrect. The fact that you have problem in few bytes, doesn't mean that all the 4k sector has to be discarded.

> 
> 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

BR
G.Baroncelli
-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: getting rid of "csum failed" on a hw raid

[Adam Borowski]

On Wed, Jun 07, 2017 at 07:01:21PM +0200, Goffredo Baroncelli wrote:
> On 2017-06-07 17:58, Chris Murphy wrote:
> > 3. My take on this would have been to use btrfs restore and go after
> > the file path if I absolutely needed a copy of this file (no backup),
> > and then copied that back to the file system.
> 
> It might be useful to have a command to handle these situations: read all
> the good data, read even the wrong data logging the range were the
> checksum is incorrect.  The fact that you have problem in few bytes,
> doesn't mean that all the 4k sector has to be discarded.

This is what we did in the DOS times: even when the OS would fail, reading
via INT 13h did usually report an error but the memory you gave as the
target buffer would have all the data with just one or a few bits flipped --
or at worst, when a sector's header was hit, 512 bytes missing.

But that's not a good idea for regular read(), even for root: it's possible
the data is not yours and contains some sensitive info from an unrelated
file.

Thus, it'd have to be a special command or a special argument.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ A tit a day keeps the vet away.
⣾⠁⢰⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ (Rejoice as my small-animal-murder-machine got unbroken after
⠈⠳⣄⠀⠀⠀⠀ nearly two years of no catch!)

Re: getting rid of "csum failed" on a hw raid

[Goffredo Baroncelli]

On 2017-06-07 20:04, Adam Borowski wrote:
> On Wed, Jun 07, 2017 at 07:01:21PM +0200, Goffredo Baroncelli wrote:
>> On 2017-06-07 17:58, Chris Murphy wrote:
>>> 3. My take on this would have been to use btrfs restore and go after
>>> the file path if I absolutely needed a copy of this file (no backup),
>>> and then copied that back to the file system.
>>
>> It might be useful to have a command to handle these situations: read all
>> the good data, read even the wrong data logging the range were the
>> checksum is incorrect.  The fact that you have problem in few bytes,
>> doesn't mean that all the 4k sector has to be discarded.
> 
> This is what we did in the DOS times: even when the OS would fail, reading
> via INT 13h did usually report an error but the memory you gave as the
> target buffer would have all the data with just one or a few bits flipped --
> or at worst, when a sector's header was hit, 512 bytes missing.
> 
> But that's not a good idea for regular read(), even for root: it's possible
> the data is not yours and contains some sensitive info from an unrelated
> file.

Depends: if the corrupted data are metadata, you are right. But the biggest part of the disk is occupied by data; so the likelihood that the corruption is in the data is greater. It would need an ioctl to read the file content without the checksum check... 
> 
> Thus, it'd have to be a special command or a special argument.
> 
> 
> Meow!
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: getting rid of "csum failed" on a hw raid

[Adam Borowski]

On Wed, Jun 07, 2017 at 10:20:53PM +0200, Goffredo Baroncelli wrote:
> On 2017-06-07 20:04, Adam Borowski wrote:
> > On Wed, Jun 07, 2017 at 07:01:21PM +0200, Goffredo Baroncelli wrote:
> >> It might be useful to have a command to handle these situations: read all
> >> the good data, read even the wrong data logging the range were the
> >> checksum is incorrect.  The fact that you have problem in few bytes,
> >> doesn't mean that all the 4k sector has to be discarded.
> > 
> > This is what we did in the DOS times: even when the OS would fail, reading
> > via INT 13h did usually report an error but the memory you gave as the
> > target buffer would have all the data with just one or a few bits flipped --
> > or at worst, when a sector's header was hit, 512 bytes missing.
> > 
> > But that's not a good idea for regular read(), even for root: it's possible
> > the data is not yours and contains some sensitive info from an unrelated
> > file.
> 
> Depends: if the corrupted data are metadata, you are right.  But the
> biggest part of the disk is occupied by data; so the likelihood that the
> corruption is in the data is greater.

Metadata also happens to be rewritten and manipulated more, thus it has a
relatively bigger chance to get mangled.  Corruption at rest is less likely
than corruption in memory or during transfer, and memory is not checksummed.

Two scenarios:
* mangled metadata points to a block belonging to pre-existing unrelated
  data (or secret metadata)
* a new write by mistake overwrites an innocent block of data

Both of these will correctly be caught by checksums, so there's no
information leak.  But there will be if you take that data anyway.  And you
don't manually read every file you managed to restore.

> It would need an ioctl to read the file content without the checksum
> check...

Do modern disks still send the partial data if they determine it to be
corrupt?  If so, every disk-backed filesystem, not just btrfs, could benefit
from such an ioctl.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ A tit a day keeps the vet away.
⣾⠁⢰⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ (Rejoice as my small-animal-murder-machine got unbroken after
⠈⠳⣄⠀⠀⠀⠀ nearly two years of no catch!)