Confusing output of --examine-badblocks1 message

Confusing output of --examine-badblocks1 message

[Roy Sigurd Karlsbakk]

Hi all

Iterating over my drives, I ran mdadm --examine-badblocks to look for clues for an issue where XFS reports

Aug 12 23:29:30 hostname kernel: [548754.725102] XFS (dm-6): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x3a85f3430 len 16 error 5

apparently repeating the same sector number over and over again. The XFS FS is on top of LVM, placed on a RAID-6, currently consisting of nine 2TB drives with two spares. The drives are of various makes and models. Running wth --examine-badblocks, it spits out a blocklist for some of the drives. I find it a bit unclear why, since none of them have anything bad reported in smart data. Well, ok, something it might be (silent errors etc). There are also no I/O errors in the kernel log when this happens.

However, back to --examine-badblocks. It seems it's reporting the same sector numbers in the list for several (up to eight) drives. If I understand this correctly, something strange has hit and damanged all drives on fixed sector numbers, such as this

Bad-blocks on /dev/sdm:
           436362944 for 128 sectors

It doesn't seem very likely, to be honest, that a lot of drives suddenly damage the same sector at once. I can see the same occur on a friend's server - sectors with identical 'bad' sector numbers been listed on individual drives.

So I just wonder:
1. How can this happen? Does it replicate the list if a drive has recorded badblocks on it?
2. Is it possible to somehow reset the list and rather do a full scan again? Something smells fishy here

There's some talk about it here, https://raid.wiki.kernel.org/index.php/The_Badblocks_controversy, and I also wonder if this is still enabled by default. It doesn't make much sense…

Vennlig hilsen

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: Confusing output of --examine-badblocks1 message

[Roy Sigurd Karlsbakk]

> However, back to --examine-badblocks. It seems it's reporting the same sector
> numbers in the list for several (up to eight) drives. If I understand this
> correctly, something strange has hit and damanged all drives on fixed sector
> numbers, such as this
> 
> Bad-blocks on /dev/sdm:
>           436362944 for 128 sectors
> 
> It doesn't seem very likely, to be honest, that a lot of drives suddenly damage
> the same sector at once. I can see the same occur on a friend's server -
> sectors with identical 'bad' sector numbers been listed on individual drives.

It seems very likely the badblocks list is just replicated to new drives. I just started

# mdadm /dev/md0 --replace /dev/sdb --with /dev/sdk

where sdk is a drive known to be good. It's about halfway through and it's already copied part of the badblocks list. No I/O errors have been reported in dmesg or otherwise.

Any idea how to remove this list and start over?

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: Confusing output of --examine-badblocks1 message

[Roy Sigurd Karlsbakk]

>> However, back to --examine-badblocks. It seems it's reporting the same sector
>> numbers in the list for several (up to eight) drives. If I understand this
>> correctly, something strange has hit and damanged all drives on fixed sector
>> numbers, such as this
>> 
>> Bad-blocks on /dev/sdm:
>>           436362944 for 128 sectors
>> 
>> It doesn't seem very likely, to be honest, that a lot of drives suddenly damage
>> the same sector at once. I can see the same occur on a friend's server -
>> sectors with identical 'bad' sector numbers been listed on individual drives.
> 
> It seems very likely the badblocks list is just replicated to new drives. I just
> started
> 
> # mdadm /dev/md0 --replace /dev/sdb --with /dev/sdk
> 
> where sdk is a drive known to be good. It's about halfway through and it's
> already copied part of the badblocks list. No I/O errors have been reported in
> dmesg or otherwise.
> 
> Any idea how to remove this list and start over?

I just tried another approach, mdadm --remove on the spares, mdadm --examine on the removed spares, no superblock. Then madm --fail for one of the drives and mdadm --add for another, now spare for a few milliseconds until recovery started. This runs as it should, slower than --replace, but I don't care. After 12% or so, I checked with --examine-badblocks, and the same sectors are popping up again. This was just a small test to see i --replace was the "bad guy" here or if a full recovery would do the same. It does.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: Confusing output of --examine-badblocks1 message

[Roy Sigurd Karlsbakk]

> I just tried another approach, mdadm --remove on the spares, mdadm --examine on
> the removed spares, no superblock. Then madm --fail for one of the drives and
> mdadm --add for another, now spare for a few milliseconds until recovery
> started. This runs as it should, slower than --replace, but I don't care. After
> 12% or so, I checked with --examine-badblocks, and the same sectors are popping
> up again. This was just a small test to see i --replace was the "bad guy" here
> or if a full recovery would do the same. It does.

For the record, I just tested mdadm --replace again on a disk in the raid. The source disk had no badblocks. The destination disk is new-ish (that is, a few years old, but hardly written to and without an md superblock). It seems the badblocks present on other drives in the raid6 are also replicated to the "new" disk. This is not really how it should be IMO.

There must be a major bug in here somewhere. If there's a bad sector somewhere, well, ok, I can handle some corruption. The filesystem will probably be able to handle it as well. But if this is all blocked because of flakey "bad" sectors not really being bad, then something is bad indeed.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: Confusing output of --examine-badblocks1 message

[Phil Turmel]

On 8/14/20 2:07 PM, Roy Sigurd Karlsbakk wrote:
>> I just tried another approach, mdadm --remove on the spares, mdadm --examine on
>> the removed spares, no superblock. Then madm --fail for one of the drives and
>> mdadm --add for another, now spare for a few milliseconds until recovery
>> started. This runs as it should, slower than --replace, but I don't care. After
>> 12% or so, I checked with --examine-badblocks, and the same sectors are popping
>> up again. This was just a small test to see i --replace was the "bad guy" here
>> or if a full recovery would do the same. It does.
> 
> For the record, I just tested mdadm --replace again on a disk in the raid. The source disk had no badblocks. The destination disk is new-ish (that is, a few years old, but hardly written to and without an md superblock). It seems the badblocks present on other drives in the raid6 are also replicated to the "new" disk. This is not really how it should be IMO.
> 
> There must be a major bug in here somewhere. If there's a bad sector somewhere, well, ok, I can handle some corruption. The filesystem will probably be able to handle it as well. But if this is all blocked because of flakey "bad" sectors not really being bad, then something is bad indeed.

In my not-so-humble opinion, the bug is the existence of the BadBlocks 
feature.  Once a badblock is recorded for a sector, redundancy is 
permanently lost at that location.  There is no tool to undo this.

I strongly recommend that you remove badblock logs on all arrays before 
the "feature" screws you.

Phil

Re: Confusing output of --examine-badblocks1 message

[Roy Sigurd Karlsbakk]

> In my not-so-humble opinion, the bug is the existence of the BadBlocks
> feature.  Once a badblock is recorded for a sector, redundancy is
> permanently lost at that location.  There is no tool to undo this.
> 
> I strongly recommend that you remove badblock logs on all arrays before
> the "feature" screws you.

I think it has screwed me a bit already, but then, I didn't check until recently. I didn't even know about this "feature". But doesn't help much when those "badblocks" are recorded already. What would be the official way to remove them apart from rebuild the whole array?

A friend of mine wrote a thing in python to remove the badblocks list from an offline array. I haven't dared to test it on a live system, but apparently it worked on his (5 drives in RAID-5 IIRC with three of them showing a list identical of badblocks). You can find the code here https://git.thehawken.org/hawken/md-badblocktool.git

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: Confusing output of --examine-badblocks1 message

[Phil Turmel]

On 8/15/20 1:33 PM, Roy Sigurd Karlsbakk wrote:
>> In my not-so-humble opinion, the bug is the existence of the BadBlocks
>> feature.  Once a badblock is recorded for a sector, redundancy is
>> permanently lost at that location.  There is no tool to undo this.
>>
>> I strongly recommend that you remove badblock logs on all arrays before
>> the "feature" screws you.
> 
> I think it has screwed me a bit already, but then, I didn't check until recently. I didn't even know about this "feature". But doesn't help much when those "badblocks" are recorded already. What would be the official way to remove them apart from rebuild the whole array?

That's the biggest problem.  There is no way to remove sectors from the 
badblocks log.  At least, I've not seen one.  Official or otherwise.

What I don't understand is how this feature ended up in mdadm without 
any way to reverse the addition of entries, particularly since *you 
silently lose redundancy*.

> A friend of mine wrote a thing in python to remove the badblocks list from an offline array. I haven't dared to test it on a live system, but apparently it worked on his (5 drives in RAID-5 IIRC with three of them showing a list identical of badblocks). You can find the code here https://git.thehawken.org/hawken/md-badblocktool.git

This would be the first. /:

> Vennlig hilsen
> 
> roy


Phil