Spurious bad blocks on RAID-5

Spurious bad blocks on RAID-5

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 3834 bytes --]

I seem to be seeing a similar problem to the one reported at
https://marc.info/?l=linux-raid&m=151492849314453&w=2

I have a bunch of sectors which I can't read on my RAID-5.

[42184.038703] Buffer I/O error on dev md127, logical block 6543126661, async page read
[42184.077157] Buffer I/O error on dev md127, logical block 6543126662, async page read
[42184.099128] Buffer I/O error on dev md127, logical block 6543126663, async page read
[42184.110579] Buffer I/O error on dev md127, logical block 6543126664, async page read
[42184.119073] Buffer I/O error on dev md127, logical block 6543126789, async page read
[42184.127457] Buffer I/O error on dev md127, logical block 6543126790, async page read
[42184.135790] Buffer I/O error on dev md127, logical block 6543126791, async page read
[42184.144212] Buffer I/O error on dev md127, logical block 6543126792, async page read
[42184.152812] Buffer I/O error on dev md127, logical block 6543126917, async page read
[42184.161392] Buffer I/O error on dev md127, logical block 6543126918, async page read
[42198.249606] Buffer I/O error on dev md127, logical block 6543126919, async page read
[42198.295172] Buffer I/O error on dev md127, logical block 6543126920, async page read

The file system is ext4, and its fsck can't *write* to these sectors
either:
Buffer I/O error on dev md127, logical block 6543126919, lost async page write

I see no actual I/O errors on the underlying drives, and S.M.A.R.T
reports them healthy. Yet MD thinks I have bad blocks on three of them
at exactly the same location:

Bad-blocks list is empty in /dev/sda3
Bad-blocks list is empty in /dev/sdb3
Bad-blocks on /dev/sdc3:
         13086517288 for 32 sectors
Bad-blocks on /dev/sdd3:
         13086517288 for 32 sectors
Bad-blocks on /dev/sde3:
         13086517288 for 32 sectors

That seems very unlikely to me. FWIW those ranges are readable on the
underlying disks, and contain all zeroes.

Is the best option still to reassemble the array with
'--update=force-no-bbl'? Will that *clear* the BBL so that I can
subsequently assemble it with '--update=bbl' without losing those
sectors again?


The pattern of offending blocks here looks remarkably similar to the
previous report. Is there any clue how it happened?

It seemed to *start* with a 'lost async page write' message just like
the above, and ext4 mounting the file system readonly. I rebooted, to
find that fsck couldn't write those blocks.

# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
     Creation Time : Thu Jun 25 20:46:52 2020
        Raid Level : raid5
        Array Size : 31245086720 (29797.64 GiB 31994.97 GB)
     Used Dev Size : 7811271680 (7449.41 GiB 7998.74 GB)
      Raid Devices : 5
     Total Devices : 5
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue May 25 00:34:40 2021
             State : active, checking 
    Active Devices : 5
   Working Devices : 5
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

      Check Status : 58% complete

              Name : desiato_root
              UUID : 29124898:0e6a5ad0:bd30e229:64129ed0
            Events : 758338

    Number   Major   Minor   RaidDevice State
       7       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       8       8       35        2      active sync   /dev/sdc3
       9       8       51        3      active sync   /dev/sdd3
       5       8       67        4      active sync   /dev/sde3

# uname -a
Linux desiato.infradead.org 5.11.20-200.fc33.x86_64 #1 SMP Wed May 12
12:48:34 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5174 bytes --]

Re: Spurious bad blocks on RAID-5

[antlists]

On 25/05/2021 01:41, David Woodhouse wrote:
> I see no actual I/O errors on the underlying drives, and S.M.A.R.T
> reports them healthy. Yet MD thinks I have bad blocks on three of them
> at exactly the same location:
> 
> Bad-blocks list is empty in /dev/sda3
> Bad-blocks list is empty in /dev/sdb3
> Bad-blocks on /dev/sdc3:
>           13086517288 for 32 sectors
> Bad-blocks on /dev/sdd3:
>           13086517288 for 32 sectors
> Bad-blocks on /dev/sde3:
>           13086517288 for 32 sectors
> 
> That seems very unlikely to me. FWIW those ranges are readable on the
> underlying disks, and contain all zeroes.
> 
> Is the best option still to reassemble the array with
> '--update=force-no-bbl'? Will that*clear*  the BBL so that I can
> subsequently assemble it with '--update=bbl' without losing those
> sectors again?

A lot of people swear AT md badblocks. If you assemble with force 
no-bbl, the recommendation will be to NOT re-enable it.

Personally, I'd recommend using dm-integrity rather than badblocks, but 
that (a) chews up some disk space, and (b) is not very well tested with 
mdraid at the moment.

For example, md-raid should NEVER have any permanent bad blocks, because 
it's a logical layer, and the physical layer will fix it behind raid's 
back. But raid seems to accumulate and never clear bad-blocks ...

Cheers,
Wol

Re: Spurious bad blocks on RAID-5

[Reindl Harald]

Am 25.05.21 um 18:49 schrieb antlists:
> On 25/05/2021 01:41, David Woodhouse wrote:
>> I see no actual I/O errors on the underlying drives, and S.M.A.R.T
>> reports them healthy. Yet MD thinks I have bad blocks on three of them
>> at exactly the same location:
>>
>> Bad-blocks list is empty in /dev/sda3
>> Bad-blocks list is empty in /dev/sdb3
>> Bad-blocks on /dev/sdc3:
>>           13086517288 for 32 sectors
>> Bad-blocks on /dev/sdd3:
>>           13086517288 for 32 sectors
>> Bad-blocks on /dev/sde3:
>>           13086517288 for 32 sectors
>>
>> That seems very unlikely to me. FWIW those ranges are readable on the
>> underlying disks, and contain all zeroes.
>>
>> Is the best option still to reassemble the array with
>> '--update=force-no-bbl'? Will that*clear*  the BBL so that I can
>> subsequently assemble it with '--update=bbl' without losing those
>> sectors again?
> 
> A lot of people swear AT md badblocks. If you assemble with force 
> no-bbl, the recommendation will be to NOT re-enable it.
> 
> Personally, I'd recommend using dm-integrity rather than badblocks, but 
> that (a) chews up some disk space, and (b) is not very well tested with 
> mdraid at the moment.
> 
> For example, md-raid should NEVER have any permanent bad blocks, because 
> it's a logical layer, and the physical layer will fix it behind raid's 
> back. But raid seems to accumulate and never clear bad-blocks ...

and why is that crap implemented in a way that replace a disk with bad 
blocks don't reset the bad blocks for the given device?

that's pervert