"--re-add for /dev/sdb1 to /dev/md0 is not possible"

"--re-add for /dev/sdb1 to /dev/md0 is not possible"

[Andy Smith]

Hi,

In my continuing goal to remove the bad blocks log from any of my
arrays and not have one on any new arrays I create, I wrote this
article:

    https://strugglers.net/~andy/blog/2020/09/13/debian-installer-mdadm-configuration-and-the-bad-blocks-controversy

Shortly afterwards someone on Hacker News¹ said that it is possible
to remove the BBL by failing and re-adding devices, like so:

# mdadm /dev/md127 --fail /dev/sda --remove /dev/sda --re-add /dev/sda --update=no-bbl

I tried that on Ubuntu 18.04:

$ mdadm --version
mdadm - v4.1-rc1 - 2018-03-22
$ sudo mdadm --fail /dev/md0 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1 --update=no-bbl
mdadm: set /dev/sdb1 faulty in /dev/md0
mdadm: hot removed /dev/sdb1 from /dev/md0
mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible
$ sudo mdadm --add /dev/md0 /dev/sdb1 --update=no-bbl
mdadm: --update in Manage mode only allowed with --re-add.
$ sudo mdadm --add /dev/md0 /dev/sdb1
mdadm: added /dev/sdb1
$ sudo mdadm --examine-badblocks /dev/sdb1
Bad-blocks list is empty in /dev/sdb1

I tried it on Debian buster:

$ mdadm --version
mdadm - v4.1 - 2018-10-01
$ sudo mdadm --fail /dev/md6 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1 --update=no-bbl
mdadm: set /dev/sdb1 faulty in /dev/md6
mdadm: hot removed /dev/sdb1 from /dev/md6
mdadm: --re-add for /dev/sdb1 to /dev/md6 is not possible

So, is that supposed to work and if so, why doesn't it work for me?

In both cases these are simple two device RAID-1 metadata version
1.2 arrays. Neither has bitmaps.

Thanks,
Andy

¹ https://news.ycombinator.com/item?id=24479235

Re: "--re-add for /dev/sdb1 to /dev/md0 is not possible"

[Andy Smith]

On Tue, Sep 15, 2020 at 10:27:36AM +0000, Andy Smith wrote:
> mdadm: --re-add for /dev/sdb1 to /dev/md6 is not possible
> 
> So, is that supposed to work and if so, why doesn't it work for me?
> 
> In both cases these are simple two device RAID-1 metadata version
> 1.2 arrays. Neither has bitmaps.

The lack of bitmaps was why I couldn't use --re-add. I was testing
with small arrays that didn't get a bitmap by default. As mentioned
on the wiki, a bitmap can be added like:

# mdadm --grow --bitmap=internal /dev/mdX

So, as pointed out to me by Jakub Wilk, it is possible to remove the
bad blocks log on a running array by failing and re-adding each
device in turn, as long as you are willing to experience lower / no
redundancy while it rebuilds.

# mdadm --fail /dev/md0 /dev/sdb1 \
        --remove /dev/sdb1 \
        --re-add /dev/sdb1 \
        --update=no-bbl

Cheers,
Andy

Re: "--re-add for /dev/sdb1 to /dev/md0 is not possible"

[Roy Sigurd Karlsbakk]

> In my continuing goal to remove the bad blocks log from any of my
> arrays and not have one on any new arrays I create, I wrote this
> article:
> 
>    https://strugglers.net/~andy/blog/2020/09/13/debian-installer-mdadm-configuration-and-the-bad-blocks-controversy
> 
> Shortly afterwards someone on Hacker News¹ said that it is possible
> to remove the BBL by failing and re-adding devices, like so:
> 
> # mdadm /dev/md127 --fail /dev/sda --remove /dev/sda --re-add /dev/sda
> --update=no-bbl
> 
> I tried that on Ubuntu 18.04:
> 
> $ mdadm --version
> mdadm - v4.1-rc1 - 2018-03-22
> $ sudo mdadm --fail /dev/md0 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1
> --update=no-bbl
> mdadm: set /dev/sdb1 faulty in /dev/md0
> mdadm: hot removed /dev/sdb1 from /dev/md0
> mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible
> $ sudo mdadm --add /dev/md0 /dev/sdb1 --update=no-bbl
> mdadm: --update in Manage mode only allowed with --re-add.
> $ sudo mdadm --add /dev/md0 /dev/sdb1
> mdadm: added /dev/sdb1
> $ sudo mdadm --examine-badblocks /dev/sdb1
> Bad-blocks list is empty in /dev/sdb1
> 
> I tried it on Debian buster:
> 
> $ mdadm --version
> mdadm - v4.1 - 2018-10-01
> $ sudo mdadm --fail /dev/md6 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1
> --update=no-bbl
> mdadm: set /dev/sdb1 faulty in /dev/md6
> mdadm: hot removed /dev/sdb1 from /dev/md6
> mdadm: --re-add for /dev/sdb1 to /dev/md6 is not possible
> 
> So, is that supposed to work and if so, why doesn't it work for me?
> 
> In both cases these are simple two device RAID-1 metadata version
> 1.2 arrays. Neither has bitmaps.

I've tried this as well, and ran

mddev=/dev/md0
diskdev=/dev/sdm

mdadm --fail $mddev $diskdev
mdadm --remove $mddev $diskdev
mdadm --re-add $mddev $diskdev

All returned ok, but sdm was listed as a slave, thus not active anymore. After this, a new --remove and --re-add didn't work, so it'll take some hours to --add it again (yet another time).

It would be very nice if someone at linux-raid could prioritise this rather obvious bug in the bbl code, where the bbl keeps replicating itself over and over, regardless of any actual failures on the disks. IMHO the whole BBL should be scrapped, as mentioned earlier, since it really has no function. Mapping out bad sectors is for the drive to decide and if it can't handle it, it should be kicked out of the array.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: "--re-add for /dev/sdb1 to /dev/md0 is not possible"

[antlists]

On 22/09/2020 11:27, Roy Sigurd Karlsbakk wrote:
> It would be very nice if someone at linux-raid could prioritise this rather obvious bug in the bbl code, where the bbl keeps replicating itself over and over, regardless of any actual failures on the disks. IMHO the whole BBL should be scrapped, as mentioned earlier, since it really has no function. Mapping out bad sectors is for the drive to decide and if it can't handle it, it should be kicked out of the array.

Well, I guess if you want a volunteer, you're it! Thanks for steping up 
to the plate!

Cheers,
Wol

Re: "--re-add for /dev/sdb1 to /dev/md0 is not possible"

[Roy Sigurd Karlsbakk]

Will anyone look into this issue, please? It really is a problem.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

----- Original Message -----
> From: "Wols Lists" <antlists@youngman.org.uk>
> To: "Roy Sigurd Karlsbakk" <roy@karlsbakk.net>, "Andy Smith" <andy@strugglers.net>
> Cc: "Linux Raid" <linux-raid@vger.kernel.org>
> Sent: Wednesday, 23 September, 2020 01:18:26
> Subject: Re: "--re-add for /dev/sdb1 to /dev/md0 is not possible"

> On 22/09/2020 11:27, Roy Sigurd Karlsbakk wrote:
>> It would be very nice if someone at linux-raid could prioritise this rather
>> obvious bug in the bbl code, where the bbl keeps replicating itself over and
>> over, regardless of any actual failures on the disks. IMHO the whole BBL should
>> be scrapped, as mentioned earlier, since it really has no function. Mapping out
>> bad sectors is for the drive to decide and if it can't handle it, it should be
>> kicked out of the array.
> 
> Well, I guess if you want a volunteer, you're it! Thanks for steping up
> to the plate!
> 
> Cheers,
> Wol

Re: "--re-add for /dev/sdb1 to /dev/md0 is not possible"

[Andy Smith]

Hi Roy,

On Sat, Nov 28, 2020 at 06:59:25PM +0100, Roy Sigurd Karlsbakk wrote:
> Will anyone look into this issue, please? It really is a problem.

You already did ask once and got the response to get on with it and
let everyone else know; asking further times isn't going to get
anywhere so I suggest you do what is within your power to do.

If the option of coding the fix yourself is not available and you
can't convince (or pay?) someone else to do it, I guess that just
leaves you with the option of disabling BBL.

At least we did go through this before and established that it's
possible to do without downtime, though with short period of loss of
redundancy.

Cheers,
Andy