From: Bob Brand <brand@wmawater.com.au>
To: "Wols Lists" <antlists@youngman.org.uk>, <linux-raid@vger.kernel.org>
Cc: "Phil Turmel" <philip@turmel.org>, "NeilBrown" <neilb@suse.com>
Subject: RE: Failed adadm RAID array after aborted Grown operation
Date: Mon, 9 May 2022 23:07:31 +1000 (AEST) [thread overview]
Message-ID: <002601d863a5$bd9f5880$38de0980$@wmawater.com.au> (raw)
In-Reply-To: <5931f716-008d-399b-2ea8-acbbc9c8d239@youngman.org.uk>
Hi Wol,
I did read the links you sent, actually I'd already trawled through them
prior to subscribing to the mailing list. They're how I learned about the
mailing list.
It seems that the conventional version of CentOS 8.5 is no longer available,
there's just the CentOS 8 Streams version and I wasn't sure how it would go
with the old style of CentOS. To be honest it didn't occur to me to go with
another flavour of Linux, I just figured that I'd use CentOS to repair
CentOS.
Anyway, I did try using "mdadm /dev/md2 -r detached" and "mdadm /dev/md2 -r
failed" to remove the removed disk to no avail. I ended up using
"mdadm --grow /dev/md125 --array-size
218789036032 --backup-file=/mnt/sysimage/grow_md125_size_grow.bak --verbose"
followed by "mdadm --grow
/dev/md125 --raid-devices=30 --backup-file=/mnt/sysimage/grow_md125_grow_disks.bak
--verbose" and it seems to be working in that it is reshaping the array
although it is apparently going to take around 16,000 minutes (would that be
because we've about 200TB of data?).
My concern now is whether or not I'll still have the mount issue once it
finally completes the reshape. If it does mount OK, does that mean I'm good
to reboot it?
With regards to your comment about downloading lsdrv, I'll try and do that
although I'm having trouble configuring my DNS servers in the running rescue
disk OS. I could run lsblk but, from what I see of lsdrv, lsblk doesn't have
the detail that lsdrv has. I'll keep working on that and let you know what I
get - it looks like I have to edit it to use the older version of Python
that this installation has.
Cheers,
Bob
-----Original Message-----
From: Wols Lists <antlists@youngman.org.uk>
Sent: Monday, 9 May 2022 4:52 PM
To: Bob Brand <brand@wmawater.com.au>; linux-raid@vger.kernel.org
Cc: Phil Turmel <philip@turmel.org>; NeilBrown <neilb@suse.com>
Subject: Re: Failed adadm RAID array after aborted Grown operation
On 09/05/2022 01:09, Bob Brand wrote:
> Hi Wol,
>
> My apologies for continually bothering you but I have a couple of
> questions:
Did you read the links I sent you?
>
> 1. How do I overcome the error message "mount: /dev/md125: can't read
> superblock." Do it use fsck?
>
> 2. The removed disk is showing as " - 0 0 30 removed". Is it
> safe
> to use "mdadm /dev/md2 -r detached" or "mdadm /dev/md2 -r failed" to
> overcome this?
I don't know :-( This is getting a bit out of my depth. But I'm SERIOUSLY
concerned you're still futzing about with CentOS 7!!!
Why didn't you download CentOS 8.5? Why didn't you download RHEL 8.5, or the
latest Fedora? Why didn't you download SUSE SLES 15?
Any and all CentOS 7 will come with either an out-of-date mdadm, or a
Frankenkernel. NEITHER are a good idea.
Go back to the links I gave you, download and run lsdrv, and post the output
here. Hopefully somebody will tell you the next steps. I will do my best.
>
> Thank you!
>
Cheers,
Wol
>
> -----Original Message-----
> From: Bob Brand <brand@wmawater.com.au>
> Sent: Monday, 9 May 2022 9:33 AM
> To: Bob Brand <brand@wmawater.com.au>; Wol <antlists@youngman.org.uk>;
> linux-raid@vger.kernel.org
> Cc: Phil Turmel <philip@turmel.org>
> Subject: RE: Failed adadm RAID array after aborted Grown operation
>
> I just tried it again with the --invalid_backup switch and it's now
> showing the State as "clean, degraded".and it's showing all the disks
> except for the suspect one that I removed.
>
> I'm unable to mount it and see the contents. I get the error "mount:
> /dev/md125: can't read superblock."
>
> Is there more that I need to do?
>
> Thanks
>
>
> -----Original Message-----
> From: Bob Brand <brand@wmawater.com.au>
> Sent: Monday, 9 May 2022 9:02 AM
> To: Bob Brand <brand@wmawater.com.au>; Wol <antlists@youngman.org.uk>;
> linux-raid@vger.kernel.org
> Cc: Phil Turmel <philip@turmel.org>
> Subject: RE: Failed adadm RAID array after aborted Grown operation
>
> Hi Wol,
>
> I've booted to the installation media and I've run the following command:
>
> mdadm
> /dev/md125 --assemble --update=revert-reshape --backup-file=/mnt/sysimage/grow_md125.bak
> --verbose --uuid= f9b65f55:5f257add:1140ccc0:46ca6c19
> /dev/md125mdadm --assemble --update=revert-reshape --backup-file=/grow_md125.bak
> --verbose --uuid=f9b65f55:5f257add:1140ccc0:46ca6c19
>
> But I'm still getting the error:
>
> mdadm: /dev/md125 has an active reshape - checking if critical section
> needs to be restored
> mdadm: No backup metadata on /mnt/sysimage/grow_md125.back
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
>
>
> Should I try the --invalid_backup switch or --force?
>
> Thanks,
> Bob
>
>
> -----Original Message-----
> From: Bob Brand <brand@wmawater.com.au>
> Sent: Monday, 9 May 2022 8:19 AM
> To: Wol <antlists@youngman.org.uk>; linux-raid@vger.kernel.org
> Cc: Phil Turmel <philip@turmel.org>
> Subject: RE: Failed adadm RAID array after aborted Grown operation
>
> OK. I've downloaded a Centos 7 - 2009 ISO from centos.org - that
> seems to be the most recent they have.
>
>
> -----Original Message-----
> From: Wol <antlists@youngman.org.uk>
> Sent: Monday, 9 May 2022 8:16 AM
> To: Bob Brand <brand@wmawater.com.au>; linux-raid@vger.kernel.org
> Cc: Phil Turmel <philip@turmel.org>
> Subject: Re: Failed adadm RAID array after aborted Grown operation
>
> How old is CentOS 7? With that kernel I guess it's quite old?
>
> Try and get a CentOS 8.5 disk. At the end of the day, the version of
> linux doesn't matter. What you need is an up-to-date rescue disk.
> Distro/whatever is unimportant - what IS important is that you are
> using the latest mdadm, and a kernel that matches.
>
> The problem you have sounds like a long-standing but now-fixed bug. An
> original CentOS disk might be okay (with matched kernel and mdadm),
> but almost certainly has what I consider to be a "dodgy" version of mdadm.
>
> If you can afford the downtime, after you've reverted the reshape, I'd
> try starting it again with the rescue disk. It'll probably run fine.
> Let it complete and then your old CentOS 7 will be fine with it.
>
> Cheers,
> Wol
>
> On 08/05/2022 23:04, Bob Brand wrote:
>> Thank Wol.
>>
>> Should I use a CentOS 7 disk or a CentOS disk?
>>
>> Thanks
>>
>> -----Original Message-----
>> From: Wols Lists <antlists@youngman.org.uk>
>> Sent: Monday, 9 May 2022 1:32 AM
>> To: Bob Brand <brand@wmawater.com.au>; linux-raid@vger.kernel.org
>> Cc: Phil Turmel <philip@turmel.org>
>> Subject: Re: Failed adadm RAID array after aborted Grown operation
>>
>> On 08/05/2022 14:18, Bob Brand wrote:
>>> If you’ve stuck with me and read all this way, thank you and I hope
>>> you can help me.
>>
>> https://raid.wiki.kernel.org/index.php/Linux_Raid
>>
>> Especially
>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrog
>> n
>>
>> What you need to do is revert the reshape. I know what may have
>> happened, and what bothers me is your kernel version, 3.10.
>>
>> The first thing to try is to boot from up-to-date rescue media and
>> see if an mdadm --revert works from there. If it does, your Centos
>> should then bring everything back no problem.
>>
>> (You've currently got what I call a Frankensetup, a very old kernel,
>> a pretty new mdadm, and a whole bunch of patches that does who knows
>> what.
>> You really need a matching kernel and mdadm, and your frankenkernel
>> won't match anything ...)
>>
>> Let us know how that goes ...
>>
>> Cheers,
>> Wol
>>
>>
>>
>> CAUTION!!! This E-mail originated from outside of WMA Water. Do not
>> click links or open attachments unless you recognize the sender and
>> know the content is safe.
>>
>>
>
>
>
> CAUTION!!! This E-mail originated from outside of WMA Water. Do not
> click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
>
CAUTION!!! This E-mail originated from outside of WMA Water. Do not click
links or open attachments unless you recognize the sender and know the
content is safe.
next prev parent reply other threads:[~2022-05-09 13:07 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-08 13:18 Failed adadm RAID array after aborted Grown operation Bob Brand
2022-05-08 15:32 ` Wols Lists
2022-05-08 22:04 ` Bob Brand
2022-05-08 22:15 ` Wol
2022-05-08 22:19 ` Bob Brand
2022-05-08 23:02 ` Bob Brand
2022-05-08 23:32 ` Bob Brand
2022-05-09 0:09 ` Bob Brand
2022-05-09 6:52 ` Wols Lists
2022-05-09 13:07 ` Bob Brand [this message]
[not found] ` <CAAMCDecTb69YY+jGzq9HVqx4xZmdVGiRa54BD55Amcz5yaZo1Q@mail.gmail.com>
2022-05-11 5:39 ` Bob Brand
2022-05-11 12:35 ` Reindl Harald
2022-05-11 13:22 ` Bob Brand
2022-05-11 14:56 ` Reindl Harald
2022-05-11 14:59 ` Reindl Harald
2022-05-13 5:32 ` Bob Brand
2022-05-13 8:18 ` Reindl Harald
2022-05-20 15:13 ` Bob Brand
2022-05-20 15:41 ` Reindl Harald
2022-05-22 4:13 ` Bob Brand
2022-05-22 11:25 ` Reindl Harald
2022-05-22 13:31 ` Wols Lists
2022-05-22 22:54 ` Bob Brand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='002601d863a5$bd9f5880$38de0980$@wmawater.com.au' \
--to=brand@wmawater.com.au \
--cc=antlists@youngman.org.uk \
--cc=linux-raid@vger.kernel.org \
--cc=neilb@suse.com \
--cc=philip@turmel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.