* Self inflicted reshape catastrophe
@ 2021-01-14 22:57 Nathan Brown
2021-01-15 16:21 ` antlists
2021-01-18 0:49 ` NeilBrown
0 siblings, 2 replies; 6+ messages in thread
From: Nathan Brown @ 2021-01-14 22:57 UTC (permalink / raw)
To: linux-raid
Scenario:
I had a 4 by 10TB raid 5 array and was adding 5 more disks and
reshaping it to a raid6. This was working just fine until I got a
little too aggressive with perf tuning and caused `mdadm` to
completely hang. I froze the rebuild and rebooted the server to wipe
away my tuning mess. The raid didn't automatically assemble so I did
`mdadm --assemble` but really screwed up and put the 5 new disks in a
different array. Not sure why the superblock on those disks didn't
stop `mdadm` from putting them into service but the end result was the
superblock on those 5 new drives got wiped. That array was missing a
disk so 4 went into spare and 1 went into service, I let that rebuild
complete as I figure I'd likely already lost any usable data there.
I now have 4 disks with proper looking superblocks, 4 disks with
garbage superblocks, and 1 disk sitting in an array that it shouldn't
be in. My primary concern is on assembling the 10TB disk array.
What I've done so far:
All this is done with an overlay to avoid modifying the disks any further.
`mdadm --assemble` if I provide all disks, it will refuse to start as
it hits the first of the new drives "superblock on ... doesn't match
others", `--force` has no effect. `--update=revert-reshape` changes
the `--examine` details but nothing happens since the other 5 drives
are absent.
`mdadm --assemble` again with all disks but the new disks super blocks
have been zero'd. Refuses once it hits the first of the new disks "No
super block found on ...", `--force` has no effect.
`mdadm --assemble` using only the 4 original disks, the md dev shows
up now but can't start. If I try to add any of the new disks I get
"Cannot get array info for /dev/md#", `--force` and super block
zeroing has no effect.
`mdadm --create` using all permutations of the new drives, I believe
know the order of the old ones. A handful of the 120 different
arrangements will allow me to see some of the files, but I do not know
how to move the reshape along in this state. Please note that 1 disk
position is using `missing`.
I believe my next best bet is to try and create an appropriate super
block and write them to each of the new disks to see if it will
assemble and continue the reshape. I wanted to get this lists
suggestions before I went down that path.
Thank you for your time.
Details:
`mdadm --version`
mdadm - v4.1 - 2018-10-01
`lsb_release -a`
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
`uname -a`
Linux nas2 5.4.0-60-generic #67-Ubuntu SMP Tue Jan 5 18:31:36 UTC 2021
x86_64 x86_64 x86_64 GNU/Linux
`mdadm -E /dev/sdk1`
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:0
Creation Time : Mon Dec 23 22:56:41 2019
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=0 sectors
State : clean
Device UUID : a247b8d7:6abdf354:8ca03a82:8681cf54
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
Delta Devices : 4 (5->9)
New Layout : left-symmetric
Update Time : Thu Jan 14 02:02:24 2021
Bad Block Log : 512 entries available at offset 48 sectors
Checksum : 4229db98 - correct
Events : 146894
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA..A.A ('A' == active, '.' == missing, 'R' == replacing)
mdadm -E /dev/sdj1
`/dev/sdj1:`
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:0
Creation Time : Mon Dec 23 22:56:41 2019
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=0 sectors
State : clean
Device UUID : 218773e0:f097e26a:10eb2032:8b0c5f2a
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
Delta Devices : 4 (5->9)
New Layout : left-symmetric
Update Time : Thu Jan 14 02:02:24 2021
Bad Block Log : 512 entries available at offset 48 sectors
Checksum : e64ccb33 - correct
Events : 146894
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
`mdadm -E /dev/sdh1`
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:0
Creation Time : Mon Dec 23 22:56:41 2019
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=0 sectors
State : clean
Device UUID : e8062d92:654dc1e0:4e28b361:eb97ccc2
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
Delta Devices : 4 (5->9)
New Layout : left-symmetric
Update Time : Thu Jan 14 02:02:24 2021
Bad Block Log : 512 entries available at offset 48 sectors
Checksum : d5e4c90f - correct
Events : 146894
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
`mdadm -E /dev/sdi1`
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:0
Creation Time : Mon Dec 23 22:56:41 2019
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=0 sectors
State : clean
Device UUID : f0612be8:dcf9d96b:1926ce52:484d9ab2
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
Delta Devices : 4 (5->9)
New Layout : left-symmetric
Update Time : Thu Jan 14 02:02:24 2021
Bad Block Log : 512 entries available at offset 48 sectors
Checksum : 97e483b8 - correct
Events : 146894
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
If assembled with only the 4 disks with appropriate super blocks I get
`mdadm --detail /dev/md0`
/dev/md0:
Version : 1.2
Raid Level : raid0
Total Devices : 4
Persistence : Superblock is persistent
State : inactive
Working Devices : 4
Delta Devices : 4, (-4->0)
New Level : raid6
New Layout : left-symmetric
New Chunksize : 512K
Name : any:0
UUID : a6914f4a:14a64337:c3546c24:42930ff9
Events : 146894
Number Major Minor RaidDevice
- 253 7 - /dev/dm-7
- 253 5 - /dev/dm-5
- 253 6 - /dev/dm-6
- 253 4 - /dev/dm-4
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Self inflicted reshape catastrophe
2021-01-14 22:57 Self inflicted reshape catastrophe Nathan Brown
@ 2021-01-15 16:21 ` antlists
2021-01-18 0:49 ` NeilBrown
1 sibling, 0 replies; 6+ messages in thread
From: antlists @ 2021-01-15 16:21 UTC (permalink / raw)
To: Nathan Brown, linux-raid, Phil Turmel, NeilBrown
On 14/01/2021 22:57, Nathan Brown wrote:
> Scenario:
>
> I had a 4 by 10TB raid 5 array and was adding 5 more disks and
> reshaping it to a raid6. This was working just fine until I got a
> little too aggressive with perf tuning and caused `mdadm` to
> completely hang. I froze the rebuild and rebooted the server to wipe
> away my tuning mess. The raid didn't automatically assemble so I did
> `mdadm --assemble` but really screwed up and put the 5 new disks in a
> different array. Not sure why the superblock on those disks didn't
> stop `mdadm` from putting them into service but the end result was the
> superblock on those 5 new drives got wiped. That array was missing a
> disk so 4 went into spare and 1 went into service, I let that rebuild
> complete as I figure I'd likely already lost any usable data there.
Okay. Trying to make sense of what you're saying ... you were trying to
convert it to a 9-disk raid-6?
Can you remember what you did? The more you can tell us, in detail
terms, the better, but if you've crashed in the middle of a reshape and
lost the superblocks, then the omens are not good.
That said, I think we might have succeeded in reconstructing a few arrays...
>
> I now have 4 disks with proper looking superblocks, 4 disks with
> garbage superblocks, and 1 disk sitting in an array that it shouldn't
> be in. My primary concern is on assembling the 10TB disk array.
Is this the original four disks?
>
> What I've done so far:
>
> All this is done with an overlay to avoid modifying the disks any further.
>
> `mdadm --assemble` if I provide all disks, it will refuse to start as
> it hits the first of the new drives "superblock on ... doesn't match
> others", `--force` has no effect. `--update=revert-reshape` changes
> the `--examine` details but nothing happens since the other 5 drives
> are absent.
>
> `mdadm --assemble` again with all disks but the new disks super blocks
> have been zero'd. Refuses once it hits the first of the new disks "No
> super block found on ...", `--force` has no effect.
>
> `mdadm --assemble` using only the 4 original disks, the md dev shows
> up now but can't start. If I try to add any of the new disks I get
> "Cannot get array info for /dev/md#", `--force` and super block
> zeroing has no effect.
>
> `mdadm --create` using all permutations of the new drives, I believe
> know the order of the old ones. A handful of the 120 different
> arrangements will allow me to see some of the files, but I do not know
> how to move the reshape along in this state. Please note that 1 disk
> position is using `missing`.
>
> I believe my next best bet is to try and create an appropriate super
> block and write them to each of the new disks to see if it will
> assemble and continue the reshape. I wanted to get this lists
> suggestions before I went down that path.
>
> Thank you for your time.
>
> Details:
>
> `mdadm --version`
> mdadm - v4.1 - 2018-10-01
>
> `lsb_release -a`
> Distributor ID: Ubuntu
> Description: Ubuntu 20.04.1 LTS
> Release: 20.04
> Codename: focal
>
> `uname -a`
> Linux nas2 5.4.0-60-generic #67-Ubuntu SMP Tue Jan 5 18:31:36 UTC 2021
> x86_64 x86_64 x86_64 GNU/Linux
>
> `mdadm -E /dev/sdk1`
> /dev/sdk1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x5
> Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
> Name : any:0
> Creation Time : Mon Dec 23 22:56:41 2019
> Raid Level : raid6
> Raid Devices : 9
> Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
> Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
> Data Offset : 264192 sectors
> Super Offset : 8 sectors
> Unused Space : before=264112 sectors, after=0 sectors
> State : clean
> Device UUID : a247b8d7:6abdf354:8ca03a82:8681cf54
> Internal Bitmap : 8 sectors from superblock
> Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
> Delta Devices : 4 (5->9)
> New Layout : left-symmetric
> Update Time : Thu Jan 14 02:02:24 2021
> Bad Block Log : 512 entries available at offset 48 sectors
> Checksum : 4229db98 - correct
> Events : 146894
> Layout : left-symmetric-6
> Chunk Size : 512K
> Device Role : Active device 0
> Array State : AAAA..A.A ('A' == active, '.' == missing, 'R' == replacing)
>
> mdadm -E /dev/sdj1
> `/dev/sdj1:`
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x5
> Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
> Name : any:0
> Creation Time : Mon Dec 23 22:56:41 2019
> Raid Level : raid6
> Raid Devices : 9
> Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
> Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
> Data Offset : 264192 sectors
> Super Offset : 8 sectors
> Unused Space : before=264112 sectors, after=0 sectors
> State : clean
> Device UUID : 218773e0:f097e26a:10eb2032:8b0c5f2a
> Internal Bitmap : 8 sectors from superblock
> Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
> Delta Devices : 4 (5->9)
> New Layout : left-symmetric
> Update Time : Thu Jan 14 02:02:24 2021
> Bad Block Log : 512 entries available at offset 48 sectors
> Checksum : e64ccb33 - correct
> Events : 146894
> Layout : left-symmetric-6
> Chunk Size : 512K
> Device Role : Active device 1
> Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> `mdadm -E /dev/sdh1`
> /dev/sdh1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x5
> Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
> Name : any:0
> Creation Time : Mon Dec 23 22:56:41 2019
> Raid Level : raid6
> Raid Devices : 9
> Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
> Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
> Data Offset : 264192 sectors
> Super Offset : 8 sectors
> Unused Space : before=264112 sectors, after=0 sectors
> State : clean
> Device UUID : e8062d92:654dc1e0:4e28b361:eb97ccc2
> Internal Bitmap : 8 sectors from superblock
> Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
> Delta Devices : 4 (5->9)
> New Layout : left-symmetric
> Update Time : Thu Jan 14 02:02:24 2021
> Bad Block Log : 512 entries available at offset 48 sectors
> Checksum : d5e4c90f - correct
> Events : 146894
> Layout : left-symmetric-6
> Chunk Size : 512K
> Device Role : Active device 2
> Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> `mdadm -E /dev/sdi1`
> /dev/sdi1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x5
> Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
> Name : any:0
> Creation Time : Mon Dec 23 22:56:41 2019
> Raid Level : raid6
> Raid Devices : 9
> Avail Dev Size : 19532605440 (9313.87 GiB 10000.69 GB)
> Array Size : 68364119040 (65197.10 GiB 70004.86 GB)
> Data Offset : 264192 sectors
> Super Offset : 8 sectors
> Unused Space : before=264112 sectors, after=0 sectors
> State : clean
> Device UUID : f0612be8:dcf9d96b:1926ce52:484d9ab2
> Internal Bitmap : 8 sectors from superblock
> Reshape pos'n : 1922360832 (1833.31 GiB 1968.50 GB)
> Delta Devices : 4 (5->9)
> New Layout : left-symmetric
> Update Time : Thu Jan 14 02:02:24 2021
> Bad Block Log : 512 entries available at offset 48 sectors
> Checksum : 97e483b8 - correct
> Events : 146894
> Layout : left-symmetric-6
> Chunk Size : 512K
> Device Role : Active device 3
> Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> If assembled with only the 4 disks with appropriate super blocks I get
> `mdadm --detail /dev/md0`
> /dev/md0:
> Version : 1.2
> Raid Level : raid0
> Total Devices : 4
> Persistence : Superblock is persistent
> State : inactive
> Working Devices : 4
> Delta Devices : 4, (-4->0)
> New Level : raid6
> New Layout : left-symmetric
> New Chunksize : 512K
> Name : any:0
> UUID : a6914f4a:14a64337:c3546c24:42930ff9
> Events : 146894
> Number Major Minor RaidDevice
> - 253 7 - /dev/dm-7
> - 253 5 - /dev/dm-5
> - 253 6 - /dev/dm-6
> - 253 4 - /dev/dm-4
>
Have you looked at
https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
You've given us a pretty good problem report, but lsdrv (over all 9
drives) would be a help, and can you give us a brief smartctl report
over the drives? That probably won't tell us anything, but you never know...
I've added Phil and Neil to the "to" line because I'm out of my depth
here. They know a lot more than I do so hopefully they'll step in and help.
Cheers,
Wol
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Self inflicted reshape catastrophe
2021-01-14 22:57 Self inflicted reshape catastrophe Nathan Brown
2021-01-15 16:21 ` antlists
@ 2021-01-18 0:49 ` NeilBrown
2021-01-18 3:12 ` Nathan Brown
1 sibling, 1 reply; 6+ messages in thread
From: NeilBrown @ 2021-01-18 0:49 UTC (permalink / raw)
To: Nathan Brown, linux-raid
[-- Attachment #1: Type: text/plain, Size: 873 bytes --]
On Thu, Jan 14 2021, Nathan Brown wrote:
> Scenario:
>
> I had a 4 by 10TB raid 5 array and was adding 5 more disks and
> reshaping it to a raid6. This was working just fine until I got a
> little too aggressive with perf tuning and caused `mdadm` to
> completely hang. I froze the rebuild and rebooted the server to wipe
> away my tuning mess. The raid didn't automatically assemble so I did
> `mdadm --assemble` but really screwed up and put the 5 new disks in a
> different array. Not sure why the superblock on those disks didn't
> stop `mdadm` from putting them into service but the end result was the
> superblock on those 5 new drives got wiped. That array was missing a
What does this mean "the superblock on those 5 new drives got wiped"?
How do you think that happened.
Can you please report the current super blocks (mdadm --examine) anyway
please.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Self inflicted reshape catastrophe
2021-01-18 0:49 ` NeilBrown
@ 2021-01-18 3:12 ` Nathan Brown
2021-01-18 22:36 ` NeilBrown
0 siblings, 1 reply; 6+ messages in thread
From: Nathan Brown @ 2021-01-18 3:12 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
It was this part of the original post
> The raid didn't automatically assemble so I did
> `mdadm --assemble` but really screwed up and put the 5 new disks in a
> different array
Basically, I did an `mdadm --assemble /dev/md1 <new disks for md0>`
instead of `mdadm --assemble /dev/md0 <new disks for md0>`. Further
complicated by the fact the md1 was missing a disk, so I let 1 of the
5 disks become a full member md1 since I didn't catch my error in time
and enough recovery on md1 had occurred to wipe out any data transfer
from the reshape on md0. The other 4 became hot spares. This wiped the
super block on those 5 new disks, the super blocks no longer contain
the correct information showing the original reshape attempt on md0.
I have yet to dive into the code but it seems likely that I can
manually reconstruct the appropriate super blocks for these 4 disks
that still contain valid data as a result of the reshape with a worst
case of ~1/5th data loss.
Here are the `mdadm --examine` for the 4 remaining drives destined for
md0, they believe they belong to md1 (a 6 x 4tb raid 5)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : 2e3e6281:d6735073:c92ffd33:cb6d1413
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 03:18:04 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : e87847d5 - correct
Events : 55628
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : 268fa8cf:0299a907:5f76dd12:94f78cf2
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 03:18:03 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 89708e55 - correct
Events : 55627
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : d8fa02f7:8dd22cea:06d0de1d:32d4039d
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 03:37:53 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 48c66f6b - correct
Events : 55861
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : f464edaa:6338d020:7541f151:678234d2
Name : big-nas:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : f6c7b906:7aa7bbe7:aec61998:d3183973
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 13:04:45 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 24b2884e - correct
Events : 62212
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
The 5th and final 10tb disk I allowed to become the 6th member of the
6 x 4tb raid 5 after my screwup
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : f464edaa:6338d020:7541f151:678234d2
Name : big-nas:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : 527db0c9:1636b7c4:06afabaf:35d6db73
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Jan 16 06:11:30 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : dcdbb6cf - correct
Events : 62885
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
On Sun, Jan 17, 2021 at 6:49 PM NeilBrown <neil@brown.name> wrote:
>
> On Thu, Jan 14 2021, Nathan Brown wrote:
>
> > Scenario:
> >
> > I had a 4 by 10TB raid 5 array and was adding 5 more disks and
> > reshaping it to a raid6. This was working just fine until I got a
> > little too aggressive with perf tuning and caused `mdadm` to
> > completely hang. I froze the rebuild and rebooted the server to wipe
> > away my tuning mess. The raid didn't automatically assemble so I did
> > `mdadm --assemble` but really screwed up and put the 5 new disks in a
> > different array. Not sure why the superblock on those disks didn't
> > stop `mdadm` from putting them into service but the end result was the
> > superblock on those 5 new drives got wiped. That array was missing a
>
> What does this mean "the superblock on those 5 new drives got wiped"?
> How do you think that happened.
> Can you please report the current super blocks (mdadm --examine) anyway
> please.
>
> NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Self inflicted reshape catastrophe
2021-01-18 3:12 ` Nathan Brown
@ 2021-01-18 22:36 ` NeilBrown
2021-01-19 0:09 ` Nathan Brown
0 siblings, 1 reply; 6+ messages in thread
From: NeilBrown @ 2021-01-18 22:36 UTC (permalink / raw)
To: Nathan Brown; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 2498 bytes --]
On Sun, Jan 17 2021, Nathan Brown wrote:
> It was this part of the original post
>
>> The raid didn't automatically assemble so I did
>> `mdadm --assemble` but really screwed up and put the 5 new disks in a
>> different array
>
> Basically, I did an `mdadm --assemble /dev/md1 <new disks for md0>`
That command wouldn't have the effect you describe (and is visible in
the --examine output - thanks).
Maybe you mean "--add" ???
> instead of `mdadm --assemble /dev/md0 <new disks for md0>`. Further
> complicated by the fact the md1 was missing a disk, so I let 1 of the
> 5 disks become a full member md1 since I didn't catch my error in time
> and enough recovery on md1 had occurred to wipe out any data transfer
> from the reshape on md0. The other 4 became hot spares. This wiped the
> super block on those 5 new disks, the super blocks no longer contain
> the correct information showing the original reshape attempt on md0.
>
> I have yet to dive into the code but it seems likely that I can
> manually reconstruct the appropriate super blocks for these 4 disks
> that still contain valid data as a result of the reshape with a worst
> case of ~1/5th data loss.
There will be fs-metadata loss as well as data loss, and that is the real
killer.
Yes the data is probably still on those "spare" devices. Probably just
the md-metadata is lost. The data that was on sdo1 is now lost, but
RAID6 protects you from losing one device, so that doesn't matter.
To reconstruct the correct metadata, the easiest approach is probably to
copy the superblock from the best drive in md0 and use a binary-editor
to change the 'Device Role' field to an appropriate number for each
different device. Maybe your kernel logs will have enough info to
confirm which device was in each role.
One approach to copying the metadata is to use "mdadm --dump=/tmp/md0 /dev/md0"
which should create sparse files in /tmp/md0 with the metadata from each
device.
Then binary-edit those files, and rename them. Then use
mdadm --restore=/tmp/md0 /dev/md0
to copy the metadata back. Maybe.
Then use "mdadm --examine --super=1.2" to check that the superblock
looks OK and to find out what the "expected" checksum is. Then edit the
superblock again to set the checksum.
Then try assembling the array with
mdadm --assemble --freeze-reshape --readonly ....
which should minimize the damage that can be done if something isn't
right.
Then try "fsck -n" the filesystem to see if it looks OK.
Good luck
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Self inflicted reshape catastrophe
2021-01-18 22:36 ` NeilBrown
@ 2021-01-19 0:09 ` Nathan Brown
0 siblings, 0 replies; 6+ messages in thread
From: Nathan Brown @ 2021-01-19 0:09 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
> That command wouldn't have the effect you describe (and is visible in
> the --examine output - thanks).
>
> Maybe you mean "--add" ???
I feel like I did `--assemble` but it is entirely possible I am not
remembering correctly and did an `--add`. It may be worth testing both
scenarios to see if `mdadm` could stop people like me from shooting
themselves in the foot. To be clear I am not placing any blame on
`mdadm` for my stupid mistakes.
> To reconstruct the correct metadata, the easiest approach is probably to
> copy the superblock from the best drive in md0 and use a binary-editor
> to change the 'Device Role' field to an appropriate number for each
> different device. Maybe your kernel logs will have enough info to
> confirm which device was in each role.
I checked all my logs and all I can see is the start of the reshape
with no indication of what is changing.
Jan 14 00:20:46 nas2 kernel: md: reshape of RAID array md0
> One approach to copying the metadata is to use "mdadm --dump=/tmp/md0 /dev/md0"
> which should create sparse files in /tmp/md0 with the metadata from each
> device.
> Then binary-edit those files, and rename them. Then use
> mdadm --restore=/tmp/md0 /dev/md0
> to copy the metadata back. Maybe.
>
> Then use "mdadm --examine --super=1.2" to check that the superblock
> looks OK and to find out what the "expected" checksum is. Then edit the
> superblock again to set the checksum.
>
> Then try assembling the array with
> mdadm --assemble --freeze-reshape --readonly ....
>
> which should minimize the damage that can be done if something isn't
> right.
> Then try "fsck -n" the filesystem to see if it looks OK.
Since I do not know the disk order I'll work up a script to use
overlays and craft these super blocks in each permutation with `fsck`
checks to see which one gets me the most data back.
I truly appreciate the help, thank you very much. I'll let you all
know how it goes :)
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-01-19 0:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-14 22:57 Self inflicted reshape catastrophe Nathan Brown
2021-01-15 16:21 ` antlists
2021-01-18 0:49 ` NeilBrown
2021-01-18 3:12 ` Nathan Brown
2021-01-18 22:36 ` NeilBrown
2021-01-19 0:09 ` Nathan Brown
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.