* Recovering an Array with inconsistent Superblocks
@ 2014-01-04 10:04 Fabian Knorr
2014-01-04 16:24 ` Phil Turmel
0 siblings, 1 reply; 15+ messages in thread
From: Fabian Knorr @ 2014-01-04 10:04 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]
Good morning folks,
I have a MD-RAID5 with 8 disks, 7 of them active, 1 spare. They are
connected to two SATA controllers, 4 disks each.
A few days ago, the disks connected to one of the controllers stopped
operating (some sort of controller hiccup, I guess). Now those disks are
marked as "possibly outdated" and the array refuses to assemble or
start, telling me there are only 4 out of 7 devices operational (See
attachment "assemble.status")
On boot, "/proc/mdstat" reports an inactive array with 7 spares, trying
to "mdadm --run" the array fails with the message mentioned above,
changing "/proc/mdstat" to now show an array of 4 disks.
I tried "--add" with a missing device, getting the message "re-added
device /dev/sd...", but failing for subsequent devices with the message
"/dev/md0 failed to start, not adding device ..., You should stop and
re-assemble the array".
Then I tried "--assemble --scan --force", which yielded the same result
as above.
The next thing I would try is recreating the array with the layout
stored in the superblocks, but I was surprised to find it to be
inconsistent between the disks. I attached the output of "--examine
--verbose" as "raid.status".
Could "--add"ing have changed one superblock, and can I safely try to
re-create the array with the layout given by /dev/sda and /dev/sdc?
Also, what parameters would I need to keep the layout (As mentioned in
the wiki at https://raid.wiki.kernel.org/index.php/RAID_Recovery)
consistent with the one I have now?
Thanks for your time!
Fabian
[-- Attachment #2: assemble.status --]
[-- Type: text/plain, Size: 2854 bytes --]
mdadm: looking for devices for /dev/md0
mdadm: /dev/dm-0 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/loop2 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/loop1 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/loop0 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdj2 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdj1 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdj is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdi is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdh is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdg is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdf is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sde is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdd1 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdd is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sda is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdb is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdc is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 5.
mdadm: added /dev/sdg1 to /dev/md0 as 1
mdadm: added /dev/sdh1 to /dev/md0 as 2
mdadm: added /dev/sdi1 to /dev/md0 as 3
mdadm: added /dev/sda1 to /dev/md0 as 4 (possibly out of date)
mdadm: added /dev/sdc1 to /dev/md0 as 5 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md0 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 4 drives - not enough to start the array.
[-- Attachment #3: raid.status --]
[-- Type: text/plain, Size: 9719 bytes --]
/dev/sda1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2a6 - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 4 8 1 4 active sync /dev/sda1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 8
Preferred Minor : 0
Update Time : Sun Oct 27 15:00:43 2013
State : active
Active Devices : 7
Working Devices : 8
Failed Devices : 0
Spare Devices : 1
Checksum : f3d95119 - correct
Events : 24319
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 4 8 17 4 active sync /dev/sdb1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 17 4 active sync /dev/sdb1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
7 7 8 1 7 spare /dev/sda1
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2c8 - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 5 8 33 5 active sync /dev/sdc1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sde1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2da - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 6 8 49 6 active sync /dev/sdd1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sdf1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f43123f2 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 0 8 65 0 active sync /dev/sde1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdg1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312404 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 1 8 81 1 active sync /dev/sdf1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdh1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312416 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 2 8 97 2 active sync /dev/sdg1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdi1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312428 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 3 8 113 3 active sync /dev/sdh1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdj1:
MBR Magic : aa55
Partition[0] : 1069056 sectors at 0 (type 00)
Partition[1] : 63488 sectors at 95576 (type ef)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-04 10:04 Recovering an Array with inconsistent Superblocks Fabian Knorr
@ 2014-01-04 16:24 ` Phil Turmel
2014-01-04 17:59 ` Can Jeuleers
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Phil Turmel @ 2014-01-04 16:24 UTC (permalink / raw)
To: Fabian Knorr, linux-raid
Good morning Fabian,
We might be able to save you here, but it isn't certain.
On 01/04/2014 05:04 AM, Fabian Knorr wrote:
> Good morning folks,
>
> I have a MD-RAID5 with 8 disks, 7 of them active, 1 spare. They are
> connected to two SATA controllers, 4 disks each.
Side note: If you have a live spare available for a raid5, there's no
good reason not to reshape to a raid6, and very good reasons to do so.
> A few days ago, the disks connected to one of the controllers stopped
> operating (some sort of controller hiccup, I guess). Now those disks are
> marked as "possibly outdated" and the array refuses to assemble or
> start, telling me there are only 4 out of 7 devices operational (See
> attachment "assemble.status")
>
> On boot, "/proc/mdstat" reports an inactive array with 7 spares, trying
> to "mdadm --run" the array fails with the message mentioned above,
> changing "/proc/mdstat" to now show an array of 4 disks.
"mdadm --assemble --force" would have fixed you up if you'd done it
right at this point. That's what "--force" is intended for.
> I tried "--add" with a missing device, getting the message "re-added
> device /dev/sd...", but failing for subsequent devices with the message
> "/dev/md0 failed to start, not adding device ..., You should stop and
> re-assemble the array".
Using "--add" changed those devices' superblocks, losing their original
identity in the array. Which is why ...
> Then I tried "--assemble --scan --force", which yielded the same result
> as above.
... this didn't work.
> The next thing I would try is recreating the array with the layout
> stored in the superblocks, but I was surprised to find it to be
> inconsistent between the disks. I attached the output of "--examine
> --verbose" as "raid.status".
Device names are not guaranteed to remain identical from one boot to
another. And often won't be if a removable device is plugged in at that
time. The linux MD driver keeps identity data in the superblock that
makes the actual device names immaterial.
It is really important that we get a "map" of device names to drive
serial numbers, and adjust all future operations to ensure we are
working with the correct names. An excerpt from "ls -l
/dev/disk/by-id/" would do. And you need to verify it after every boot
until this crisis is resolved.
> Could "--add"ing have changed one superblock, and can I safely try to
> re-create the array with the layout given by /dev/sda and /dev/sdc?
> Also, what parameters would I need to keep the layout (As mentioned in
> the wiki at https://raid.wiki.kernel.org/index.php/RAID_Recovery)
> consistent with the one I have now?
Some additional questions/notes before proceeding:
1) raid.status appears to be from *after* your --add attempts. That
means anything in those reports from those devices is useless. So we
will have to figure out what that data was.
2) You attempted to recreate the array. If you left out
"--assume-clean", your data is toast. Please show the precise command
line you used in your re-create attempt. Also generate a fresh
"raid.status" for the current situation.
3) The array seems to think it's member devices were /dev/sda through
/dev/sdh (not in that order). Your "raid.status" has /dev/sd[abcefghi],
suggesting a rescue usb or some such is /dev/sdd. So device names have
to be re-interpreted between old metadata and recovery attempts.
4) Please describe the structure of the *content* of the array, so we
can suggest strategies to *safely* recognize when our future attempts to
--create --assume-clean have succeeded. LVM? Partitioned? One big
filesystem?
Phil
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-04 16:24 ` Phil Turmel
@ 2014-01-04 17:59 ` Can Jeuleers
2014-01-04 19:16 ` Phil Turmel
2014-01-04 22:05 ` Fabian Knorr
2014-01-04 22:08 ` Fabian Knorr
2 siblings, 1 reply; 15+ messages in thread
From: Can Jeuleers @ 2014-01-04 17:59 UTC (permalink / raw)
To: linux-raid
On 01/04/2014 05:24 PM, Phil Turmel wrote:
> Side note: If you have a live spare available for a raid5, there's no
> good reason not to reshape to a raid6, and very good reasons to do so.
I used to have a setup like that (i.e. RAID5 + spare), and my reason was
power consumption and heat dissipation, since the spare was spun down.
Jan
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-04 17:59 ` Can Jeuleers
@ 2014-01-04 19:16 ` Phil Turmel
0 siblings, 0 replies; 15+ messages in thread
From: Phil Turmel @ 2014-01-04 19:16 UTC (permalink / raw)
To: Can Jeuleers, linux-raid
On 01/04/2014 12:59 PM, Can Jeuleers wrote:
> On 01/04/2014 05:24 PM, Phil Turmel wrote:
>> Side note: If you have a live spare available for a raid5, there's no
>> good reason not to reshape to a raid6, and very good reasons to do so.
>
> I used to have a setup like that (i.e. RAID5 + spare), and my reason was
> power consumption and heat dissipation, since the spare was spun down.
Ok. I can understand that. It wouldn't meet *my* definition of a
"good" reason, though.
Phil
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-04 16:24 ` Phil Turmel
2014-01-04 17:59 ` Can Jeuleers
@ 2014-01-04 22:05 ` Fabian Knorr
2014-01-05 2:32 ` Phil Turmel
2014-01-14 8:54 ` David Brown
2014-01-04 22:08 ` Fabian Knorr
2 siblings, 2 replies; 15+ messages in thread
From: Fabian Knorr @ 2014-01-04 22:05 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]
Hi, Phil,
thank you very much for your reply.
> Side note: If you have a live spare available for a raid5, there's no
> good reason not to reshape to a raid6, and very good reasons to do so.
I was worried that RAID6 would incur a significant load on the CPU,
especially if one disk fails. The system is a single-core Intel Atom.
> Device names are not guaranteed to remain identical from one boot to
> another. And often won't be if a removable device is plugged in at that
> time. The linux MD driver keeps identity data in the superblock that
> makes the actual device names immaterial.
>
> It is really important that we get a "map" of device names to drive
> serial numbers, and adjust all future operations to ensure we are
> working with the correct names. An excerpt from "ls -l
> /dev/disk/by-id/" would do. And you need to verify it after every boot
> until this crisis is resolved.
See the attachment "partitions". I grep'ed for raid partitions.
> 1) raid.status appears to be from *after* your --add attempts. That
> means anything in those reports from those devices is useless. So we
> will have to figure out what that data was.
Could it be that --add only changed the superblock of one disk,
namely /dev/sdb in file from my first e-mail?
> 2) You attempted to recreate the array. If you left out
> "--assume-clean", your data is toast. Please show the precise command
> line you used in your re-create attempt. Also generate a fresh
> "raid.status" for the current situation.
The only commands I used were --add /dev/sdb, --run, --assemble --scan,
--assemble --scan --force and --stop. I didn't try to re-create it, at
least not now. Also, the timestamp from raid.status (2011) is incorrect,
the array was re-created from scratch in the summer of 2012. I can't
tell why disks other than /dev/sdb1 have an invalid superblock.
> 3) The array seems to think it's member devices were /dev/sda through
> /dev/sdh (not in that order). Your "raid.status" has /dev/sd[abcefghi],
> suggesting a rescue usb or some such is /dev/sdd.
Yes, that's correct.
> 4) Please describe the structure of the *content* of the array, so we
> can suggest strategies to *safely* recognize when our future attempts to
> --create --assume-clean have succeeded. LVM? Partitioned? One big
> filesystem?
I'm using the array as a physical volume for LVM.
Thanks for your help.
Fabian
[-- Attachment #2: uuids --]
[-- Type: text/plain, Size: 832 bytes --]
/dev/sda1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000549c5-01"
/dev/sdb1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="0004ab8d-01"
/dev/sdc1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="0003ba2a-01"
/dev/sde1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000f04e8-01"
/dev/sdf1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="0002f1df-01"
/dev/sdg1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000d882d-01"
/dev/sdh1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000658a7-01"
/dev/sdi1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000d8fc8-01"
[-- Attachment #3: raid.status --]
[-- Type: text/plain, Size: 10732 bytes --]
/dev/sda:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sda1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2a6 - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 4 8 1 4 active sync /dev/sda1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 8
Preferred Minor : 0
Update Time : Sun Oct 27 15:00:43 2013
State : active
Active Devices : 7
Working Devices : 8
Failed Devices : 0
Spare Devices : 1
Checksum : f3d95119 - correct
Events : 24319
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 4 8 17 4 active sync /dev/sdb1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 17 4 active sync /dev/sdb1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
7 7 8 1 7 spare /dev/sda1
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2c8 - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 5 8 33 5 active sync /dev/sdc1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sdd:
MBR Magic : aa55
Partition[0] : 1001472 sectors at 2048 (type 83)
/dev/sde:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sde1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2da - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 6 8 49 6 active sync /dev/sdd1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sdf:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sdf1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f43123f2 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 0 8 65 0 active sync /dev/sde1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdg:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sdg1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312404 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 1 8 81 1 active sync /dev/sdf1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdh:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sdh1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312416 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 2 8 97 2 active sync /dev/sdg1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdi:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type 83)
/dev/sdi1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312428 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 3 8 113 3 active sync /dev/sdh1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdj:
MBR Magic : aa55
Partition[0] : 1069056 sectors at 0 (type 00)
Partition[1] : 63488 sectors at 95576 (type ef)
/dev/sdj1:
MBR Magic : aa55
Partition[0] : 1069056 sectors at 0 (type 00)
Partition[1] : 63488 sectors at 95576 (type ef)
/dev/sdj2:
MBR Magic : aa55
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-04 16:24 ` Phil Turmel
2014-01-04 17:59 ` Can Jeuleers
2014-01-04 22:05 ` Fabian Knorr
@ 2014-01-04 22:08 ` Fabian Knorr
2 siblings, 0 replies; 15+ messages in thread
From: Fabian Knorr @ 2014-01-04 22:08 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 79 bytes --]
Hi Phil,
I attached the wrong file in the other reply to your e-mail.
Fabian
[-- Attachment #2: partitions --]
[-- Type: text/plain, Size: 1679 bytes --]
$ ls /dev/disk/by-id | grep ata | grep sd[a-z]1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-SAMSUNG_HD204UI_S2JGJ1BZA02506-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-SAMSUNG_HD204UI_S2JGJ1BZA02507-part1 -> ../../sdh1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-SAMSUNG_HD204UI_S2JGJ1BZA02510-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-SAMSUNG_HD204UI_S2JGJ1BZA02513-part1 -> ../../sdi1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-ST2000DL004_HD204UI_S2H7J90C513540-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-ST2000DL004_HD204UI_S2H7J90C569210-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-ST2000DL004_HD204UI_S2H7JX0CA00102-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 4 21:29 ata-ST2000DL004_HD204UI_S2H7JX0CA00103-part1 -> ../../sdb1
$ blkid /dev/sd* | grep raid
/dev/sda1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000549c5-01"
/dev/sdb1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="0004ab8d-01"
/dev/sdc1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="0003ba2a-01"
/dev/sde1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000f04e8-01"
/dev/sdf1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="0002f1df-01"
/dev/sdg1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000d882d-01"
/dev/sdh1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000658a7-01"
/dev/sdi1: UUID="cb5a6670-1b5d-391f-edd8-d016611227ea" TYPE="linux_raid_member" PARTUUID="000d8fc8-01"
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-04 22:05 ` Fabian Knorr
@ 2014-01-05 2:32 ` Phil Turmel
2014-01-05 9:07 ` Fabian Knorr
2014-01-14 8:54 ` David Brown
1 sibling, 1 reply; 15+ messages in thread
From: Phil Turmel @ 2014-01-05 2:32 UTC (permalink / raw)
To: Fabian Knorr; +Cc: linux-raid
On 01/04/2014 05:05 PM, Fabian Knorr wrote:
> Hi, Phil,
>
> thank you very much for your reply.
>
>> Side note: If you have a live spare available for a raid5, there's no
>> good reason not to reshape to a raid6, and very good reasons to do so.
>
> I was worried that RAID6 would incur a significant load on the CPU,
> especially if one disk fails. The system is a single-core Intel Atom.
It does add more load, especially when degraded. I guess it depends on
your usage pattern. I would try it before I gave up on the idea.
>> Device names are not guaranteed to remain identical from one boot to
>> another. And often won't be if a removable device is plugged in at that
>> time. The linux MD driver keeps identity data in the superblock that
>> makes the actual device names immaterial.
>>
>> It is really important that we get a "map" of device names to drive
>> serial numbers, and adjust all future operations to ensure we are
>> working with the correct names. An excerpt from "ls -l
>> /dev/disk/by-id/" would do. And you need to verify it after every boot
>> until this crisis is resolved.
>
> See the attachment "partitions". I grep'ed for raid partitions.
>
>> 1) raid.status appears to be from *after* your --add attempts. That
>> means anything in those reports from those devices is useless. So we
>> will have to figure out what that data was.
>
> Could it be that --add only changed the superblock of one disk,
> namely /dev/sdb in file from my first e-mail?
/dev/sda actually.
>> 2) You attempted to recreate the array. If you left out
>> "--assume-clean", your data is toast. Please show the precise command
>> line you used in your re-create attempt. Also generate a fresh
>> "raid.status" for the current situation.
>
> The only commands I used were --add /dev/sdb, --run, --assemble --scan,
> --assemble --scan --force and --stop. I didn't try to re-create it, at
> least not now. Also, the timestamp from raid.status (2011) is incorrect,
> the array was re-created from scratch in the summer of 2012. I can't
> tell why disks other than /dev/sdb1 have an invalid superblock.
This is very good news. In fact, I think --assemble --force can still
be made to work....
>> 3) The array seems to think it's member devices were /dev/sda through
>> /dev/sdh (not in that order). Your "raid.status" has /dev/sd[abcefghi],
>> suggesting a rescue usb or some such is /dev/sdd.
>
> Yes, that's correct.
Very good.
>> 4) Please describe the structure of the *content* of the array, so we
>> can suggest strategies to *safely* recognize when our future attempts to
>> --create --assume-clean have succeeded. LVM? Partitioned? One big
>> filesystem?
>
> I'm using the array as a physical volume for LVM.
Ok.
Try this:
mdadm --stop /dev/md0
mdadm -Afv /dev/md0 /dev/sd[bcefghi]1
It leaves out /dev/sda, which appears to have been the spare in the
original setup.
If MD is happy after that, use fsck -n on your logical volumes to verify
your FS integrity, and/or see the extent of the damage (little or none,
I think).
If that works, you can --add /dev/sda1 again, and it will become the
spare again.
If it doesn't work, show everything printed by "mdadm -Afv" above.
HTH,
Phil
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-05 2:32 ` Phil Turmel
@ 2014-01-05 9:07 ` Fabian Knorr
2014-01-05 9:56 ` NeilBrown
0 siblings, 1 reply; 15+ messages in thread
From: Fabian Knorr @ 2014-01-05 9:07 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 674 bytes --]
> Try this:
>
> mdadm --stop /dev/md0
>
> mdadm -Afv /dev/md0 /dev/sd[bcefghi]1
>
> It leaves out /dev/sda, which appears to have been the spare in the
> original setup.
>
> If MD is happy after that, use fsck -n on your logical volumes to verify
> your FS integrity, and/or see the extent of the damage (little or none,
> I think).
This had the same result as simply trying --assemble --scan, see
"assemble.log". I tried using sda1 instead of sdb1 as well as the both
seem to have ID 4 ("assemble2.log"), same thing again.
Do you have any idea why mdadm seems to ignore --force and does not add
the outdated disks? (Or am I misinterpreting --force?)
Thanks.
Fabian
[-- Attachment #2: assemble.log --]
[-- Type: text/x-log, Size: 910 bytes --]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdg1 to /dev/md0 as 1
mdadm: added /dev/sdh1 to /dev/md0 as 2
mdadm: added /dev/sdi1 to /dev/md0 as 3
mdadm: added /dev/sdb1 to /dev/md0 as 4 (possibly out of date)
mdadm: added /dev/sdc1 to /dev/md0 as 5 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md0 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 4 drives - not enough to start the array.
[-- Attachment #3: assemble2.log --]
[-- Type: text/x-log, Size: 910 bytes --]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdg1 to /dev/md0 as 1
mdadm: added /dev/sdh1 to /dev/md0 as 2
mdadm: added /dev/sdi1 to /dev/md0 as 3
mdadm: added /dev/sda1 to /dev/md0 as 4 (possibly out of date)
mdadm: added /dev/sdc1 to /dev/md0 as 5 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md0 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 4 drives - not enough to start the array.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-05 9:07 ` Fabian Knorr
@ 2014-01-05 9:56 ` NeilBrown
2014-01-05 10:40 ` Fabian Knorr
[not found] ` <1388918703.3591.20.camel@vessel>
0 siblings, 2 replies; 15+ messages in thread
From: NeilBrown @ 2014-01-05 9:56 UTC (permalink / raw)
To: Fabian Knorr; +Cc: Phil Turmel, linux-raid
[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]
On Sun, 05 Jan 2014 10:07:50 +0100 Fabian Knorr <knorrfab@fim.uni-passau.de>
wrote:
> > Try this:
> >
> > mdadm --stop /dev/md0
> >
> > mdadm -Afv /dev/md0 /dev/sd[bcefghi]1
> >
> > It leaves out /dev/sda, which appears to have been the spare in the
> > original setup.
> >
> > If MD is happy after that, use fsck -n on your logical volumes to verify
> > your FS integrity, and/or see the extent of the damage (little or none,
> > I think).
>
> This had the same result as simply trying --assemble --scan, see
> "assemble.log". I tried using sda1 instead of sdb1 as well as the both
> seem to have ID 4 ("assemble2.log"), same thing again.
>
> Do you have any idea why mdadm seems to ignore --force and does not add
> the outdated disks? (Or am I misinterpreting --force?)
I haven't read the whole thread, but this sounds like the bug fixed by:
commit f81a2b56c4b437f66aaf5582a9c6b7f5ab2103c4
Author: NeilBrown <neilb@suse.de>
Date: Tue Oct 22 09:55:04 2013 +1100
Assembe: fix bug in force_array - it wasn't forcing properly.
try getting mdadm from git and building that
git clone git://neil.brown.name/mdadm
cd mdadm
make
NeilBrown
>
> Thanks.
>
> Fabian
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-05 9:56 ` NeilBrown
@ 2014-01-05 10:40 ` Fabian Knorr
[not found] ` <1388918703.3591.20.camel@vessel>
1 sibling, 0 replies; 15+ messages in thread
From: Fabian Knorr @ 2014-01-05 10:40 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, linux-raid
> I haven't read the whole thread, but this sounds like the bug fixed by:
>
> commit f81a2b56c4b437f66aaf5582a9c6b7f5ab2103c4
> Author: NeilBrown <neilb@suse.de>
> Date: Tue Oct 22 09:55:04 2013 +1100
>
> Assembe: fix bug in force_array - it wasn't forcing properly.
>
>
> try getting mdadm from git and building that
> git clone git://neil.brown.name/mdadm
> cd mdadm
> make
Thank you Neil, that did it.
With the git version, I was able to re-assemble the array as Phil told
me:
mdadm --assemble /dev/md0 /dev/sd[bcefghi] --force --verbose
mdadm told me that "7 devices are not enough to start the array" (see
assemble.log). /proc/mdstat still showed /dev/md127 instead of md0, now
correctly with 7 drives and r/o (mdstat.log).
Still, I was able to see my LVM volumes after "vgchange -ay" and do a
(read-only) fsck (fsck.log). It seemed to pass flawlessly for the volume
"lvm-storage", which is the partition containing my data. "lvm-root"
failed horribly, but this is just the system partition which I planned
on re-building from scratch anyway.
I'd like to keep the state of lvm-storage as it is and re-install my
system. Still, a few things I'm not sure about:
- How can I get mdadm to initiate resync and get write access
to my data?
- As lvm-root seems to be severly damaged, is there any chance
of errors in the lvm-storage FS that fsck -fn does not
detect? Could re-syncing therefore destroy data I have access
to now?
- Can I make sure that the working setup is written to all
superblocks, and zero the spare's superblock so that there is
no such confusion when trying to assemble the array in the
future?
Thanks.
Fabian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
[not found] ` <1388918703.3591.20.camel@vessel>
@ 2014-01-05 18:25 ` Phil Turmel
2014-01-05 23:50 ` NeilBrown
2014-01-06 14:00 ` Fabian Knorr
0 siblings, 2 replies; 15+ messages in thread
From: Phil Turmel @ 2014-01-05 18:25 UTC (permalink / raw)
To: Fabian Knorr, NeilBrown; +Cc: linux-raid
Hi Fabian,
I see good news to greet me this morning!
On 01/05/2014 05:45 AM, Fabian Knorr wrote:
> $ cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md127 : active (auto-read-only) raid5 sde1[6] sdh1[2] sdi1[3] sdb1[4] sdg1[1] sdc1[5] sdf1[0]
> 11721074688 blocks level 5, 1024k chunk, algorithm 2 [7/7] [UUUUUUU]
> resync=PENDING
Auto-read-only will switch to read-write as soon as you actually write
anything to the array. Which will also kick off the resync.
> unused devices: <none>
>
> $ mdadm --stop /dev/md127
> mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process, mounted filesystem or active volume group?
LVM has the array open--but you don't need to stop it. Just mount your
storage filesystem.
The damaged root filesystem undoubtedly had much info in RAM cache that
couldn't be written after the controller hiccup.
> I'd like to keep the state of lvm-storage as it is and re-install my
> system. Still, a few things I'm not sure about:
>
> - How can I get mdadm to initiate resync and get write access
> to my data?
See above.
> - As lvm-root seems to be severly damaged, is there any chance
> of errors in the lvm-storage FS that fsck -fn does not
> detect? Could re-syncing therefore destroy data I have access
> to now?
Possible, but not any help for it. The file allocation structures are
Ok (that's what fsck can see), but if any contents had been just written
at the time of the crash, parts of those writes might not have reached
the dropped array members.
> - Can I make sure that the working setup is written to all
> superblocks, and zero the spare's superblock so that there is
> no such confusion when trying to assemble the array in the
> future?
Yes, you can use --zero-superblock on the spare before you --add it back
the array. It might save you some warnings.
Long term, consider creating a new array with metadata type 1.x so you
can use a bitmap. If you ever have an event like this, where one or
more devices are disconnected then reconnected, it'll greatly shorten
the recovery time.
You should also learn to use --re-add instead of --add for devices that
are supposed to already be part of an array. Newer versions of mdadm
try to help you with this.
HTH,
Phil
(And thank you too, Neil, for a timely assist!)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-05 18:25 ` Phil Turmel
@ 2014-01-05 23:50 ` NeilBrown
2014-01-06 14:00 ` Fabian Knorr
1 sibling, 0 replies; 15+ messages in thread
From: NeilBrown @ 2014-01-05 23:50 UTC (permalink / raw)
To: Phil Turmel; +Cc: Fabian Knorr, linux-raid
[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]
On Sun, 05 Jan 2014 13:25:29 -0500 Phil Turmel <philip@turmel.org> wrote:
> Hi Fabian,
>
> I see good news to greet me this morning!
>
> On 01/05/2014 05:45 AM, Fabian Knorr wrote:
> > $ cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4]
> > md127 : active (auto-read-only) raid5 sde1[6] sdh1[2] sdi1[3] sdb1[4] sdg1[1] sdc1[5] sdf1[0]
> > 11721074688 blocks level 5, 1024k chunk, algorithm 2 [7/7] [UUUUUUU]
> > resync=PENDING
>
> Auto-read-only will switch to read-write as soon as you actually write
> anything to the array. Which will also kick off the resync.
You can also "mdadm --readwrite /dev/md127" to allow resync to continue.
> Long term, consider creating a new array with metadata type 1.x so you
> can use a bitmap. If you ever have an event like this, where one or
> more devices are disconnected then reconnected, it'll greatly shorten
> the recovery time.
While I would encourage the use for 1.x metadata, you don't strictly need it
for a bitmap. 0.90 metadata supports bitmaps too, though it is slightly less
flexibly in the granularity of the bitmap (not that you would probably
notice).
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-05 18:25 ` Phil Turmel
2014-01-05 23:50 ` NeilBrown
@ 2014-01-06 14:00 ` Fabian Knorr
2014-01-07 0:26 ` NeilBrown
1 sibling, 1 reply; 15+ messages in thread
From: Fabian Knorr @ 2014-01-06 14:00 UTC (permalink / raw)
To: Phil Turmel; +Cc: NeilBrown, linux-raid
Many thanks to both of you, Phil and Neil!
Everything works flawlessly now and I had no data loss whatsoever.
> Long term, consider creating a new array with metadata type 1.x so you
> can use a bitmap. If you ever have an event like this, where one or
> more devices are disconnected then reconnected, it'll greatly shorten
> the recovery time.
But it's not possible to upgrade the metadata version of an existing
array, is it?
Fabian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-06 14:00 ` Fabian Knorr
@ 2014-01-07 0:26 ` NeilBrown
0 siblings, 0 replies; 15+ messages in thread
From: NeilBrown @ 2014-01-07 0:26 UTC (permalink / raw)
To: Fabian Knorr; +Cc: Phil Turmel, linux-raid
[-- Attachment #1: Type: text/plain, Size: 749 bytes --]
On Mon, 06 Jan 2014 15:00:46 +0100 Fabian Knorr <knorrfab@fim.uni-passau.de>
wrote:
> Many thanks to both of you, Phil and Neil!
>
> Everything works flawlessly now and I had no data loss whatsoever.
>
> > Long term, consider creating a new array with metadata type 1.x so you
> > can use a bitmap. If you ever have an event like this, where one or
> > more devices are disconnected then reconnected, it'll greatly shorten
> > the recovery time.
>
> But it's not possible to upgrade the metadata version of an existing
> array, is it?
With mdadm-3.3 you can assemble an array with 0.90 metadata using
mdadm --assemble --update=metadata /dev/mdWHATEVER /dev/.....
and it will be converted to 1.0 metadata.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Recovering an Array with inconsistent Superblocks
2014-01-04 22:05 ` Fabian Knorr
2014-01-05 2:32 ` Phil Turmel
@ 2014-01-14 8:54 ` David Brown
1 sibling, 0 replies; 15+ messages in thread
From: David Brown @ 2014-01-14 8:54 UTC (permalink / raw)
To: Fabian Knorr, Phil Turmel; +Cc: linux-raid
On 04/01/14 23:05, Fabian Knorr wrote:
> Hi, Phil,
>
> thank you very much for your reply.
>
>> Side note: If you have a live spare available for a raid5, there's no
>> good reason not to reshape to a raid6, and very good reasons to do so.
>
> I was worried that RAID6 would incur a significant load on the CPU,
> especially if one disk fails. The system is a single-core Intel Atom.
>
Recovery of a single disk failure with RAID6 is the same as for RAID5 -
if a data block is missing, md will use the other data blocks and the
RAID5 xor parity for the recovery. It only needs to do "hard" RAID6
recovery if there are /two/ disks (or blocks) that have failed.
RAID6 parity generation is harder than RAID5 generation, but it is not
/that/ hard, even for a single core Atom - especially if the cpu is not
doing many other tasks. You might feel it on big writes, or during a
resync / rebuild.
Other than that, the only disadvantage of RAID6 is that you don't get
the RMW behaviour on partial stripe writes (I believe this is under
development, but I don't know the current status). With RAID5, if you
write to a single block in a stripe then md can read the old block and
the old parity, then write the new block and the new parity rather than
having to read in the whole stripe. RAID6 always has to read the whole
stripe to create new parities. This means that small writes will be
slower on RAID6 than RAID5.
However, if you are doing so many small writes that this is an important
factor, you are probably better off with RAID10.
So IMHO if you have the drives for RAID5 + spare, you are almost always
better off using RAID6.
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2014-01-14 8:54 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-04 10:04 Recovering an Array with inconsistent Superblocks Fabian Knorr
2014-01-04 16:24 ` Phil Turmel
2014-01-04 17:59 ` Can Jeuleers
2014-01-04 19:16 ` Phil Turmel
2014-01-04 22:05 ` Fabian Knorr
2014-01-05 2:32 ` Phil Turmel
2014-01-05 9:07 ` Fabian Knorr
2014-01-05 9:56 ` NeilBrown
2014-01-05 10:40 ` Fabian Knorr
[not found] ` <1388918703.3591.20.camel@vessel>
2014-01-05 18:25 ` Phil Turmel
2014-01-05 23:50 ` NeilBrown
2014-01-06 14:00 ` Fabian Knorr
2014-01-07 0:26 ` NeilBrown
2014-01-14 8:54 ` David Brown
2014-01-04 22:08 ` Fabian Knorr
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.