* power outage while raid5->raid6 was in progress
@ 2010-07-07 20:41 Sebastian Reichel
2010-07-07 22:44 ` Neil Brown
0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Reichel @ 2010-07-07 20:41 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 3739 bytes --]
Hi,
I have some problems with my raid. I tried updating from 5 disks raid5 to 8 disks
raid6 as described on http://neil.brown.name/blog/20090817000931#2. The command I
used was: mdadm --grow /dev/md0 --level=6 --raid-disk=8
While the rebuild was in progress my system hung, so I had to force power down it.
After rebooting the system I reassembled the raid. You can see the resulting mess
below. How can I recover from this state?
root@mars ~ # mdadm -Af /dev/md0
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
root@mars ~ # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdk1[0] sdh1[8](S) sdg1[7](S) sdf1[6](S) sdi1[4] sdj1[3] sdl1[2] sde1[1]
11721087488 blocks
md1 : active raid5 sdb1[0] sdd1[2] sdc1[1]
1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>
root@mars ~ # mdadm -D /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jul 7 00:21:00 2010
State : active, degraded, Not Started
Active Devices : 5
Working Devices : 8
Failed Devices : 0
Spare Devices : 3
Layout : left-symmetric-6
Chunk Size : 64K
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Events : 0.991518
Number Major Minor RaidDevice State
0 8 161 0 active sync /dev/sdk1
1 8 65 1 active sync /dev/sde1
2 8 177 2 active sync /dev/sdl1
3 8 145 3 active sync /dev/sdj1
4 8 129 4 active sync /dev/sdi1
5 0 0 5 removed
6 8 81 - spare /dev/sdf1
7 8 97 - spare /dev/sdg1
8 8 113 - spare /dev/sdh1
root@mars ~ # dmesg | tail -n 30
[ 7896.306100] md: bind<sde1>
[ 7896.306240] md: bind<sdl1>
[ 7896.306368] md: bind<sdj1>
[ 7896.306495] md: bind<sdi1>
[ 7896.306666] md: bind<sdf1>
[ 7896.306795] md: bind<sdg1>
[ 7896.306921] md: bind<sdh1>
[ 7896.307048] md: bind<sdk1>
[ 7896.308745] raid5: md0 is not clean -- starting background reconstruction
[ 7896.308763] raid5: device sdk1 operational as raid disk 0
[ 7896.308765] raid5: device sdi1 operational as raid disk 4
[ 7896.308768] raid5: device sdj1 operational as raid disk 3
[ 7896.308770] raid5: device sdl1 operational as raid disk 2
[ 7896.308772] raid5: device sde1 operational as raid disk 1
[ 7896.309420] raid5: allocated 6386kB for md0
[ 7896.309465] 0: w=1 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 7896.309468] 4: w=2 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 7896.309471] 3: w=3 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 7896.309473] 2: w=4 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 7896.309476] 1: w=5 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 7896.309478] raid5: cannot start dirty degraded array for md0
[ 7896.309503] RAID5 conf printout:
[ 7896.309504] --- rd:6 wd:5
[ 7896.309506] disk 0, o:1, dev:sdk1
[ 7896.309508] disk 1, o:1, dev:sde1
[ 7896.309510] disk 2, o:1, dev:sdl1
[ 7896.309512] disk 3, o:1, dev:sdj1
[ 7896.309514] disk 4, o:1, dev:sdi1
[ 7896.309956] raid5: failed to run raid set md0
[ 7896.309970] md: pers->run() failed ...
root@mars ~ # uname -a
Linux mars 2.6.34-1-amd64 #1 SMP Sun Jun 6 22:29:25 UTC 2010 x86_64 GNU/Linux
root@mars ~ # mdadm --version
mdadm - v3.1.2 - 10th March 2010
-- Sebastian
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: power outage while raid5->raid6 was in progress
2010-07-07 20:41 power outage while raid5->raid6 was in progress Sebastian Reichel
@ 2010-07-07 22:44 ` Neil Brown
2010-07-07 23:13 ` Sebastian Reichel
0 siblings, 1 reply; 8+ messages in thread
From: Neil Brown @ 2010-07-07 22:44 UTC (permalink / raw)
To: Sebastian Reichel; +Cc: linux-raid
On Wed, 7 Jul 2010 22:41:10 +0200
Sebastian Reichel <elektranox@gmail.com> wrote:
> Hi,
>
> I have some problems with my raid. I tried updating from 5 disks raid5 to 8 disks
> raid6 as described on http://neil.brown.name/blog/20090817000931#2. The command I
> used was: mdadm --grow /dev/md0 --level=6 --raid-disk=8
>
> While the rebuild was in progress my system hung, so I had to force power down it.
> After rebooting the system I reassembled the raid. You can see the resulting mess
> below. How can I recover from this state?
Please report the output of
mdadm -E /dev/sd[efghijkl]1
then I'll see what can be done.
NeilBrown
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: power outage while raid5->raid6 was in progress
2010-07-07 22:44 ` Neil Brown
@ 2010-07-07 23:13 ` Sebastian Reichel
2010-07-08 1:21 ` Neil Brown
0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Reichel @ 2010-07-07 23:13 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 11704 bytes --]
On Thu, Jul 08, 2010 at 08:44:50AM +1000, Neil Brown wrote:
> On Wed, 7 Jul 2010 22:41:10 +0200
> Sebastian Reichel <elektranox@gmail.com> wrote:
>
> > Hi,
> >
> > I have some problems with my raid. I tried updating from 5 disks raid5 to 8 disks
> > raid6 as described on http://neil.brown.name/blog/20090817000931#2. The command I
> > used was: mdadm --grow /dev/md0 --level=6 --raid-disk=8
> >
> > While the rebuild was in progress my system hung, so I had to force power down it.
> > After rebooting the system I reassembled the raid. You can see the resulting mess
> > below. How can I recover from this state?
>
> Please report the output of
>
> mdadm -E /dev/sd[efghijkl]1
>
> then I'll see what can be done.
thank you for having a look at it :)
root@mars ~ # mdadm -E /dev/sd[efghijkl]1
/dev/sde1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Wed Jul 7 00:21:00 2010
State : active
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4439feb9 - correct
Events : 991519
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 65 1 active sync /dev/sde1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
/dev/sdf1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Wed Jul 7 00:21:00 2010
State : active
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4439fecd - correct
Events : 991519
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 81 6 spare /dev/sdf1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
/dev/sdg1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Wed Jul 7 00:21:00 2010
State : active
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4439fedf - correct
Events : 991519
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 8 97 7 spare /dev/sdg1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
/dev/sdh1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Wed Jul 7 00:21:00 2010
State : active
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4439fef1 - correct
Events : 991519
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 113 8 spare /dev/sdh1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
/dev/sdi1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Wed Jul 7 00:21:00 2010
State : active
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4439feff - correct
Events : 991519
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 129 4 active sync /dev/sdi1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
/dev/sdj1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Wed Jul 7 00:21:00 2010
State : active
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4439ff0d - correct
Events : 991519
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 145 3 active sync /dev/sdj1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
/dev/sdk1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Tue Jul 6 23:37:42 2010
State : clean
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4449160f - correct
Events : 991518
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 161 0 active sync /dev/sdk1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
/dev/sdl1:
Magic : a92b4efc
Version : 0.90.00
UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
Creation Time : Fri Apr 9 19:24:51 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Raid Devices : 6
Total Devices : 8
Preferred Minor : 0
Update Time : Wed Jul 7 00:21:00 2010
State : active
Active Devices : 5
Working Devices : 8
Failed Devices : 1
Spare Devices : 3
Checksum : 4439ff2b - correct
Events : 991519
Layout : left-symmetric-6
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 177 2 active sync /dev/sdl1
0 0 8 161 0 active sync /dev/sdk1
1 1 8 65 1 active sync /dev/sde1
2 2 8 177 2 active sync /dev/sdl1
3 3 8 145 3 active sync /dev/sdj1
4 4 8 129 4 active sync /dev/sdi1
5 5 0 0 5 faulty removed
6 6 8 81 6 spare /dev/sdf1
7 7 8 97 7 spare /dev/sdg1
8 8 8 113 8 spare /dev/sdh1
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: power outage while raid5->raid6 was in progress
2010-07-07 23:13 ` Sebastian Reichel
@ 2010-07-08 1:21 ` Neil Brown
2010-07-08 8:48 ` Sebastian Reichel
0 siblings, 1 reply; 8+ messages in thread
From: Neil Brown @ 2010-07-08 1:21 UTC (permalink / raw)
To: Sebastian Reichel; +Cc: linux-raid
On Thu, 8 Jul 2010 01:13:16 +0200
Sebastian Reichel <elektranox@gmail.com> wrote:
> On Thu, Jul 08, 2010 at 08:44:50AM +1000, Neil Brown wrote:
> > On Wed, 7 Jul 2010 22:41:10 +0200
> > Sebastian Reichel <elektranox@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I have some problems with my raid. I tried updating from 5 disks raid5 to 8 disks
> > > raid6 as described on http://neil.brown.name/blog/20090817000931#2. The command I
> > > used was: mdadm --grow /dev/md0 --level=6 --raid-disk=8
> > >
> > > While the rebuild was in progress my system hung, so I had to force power down it.
> > > After rebooting the system I reassembled the raid. You can see the resulting mess
> > > below. How can I recover from this state?
> >
> > Please report the output of
> >
> > mdadm -E /dev/sd[efghijkl]1
> >
> > then I'll see what can be done.
>
> thank you for having a look at it :)
It appears that the RAID5 -> RAID6 conversion (which is instantaneous, but
results in a non-standard RAID6 parity layout) happened, but the
6disk -> 8disk reshape which would have been combined with producing a
more standard RAID6 parity layout did not even begin.
I don't know why that would be. Do you remember seeing the reshape being
under-way in /proc/mdstat at all??
If you didn't them I am very confused and the following is not at all
reliable. If you didn't and you only assumed a reshape was happening, then
read on.
So it appear you have an active (dirty), degraded RAID6 array with 3 spares.
md will not normally start such arrays as they could potentially contain
corruption (the so-called RAID5 write hole), though the chance is rather
small.
You need to explicitly request that the array be started anyway using
"--force" to --assemble.
However you have done this and it doesn't seem to have worked. I cannot
work out why. There is clear evidence that you tried this as sdk1 has a
status of 'clean' rather than 'active', and the kernel log showed it being
added to the array last, so (as the event counts are all equal) its 'clean'
status will have over-ruled.
However it seems (again from the kernel logs) that raid5 still thinks the
array is dirty and so will not start it.
You can over-ride this with
echo 1 > /sys/module/md_mod/parameters/start_dirty_degraded
that tells raid5 to start a degraded array even if it is dirty (i.e. active).
You should probably also
echo 1 > /sys/module/md_mod/parameters/start_ro
so that the array is started read-only, and doesn't immediately try a resync.
Then you can "fsck -n" the array to make sure your data looks safe.
If it doesn't stop the array immediately and we will have to go over the
details again.
If it does look good, you can try the reshape again:
mdadm -G /dev/md0 -n 8 --layout normalise
and hope it works this time.
It might be best to convert it back to RAID5 first
mdadm -G /dev/md0 --level raid5
then repeat the command you started with (after making sure all the spares
are attached).
But if you are sure the reshape actually started the first time, don't do
any of this. Rather try to find some earlier kernel logs that show the
reshape starting, and maybe show what caused the crash.
good luck,
NeilBrown
>
> root@mars ~ # mdadm -E /dev/sd[efghijkl]1
> /dev/sde1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Wed Jul 7 00:21:00 2010
> State : active
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4439feb9 - correct
> Events : 991519
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 1 8 65 1 active sync /dev/sde1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
> /dev/sdf1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Wed Jul 7 00:21:00 2010
> State : active
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4439fecd - correct
> Events : 991519
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 6 8 81 6 spare /dev/sdf1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
> /dev/sdg1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Wed Jul 7 00:21:00 2010
> State : active
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4439fedf - correct
> Events : 991519
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 7 8 97 7 spare /dev/sdg1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
> /dev/sdh1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Wed Jul 7 00:21:00 2010
> State : active
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4439fef1 - correct
> Events : 991519
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 8 8 113 8 spare /dev/sdh1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
> /dev/sdi1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Wed Jul 7 00:21:00 2010
> State : active
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4439feff - correct
> Events : 991519
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 4 8 129 4 active sync /dev/sdi1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
> /dev/sdj1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Wed Jul 7 00:21:00 2010
> State : active
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4439ff0d - correct
> Events : 991519
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 3 8 145 3 active sync /dev/sdj1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
> /dev/sdk1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Tue Jul 6 23:37:42 2010
> State : clean
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4449160f - correct
> Events : 991518
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 0 8 161 0 active sync /dev/sdk1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
> /dev/sdl1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a1eb26ff:0d33b804:1c7aa044:e01dc78c (local to host mars)
> Creation Time : Fri Apr 9 19:24:51 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
> Raid Devices : 6
> Total Devices : 8
> Preferred Minor : 0
>
> Update Time : Wed Jul 7 00:21:00 2010
> State : active
> Active Devices : 5
> Working Devices : 8
> Failed Devices : 1
> Spare Devices : 3
> Checksum : 4439ff2b - correct
> Events : 991519
>
> Layout : left-symmetric-6
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 2 8 177 2 active sync /dev/sdl1
>
> 0 0 8 161 0 active sync /dev/sdk1
> 1 1 8 65 1 active sync /dev/sde1
> 2 2 8 177 2 active sync /dev/sdl1
> 3 3 8 145 3 active sync /dev/sdj1
> 4 4 8 129 4 active sync /dev/sdi1
> 5 5 0 0 5 faulty removed
> 6 6 8 81 6 spare /dev/sdf1
> 7 7 8 97 7 spare /dev/sdg1
> 8 8 8 113 8 spare /dev/sdh1
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: power outage while raid5->raid6 was in progress
2010-07-08 1:21 ` Neil Brown
@ 2010-07-08 8:48 ` Sebastian Reichel
2010-07-08 22:48 ` Neil Brown
0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Reichel @ 2010-07-08 8:48 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 11795 bytes --]
On Thu, Jul 08, 2010 at 11:21:44AM +1000, Neil Brown wrote:
> On Thu, 8 Jul 2010 01:13:16 +0200
> Sebastian Reichel <elektranox@gmail.com> wrote:
>
> > On Thu, Jul 08, 2010 at 08:44:50AM +1000, Neil Brown wrote:
> > > On Wed, 7 Jul 2010 22:41:10 +0200
> > > Sebastian Reichel <elektranox@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I have some problems with my raid. I tried updating from 5 disks raid5 to 8 disks
> > > > raid6 as described on http://neil.brown.name/blog/20090817000931#2. The command I
> > > > used was: mdadm --grow /dev/md0 --level=6 --raid-disk=8
> > > >
> > > > While the rebuild was in progress my system hung, so I had to force power down it.
> > > > After rebooting the system I reassembled the raid. You can see the resulting mess
> > > > below. How can I recover from this state?
> > >
> > > Please report the output of
> > >
> > > mdadm -E /dev/sd[efghijkl]1
> > >
> > > then I'll see what can be done.
> >
> > thank you for having a look at it :)
>
>
> It appears that the RAID5 -> RAID6 conversion (which is instantaneous, but
> results in a non-standard RAID6 parity layout) happened, but the
> 6disk -> 8disk reshape which would have been combined with producing a
> more standard RAID6 parity layout did not even begin.
> I don't know why that would be. Do you remember seeing the reshape being
> under-way in /proc/mdstat at all??
>
> If you didn't them I am very confused and the following is not at all
> reliable. If you didn't and you only assumed a reshape was happening, then
> read on.
>
> [...]
>
> But if you are sure the reshape actually started the first time, don't do
> any of this. Rather try to find some earlier kernel logs that show the
> reshape starting, and maybe show what caused the crash.
Yes, I saw the reshape via /proc/mdstat, it already passed 30% iirc.
Here are the interesting parts of the kernel log:
Jul 6 21:09:14 mars kernel: [ 7716.935683] RAID5 conf printout:
Jul 6 21:09:14 mars kernel: [ 7716.935688] --- rd:5 wd:5
Jul 6 21:09:14 mars kernel: [ 7716.935691] disk 0, o:1, dev:sdk1
Jul 6 21:09:14 mars kernel: [ 7716.935693] disk 1, o:1, dev:sde1
Jul 6 21:09:14 mars kernel: [ 7716.935695] disk 2, o:1, dev:sdl1
Jul 6 21:09:14 mars kernel: [ 7716.935696] disk 3, o:1, dev:sdj1
Jul 6 21:09:14 mars kernel: [ 7716.935698] disk 4, o:1, dev:sdi1
Jul 6 21:10:04 mars kernel: [ 7766.940183] raid5: device sdk1 operational as raid disk 0
Jul 6 21:10:04 mars kernel: [ 7766.940186] raid5: device sdi1 operational as raid disk 4
Jul 6 21:10:04 mars kernel: [ 7766.940189] raid5: device sdj1 operational as raid disk 3
Jul 6 21:10:04 mars kernel: [ 7766.940191] raid5: device sdl1 operational as raid disk 2
Jul 6 21:10:04 mars kernel: [ 7766.940193] raid5: device sde1 operational as raid disk 1
Jul 6 21:10:04 mars kernel: [ 7766.940840] raid5: allocated 6386kB for md0
Jul 6 21:10:04 mars kernel: [ 7766.952476] 0: w=1 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
Jul 6 21:10:04 mars kernel: [ 7766.952480] 4: w=2 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
Jul 6 21:10:04 mars kernel: [ 7766.952482] 3: w=3 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
Jul 6 21:10:04 mars kernel: [ 7766.952485] 2: w=4 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
Jul 6 21:10:04 mars kernel: [ 7766.952487] 1: w=5 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
Jul 6 21:10:04 mars kernel: [ 7766.952490] raid5: raid level 6 set md0 active with 5 out of 6 devices, algorithm 18
Jul 6 21:10:04 mars kernel: [ 7766.952516] RAID5 conf printout:
Jul 6 21:10:04 mars kernel: [ 7766.952517] --- rd:6 wd:5
Jul 6 21:10:04 mars kernel: [ 7766.952520] disk 0, o:1, dev:sdk1
Jul 6 21:10:04 mars kernel: [ 7766.952522] disk 1, o:1, dev:sde1
Jul 6 21:10:04 mars kernel: [ 7766.952523] disk 2, o:1, dev:sdl1
Jul 6 21:10:04 mars kernel: [ 7766.952525] disk 3, o:1, dev:sdj1
Jul 6 21:10:04 mars kernel: [ 7766.952527] disk 4, o:1, dev:sdi1
Jul 6 21:10:04 mars kernel: [ 7766.952536] ------------[ cut here ]------------
Jul 6 21:10:04 mars kernel: [ 7766.952542] WARNING: at /build/buildd-linux-2.6_2.6.34-1~experimental.2-amd64-zn0ozk/linux-2.6-2.6.34/debian/build/source_amd64_none/fs/sysfs/dir.c:451 sysfs_add_one+0xcc/0xe3()
Jul 6 21:10:04 mars kernel: [ 7766.952545] Hardware name: System Product Name
Jul 6 21:10:04 mars kernel: [ 7766.952547] sysfs: cannot create duplicate filename '/devices/virtual/block/md0/md/stripe_cache_size'
Jul 6 21:10:04 mars kernel: [ 7766.952549] Modules linked in: ip6t_LOG xt_hl nf_conntrack_ipv6 ipt_REJECT ipt_LOG xt_limit xt_tcpudp ipt_addrtype ipt_MASQUERADE iptable_nat xt_state ip6table_filter ip6_tables nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables nfsd nfs lockd fscache nfs_acl auth_rpcgss sunrpc pppoe pppox ppp_generic xfs exportfs rr272x_1x(P) hwmon_vid loop sha256_generic aes_x86_64 aes_generic cbc dm_crypt dm_mod arc4 ecb dvb_pll ath5k hfcpci mac80211 stv0299 b2c2_flexcop_pci mISDN_core ath b2c2_flexcop hisax dvb_core snd_pcm cx24123 snd_timer crc_ccitt cfg80211 cx24113 isdn snd rfkill s5h1420 soundcore i2c_nforce2 evdev pcspkr led_class snd_page_alloc edac_core edac_mce_amd i2c_core k8temp slhc tpm_tis tpm tpm_bios processor button asus_atk0110 ext3 jbd mbcache raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx ohci_hcd sd_mod crc_t10dif ata_generic fan firewire_ohci
Jul 6 21:10:04 mars kernel: ehci_hcd pata_amd firewire_core crc_itu_t thermal thermal_sys sata_nv usbcore nls_base libata forcedeth scsi_mod [last unloaded: scsi_wait_scan]
Jul 6 21:10:04 mars kernel: [ 7766.952622] Pid: 6844, comm: mdadm Tainted: P W 2.6.34-1-amd64 #1
Jul 6 21:10:04 mars kernel: [ 7766.952624] Call Trace:
Jul 6 21:10:04 mars kernel: [ 7766.952630] [<ffffffff81045655>] ? warn_slowpath_common+0x76/0x8c
Jul 6 21:10:04 mars kernel: [ 7766.952635] [<ffffffff810456bd>] ? warn_slowpath_fmt+0x40/0x45
Jul 6 21:10:04 mars kernel: [ 7766.952638] [<ffffffff81138209>] ? sysfs_add_one+0xcc/0xe3
Jul 6 21:10:04 mars kernel: [ 7766.952642] [<ffffffff81137416>] ? sysfs_add_file_mode+0x4b/0x7d
Jul 6 21:10:04 mars kernel: [ 7766.952646] [<ffffffff8113933c>] ? internal_create_group+0xdd/0x16b
Jul 6 21:10:04 mars kernel: [ 7766.952652] [<ffffffffa01a0f67>] ? run+0x4fa/0x685 [raid456]
Jul 6 21:10:04 mars kernel: [ 7766.952660] [<ffffffffa0181b20>] ? level_store+0x3b7/0x42e [md_mod]
Jul 6 21:10:04 mars kernel: [ 7766.952667] [<ffffffffa017eb1f>] ? md_attr_store+0x77/0x96 [md_mod]
Jul 6 21:10:04 mars kernel: [ 7766.952670] [<ffffffff81136ede>] ? sysfs_write_file+0xe3/0x11f
Jul 6 21:10:04 mars kernel: [ 7766.952674] [<ffffffff810e7bc6>] ? vfs_write+0xa4/0x101
Jul 6 21:10:04 mars kernel: [ 7766.952677] [<ffffffff810e7cd6>] ? sys_write+0x45/0x6b
Jul 6 21:10:04 mars kernel: [ 7766.952680] [<ffffffff810089c2>] ? system_call_fastpath+0x16/0x1b
Jul 6 21:10:04 mars kernel: [ 7766.952682] ---[ end trace 2d8a2ef8dd7ca7b7 ]---
Jul 6 21:10:04 mars kernel: [ 7766.952687] raid5: failed to create sysfs attributes for md0
Jul 6 21:21:10 mars kernel: [ 8432.471242] md: md0 stopped.
[...]
Jul 6 21:25:23 mars kernel: [ 8685.659605] raid5: raid level 6 set md0 active with 5 out of 6 devices, algorithm 18
Jul 6 21:25:23 mars kernel: [ 8685.659631] RAID5 conf printout:
Jul 6 21:25:23 mars kernel: [ 8685.659632] --- rd:6 wd:5
Jul 6 21:25:23 mars kernel: [ 8685.659634] disk 0, o:1, dev:sdk1
Jul 6 21:25:23 mars kernel: [ 8685.659636] disk 1, o:1, dev:sde1
Jul 6 21:25:23 mars kernel: [ 8685.659638] disk 2, o:1, dev:sdl1
Jul 6 21:25:23 mars kernel: [ 8685.659640] disk 3, o:1, dev:sdj1
Jul 6 21:25:23 mars kernel: [ 8685.659641] disk 4, o:1, dev:sdi1
Jul 6 21:25:23 mars kernel: [ 8685.659677] md0: detected capacity change from 0 to 6001196793856
Jul 6 21:25:23 mars kernel: [ 8685.659813] md0: unknown partition table
[...]
Jul 6 21:28:16 mars kernel: [ 8858.893402] md: recovery of RAID array md0
Jul 6 21:28:16 mars kernel: [ 8858.893405] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jul 6 21:28:16 mars kernel: [ 8858.893407] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Jul 6 21:28:16 mars kernel: [ 8858.893422] md: using 128k window, over a total of 1465135936 blocks.
[...]
Jul 6 21:31:18 mars kernel: [ 9040.089434] rr272x_1x:Device error information 0x1000000
Jul 6 21:31:18 mars kernel: [ 9040.089440] rr272x_1x:Task file error, StatusReg=0x41, ErrReg=0x4, LBA[0-3]=0x17fbec6,LBA[4-7]=0x0.
[...] (more of these messages)
Jul 6 23:54:20 mars kernel: [17622.164846] sd 8:0:6:0: [sdk] Unhandled error code
Jul 6 23:54:20 mars kernel: [17622.164848] sd 8:0:6:0: [sdk] Result: hostbyte=DID_ABORT driverbyte=DRIVER_INVALID
Jul 6 23:54:20 mars kernel: [17622.164851] sd 8:0:6:0: [sdk] CDB: Read(10): 28 00 49 ff 72 3f 00 00 38 00
Jul 6 23:54:20 mars kernel: [17622.164858] end_request: I/O error, dev sdk, sector 1241477695
Jul 6 23:54:20 mars kernel: [17622.181377] raid5:md0: read error corrected (8 sectors at 1241477632 on sdk1)
Jul 6 23:54:20 mars kernel: [17622.181503] raid5:md0: read error corrected (8 sectors at 1241477640 on sdk1)
Jul 6 23:54:20 mars kernel: [17622.181507] raid5:md0: read error corrected (8 sectors at 1241477648 on sdk1)
Jul 6 23:54:20 mars kernel: [17622.181510] raid5:md0: read error corrected (8 sectors at 1241477656 on sdk1)
Jul 6 23:54:20 mars kernel: [17622.181512] raid5:md0: read error corrected (8 sectors at 1241477664 on sdk1)
Jul 6 23:54:20 mars kernel: [17622.181515] raid5:md0: read error corrected (8 sectors at 1241477672 on sdk1)
Jul 6 23:54:20 mars kernel: [17622.181518] raid5:md0: read error corrected (8 sectors at 1241477680 on sdk1)
[...] (more sdk failures)
Jul 6 23:58:38 mars kernel: [17880.544030] INFO: task md0_resync:13565 blocked for more than 120 seconds.
Jul 6 23:58:38 mars kernel: [17880.544109] md0_resync D ffff88007cdf9500 0 13565 2 0x00000000
Jul 6 23:58:38 mars kernel: [17880.544114] ffff88007cdf9500 0000000000000046 0000000000000086 ffff88007400a800
Jul 6 23:58:38 mars kernel: [17880.544117] ffff88007ce25400 0000000000015200 0000000000015200 0000000000015200
Jul 6 23:58:38 mars kernel: [17880.544120] ffff88006baf5fd8 0000000000015200 ffff88007cdf9500 ffff88006baf5fd8
Jul 6 23:58:38 mars kernel: [17880.544124] Call Trace:
Jul 6 23:58:38 mars kernel: [17880.544145] [<ffffffffa019bdef>] ? unplug_slaves+0x70/0xa6 [raid456]
Jul 6 23:58:38 mars kernel: [17880.544150] [<ffffffffa019c17e>] ? get_active_stripe+0x26f/0x54b [raid456]
Jul 6 23:58:38 mars kernel: [17880.544154] [<ffffffff81040b38>] ? default_wake_function+0x0/0xf
Jul 6 23:58:38 mars kernel: [17880.544159] [<ffffffffa019ef3b>] ? sync_request+0x245/0x2d1 [raid456]
Jul 6 23:58:38 mars kernel: [17880.544166] [<ffffffffa01809d4>] ? is_mddev_idle+0xa2/0xf5 [md_mod]
Jul 6 23:58:38 mars kernel: [17880.544172] [<ffffffffa018113a>] ? md_do_sync+0x713/0xb0f [md_mod]
Jul 6 23:58:38 mars kernel: [17880.544177] [<ffffffff8105d495>] ? autoremove_wake_function+0x0/0x2a
Jul 6 23:58:38 mars kernel: [17880.544182] [<ffffffffa0181c89>] ? md_thread+0xf2/0x110 [md_mod]
Jul 6 23:58:38 mars kernel: [17880.544188] [<ffffffffa0181b97>] ? md_thread+0x0/0x110 [md_mod]
Jul 6 23:58:38 mars kernel: [17880.544190] [<ffffffff8105d061>] ? kthread+0x75/0x7d
Jul 6 23:58:38 mars kernel: [17880.544194] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
Jul 6 23:58:38 mars kernel: [17880.544197] [<ffffffff8105cfec>] ? kthread+0x0/0x7d
Jul 6 23:58:38 mars kernel: [17880.544199] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: power outage while raid5->raid6 was in progress
2010-07-08 8:48 ` Sebastian Reichel
@ 2010-07-08 22:48 ` Neil Brown
2010-07-08 23:50 ` Sebastian Reichel
2010-07-13 22:14 ` Sebastian Reichel
0 siblings, 2 replies; 8+ messages in thread
From: Neil Brown @ 2010-07-08 22:48 UTC (permalink / raw)
To: Sebastian Reichel; +Cc: linux-raid
On Thu, 8 Jul 2010 10:48:37 +0200
Sebastian Reichel <elektranox@gmail.com> wrote:
> On Thu, Jul 08, 2010 at 11:21:44AM +1000, Neil Brown wrote:
> > On Thu, 8 Jul 2010 01:13:16 +0200
> > Sebastian Reichel <elektranox@gmail.com> wrote:
> >
> > > On Thu, Jul 08, 2010 at 08:44:50AM +1000, Neil Brown wrote:
> > > > On Wed, 7 Jul 2010 22:41:10 +0200
> > > > Sebastian Reichel <elektranox@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have some problems with my raid. I tried updating from 5 disks raid5 to 8 disks
> > > > > raid6 as described on http://neil.brown.name/blog/20090817000931#2. The command I
> > > > > used was: mdadm --grow /dev/md0 --level=6 --raid-disk=8
> > > > >
> > > > > While the rebuild was in progress my system hung, so I had to force power down it.
> > > > > After rebooting the system I reassembled the raid. You can see the resulting mess
> > > > > below. How can I recover from this state?
> > > >
> > > > Please report the output of
> > > >
> > > > mdadm -E /dev/sd[efghijkl]1
> > > >
> > > > then I'll see what can be done.
> > >
> > > thank you for having a look at it :)
> >
> >
> > It appears that the RAID5 -> RAID6 conversion (which is instantaneous, but
> > results in a non-standard RAID6 parity layout) happened, but the
> > 6disk -> 8disk reshape which would have been combined with producing a
> > more standard RAID6 parity layout did not even begin.
> > I don't know why that would be. Do you remember seeing the reshape being
> > under-way in /proc/mdstat at all??
> >
> > If you didn't them I am very confused and the following is not at all
> > reliable. If you didn't and you only assumed a reshape was happening, then
> > read on.
> >
> > [...]
> >
> > But if you are sure the reshape actually started the first time, don't do
> > any of this. Rather try to find some earlier kernel logs that show the
> > reshape starting, and maybe show what caused the crash.
>
> Yes, I saw the reshape via /proc/mdstat, it already passed 30% iirc.
> Here are the interesting parts of the kernel log:
Ahhh, that helps a lot - thanks.
mdadm successfully converted RAID5 -> RAID6 as I thought, but it hit a bug
(that is aleady fixed in the -stable series - sorry about that). So the
6disk -> 8disk conversion couldn't be requested.
You stopped and restarted the array which worked and you now have a
degraded RAID6 over 6 devices with a non-standard layout.
As it was degraded and there was a spare it started recovering the missing
devices to a spare. You thought this was the reshape progressing and in a
sense it was, just not exactly the reshape you were expecting.
You then hit read errors on sdk - at about the 42% mark if my sums are right.
Then it threw a warning about spending too much time in the thread that was
trying to correct the read errors. Then .... did it hang?
Anyway, this all confirms that I was right about the state of your array.
It is still a degraded 6disk RAID6 with a non-standard layout.
However sdk is clearly questionable. You might want to exclude it from the
array.
The rest of my advice about what to do holds.
However you might want to get the latest 2.6.34.y -stable kernel if you can.
BTW, I'm curious about your hang. You replaced a bunch of errors with
> [...] (more sdk failures)
2 questions:
1/ was this all "read error corrected" with consecutive addresses? what
did the addresses go up to?
2/ did this continue for the full 4 minutes until the "INFO:" hang message?
Thanks,
NeilBrown
>
> Jul 6 21:09:14 mars kernel: [ 7716.935683] RAID5 conf printout:
> Jul 6 21:09:14 mars kernel: [ 7716.935688] --- rd:5 wd:5
> Jul 6 21:09:14 mars kernel: [ 7716.935691] disk 0, o:1, dev:sdk1
> Jul 6 21:09:14 mars kernel: [ 7716.935693] disk 1, o:1, dev:sde1
> Jul 6 21:09:14 mars kernel: [ 7716.935695] disk 2, o:1, dev:sdl1
> Jul 6 21:09:14 mars kernel: [ 7716.935696] disk 3, o:1, dev:sdj1
> Jul 6 21:09:14 mars kernel: [ 7716.935698] disk 4, o:1, dev:sdi1
> Jul 6 21:10:04 mars kernel: [ 7766.940183] raid5: device sdk1 operational as raid disk 0
> Jul 6 21:10:04 mars kernel: [ 7766.940186] raid5: device sdi1 operational as raid disk 4
> Jul 6 21:10:04 mars kernel: [ 7766.940189] raid5: device sdj1 operational as raid disk 3
> Jul 6 21:10:04 mars kernel: [ 7766.940191] raid5: device sdl1 operational as raid disk 2
> Jul 6 21:10:04 mars kernel: [ 7766.940193] raid5: device sde1 operational as raid disk 1
> Jul 6 21:10:04 mars kernel: [ 7766.940840] raid5: allocated 6386kB for md0
> Jul 6 21:10:04 mars kernel: [ 7766.952476] 0: w=1 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> Jul 6 21:10:04 mars kernel: [ 7766.952480] 4: w=2 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> Jul 6 21:10:04 mars kernel: [ 7766.952482] 3: w=3 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> Jul 6 21:10:04 mars kernel: [ 7766.952485] 2: w=4 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> Jul 6 21:10:04 mars kernel: [ 7766.952487] 1: w=5 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> Jul 6 21:10:04 mars kernel: [ 7766.952490] raid5: raid level 6 set md0 active with 5 out of 6 devices, algorithm 18
> Jul 6 21:10:04 mars kernel: [ 7766.952516] RAID5 conf printout:
> Jul 6 21:10:04 mars kernel: [ 7766.952517] --- rd:6 wd:5
> Jul 6 21:10:04 mars kernel: [ 7766.952520] disk 0, o:1, dev:sdk1
> Jul 6 21:10:04 mars kernel: [ 7766.952522] disk 1, o:1, dev:sde1
> Jul 6 21:10:04 mars kernel: [ 7766.952523] disk 2, o:1, dev:sdl1
> Jul 6 21:10:04 mars kernel: [ 7766.952525] disk 3, o:1, dev:sdj1
> Jul 6 21:10:04 mars kernel: [ 7766.952527] disk 4, o:1, dev:sdi1
> Jul 6 21:10:04 mars kernel: [ 7766.952536] ------------[ cut here ]------------
> Jul 6 21:10:04 mars kernel: [ 7766.952542] WARNING: at /build/buildd-linux-2.6_2.6.34-1~experimental.2-amd64-zn0ozk/linux-2.6-2.6.34/debian/build/source_amd64_none/fs/sysfs/dir.c:451 sysfs_add_one+0xcc/0xe3()
> Jul 6 21:10:04 mars kernel: [ 7766.952545] Hardware name: System Product Name
> Jul 6 21:10:04 mars kernel: [ 7766.952547] sysfs: cannot create duplicate filename '/devices/virtual/block/md0/md/stripe_cache_size'
> Jul 6 21:10:04 mars kernel: [ 7766.952549] Modules linked in: ip6t_LOG xt_hl nf_conntrack_ipv6 ipt_REJECT ipt_LOG xt_limit xt_tcpudp ipt_addrtype ipt_MASQUERADE iptable_nat xt_state ip6table_filter ip6_tables nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables nfsd nfs lockd fscache nfs_acl auth_rpcgss sunrpc pppoe pppox ppp_generic xfs exportfs rr272x_1x(P) hwmon_vid loop sha256_generic aes_x86_64 aes_generic cbc dm_crypt dm_mod arc4 ecb dvb_pll ath5k hfcpci mac80211 stv0299 b2c2_flexcop_pci mISDN_core ath b2c2_flexcop hisax dvb_core snd_pcm cx24123 snd_timer crc_ccitt cfg80211 cx24113 isdn snd rfkill s5h1420 soundcore i2c_nforce2 evdev pcspkr led_class snd_page_alloc edac_core edac_mce_amd i2c_c
ore k8temp slhc tpm_tis tpm tpm_bios processor button asus_atk0110 ext3 jbd mbcache raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx ohci_hcd sd_mod crc
_t10dif ata_generic fan firewire_ohci
> Jul 6 21:10:04 mars kernel: ehci_hcd pata_amd firewire_core crc_itu_t thermal thermal_sys sata_nv usbcore nls_base libata forcedeth scsi_mod [last unloaded: scsi_wait_scan]
> Jul 6 21:10:04 mars kernel: [ 7766.952622] Pid: 6844, comm: mdadm Tainted: P W 2.6.34-1-amd64 #1
> Jul 6 21:10:04 mars kernel: [ 7766.952624] Call Trace:
> Jul 6 21:10:04 mars kernel: [ 7766.952630] [<ffffffff81045655>] ? warn_slowpath_common+0x76/0x8c
> Jul 6 21:10:04 mars kernel: [ 7766.952635] [<ffffffff810456bd>] ? warn_slowpath_fmt+0x40/0x45
> Jul 6 21:10:04 mars kernel: [ 7766.952638] [<ffffffff81138209>] ? sysfs_add_one+0xcc/0xe3
> Jul 6 21:10:04 mars kernel: [ 7766.952642] [<ffffffff81137416>] ? sysfs_add_file_mode+0x4b/0x7d
> Jul 6 21:10:04 mars kernel: [ 7766.952646] [<ffffffff8113933c>] ? internal_create_group+0xdd/0x16b
> Jul 6 21:10:04 mars kernel: [ 7766.952652] [<ffffffffa01a0f67>] ? run+0x4fa/0x685 [raid456]
> Jul 6 21:10:04 mars kernel: [ 7766.952660] [<ffffffffa0181b20>] ? level_store+0x3b7/0x42e [md_mod]
> Jul 6 21:10:04 mars kernel: [ 7766.952667] [<ffffffffa017eb1f>] ? md_attr_store+0x77/0x96 [md_mod]
> Jul 6 21:10:04 mars kernel: [ 7766.952670] [<ffffffff81136ede>] ? sysfs_write_file+0xe3/0x11f
> Jul 6 21:10:04 mars kernel: [ 7766.952674] [<ffffffff810e7bc6>] ? vfs_write+0xa4/0x101
> Jul 6 21:10:04 mars kernel: [ 7766.952677] [<ffffffff810e7cd6>] ? sys_write+0x45/0x6b
> Jul 6 21:10:04 mars kernel: [ 7766.952680] [<ffffffff810089c2>] ? system_call_fastpath+0x16/0x1b
> Jul 6 21:10:04 mars kernel: [ 7766.952682] ---[ end trace 2d8a2ef8dd7ca7b7 ]---
> Jul 6 21:10:04 mars kernel: [ 7766.952687] raid5: failed to create sysfs attributes for md0
> Jul 6 21:21:10 mars kernel: [ 8432.471242] md: md0 stopped.
> [...]
> Jul 6 21:25:23 mars kernel: [ 8685.659605] raid5: raid level 6 set md0 active with 5 out of 6 devices, algorithm 18
> Jul 6 21:25:23 mars kernel: [ 8685.659631] RAID5 conf printout:
> Jul 6 21:25:23 mars kernel: [ 8685.659632] --- rd:6 wd:5
> Jul 6 21:25:23 mars kernel: [ 8685.659634] disk 0, o:1, dev:sdk1
> Jul 6 21:25:23 mars kernel: [ 8685.659636] disk 1, o:1, dev:sde1
> Jul 6 21:25:23 mars kernel: [ 8685.659638] disk 2, o:1, dev:sdl1
> Jul 6 21:25:23 mars kernel: [ 8685.659640] disk 3, o:1, dev:sdj1
> Jul 6 21:25:23 mars kernel: [ 8685.659641] disk 4, o:1, dev:sdi1
> Jul 6 21:25:23 mars kernel: [ 8685.659677] md0: detected capacity change from 0 to 6001196793856
> Jul 6 21:25:23 mars kernel: [ 8685.659813] md0: unknown partition table
> [...]
> Jul 6 21:28:16 mars kernel: [ 8858.893402] md: recovery of RAID array md0
> Jul 6 21:28:16 mars kernel: [ 8858.893405] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> Jul 6 21:28:16 mars kernel: [ 8858.893407] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> Jul 6 21:28:16 mars kernel: [ 8858.893422] md: using 128k window, over a total of 1465135936 blocks.
> [...]
> Jul 6 21:31:18 mars kernel: [ 9040.089434] rr272x_1x:Device error information 0x1000000
> Jul 6 21:31:18 mars kernel: [ 9040.089440] rr272x_1x:Task file error, StatusReg=0x41, ErrReg=0x4, LBA[0-3]=0x17fbec6,LBA[4-7]=0x0.
> [...] (more of these messages)
> Jul 6 23:54:20 mars kernel: [17622.164846] sd 8:0:6:0: [sdk] Unhandled error code
> Jul 6 23:54:20 mars kernel: [17622.164848] sd 8:0:6:0: [sdk] Result: hostbyte=DID_ABORT driverbyte=DRIVER_INVALID
> Jul 6 23:54:20 mars kernel: [17622.164851] sd 8:0:6:0: [sdk] CDB: Read(10): 28 00 49 ff 72 3f 00 00 38 00
> Jul 6 23:54:20 mars kernel: [17622.164858] end_request: I/O error, dev sdk, sector 1241477695
> Jul 6 23:54:20 mars kernel: [17622.181377] raid5:md0: read error corrected (8 sectors at 1241477632 on sdk1)
> Jul 6 23:54:20 mars kernel: [17622.181503] raid5:md0: read error corrected (8 sectors at 1241477640 on sdk1)
> Jul 6 23:54:20 mars kernel: [17622.181507] raid5:md0: read error corrected (8 sectors at 1241477648 on sdk1)
> Jul 6 23:54:20 mars kernel: [17622.181510] raid5:md0: read error corrected (8 sectors at 1241477656 on sdk1)
> Jul 6 23:54:20 mars kernel: [17622.181512] raid5:md0: read error corrected (8 sectors at 1241477664 on sdk1)
> Jul 6 23:54:20 mars kernel: [17622.181515] raid5:md0: read error corrected (8 sectors at 1241477672 on sdk1)
> Jul 6 23:54:20 mars kernel: [17622.181518] raid5:md0: read error corrected (8 sectors at 1241477680 on sdk1)
> [...] (more sdk failures)
> Jul 6 23:58:38 mars kernel: [17880.544030] INFO: task md0_resync:13565 blocked for more than 120 seconds.
> Jul 6 23:58:38 mars kernel: [17880.544109] md0_resync D ffff88007cdf9500 0 13565 2 0x00000000
> Jul 6 23:58:38 mars kernel: [17880.544114] ffff88007cdf9500 0000000000000046 0000000000000086 ffff88007400a800
> Jul 6 23:58:38 mars kernel: [17880.544117] ffff88007ce25400 0000000000015200 0000000000015200 0000000000015200
> Jul 6 23:58:38 mars kernel: [17880.544120] ffff88006baf5fd8 0000000000015200 ffff88007cdf9500 ffff88006baf5fd8
> Jul 6 23:58:38 mars kernel: [17880.544124] Call Trace:
> Jul 6 23:58:38 mars kernel: [17880.544145] [<ffffffffa019bdef>] ? unplug_slaves+0x70/0xa6 [raid456]
> Jul 6 23:58:38 mars kernel: [17880.544150] [<ffffffffa019c17e>] ? get_active_stripe+0x26f/0x54b [raid456]
> Jul 6 23:58:38 mars kernel: [17880.544154] [<ffffffff81040b38>] ? default_wake_function+0x0/0xf
> Jul 6 23:58:38 mars kernel: [17880.544159] [<ffffffffa019ef3b>] ? sync_request+0x245/0x2d1 [raid456]
> Jul 6 23:58:38 mars kernel: [17880.544166] [<ffffffffa01809d4>] ? is_mddev_idle+0xa2/0xf5 [md_mod]
> Jul 6 23:58:38 mars kernel: [17880.544172] [<ffffffffa018113a>] ? md_do_sync+0x713/0xb0f [md_mod]
> Jul 6 23:58:38 mars kernel: [17880.544177] [<ffffffff8105d495>] ? autoremove_wake_function+0x0/0x2a
> Jul 6 23:58:38 mars kernel: [17880.544182] [<ffffffffa0181c89>] ? md_thread+0xf2/0x110 [md_mod]
> Jul 6 23:58:38 mars kernel: [17880.544188] [<ffffffffa0181b97>] ? md_thread+0x0/0x110 [md_mod]
> Jul 6 23:58:38 mars kernel: [17880.544190] [<ffffffff8105d061>] ? kthread+0x75/0x7d
> Jul 6 23:58:38 mars kernel: [17880.544194] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> Jul 6 23:58:38 mars kernel: [17880.544197] [<ffffffff8105cfec>] ? kthread+0x0/0x7d
> Jul 6 23:58:38 mars kernel: [17880.544199] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: power outage while raid5->raid6 was in progress
2010-07-08 22:48 ` Neil Brown
@ 2010-07-08 23:50 ` Sebastian Reichel
2010-07-13 22:14 ` Sebastian Reichel
1 sibling, 0 replies; 8+ messages in thread
From: Sebastian Reichel @ 2010-07-08 23:50 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 16456 bytes --]
On Fri, Jul 09, 2010 at 08:48:08AM +1000, Neil Brown wrote:
> On Thu, 8 Jul 2010 10:48:37 +0200
> Sebastian Reichel <elektranox@gmail.com> wrote:
>
> > On Thu, Jul 08, 2010 at 11:21:44AM +1000, Neil Brown wrote:
> > > On Thu, 8 Jul 2010 01:13:16 +0200
> > > Sebastian Reichel <elektranox@gmail.com> wrote:
> > >
> > > > On Thu, Jul 08, 2010 at 08:44:50AM +1000, Neil Brown wrote:
> > > > > On Wed, 7 Jul 2010 22:41:10 +0200
> > > > > Sebastian Reichel <elektranox@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have some problems with my raid. I tried updating from 5 disks raid5 to 8 disks
> > > > > > raid6 as described on http://neil.brown.name/blog/20090817000931#2. The command I
> > > > > > used was: mdadm --grow /dev/md0 --level=6 --raid-disk=8
> > > > > >
> > > > > > While the rebuild was in progress my system hung, so I had to force power down it.
> > > > > > After rebooting the system I reassembled the raid. You can see the resulting mess
> > > > > > below. How can I recover from this state?
> > > > >
> > > > > Please report the output of
> > > > >
> > > > > mdadm -E /dev/sd[efghijkl]1
> > > > >
> > > > > then I'll see what can be done.
> > > >
> > > > thank you for having a look at it :)
> > >
> > >
> > > It appears that the RAID5 -> RAID6 conversion (which is instantaneous, but
> > > results in a non-standard RAID6 parity layout) happened, but the
> > > 6disk -> 8disk reshape which would have been combined with producing a
> > > more standard RAID6 parity layout did not even begin.
> > > I don't know why that would be. Do you remember seeing the reshape being
> > > under-way in /proc/mdstat at all??
> > >
> > > If you didn't them I am very confused and the following is not at all
> > > reliable. If you didn't and you only assumed a reshape was happening, then
> > > read on.
> > >
> > > [...]
> > >
> > > But if you are sure the reshape actually started the first time, don't do
> > > any of this. Rather try to find some earlier kernel logs that show the
> > > reshape starting, and maybe show what caused the crash.
> >
> > Yes, I saw the reshape via /proc/mdstat, it already passed 30% iirc.
> > Here are the interesting parts of the kernel log:
>
> Ahhh, that helps a lot - thanks.
>
> mdadm successfully converted RAID5 -> RAID6 as I thought, but it hit a bug
> (that is aleady fixed in the -stable series - sorry about that). So the
> 6disk -> 8disk conversion couldn't be requested.
> You stopped and restarted the array which worked and you now have a
> degraded RAID6 over 6 devices with a non-standard layout.
Yes, I read about this problem before, so I restarted it ;)
> As it was degraded and there was a spare it started recovering the missing
> devices to a spare. You thought this was the reshape progressing and in a
> sense it was, just not exactly the reshape you were expecting.
>
> You then hit read errors on sdk - at about the 42% mark if my sums are right.
>
> Then it threw a warning about spending too much time in the thread that was
> trying to correct the read errors. Then .... did it hang?
iirc it was like this:
- mdadm reshap hang
- system itself was still working
- I couldn't get rid of the raid
- clean shutdown did not work (probably because shutdown tries to cleanly umount?)
> Anyway, this all confirms that I was right about the state of your array.
> It is still a degraded 6disk RAID6 with a non-standard layout.
>
> However sdk is clearly questionable. You might want to exclude it from the
> array.
I guess it's the fault of the backplane, which already made some problems before.
I will check after sleeping.
> The rest of my advice about what to do holds.
> However you might want to get the latest 2.6.34.y -stable kernel if you can.
Ok, I will compile it after checking the backplane.
>> BTW, I'm curious about your hang. You replaced a bunch of errors with
> > [...] (more sdk failures)
>
> 2 questions:
> 1/ was this all "read error corrected" with consecutive addresses? what
> did the addresses go up to?
> 2/ did this continue for the full 4 minutes until the "INFO:" hang message?
Here is the part I replaced at that point:
Jul 6 23:54:20 mars kernel: [17622.181518] raid5:md0: read error corrected (8 sectors at 1241477680 on sdk1)
Jul 6 23:56:03 mars kernel: [17715.250058] rr272x_1x:Device error information 0x1000000
Jul 6 23:56:03 mars kernel: [17715.250064] rr272x_1x:Task file error, StatusReg=0x40, ErrReg=0x0, LBA[0-3]=0x4ac51577,LBA[4-7]=0x0.
Jul 6 23:56:03 mars kernel: [17715.250453] rr272x_1x:Device error information 0x1000000
Jul 6 23:56:03 mars kernel: [17715.250456] rr272x_1x:Task file error, StatusReg=0x51, ErrReg=0x4, LBA[0-3]=0x4ac51576,LBA[4-7]=0x0.
Jul 6 23:56:03 mars kernel: [17725.226318] sd 8:0:6:0: [sdk] Unhandled error code
Jul 6 23:56:03 mars kernel: [17725.226322] sd 8:0:6:0: [sdk] Result: hostbyte=DID_ABORT driverbyte=DRIVER_INVALID
Jul 6 23:56:03 mars kernel: [17725.226325] sd 8:0:6:0: [sdk] CDB: Read(10): 28 00 4a c5 15 77 00 00 48 00
Jul 6 23:56:03 mars kernel: [17725.226331] end_request: I/O error, dev sdk, sector 1254430071
[...] more sdk unhandled error code with increasing sector, see next line for the last one
Jul 6 23:56:03 mars kernel: [17725.227090] sd 8:0:6:0: [sdk] Unhandled error code
Jul 6 23:56:03 mars kernel: [17725.227091] sd 8:0:6:0: [sdk] Result: hostbyte=DID_ABORT driverbyte=DRIVER_INVALID
Jul 6 23:56:03 mars kernel: [17725.227094] sd 8:0:6:0: [sdk] CDB: Read(10): 28 00 4a c5 1a bf 00 00 38 00
Jul 6 23:56:03 mars kernel: [17725.227099] end_request: I/O error, dev sdk, sector 1254431423
Jul 6 23:56:03 mars kernel: [17725.265533] rr272x_1x:Device error information 0x1000000
Jul 6 23:56:03 mars kernel: [17725.265539] rr272x_1x:Task file error, StatusReg=0x41, ErrReg=0x4, LBA[0-3]=0x4ac51a8e,LBA[4-7]=0x0.
Jul 6 23:56:03 mars kernel: [17725.347683] rr272x_1x:Device error information 0x8000000080000000
Jul 6 23:58:38 mars kernel: [17880.544030] INFO: task md0_resync:13565 blocked for more than 120 seconds.
-- Sebastian
> > Jul 6 21:09:14 mars kernel: [ 7716.935683] RAID5 conf printout:
> > Jul 6 21:09:14 mars kernel: [ 7716.935688] --- rd:5 wd:5
> > Jul 6 21:09:14 mars kernel: [ 7716.935691] disk 0, o:1, dev:sdk1
> > Jul 6 21:09:14 mars kernel: [ 7716.935693] disk 1, o:1, dev:sde1
> > Jul 6 21:09:14 mars kernel: [ 7716.935695] disk 2, o:1, dev:sdl1
> > Jul 6 21:09:14 mars kernel: [ 7716.935696] disk 3, o:1, dev:sdj1
> > Jul 6 21:09:14 mars kernel: [ 7716.935698] disk 4, o:1, dev:sdi1
> > Jul 6 21:10:04 mars kernel: [ 7766.940183] raid5: device sdk1 operational as raid disk 0
> > Jul 6 21:10:04 mars kernel: [ 7766.940186] raid5: device sdi1 operational as raid disk 4
> > Jul 6 21:10:04 mars kernel: [ 7766.940189] raid5: device sdj1 operational as raid disk 3
> > Jul 6 21:10:04 mars kernel: [ 7766.940191] raid5: device sdl1 operational as raid disk 2
> > Jul 6 21:10:04 mars kernel: [ 7766.940193] raid5: device sde1 operational as raid disk 1
> > Jul 6 21:10:04 mars kernel: [ 7766.940840] raid5: allocated 6386kB for md0
> > Jul 6 21:10:04 mars kernel: [ 7766.952476] 0: w=1 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> > Jul 6 21:10:04 mars kernel: [ 7766.952480] 4: w=2 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> > Jul 6 21:10:04 mars kernel: [ 7766.952482] 3: w=3 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> > Jul 6 21:10:04 mars kernel: [ 7766.952485] 2: w=4 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> > Jul 6 21:10:04 mars kernel: [ 7766.952487] 1: w=5 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
> > Jul 6 21:10:04 mars kernel: [ 7766.952490] raid5: raid level 6 set md0 active with 5 out of 6 devices, algorithm 18
> > Jul 6 21:10:04 mars kernel: [ 7766.952516] RAID5 conf printout:
> > Jul 6 21:10:04 mars kernel: [ 7766.952517] --- rd:6 wd:5
> > Jul 6 21:10:04 mars kernel: [ 7766.952520] disk 0, o:1, dev:sdk1
> > Jul 6 21:10:04 mars kernel: [ 7766.952522] disk 1, o:1, dev:sde1
> > Jul 6 21:10:04 mars kernel: [ 7766.952523] disk 2, o:1, dev:sdl1
> > Jul 6 21:10:04 mars kernel: [ 7766.952525] disk 3, o:1, dev:sdj1
> > Jul 6 21:10:04 mars kernel: [ 7766.952527] disk 4, o:1, dev:sdi1
> > Jul 6 21:10:04 mars kernel: [ 7766.952536] ------------[ cut here ]------------
> > Jul 6 21:10:04 mars kernel: [ 7766.952542] WARNING: at /build/buildd-linux-2.6_2.6.34-1~experimental.2-amd64-zn0ozk/linux-2.6-2.6.34/debian/build/source_amd64_none/fs/sysfs/dir.c:451 sysfs_add_one+0xcc/0xe3()
> > Jul 6 21:10:04 mars kernel: [ 7766.952545] Hardware name: System Product Name
> > Jul 6 21:10:04 mars kernel: [ 7766.952547] sysfs: cannot create duplicate filename '/devices/virtual/block/md0/md/stripe_cache_size'
> > Jul 6 21:10:04 mars kernel: [ 7766.952549] Modules linked in: ip6t_LOG xt_hl nf_conntrack_ipv6 ipt_REJECT ipt_LOG xt_limit xt_tcpudp ipt_addrtype ipt_MASQUERADE iptable_nat xt_state ip6table_filter ip6_tables nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables nfsd nfs lockd fscache nfs_acl auth_rpcgss sunrpc pppoe pppox ppp_generic xfs exportfs rr272x_1x(P) hwmon_vid loop sha256_generic aes_x86_64 aes_generic cbc dm_crypt dm_mod arc4 ecb dvb_pll ath5k hfcpci mac80211 stv0299 b2c2_flexcop_pci mISDN_core ath b2c2_flexcop hisax dvb_core snd_pcm cx24123 snd_timer crc_ccitt cfg80211 cx24113 isdn snd rfkill s5h1420 soundcore i2c_nforce2 evdev pcspkr led_class snd_page_alloc edac_core edac_mce_amd i2c_core k8temp slhc tpm_tis tpm tpm_bios processor button asus_atk0110 ext3 jbd mbcache raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx ohci_hcd sd_mod crc
> _t10dif ata_generic fan firewire_ohci
> > Jul 6 21:10:04 mars kernel: ehci_hcd pata_amd firewire_core crc_itu_t thermal thermal_sys sata_nv usbcore nls_base libata forcedeth scsi_mod [last unloaded: scsi_wait_scan]
> > Jul 6 21:10:04 mars kernel: [ 7766.952622] Pid: 6844, comm: mdadm Tainted: P W 2.6.34-1-amd64 #1
> > Jul 6 21:10:04 mars kernel: [ 7766.952624] Call Trace:
> > Jul 6 21:10:04 mars kernel: [ 7766.952630] [<ffffffff81045655>] ? warn_slowpath_common+0x76/0x8c
> > Jul 6 21:10:04 mars kernel: [ 7766.952635] [<ffffffff810456bd>] ? warn_slowpath_fmt+0x40/0x45
> > Jul 6 21:10:04 mars kernel: [ 7766.952638] [<ffffffff81138209>] ? sysfs_add_one+0xcc/0xe3
> > Jul 6 21:10:04 mars kernel: [ 7766.952642] [<ffffffff81137416>] ? sysfs_add_file_mode+0x4b/0x7d
> > Jul 6 21:10:04 mars kernel: [ 7766.952646] [<ffffffff8113933c>] ? internal_create_group+0xdd/0x16b
> > Jul 6 21:10:04 mars kernel: [ 7766.952652] [<ffffffffa01a0f67>] ? run+0x4fa/0x685 [raid456]
> > Jul 6 21:10:04 mars kernel: [ 7766.952660] [<ffffffffa0181b20>] ? level_store+0x3b7/0x42e [md_mod]
> > Jul 6 21:10:04 mars kernel: [ 7766.952667] [<ffffffffa017eb1f>] ? md_attr_store+0x77/0x96 [md_mod]
> > Jul 6 21:10:04 mars kernel: [ 7766.952670] [<ffffffff81136ede>] ? sysfs_write_file+0xe3/0x11f
> > Jul 6 21:10:04 mars kernel: [ 7766.952674] [<ffffffff810e7bc6>] ? vfs_write+0xa4/0x101
> > Jul 6 21:10:04 mars kernel: [ 7766.952677] [<ffffffff810e7cd6>] ? sys_write+0x45/0x6b
> > Jul 6 21:10:04 mars kernel: [ 7766.952680] [<ffffffff810089c2>] ? system_call_fastpath+0x16/0x1b
> > Jul 6 21:10:04 mars kernel: [ 7766.952682] ---[ end trace 2d8a2ef8dd7ca7b7 ]---
> > Jul 6 21:10:04 mars kernel: [ 7766.952687] raid5: failed to create sysfs attributes for md0
> > Jul 6 21:21:10 mars kernel: [ 8432.471242] md: md0 stopped.
> > [...]
> > Jul 6 21:25:23 mars kernel: [ 8685.659605] raid5: raid level 6 set md0 active with 5 out of 6 devices, algorithm 18
> > Jul 6 21:25:23 mars kernel: [ 8685.659631] RAID5 conf printout:
> > Jul 6 21:25:23 mars kernel: [ 8685.659632] --- rd:6 wd:5
> > Jul 6 21:25:23 mars kernel: [ 8685.659634] disk 0, o:1, dev:sdk1
> > Jul 6 21:25:23 mars kernel: [ 8685.659636] disk 1, o:1, dev:sde1
> > Jul 6 21:25:23 mars kernel: [ 8685.659638] disk 2, o:1, dev:sdl1
> > Jul 6 21:25:23 mars kernel: [ 8685.659640] disk 3, o:1, dev:sdj1
> > Jul 6 21:25:23 mars kernel: [ 8685.659641] disk 4, o:1, dev:sdi1
> > Jul 6 21:25:23 mars kernel: [ 8685.659677] md0: detected capacity change from 0 to 6001196793856
> > Jul 6 21:25:23 mars kernel: [ 8685.659813] md0: unknown partition table
> > [...]
> > Jul 6 21:28:16 mars kernel: [ 8858.893402] md: recovery of RAID array md0
> > Jul 6 21:28:16 mars kernel: [ 8858.893405] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > Jul 6 21:28:16 mars kernel: [ 8858.893407] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> > Jul 6 21:28:16 mars kernel: [ 8858.893422] md: using 128k window, over a total of 1465135936 blocks.
> > [...]
> > Jul 6 21:31:18 mars kernel: [ 9040.089434] rr272x_1x:Device error information 0x1000000
> > Jul 6 21:31:18 mars kernel: [ 9040.089440] rr272x_1x:Task file error, StatusReg=0x41, ErrReg=0x4, LBA[0-3]=0x17fbec6,LBA[4-7]=0x0.
> > [...] (more of these messages)
> > Jul 6 23:54:20 mars kernel: [17622.164846] sd 8:0:6:0: [sdk] Unhandled error code
> > Jul 6 23:54:20 mars kernel: [17622.164848] sd 8:0:6:0: [sdk] Result: hostbyte=DID_ABORT driverbyte=DRIVER_INVALID
> > Jul 6 23:54:20 mars kernel: [17622.164851] sd 8:0:6:0: [sdk] CDB: Read(10): 28 00 49 ff 72 3f 00 00 38 00
> > Jul 6 23:54:20 mars kernel: [17622.164858] end_request: I/O error, dev sdk, sector 1241477695
> > Jul 6 23:54:20 mars kernel: [17622.181377] raid5:md0: read error corrected (8 sectors at 1241477632 on sdk1)
> > Jul 6 23:54:20 mars kernel: [17622.181503] raid5:md0: read error corrected (8 sectors at 1241477640 on sdk1)
> > Jul 6 23:54:20 mars kernel: [17622.181507] raid5:md0: read error corrected (8 sectors at 1241477648 on sdk1)
> > Jul 6 23:54:20 mars kernel: [17622.181510] raid5:md0: read error corrected (8 sectors at 1241477656 on sdk1)
> > Jul 6 23:54:20 mars kernel: [17622.181512] raid5:md0: read error corrected (8 sectors at 1241477664 on sdk1)
> > Jul 6 23:54:20 mars kernel: [17622.181515] raid5:md0: read error corrected (8 sectors at 1241477672 on sdk1)
> > Jul 6 23:54:20 mars kernel: [17622.181518] raid5:md0: read error corrected (8 sectors at 1241477680 on sdk1)
> > [...] (more sdk failures)
> > Jul 6 23:58:38 mars kernel: [17880.544030] INFO: task md0_resync:13565 blocked for more than 120 seconds.
> > Jul 6 23:58:38 mars kernel: [17880.544109] md0_resync D ffff88007cdf9500 0 13565 2 0x00000000
> > Jul 6 23:58:38 mars kernel: [17880.544114] ffff88007cdf9500 0000000000000046 0000000000000086 ffff88007400a800
> > Jul 6 23:58:38 mars kernel: [17880.544117] ffff88007ce25400 0000000000015200 0000000000015200 0000000000015200
> > Jul 6 23:58:38 mars kernel: [17880.544120] ffff88006baf5fd8 0000000000015200 ffff88007cdf9500 ffff88006baf5fd8
> > Jul 6 23:58:38 mars kernel: [17880.544124] Call Trace:
> > Jul 6 23:58:38 mars kernel: [17880.544145] [<ffffffffa019bdef>] ? unplug_slaves+0x70/0xa6 [raid456]
> > Jul 6 23:58:38 mars kernel: [17880.544150] [<ffffffffa019c17e>] ? get_active_stripe+0x26f/0x54b [raid456]
> > Jul 6 23:58:38 mars kernel: [17880.544154] [<ffffffff81040b38>] ? default_wake_function+0x0/0xf
> > Jul 6 23:58:38 mars kernel: [17880.544159] [<ffffffffa019ef3b>] ? sync_request+0x245/0x2d1 [raid456]
> > Jul 6 23:58:38 mars kernel: [17880.544166] [<ffffffffa01809d4>] ? is_mddev_idle+0xa2/0xf5 [md_mod]
> > Jul 6 23:58:38 mars kernel: [17880.544172] [<ffffffffa018113a>] ? md_do_sync+0x713/0xb0f [md_mod]
> > Jul 6 23:58:38 mars kernel: [17880.544177] [<ffffffff8105d495>] ? autoremove_wake_function+0x0/0x2a
> > Jul 6 23:58:38 mars kernel: [17880.544182] [<ffffffffa0181c89>] ? md_thread+0xf2/0x110 [md_mod]
> > Jul 6 23:58:38 mars kernel: [17880.544188] [<ffffffffa0181b97>] ? md_thread+0x0/0x110 [md_mod]
> > Jul 6 23:58:38 mars kernel: [17880.544190] [<ffffffff8105d061>] ? kthread+0x75/0x7d
> > Jul 6 23:58:38 mars kernel: [17880.544194] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> > Jul 6 23:58:38 mars kernel: [17880.544197] [<ffffffff8105cfec>] ? kthread+0x0/0x7d
> > Jul 6 23:58:38 mars kernel: [17880.544199] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: power outage while raid5->raid6 was in progress
2010-07-08 22:48 ` Neil Brown
2010-07-08 23:50 ` Sebastian Reichel
@ 2010-07-13 22:14 ` Sebastian Reichel
1 sibling, 0 replies; 8+ messages in thread
From: Sebastian Reichel @ 2010-07-13 22:14 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 4426 bytes --]
Hi,
I just did as you told:
* upgrading to 2.6.34.1
* go back to 5 disk raid5
* start raid5 -> raid6 again
This time I got no error and it began reshaping:
md0 : active raid6 sdk1[0] sdf1[5] sdg1[6] sdh1[7] sdi1[4] sdj1[3] sdl1[2] sde1[1]
5860543744 blocks super 0.91 level 6, 64k chunk, algorithm 18 [8/7] [UUUUU_UU]
[>....................] reshape = 2.8% (41047020/1465135936) finish=32916.5min speed=721K/sec
The reshaping got stuck again. Checking top I can see ksoftirqd uses 100% CPU on one
of the 2 cores. But there is one difference. This time there are no complains about any
disk device(s). Here is the dmesg output - The lower part (from 'blocked for more than
120 seconds') is repeated multiple times.
[ 612.736649] raid5: device sdk1 operational as raid disk 0
[ 612.736653] raid5: device sdi1 operational as raid disk 4
[ 612.736656] raid5: device sdj1 operational as raid disk 3
[ 612.736658] raid5: device sdl1 operational as raid disk 2
[ 612.736660] raid5: device sde1 operational as raid disk 1
[ 612.737301] raid5: allocated 6386kB for md0
[ 612.749007] 0: w=1 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 612.749011] 4: w=2 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 612.749014] 3: w=3 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 612.749016] 2: w=4 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 612.749018] 1: w=5 pa=0 pr=6 m=2 a=18 r=6 op1=0 op2=0
[ 612.749021] raid5: raid level 6 set md0 active with 5 out of 6 devices, algorithm 18
[ 612.749065] RAID5 conf printout:
[ 612.749067] --- rd:6 wd:5
[ 612.749069] disk 0, o:1, dev:sdk1
[ 612.749071] disk 1, o:1, dev:sde1
[ 612.749073] disk 2, o:1, dev:sdl1
[ 612.749075] disk 3, o:1, dev:sdj1
[ 612.749077] disk 4, o:1, dev:sdi1
[ 612.779095] RAID5 conf printout:
[ 612.779099] --- rd:8 wd:7
[ 612.779102] disk 0, o:1, dev:sdk1
[ 612.779104] disk 1, o:1, dev:sde1
[ 612.779105] disk 2, o:1, dev:sdl1
[ 612.779107] disk 3, o:1, dev:sdj1
[ 612.779109] disk 4, o:1, dev:sdi1
[ 612.779111] disk 5, o:1, dev:sdf1
[ 612.779117] RAID5 conf printout:
[ 612.779118] --- rd:8 wd:7
[ 612.779120] disk 0, o:1, dev:sdk1
[ 612.779122] disk 1, o:1, dev:sde1
[ 612.779123] disk 2, o:1, dev:sdl1
[ 612.779125] disk 3, o:1, dev:sdj1
[ 612.779127] disk 4, o:1, dev:sdi1
[ 612.779129] disk 5, o:1, dev:sdf1
[ 612.779130] disk 6, o:1, dev:sdg1
[ 612.779134] RAID5 conf printout:
[ 612.779135] --- rd:8 wd:7
[ 612.779136] disk 0, o:1, dev:sdk1
[ 612.779138] disk 1, o:1, dev:sde1
[ 612.779140] disk 2, o:1, dev:sdl1
[ 612.779142] disk 3, o:1, dev:sdj1
[ 612.779143] disk 4, o:1, dev:sdi1
[ 612.779145] disk 5, o:1, dev:sdf1
[ 612.779147] disk 6, o:1, dev:sdg1
[ 612.779149] disk 7, o:1, dev:sdh1
[ 612.779233] md: reshape of RAID array md0
[ 612.779235] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 612.779237] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[ 612.779244] md: using 128k window, over a total of 1465135936 blocks.
[ 2846.784515] Clocksource tsc unstable (delta = 4398042075725 ns)
[ 3000.536656] INFO: task md0_reshape:5554 blocked for more than 120 seconds.
[ 3000.536675] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.536699] md0_reshape D 0000000100099eb1 0 5554 2 0x00000000
[ 3000.536704] ffff88007e08f700 0000000000000046 0000000000000082 ffff88006d3d1290
[ 3000.536707] ffffffff81632020 0000000000015280 0000000000015280 0000000000015280
[ 3000.536711] ffff88007e1f3fd8 0000000000015280 ffff88007e08f700 ffff88007e1f3fd8
[ 3000.536714] Call Trace:
[ 3000.536735] [<ffffffffa019e735>] ? reshape_request+0x1e7/0x7a8 [raid456]
[ 3000.536741] [<ffffffff8105d519>] ? autoremove_wake_function+0x0/0x2a
[ 3000.536746] [<ffffffffa019ee87>] ? sync_request+0x191/0x2d1 [raid456]
[ 3000.536754] [<ffffffffa0180a0b>] ? is_mddev_idle+0xa2/0xf5 [md_mod]
[ 3000.536760] [<ffffffffa0181171>] ? md_do_sync+0x713/0xb0f [md_mod]
[ 3000.536764] [<ffffffff8103767e>] ? update_curr+0xa2/0x145
[ 3000.536771] [<ffffffffa0181d4a>] ? md_thread+0xf2/0x110 [md_mod]
[ 3000.536777] [<ffffffffa0181c58>] ? md_thread+0x0/0x110 [md_mod]
[ 3000.536779] [<ffffffff8105d0e5>] ? kthread+0x75/0x7d
[ 3000.536783] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[ 3000.536786] [<ffffffff8105d070>] ? kthread+0x0/0x7d
[ 3000.536788] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
-- Sebastian
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2010-07-13 22:14 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-07-07 20:41 power outage while raid5->raid6 was in progress Sebastian Reichel
2010-07-07 22:44 ` Neil Brown
2010-07-07 23:13 ` Sebastian Reichel
2010-07-08 1:21 ` Neil Brown
2010-07-08 8:48 ` Sebastian Reichel
2010-07-08 22:48 ` Neil Brown
2010-07-08 23:50 ` Sebastian Reichel
2010-07-13 22:14 ` Sebastian Reichel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.