* Assembly of RAID6 with 48 disk fails
@ 2016-06-30 15:33 Grunewald, Soeren
2016-06-30 21:06 ` Phil Turmel
0 siblings, 1 reply; 3+ messages in thread
From: Grunewald, Soeren @ 2016-06-30 15:33 UTC (permalink / raw)
To: linux-raid mailinglist
[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]
Hi All,
After a crash, probably caused by some raid issue, the array can't be
assembled again. It looks like 8 disk were disappearing form the raid at
the same time, which then lead to the system crash. Now I'm unable to
get the array back to live again. Because the whole system was on the
failing array, I can't access any additional information to check what
happened. The system is a 8 year old SUN Fire X4540 system running under
Ubuntu 12.04.5 (kernel-3.13.0-91 and mdadm-3.2.5-1ubuntu0.3), which is
mainly used for data conversion (converting very large files from one
format into another). The boot partition is placed on a 2GB ssd and the
rest is running on the 48disk raid6 array. I have booted the system from
a usb stick with clonezilla (kernel 4.4.x + mdadm 3.3) and started
digging...
Since this is the first time, that a I face such a large array and such
an issue, I better follow the advice from the Linux-Raid-Wiki and
request for help.
As one can see in the attached log, the 'Events' count of 40 drives is
4391 and 4385 for the 8 others. We have the same picture for the
'State', 40 drives state 'clean' and 8 drives 'active'. So I tried
'mdadm --assemble --force ...'. This fixed the event count and faulty
flag on 4 of the 8 disks. But still the array can't be created.
Except one smart error (1 Currently unreadable (pending) sectors) do all
other drives pass the extended smart test. I know, this does not mean
that the drives are not damaged or broken.
Anyway, how should I process to get the array working again?
--
Thanks and regards,
Soeren
[-- Attachment #2: flashdaqsvr1-md-recovery.txt --]
[-- Type: text/plain, Size: 56086 bytes --]
user@flashdaqsvr1:~$ uname -a
Linux flashdaqsvr1 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
user@flashdaqsvr1:~$ lspci | grep SCSI
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 04)
03:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 04)
04:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 04)
41:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 04)
42:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 04)
43:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 04)
user@flashdaqsvr1:~$ lspci -s '02:00.0' -vvvv
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 04)
Subsystem: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 19
Region 0: I/O ports at b800 [size=256]
Region 1: Memory at df7fc000 (64-bit, non-prefetchable) [size=16K]
Region 3: Memory at df7e0000 (64-bit, non-prefetchable) [size=64K]
Expansion ROM at df400000 [disabled] [size=2M]
Capabilities: <access denied>
Kernel driver in use: mptsas
Kernel modules: mptsas
user@flashdaqsvr1:~$ mdadm --version
mdadm - v3.3 - 3rd September 2013
user@flashdaqsvr1:~$ cat mdadm.conf
ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=47 UUID=9b5294f9:a73d4576:301ab56d:af23c26d name=flashdaqsvr1:0
spares=1 devices=/dev/sdax1,/dev/sdaw1,/dev/sdau1,/dev/sdav1,/dev/sdat1,/dev/sdar1,/dev/sdas1,/dev/sdaq1,/dev/sdap1,/dev/sdao1,/dev/sdam1,/dev/sdan1,/dev/sdal1,/dev/sdak1,/dev/sdaj1,/dev/sdai1,/dev/sdah1,/dev/sdaf1,/dev/sdag1,/dev/sdae1,/dev/sdad1,/dev/sdac1,/dev/sdab1,/dev/sdaa1,/dev/sdy1,/dev/sdz1,/dev/sdx1,/dev/sdw1,/dev/sdu1,/dev/sdv1,/dev/sdt1,/dev/sds1,/dev/sdq1,/dev/sdr1,/dev/sdp1,/dev/sdo1,/dev/sdn1,/dev/sdm1,/dev/sdk1,/dev/sdl1,/dev/sdi1,/dev/sdj1,/dev/sdg1,/dev/sdh1,/dev/sde1,/dev/sdf1,/dev/sdd1,/dev/sdc1
user@flashdaqsvr1:~$ sudo mdadm --examine /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1 /dev/sdaa1 /dev/sdab1 /dev/sdac1 /dev/sdad1 /dev/sdae1 /dev/sdaf1 /dev/sdag1 /dev/sdah1 /dev/sdai1 /dev/sdaj1 /dev/sdak1 /dev/sdal1 /dev/sdam1 /dev/sdan1 /dev/sdao1 /dev/sdap1 /dev/sdaq1 /dev/sdar1 /dev/sdas1 /dev/sdat1 /dev/sdau1 /dev/sdav1 /dev/sdaw1 /dev/sdax1
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 9508f7a6:29c6051d:6d32cb05:804fb915
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 81bd39e3 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : ebc3c9ba:d55fd8ee:fd085a43:b88bfef5
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 8536a1bb - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 23
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : b16e707c:e7865b2e:fc53747c:8dfcf2c0
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 8a6f2f72 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 35
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 1435b6f5:9274bdb6:353f17d4:4fcc3e31
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 54059e59 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 080ceba1:4679ae77:990323d0:ef7cd0bd
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 49c8ef11 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : fa4206a0:7c1474fa:ea17f433:0f4501fe
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 6eab9db6 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 24
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 0cc73360:996fbd4c:9af2b019:c794a4f9
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 6282a758 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 36
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 721dbdd6:f0160888:240be7c0:20a8cd94
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 56b5d0d6 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 89cc9294:29bdd61f:b51fa68e:e26557fb
Update Time : Wed Jun 29 12:40:45 2016
Checksum : e0a2f884 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 13
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : e3ab47d2:792cbbed:8ff2680a:dc25ce6c
Update Time : Wed Jun 29 12:40:45 2016
Checksum : d975da0e - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 25
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 10a32d77:7e569ae1:8fcf5c26:fd815ae4
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 5bb346e - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 37
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : f68e8f68:7b067eb5:8c83a14e:0c0540cf
Update Time : Wed Jun 29 12:40:45 2016
Checksum : de2b0739 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : e3003a59:407e6a95:69a702a1:88c97df1
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 2360d951 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 14
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 0148ea90:cfe79ab4:854e469f:e47f0605
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 8c0de781 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 26
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 7f727449:05b1824c:8ebb0ec1:89710cf1
Update Time : Wed Jun 29 12:40:45 2016
Checksum : ea4e39ef - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 38
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdr1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : a822d82e:9e04abf9:b83cbe1c:7d1cfc61
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 497969ac - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sds1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 344e6cad:e71c7530:e4b73a21:9e80b87d
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 1f108cda - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 15
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdt1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : d5b1212e:8d53d328:afd861ea:d097d428
Update Time : Wed Jun 29 12:40:45 2016
Checksum : c675f2a - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 27
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdu1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : d18d6fea:ed2cfc6d:59b0c5a0:8618566a
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 5c36cf3 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 39
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdv1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 2ffb0541:74b517d2:58841b77:2dba0fb3
Update Time : Wed Jun 29 12:40:45 2016
Checksum : df84d85a - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdw1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 3d5b5e68:9a367a8e:3c5a61a0:db732447
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 809a492c - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 16
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdx1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : c24f63e8:0ee12608:c5aa1f99:1d67b324
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 50992bfc - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 28
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdy1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : c91efeb9:7cc23027:5acc0e10:2cb46cae
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 41e64b21 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 40
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdz1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 5394ef42:3f50fc37:ef211747:00b4dd06
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 6b1ca3b3 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaa1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 7ce483aa:1a0c43f6:587f52cc:6801b9dd
Update Time : Wed Jun 29 12:40:12 2016
Checksum : ed705ad3 - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 17
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdab1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 4935ac0e:ab67256b:720418af:ac15ee90
Update Time : Wed Jun 29 12:40:12 2016
Checksum : 5c75a09a - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 29
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdac1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 3ef8ee26:344c9fbe:ec657996:ed634c32
Update Time : Wed Jun 29 12:40:12 2016
Checksum : 50f1f7df - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 41
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdad1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 04be66f0:b5a64ae6:cbd6d6a4:12221d11
Update Time : Wed Jun 29 12:40:12 2016
Checksum : 2f434708 - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdae1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 69ded561:3a8ce15d:5c12f354:9f24407b
Update Time : Wed Jun 29 12:40:12 2016
Checksum : 32888b1b - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 18
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 8fccfecf:d732f70f:3b74565a:67d351d3
Update Time : Wed Jun 29 12:40:12 2016
Checksum : b03c3091 - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 30
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdag1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 1d14348b:cb328952:3fb558b9:10dfbc0d
Update Time : Wed Jun 29 12:40:12 2016
Checksum : 4770c4cc - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 42
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdah1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : active
Device UUID : 6edecf6f:8ba7caff:b9257ae4:0861f7b1
Update Time : Wed Jun 29 12:40:12 2016
Checksum : a8a9f6fd - correct
Events : 4385
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdai1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 4b7c0175:40aedae5:7179c5e0:2c868606
Update Time : Wed Jun 29 12:40:45 2016
Checksum : e4641369 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 19
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 936f6d2d:000f5a85:712eee6e:e2607a10
Update Time : Wed Jun 29 12:40:45 2016
Checksum : d46bf732 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 31
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdak1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 61bd6e77:40164fbb:2fbcf802:0326ae71
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 49a09f2c - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 43
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdal1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 91fdf97e:fcfe2c7d:7c3203d4:65099c00
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 730221a4 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdam1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : ce720042:1015321e:f235f593:2b1d6f5f
Update Time : Wed Jun 29 12:40:45 2016
Checksum : f5d2c43c - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 20
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdan1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 5686c434:bbfa820a:1ca35c82:82b6fd68
Update Time : Wed Jun 29 12:40:45 2016
Checksum : ccddc3fc - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 32
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdao1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : f82b9076:a49f0cbc:eb2f81d1:1ce3ca47
Update Time : Wed Jun 29 12:40:45 2016
Checksum : ee24c7fd - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 44
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdap1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : cb756108:807a30a6:a4b786f3:3b48c6a3
Update Time : Wed Jun 29 12:40:45 2016
Checksum : e81ad961 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 3ad8f9c2:41d45cf1:d54898d0:93e2ed32
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 5a18c127 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 21
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdar1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 0a2d2823:cc58caf8:55e29aba:6128b494
Update Time : Wed Jun 29 12:40:45 2016
Checksum : d7d79dc - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 33
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdas1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : b208b8c0:e76f210c:a91e7b4c:bf40ee6d
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 297ec15c - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 45
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdat1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 0318e575:036b453c:d50db9aa:0854fb1f
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 1f1ace1b - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 10
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdau1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : fed64cdb:a9dc3590:64790b84:44ca46a0
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 3210e094 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 22
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdav1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : deb5d15a:1ec3ac10:90cff353:cd412c08
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 69da73a8 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 34
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaw1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : 32ce415e:b11e7100:1ed7e7fb:d5c2d398
Update Time : Wed Jun 29 12:40:45 2016
Checksum : 95aa7032 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 46
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdax1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 9b5294f9:a73d4576:301ab56d:af23c26d
Name : flashdaqsvr1:0 (local to host flashdaqsvr1)
Creation Time : Wed Aug 19 13:06:06 2015
Raid Level : raid6
Raid Devices : 47
Avail Dev Size : 1952186368 (930.88 GiB 999.52 GB)
Array Size : 43924170240 (41889.35 GiB 44978.35 GB)
Used Dev Size : 1952185344 (930.87 GiB 999.52 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1024 sectors
State : clean
Device UUID : ae6b229f:9501ad27:0ddb9259:f901670a
Update Time : Wed Jun 29 12:40:45 2016
Checksum : cd0533a5 - correct
Events : 4391
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA..AAAAAAAAA..AAAAAAAAAA..AAAAAAAAAA..AAAA ('A' == active, '.' == missing, 'R' == replacing)
user@flashdaqsvr1:~$ sudo mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1 /dev/sdaa1 /dev/sdab1 /dev/sdac1 /dev/sdad1 /dev/sdae1 /dev/sdaf1 /dev/sdag1 /dev/sdah1 /dev/sdai1 /dev/sdaj1 /dev/sdak1 /dev/sdal1 /dev/sdam1 /dev/sdan1 /dev/sdao1 /dev/sdap1 /dev/sdaq1 /dev/sdar1 /dev/sdas1 /dev/sdat1 /dev/sdau1 /dev/sdav1 /dev/sdaw1 /dev/sdax1
mdadm: forcing event count in /dev/sdad1(6) from 4385 upto 4391
mdadm: forcing event count in /dev/sdah1(7) from 4385 upto 4391
mdadm: forcing event count in /dev/sdaa1(17) from 4385 upto 4391
mdadm: forcing event count in /dev/sdae1(18) from 4385 upto 4391
mdadm: clearing FAULTY flag for device 27 in /dev/md0 for /dev/sdad1
mdadm: clearing FAULTY flag for device 31 in /dev/md0 for /dev/sdah1
mdadm: clearing FAULTY flag for device 24 in /dev/md0 for /dev/sdaa1
mdadm: clearing FAULTY flag for device 28 in /dev/md0 for /dev/sdae1
mdadm: Marking array /dev/md0 as 'clean'
mdadm: /dev/md0 assembled from 43 drives and 1 spare - not enough to start the array.
user@flashdaqsvr1:~$ sudo mdadm --assemble --verbose /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1 /dev/sdaa1 /dev/sdab1 /dev/sdac1 /dev/sdad1 /dev/sdae1 /dev/sdaf1 /dev/sdag1 /dev/sdah1 /dev/sdai1 /dev/sdaj1 /dev/sdak1 /dev/sdal1 /dev/sdam1 /dev/sdan1 /dev/sdao1 /dev/sdap1 /dev/sdaq1 /dev/sdar1 /dev/sdas1 /dev/sdat1 /dev/sdau1 /dev/sdav1 /dev/sdaw1 /dev/sdax1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 11.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 23.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 35.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 12.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 24.
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 36.
mdadm: /dev/sdj1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdk1 is identified as a member of /dev/md0, slot 13.
mdadm: /dev/sdl1 is identified as a member of /dev/md0, slot 25.
mdadm: /dev/sdm1 is identified as a member of /dev/md0, slot 37.
mdadm: /dev/sdn1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdo1 is identified as a member of /dev/md0, slot 14.
mdadm: /dev/sdp1 is identified as a member of /dev/md0, slot 26.
mdadm: /dev/sdq1 is identified as a member of /dev/md0, slot 38.
mdadm: /dev/sdr1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sds1 is identified as a member of /dev/md0, slot 15.
mdadm: /dev/sdt1 is identified as a member of /dev/md0, slot 27.
mdadm: /dev/sdu1 is identified as a member of /dev/md0, slot 39.
mdadm: /dev/sdv1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdw1 is identified as a member of /dev/md0, slot 16.
mdadm: /dev/sdx1 is identified as a member of /dev/md0, slot 28.
mdadm: /dev/sdy1 is identified as a member of /dev/md0, slot 40.
mdadm: /dev/sdz1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdaa1 is identified as a member of /dev/md0, slot 17.
mdadm: /dev/sdab1 is identified as a member of /dev/md0, slot 29.
mdadm: /dev/sdac1 is identified as a member of /dev/md0, slot 41.
mdadm: /dev/sdad1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdae1 is identified as a member of /dev/md0, slot 18.
mdadm: /dev/sdaf1 is identified as a member of /dev/md0, slot 30.
mdadm: /dev/sdag1 is identified as a member of /dev/md0, slot 42.
mdadm: /dev/sdah1 is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdai1 is identified as a member of /dev/md0, slot 19.
mdadm: /dev/sdaj1 is identified as a member of /dev/md0, slot 31.
mdadm: /dev/sdak1 is identified as a member of /dev/md0, slot 43.
mdadm: /dev/sdal1 is identified as a member of /dev/md0, slot 8.
mdadm: /dev/sdam1 is identified as a member of /dev/md0, slot 20.
mdadm: /dev/sdan1 is identified as a member of /dev/md0, slot 32.
mdadm: /dev/sdao1 is identified as a member of /dev/md0, slot 44.
mdadm: /dev/sdap1 is identified as a member of /dev/md0, slot 9.
mdadm: /dev/sdaq1 is identified as a member of /dev/md0, slot 21.
mdadm: /dev/sdar1 is identified as a member of /dev/md0, slot 33.
mdadm: /dev/sdas1 is identified as a member of /dev/md0, slot 45.
mdadm: /dev/sdat1 is identified as a member of /dev/md0, slot 10.
mdadm: /dev/sdau1 is identified as a member of /dev/md0, slot 22.
mdadm: /dev/sdav1 is identified as a member of /dev/md0, slot 34.
mdadm: /dev/sdaw1 is identified as a member of /dev/md0, slot 46.
mdadm: /dev/sdax1 is identified as a member of /dev/md0, slot -1.
mdadm: added /dev/sdj1 to /dev/md0 as 1
mdadm: added /dev/sdn1 to /dev/md0 as 2
mdadm: added /dev/sdr1 to /dev/md0 as 3
mdadm: added /dev/sdv1 to /dev/md0 as 4
mdadm: added /dev/sdz1 to /dev/md0 as 5
mdadm: added /dev/sdad1 to /dev/md0 as 6
mdadm: added /dev/sdah1 to /dev/md0 as 7
mdadm: added /dev/sdal1 to /dev/md0 as 8
mdadm: added /dev/sdap1 to /dev/md0 as 9
mdadm: added /dev/sdat1 to /dev/md0 as 10
mdadm: added /dev/sdc1 to /dev/md0 as 11
mdadm: added /dev/sdg1 to /dev/md0 as 12
mdadm: added /dev/sdk1 to /dev/md0 as 13
mdadm: added /dev/sdo1 to /dev/md0 as 14
mdadm: added /dev/sds1 to /dev/md0 as 15
mdadm: added /dev/sdw1 to /dev/md0 as 16
mdadm: added /dev/sdaa1 to /dev/md0 as 17
mdadm: added /dev/sdae1 to /dev/md0 as 18
mdadm: added /dev/sdai1 to /dev/md0 as 19
mdadm: added /dev/sdam1 to /dev/md0 as 20
mdadm: added /dev/sdaq1 to /dev/md0 as 21
mdadm: added /dev/sdau1 to /dev/md0 as 22
mdadm: added /dev/sdd1 to /dev/md0 as 23
mdadm: added /dev/sdh1 to /dev/md0 as 24
mdadm: added /dev/sdl1 to /dev/md0 as 25
mdadm: added /dev/sdp1 to /dev/md0 as 26
mdadm: added /dev/sdt1 to /dev/md0 as 27
mdadm: added /dev/sdx1 to /dev/md0 as 28
mdadm: added /dev/sdab1 to /dev/md0 as 29 (possibly out of date)
mdadm: added /dev/sdaf1 to /dev/md0 as 30 (possibly out of date)
mdadm: added /dev/sdaj1 to /dev/md0 as 31
mdadm: added /dev/sdan1 to /dev/md0 as 32
mdadm: added /dev/sdar1 to /dev/md0 as 33
mdadm: added /dev/sdav1 to /dev/md0 as 34
mdadm: added /dev/sde1 to /dev/md0 as 35
mdadm: added /dev/sdi1 to /dev/md0 as 36
mdadm: added /dev/sdm1 to /dev/md0 as 37
mdadm: added /dev/sdq1 to /dev/md0 as 38
mdadm: added /dev/sdu1 to /dev/md0 as 39
mdadm: added /dev/sdy1 to /dev/md0 as 40
mdadm: added /dev/sdac1 to /dev/md0 as 41 (possibly out of date)
mdadm: added /dev/sdag1 to /dev/md0 as 42 (possibly out of date)
mdadm: added /dev/sdak1 to /dev/md0 as 43
mdadm: added /dev/sdao1 to /dev/md0 as 44
mdadm: added /dev/sdas1 to /dev/md0 as 45
mdadm: added /dev/sdaw1 to /dev/md0 as 46
mdadm: added /dev/sdax1 to /dev/md0 as -1
mdadm: added /dev/sdf1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 43 drives and 1 spare - not enough to start the array.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Assembly of RAID6 with 48 disk fails
2016-06-30 15:33 Assembly of RAID6 with 48 disk fails Grunewald, Soeren
@ 2016-06-30 21:06 ` Phil Turmel
2016-07-01 7:24 ` Soeren Grunewald
0 siblings, 1 reply; 3+ messages in thread
From: Phil Turmel @ 2016-06-30 21:06 UTC (permalink / raw)
To: Grunewald, Soeren, linux-raid mailinglist
On 06/30/2016 11:33 AM, Grunewald, Soeren wrote:
> Hi All,
>
> After a crash, probably caused by some raid issue, the array can't be
> assembled again. It looks like 8 disk were disappearing form the raid at
> the same time, which then lead to the system crash. Now I'm unable to
> get the array back to live again. Because the whole system was on the
> failing array, I can't access any additional information to check what
> happened. The system is a 8 year old SUN Fire X4540 system running under
> Ubuntu 12.04.5 (kernel-3.13.0-91 and mdadm-3.2.5-1ubuntu0.3), which is
> mainly used for data conversion (converting very large files from one
> format into another). The boot partition is placed on a 2GB ssd and the
> rest is running on the 48disk raid6 array. I have booted the system from
> a usb stick with clonezilla (kernel 4.4.x + mdadm 3.3) and started
> digging...
>
> Since this is the first time, that a I face such a large array and such
> an issue, I better follow the advice from the Linux-Raid-Wiki and
> request for help.
>
> As one can see in the attached log, the 'Events' count of 40 drives is
> 4391 and 4385 for the 8 others. We have the same picture for the
> 'State', 40 drives state 'clean' and 8 drives 'active'. So I tried
> 'mdadm --assemble --force ...'. This fixed the event count and faulty
> flag on 4 of the 8 disks. But still the array can't be created.
>
> Except one smart error (1 Currently unreadable (pending) sectors) do all
> other drives pass the extended smart test. I know, this does not mean
> that the drives are not damaged or broken.
>
> Anyway, how should I process to get the array working again?
>
You're the second person in the last month to run into this... There
was a bug in mdadm that makes it fail with --assemble --force when there
are several out-of-date devices.
The fix was to clone the mdadm git tree, compile the latest, and run
that stand-alone binary to perform the forced assembly. I haven't tried
to determine which released version has the fix.
Phil
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Assembly of RAID6 with 48 disk fails
2016-06-30 21:06 ` Phil Turmel
@ 2016-07-01 7:24 ` Soeren Grunewald
0 siblings, 0 replies; 3+ messages in thread
From: Soeren Grunewald @ 2016-07-01 7:24 UTC (permalink / raw)
To: Phil Turmel, linux-raid mailinglist
Hi Phil,
Thanks for your advice, that actually helped. I just build git head,
used '--assemble --force' and it was working. All filesystems on the
array are still valid, no data loss :) But this is the end for this
machine, since it was clearly a controller fault.
So thank you very very much, also from the users.
--
Cheers,
Soeren
On 06/30/2016 11:06 PM, Phil Turmel wrote:
> On 06/30/2016 11:33 AM, Grunewald, Soeren wrote:
>> Hi All,
>>
>> After a crash, probably caused by some raid issue, the array can't be
>> assembled again. It looks like 8 disk were disappearing form the raid at
>> the same time, which then lead to the system crash. Now I'm unable to
>> get the array back to live again. Because the whole system was on the
>> failing array, I can't access any additional information to check what
>> happened. The system is a 8 year old SUN Fire X4540 system running under
>> Ubuntu 12.04.5 (kernel-3.13.0-91 and mdadm-3.2.5-1ubuntu0.3), which is
>> mainly used for data conversion (converting very large files from one
>> format into another). The boot partition is placed on a 2GB ssd and the
>> rest is running on the 48disk raid6 array. I have booted the system from
>> a usb stick with clonezilla (kernel 4.4.x + mdadm 3.3) and started
>> digging...
>>
>> Since this is the first time, that a I face such a large array and such
>> an issue, I better follow the advice from the Linux-Raid-Wiki and
>> request for help.
>>
>> As one can see in the attached log, the 'Events' count of 40 drives is
>> 4391 and 4385 for the 8 others. We have the same picture for the
>> 'State', 40 drives state 'clean' and 8 drives 'active'. So I tried
>> 'mdadm --assemble --force ...'. This fixed the event count and faulty
>> flag on 4 of the 8 disks. But still the array can't be created.
>>
>> Except one smart error (1 Currently unreadable (pending) sectors) do all
>> other drives pass the extended smart test. I know, this does not mean
>> that the drives are not damaged or broken.
>>
>> Anyway, how should I process to get the array working again?
>>
> You're the second person in the last month to run into this... There
> was a bug in mdadm that makes it fail with --assemble --force when there
> are several out-of-date devices.
>
> The fix was to clone the mdadm git tree, compile the latest, and run
> that stand-alone binary to perform the forced assembly. I haven't tried
> to determine which released version has the fix.
>
> Phil
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2016-07-01 7:24 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-30 15:33 Assembly of RAID6 with 48 disk fails Grunewald, Soeren
2016-06-30 21:06 ` Phil Turmel
2016-07-01 7:24 ` Soeren Grunewald
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.