* force raid reassembly
[not found] <DB4PR07MB409B56E9B9173E33079902ED58A0@DB4PR07MB409.eurprd07.prod.outlook.com>
@ 2017-08-08 22:27 ` Bryan Hepworth
2017-08-08 22:44 ` Joe Landman
2017-08-09 10:34 ` Joe Landman
0 siblings, 2 replies; 6+ messages in thread
From: Bryan Hepworth @ 2017-08-08 22:27 UTC (permalink / raw)
To: linux-raid
Hi all
I'd be grateful of another pair of eyes just to check and verify what I'm thinking. I've had an array fail and I've checked the components with the mdadm --examine command. Five out of the six event counts are correct and I'm hoping I can get it to assemble with the force option: -
"mdadm --assemble --force /dev/mdX <list of devices>"
mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1
Output from --examine is as follows: -
[root@inscribe4 dev]# mdadm --examine /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 44b7b461 - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 dev]# mdadm --examine /dev/sdg1
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 28758d54 - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 dev]# mdadm --examine /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : b8cf8981 - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 dev]# mdadm --examine /dev/sdf1
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 8e37d25d - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 dev]# mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Jul 31 22:50:37 2017
Checksum : 74d320d6 - correct
Events : 119879
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 dev]# mdadm --examine /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : be5ec20e:249645e5:22aeca25:a60c7599
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 2e529fae - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
Thanks
Bryan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: force raid reassembly
2017-08-08 22:27 ` force raid reassembly Bryan Hepworth
@ 2017-08-08 22:44 ` Joe Landman
[not found] ` <WM!c7e5d36941a99d370e99c9ecff040c34199c6ab48c7c1548590a2e9075a62b0e4a8a44d8c43afabb53f77be4c9108e21!@mailhub-mx3.ncl.ac.uk>
2017-08-09 10:34 ` Joe Landman
1 sibling, 1 reply; 6+ messages in thread
From: Joe Landman @ 2017-08-08 22:44 UTC (permalink / raw)
To: Bryan Hepworth, linux-raid
On 8/8/17 3:27 PM, Bryan Hepworth wrote:
> Hi all
>
> I'd be grateful of another pair of eyes just to check and verify what I'm thinking. I've had an array fail and I've checked the components with the mdadm --examine command. Five out of the six event counts are correct and I'm hoping I can get it to assemble with the force option: -
>
> "mdadm --assemble --force /dev/mdX <list of devices>"
>
> mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1
I don't think you need to explicitly list the devices (/dev/sd(..)) in
the force
You could do something like this:
mdadm --examine --scan | grep md0 > /tmp/md0.conf
mdadm -As -c /tmp/md0.conf
if this doesn't work, then the force *may* be an option, but be very
careful with this.
>
> Output from --examine is as follows: -
>
>
> [root@inscribe4 dev]# mdadm --examine /dev/sde1
> /dev/sde1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 44b7b461 - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : spare
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdg1
> /dev/sdg1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 28758d54 - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdd1
> /dev/sdd1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : b8cf8981 - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdf1
> /dev/sdf1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 8e37d25d - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdb1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Jul 31 22:50:37 2017
> Checksum : 74d320d6 - correct
> Events : 119879
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 5
> Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdc1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : be5ec20e:249645e5:22aeca25:a60c7599
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 2e529fae - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 4
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> Thanks
>
> Bryan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
c: +1 734 612 4615
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: force raid reassembly
[not found] ` <DB4PR07MB40970AB64E677A452895432D58B0@DB4PR07MB409.eurprd07.prod.outlook.com>
@ 2017-08-09 7:40 ` Bryan Hepworth
0 siblings, 0 replies; 6+ messages in thread
From: Bryan Hepworth @ 2017-08-09 7:40 UTC (permalink / raw)
To: Joe Landman, linux-raid
>-----Original Message-----
>From: Joe Landman [mailto:joe.landman@gmail.com]
>Sent: 08 August 2017 23:44
>To: Bryan Hepworth; linux-raid@vger.kernel.org
>Subject: Re: force raid reassembly
>
>
>
>On 8/8/17 3:27 PM, Bryan Hepworth wrote:
>> Hi all
>>
>> I'd be grateful of another pair of eyes just to check and verify what I'm
>thinking. I've had an array fail and I've checked the components with the
>mdadm --examine command. Five out of the six event counts are correct and
>I'm hoping I can get it to assemble with the force option: -
>>
>> "mdadm --assemble --force /dev/mdX <list of devices>"
>>
>> mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1
>/dev/sde1 /dev/sdb1 /dev/sdc1
>
>I don't think you need to explicitly list the devices (/dev/sd(..)) in
>the force
>
>You could do something like this:
>
>mdadm --examine --scan | grep md0 > /tmp/md0.conf
>mdadm -As -c /tmp/md0.conf
>
>if this doesn't work, then the force *may* be an option, but be very
>careful with this.
>>
Hi Joe
Thanks for the reply, unfortunately that came back as follows: -
[root@inscribe4 ~]# mdadm --examine --scan
ARRAY /dev/md/intermediate1 metadata=1.2 UUID=64a34aa3:3703e1b0:37d3173a:bec0c874 name=intermediate1
ARRAY /dev/md/intermediate0 metadata=1.2 UUID=ace00fb5:8771d55a:cf36728b:ddc3ce89 name=intermediate0
spares=1
ARRAY /dev/md/128 metadata=1.2 UUID=d59b4fac:ade4fdbf:13ebe850:5e70ba42 name=igmimager.ncl.ac.uk:128
[root@inscribe4 md]# mdadm --examine --scan | grep intermediate0 > /tmp/intermediate0.conf
[root@inscribe4 md]# cat /tmp/intermediate0.conf
ARRAY /dev/md/intermediate0 metadata=1.2 UUID=ace00fb5:8771d55a:cf36728b:ddc3ce89 name=intermediate0
[root@inscribe4 md]# mdadm --assemble --scan -c /tmp/intermediate0.conf
mdadm: /dev/md/intermediate0 assembled from 4 drives and 1 spare - not enough to start the array.
[root@inscribe4 md]#
Looks like I'll have to use the force option unless you can think of a better way
Thanks
Bryan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: force raid reassembly
2017-08-08 22:27 ` force raid reassembly Bryan Hepworth
2017-08-08 22:44 ` Joe Landman
@ 2017-08-09 10:34 ` Joe Landman
2017-08-12 17:44 ` Anthony Youngman
1 sibling, 1 reply; 6+ messages in thread
From: Joe Landman @ 2017-08-09 10:34 UTC (permalink / raw)
To: Bryan Hepworth, linux-raid
Ok, you are missing 2 drives in a 6 drive raid. This is generally bad
news. Are you sure all your drives are accessible? Can you read partition
tables from each? If not, f2f hen a force won't likely do anything.
Can you give context and logs entries on what happend to cause this? Also
do you have backups of the data... raid is not a backup itself ... if need
be, can you recover from a recent backup?
On August 8, 2017 3:39:46 PM Bryan Hepworth
<bryan.hepworth@newcastle.ac.uk> wrote:
> Hi all
>
> I'd be grateful of another pair of eyes just to check and verify what I'm
> thinking. I've had an array fail and I've checked the components with the
> mdadm --examine command. Five out of the six event counts are correct and
> I'm hoping I can get it to assemble with the force option: -
>
> "mdadm --assemble --force /dev/mdX <list of devices>"
>
> mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sde1
> /dev/sdb1 /dev/sdc1
>
> Output from --examine is as follows: -
>
>
> [root@inscribe4 dev]# mdadm --examine /dev/sde1
> /dev/sde1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 44b7b461 - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : spare
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdg1
> /dev/sdg1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 28758d54 - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdd1
> /dev/sdd1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : b8cf8981 - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdf1
> /dev/sdf1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 8e37d25d - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdb1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Jul 31 22:50:37 2017
> Checksum : 74d320d6 - correct
> Events : 119879
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 5
> Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@inscribe4 dev]# mdadm --examine /dev/sdc1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
> Name : intermediate0
> Creation Time : Tue Oct 15 21:03:50 2013
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1154 sectors
> State : clean
> Device UUID : be5ec20e:249645e5:22aeca25:a60c7599
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Aug 8 14:16:53 2017
> Checksum : 2e529fae - correct
> Events : 119894
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 4
> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>
> Thanks
>
> Bryan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: force raid reassembly
2017-08-09 10:34 ` Joe Landman
@ 2017-08-12 17:44 ` Anthony Youngman
[not found] ` <WM!13c5dc8d433950e8d178eb39ac57b28caef2d362792181894459f3aa56870cedd0dcd8b5aa75be7d1b060d3021a4afe9!@mailhub-mx3.ncl.ac.uk>
0 siblings, 1 reply; 6+ messages in thread
From: Anthony Youngman @ 2017-08-12 17:44 UTC (permalink / raw)
To: Joe Landman, Bryan Hepworth, linux-raid
Only thing is, sdb1 says "AAA.AA", which is a bit weird, unless sdb1 is
the very last one, which is the last dot in AAA.A. of all the others.
If five drives all have almost identical event counts, I'd be inclined
to ddrescue the dodgy one (probably sdb1) and hope it all works out.
The alternative is to ddrescue both dodgy drives, and write the program
that marks blocks as dodgy. Then you can hope that a scan can rebuild
everything.
Cheers,
Wol
On 09/08/17 11:34, Joe Landman wrote:
> Ok, you are missing 2 drives in a 6 drive raid. This is generally bad
> news. Are you sure all your drives are accessible? Can you read
> partition tables from each? If not, f2f hen a force won't likely do
> anything.
>
> Can you give context and logs entries on what happend to cause this?
> Also do you have backups of the data... raid is not a backup itself ...
> if need be, can you recover from a recent backup?
>
>
> On August 8, 2017 3:39:46 PM Bryan Hepworth
> <bryan.hepworth@newcastle.ac.uk> wrote:
>
>> Hi all
>>
>> I'd be grateful of another pair of eyes just to check and verify what
>> I'm thinking. I've had an array fail and I've checked the components
>> with the mdadm --examine command. Five out of the six event counts are
>> correct and I'm hoping I can get it to assemble with the force option: -
>>
>> "mdadm --assemble --force /dev/mdX <list of devices>"
>>
>> mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1
>> /dev/sde1 /dev/sdb1 /dev/sdc1
>>
>> Output from --examine is as follows: -
>>
>>
>> [root@inscribe4 dev]# mdadm --examine /dev/sde1
>> /dev/sde1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
>> Name : intermediate0
>> Creation Time : Tue Oct 15 21:03:50 2013
>> Raid Level : raid5
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
>> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
>> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1154 sectors
>> State : clean
>> Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Tue Aug 8 14:16:53 2017
>> Checksum : 44b7b461 - correct
>> Events : 119894
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : spare
>> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@inscribe4 dev]# mdadm --examine /dev/sdg1
>> /dev/sdg1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
>> Name : intermediate0
>> Creation Time : Tue Oct 15 21:03:50 2013
>> Raid Level : raid5
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
>> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
>> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1154 sectors
>> State : clean
>> Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Tue Aug 8 14:16:53 2017
>> Checksum : 28758d54 - correct
>> Events : 119894
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@inscribe4 dev]# mdadm --examine /dev/sdd1
>> /dev/sdd1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
>> Name : intermediate0
>> Creation Time : Tue Oct 15 21:03:50 2013
>> Raid Level : raid5
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
>> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
>> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1154 sectors
>> State : clean
>> Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Tue Aug 8 14:16:53 2017
>> Checksum : b8cf8981 - correct
>> Events : 119894
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@inscribe4 dev]# mdadm --examine /dev/sdf1
>> /dev/sdf1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
>> Name : intermediate0
>> Creation Time : Tue Oct 15 21:03:50 2013
>> Raid Level : raid5
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
>> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
>> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1154 sectors
>> State : clean
>> Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Tue Aug 8 14:16:53 2017
>> Checksum : 8e37d25d - correct
>> Events : 119894
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@inscribe4 dev]# mdadm --examine /dev/sdb1
>> /dev/sdb1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
>> Name : intermediate0
>> Creation Time : Tue Oct 15 21:03:50 2013
>> Raid Level : raid5
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
>> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
>> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1154 sectors
>> State : clean
>> Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Mon Jul 31 22:50:37 2017
>> Checksum : 74d320d6 - correct
>> Events : 119879
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 5
>> Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@inscribe4 dev]# mdadm --examine /dev/sdc1
>> /dev/sdc1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
>> Name : intermediate0
>> Creation Time : Tue Oct 15 21:03:50 2013
>> Raid Level : raid5
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
>> Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
>> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1154 sectors
>> State : clean
>> Device UUID : be5ec20e:249645e5:22aeca25:a60c7599
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Tue Aug 8 14:16:53 2017
>> Checksum : 2e529fae - correct
>> Events : 119894
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 4
>> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> Thanks
>>
>> Bryan
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: force raid reassembly
[not found] ` <DBXPR07MB415CB58355ECBF4DB0D6632D59B0@DBXPR07MB415.eurprd07.prod.outlook.com>
@ 2017-08-25 8:03 ` Bryan Hepworth
0 siblings, 0 replies; 6+ messages in thread
From: Bryan Hepworth @ 2017-08-25 8:03 UTC (permalink / raw)
To: Anthony Youngman, Joe Landman, linux-raid
>-----Original Message-----
>From: Anthony Youngman [mailto:antlists@youngman.org.uk]
>Sent: 12 August 2017 18:44
>To: Joe Landman; Bryan Hepworth; linux-raid@vger.kernel.org
>Subject: Re: force raid reassembly
>
>Only thing is, sdb1 says "AAA.AA", which is a bit weird, unless sdb1 is
>the very last one, which is the last dot in AAA.A. of all the others.
>
>If five drives all have almost identical event counts, I'd be inclined
>to ddrescue the dodgy one (probably sdb1) and hope it all works out.
>
>The alternative is to ddrescue both dodgy drives, and write the program
>that marks blocks as dodgy. Then you can hope that a scan can rebuild
>everything.
>
>Cheers,
>Wol
Hi Anthony
I've got another quick question, hope you can see what you reckon...
The disks are in an md1200 storage shelf and all seem to have settled down now. Could you take a quick look at the output and see if you think the same advice stands.
Thanks so much.
Bryan
[root@inscribe4 ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md125 : active (auto-read-only) raid5 sdh1[4] sdg1[6] sdi1[3] sdk1[1] sdl1[0] sdj1[7]
9767552000 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
md126 : inactive sdb1[4] sde1[7] sdc1[0] sdf1[1] sdd1[8](S)
9766904645 blocks super 1.2
md127 : active (auto-read-only) raid6 sdo[2] sdq[4] sdm[0] sdw[10] sdt[12] sds[6] sdn[1] sdr[5] sdu[8] sdp[3] sdx[11] sdv[9]
19535129600 blocks super 1.2 level 6, 512k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
unused devices: <none>
[root@inscribe4 ~]# mdadm --examine /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Jul 31 22:50:37 2017
Checksum : 74d320d6 - correct
Events : 119879
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 ~]# mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : be5ec20e:249645e5:22aeca25:a60c7599
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 2e529fae - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 ~]# mdadm --examine /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : b8cf8981 - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 ~]# mdadm --examine /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 44b7b461 - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 ~]# mdadm --examine /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 8e37d25d - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 ~]# mdadm --examine /dev/sdf1
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89
Name : intermediate0
Creation Time : Tue Oct 15 21:03:50 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 9766901760 (9314.44 GiB 10001.31 GB)
Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1154 sectors
State : clean
Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Aug 8 14:16:53 2017
Checksum : 28758d54 - correct
Events : 119894
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@inscribe4 ~]#
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-08-25 8:03 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <DB4PR07MB409B56E9B9173E33079902ED58A0@DB4PR07MB409.eurprd07.prod.outlook.com>
2017-08-08 22:27 ` force raid reassembly Bryan Hepworth
2017-08-08 22:44 ` Joe Landman
[not found] ` <WM!c7e5d36941a99d370e99c9ecff040c34199c6ab48c7c1548590a2e9075a62b0e4a8a44d8c43afabb53f77be4c9108e21!@mailhub-mx3.ncl.ac.uk>
[not found] ` <DB4PR07MB40970AB64E677A452895432D58B0@DB4PR07MB409.eurprd07.prod.outlook.com>
2017-08-09 7:40 ` Bryan Hepworth
2017-08-09 10:34 ` Joe Landman
2017-08-12 17:44 ` Anthony Youngman
[not found] ` <WM!13c5dc8d433950e8d178eb39ac57b28caef2d362792181894459f3aa56870cedd0dcd8b5aa75be7d1b060d3021a4afe9!@mailhub-mx3.ncl.ac.uk>
[not found] ` <DBXPR07MB415CB58355ECBF4DB0D6632D59B0@DBXPR07MB415.eurprd07.prod.outlook.com>
2017-08-25 8:03 ` Bryan Hepworth
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.