broken raid level 5 array caused by user error

broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Folks,

I'm running a raid level 5 with 4 devices for some years and tried to 
grow my array yesterday. I wanted to add two more devices and used the 
following commands:

mdadm --add /dev/md0 /dev/sdf1 /dev/sdg1
mdadm --grow --raid-devices=6 /dev/md0

So far, so good. Everything seems to work, but after about 2 hours, the 
reshape progress was still at 0.0% and now, my own stupidity kicked in. 
I checked the logs via journalctl (I'm running Centos 7) and read 
something about "main process died" or similar... then I decided to 
reboot.

After reboot, assembling the array failed:

mdadm: Failed to restore critical section for reshape, sorry. Possibly 
you needed to specify the --backup-file

But I did not have a backup file and so I panicked and made even worse 
decisions.

First I tried to assemble the array using --invalid-backup but it did 
not work. I should stop here and ask but I didn't. I read at some board, 
that rebuilding the original array with 4 devices will fix my problem. I 
did not validate this and entered the suggested command:

mdadm -CR /dev/md0 --metadata=1.2 -n4 -l5 -c512 /dev/sd[bcde]1 
--assume-clean

But this did not work (the array assembled but I could not access the 
ext4 filesystem), it seems that I assembled it in the wrong device 
order, so I also tried different (i.e. all possible) orders, but nothing 
helped ( I always used --assume-clean).

I guess this is the perfect guide for how _not_ to do it :(

I continued reading and found this:

http://serverfault.com/questions/347606/recover-raid-5-data-after-created-new-array-instead-of-re-using

This gave me some hope and now I wonder, if there is a way to get my 
data back, maybe the offset is wrong?

Things I know about the array:

metadata: 1.2
Left-symetric
chunk-size: 512

When I run mdadm --detail /dev/md0 it still shows an array size of 6TB, 
the UUID is also still the same

         Version : 1.2
   Creation Time : Mon Nov  9 00:00:40 2015
      Raid Level : raid5
      Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
   Used Dev Size : 1953380864 (1862.89 GiB 2000.26 GB)
    Raid Devices : 4
   Total Devices : 4
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Mon Nov  9 00:00:45 2015
           State : active
  Active Devices : 4
Working Devices : 4
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-symmetric
      Chunk Size : 512K

            Name : xxxx
            UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
          Events : 1

     Number   Major   Minor   RaidDevice State
        0       8       49        0      active sync   /dev/sdd1
        1       8       65        1      active sync   /dev/sde1
        2       8       33        2      active sync   /dev/sdc1
        3       8       17        3      active sync   /dev/sdb1



an mdadm --examine gives this results:

/dev/sdb1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
            Name : xxxx
   Creation Time : Mon Nov  9 00:00:40 2015
      Raid Level : raid5
    Raid Devices : 4

  Avail Dev Size : 3906766961 (1862.89 GiB 2000.26 GB)
      Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
   Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=5233 sectors
           State : clean
     Device UUID : e14f0e2d:a26a7b90:d7dbf780:e2218327

Internal Bitmap : 8 sectors from superblock
     Update Time : Mon Nov  9 00:00:45 2015
   Bad Block Log : 512 entries available at offset 72 sectors
        Checksum : 7a76b0d6 - correct
          Events : 1

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 3
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdc1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
            Name : xxxx
   Creation Time : Mon Nov  9 00:00:40 2015
      Raid Level : raid5
    Raid Devices : 4

  Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
      Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
   Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=3248 sectors
           State : clean
     Device UUID : d408e617:37f3f0f5:feb5d77f:07e57668

Internal Bitmap : 8 sectors from superblock
     Update Time : Mon Nov  9 00:00:45 2015
   Bad Block Log : 512 entries available at offset 72 sectors
        Checksum : a9787e9 - correct
          Events : 1

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 2
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd:
    MBR Magic : aa55
Partition[0] :   3907024002 sectors at           63 (type fd)

/dev/sdd1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
            Name : xxxx
   Creation Time : Mon Nov  9 00:00:40 2015
      Raid Level : raid5
    Raid Devices : 4

  Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
      Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
   Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=130 sectors
           State : clean
     Device UUID : faf7ec39:e7c0cb77:770a439d:18dc65a0

Internal Bitmap : 8 sectors from superblock
     Update Time : Mon Nov  9 00:00:45 2015
   Bad Block Log : 512 entries available at offset 72 sectors
        Checksum : 3d38419 - correct
          Events : 1

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 0
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sde1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
            Name : xxx
   Creation Time : Mon Nov  9 00:00:40 2015
      Raid Level : raid5
    Raid Devices : 4

  Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
      Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
   Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=3248 sectors
           State : clean
     Device UUID : fe31b351:3559f949:978035ae:616ae615

Internal Bitmap : 8 sectors from superblock
     Update Time : Mon Nov  9 00:00:45 2015
   Bad Block Log : 512 entries available at offset 72 sectors
        Checksum : 743a6702 - correct
          Events : 1

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 1
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)



I guess I know the old device order as well, I saved an old boot-log:

md: bind<sde1>
md: bind<sdb1>
md: bind<sdc1>
md: bind<sdd1>
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md/raid:md127: device sdd1 operational as raid disk 0
md/raid:md127: device sdc1 operational as raid disk 1
md/raid:md127: device sdb1 operational as raid disk 2
md/raid:md127: device sde1 operational as raid disk 3
md/raid:md127: allocated 4314kB
md/raid:md127: raid level 5 active with 4 out of 4 devices, algorithm 2
created bitmap (15 pages) for device md127
md127: bitmap initialized from disk: read 1 pages, set 0 of 29809 bits
md127: detected capacity change from 0 to 6001188667392



Please help me, I know I'm stupid and don't deserve it. I really hope, 
there is a chance for reovering the array.

Thanks a lot in advance

Mathias

Re: broken raid level 5 array caused by user error

[Mikael Abrahamsson]

On Mon, 9 Nov 2015, Mathias Mueller wrote:

> Please help me, I know I'm stupid and don't deserve it. I really hope, 
> there is a chance for reovering the array.

It's possible. There are numerous threads in the mailing list archives 
with people who have been in the same situation as you are in now. Without 
having a backup of the original superblocks, you're going to have to guess 
offsets. It might make sense to investigate what the defaults were for the 
mdadm you originally created the array with.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 11/09/2015 06:56 AM, Mikael Abrahamsson wrote:
> On Mon, 9 Nov 2015, Mathias Mueller wrote:
> 
>> Please help me, I know I'm stupid and don't deserve it. I really hope,
>> there is a chance for reovering the array.
> 
> It's possible. There are numerous threads in the mailing list archives
> with people who have been in the same situation as you are in now.
> Without having a backup of the original superblocks, you're going to
> have to guess offsets. It might make sense to investigate what the
> defaults were for the mdadm you originally created the array with.

Chunks size defaults have varied, too.

I would search the known first drive for an ext4 filesystem signature
and work backwards from there.  Show the output of this search:

dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep '  53 ef ' |head -n 1000

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

> I trimmed the rest because a valid signature was at the very beginning
> of the device.  So you had either v0.90 or v1.0 metadata.  I would bet
> it's v0.90.  I would also bet that the chunk size is 64k.
> 
> I suggest you try that first:
> 
> mdadm --create --assume-clean --level=5 --raid-devices=4 --chunk=64 \
>   --metadata=0.90 /dev/md1 /dev/sd{d,c,b,e}1
> 
> Do *not* mount it yet!
> 
> Do a non-destructive fsck to see if my guess was accurate:
> 
> fsck -n /dev/md1
> 
> Generally, if this has many errors, you would stop the array, and try
> the other order combinations (but always with sdd1 first) in turn.
> 
> However, you don't say, and we can't tell from the new metadata, how
> far into the reshape mdadm had gotten.  If very far, there will be
> many errors no matter what order you try, and your array is toast.
> 
> But a result with few fsck errors is almost certainly success,
> and you can mount your array.
> 
> BTW, if you recall where you found the suggestion to re-create your
> array, please go back and report there how crazy that is, and point
> them at this mailing list.
> 
> Phil


Hi Phil,

thanks for your reply and sorry for my mistakes, I'm new with the 
mailinglist.

I tried your suggestion but it did not work, fsck could not find a 
superblock. I'm very sure, that I used the metadata v1.2 (and also kind 
of sure that I used a chunk size of 512k). Sadly I tried to mount the 
array yesterday when I was trying the different device orders.

Is it helpful to post the hexdump of the other devices as well?

Thanks for your Help!

Re: broken raid level 5 array caused by user error

[Phil Turmel]

{Resend with list address corrected -- ignore the duplicate, Mathias }

Hi Matthias,

On 11/09/2015 02:36 PM, Mathias Mueller wrote:
>> for x in /dev/sd[b-e]1 ; do echo $x ; dd if=$x bs=4k count=2 |hexdump
>> -C ; done
> 
> /dev/sdb1
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
> *
> 00000400  00 20 47 07 16 10 1c 1d  67 9a 74 01 8b 5d a6 1c  |. G.....g.t..]..|
> 00000410  f5 1f 47 07 00 00 00 00  02 00 00 00 02 00 00 00  |..G.............|
> 00000420  00 80 00 00 00 80 00 00  00 20 00 00 00 00 00 00  |......... ......|
> 00000430  d9 45 8b 4c 00 00 27 00  53 ef 01 00 01 00 00 00  |.E.L..'.S.......|
> 00000440  2e 44 8b 4c 00 4e ed 00  00 00 00 00 01 00 00 00  |.D.L.N..........|
> 00000450  00 00 00 00 0b 00 00 00  00 01 00 00 3c 00 00 00  |............<...|

False positive. (Not an ext4 signature).

> /dev/sdc1
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
> *
> 00000400  00 20 47 07 16 10 1c 1d  67 9a 74 01 8b 5d a6 1c  |. G.....g.t..]..|
> 00000410  f5 1f 47 07 00 00 00 00  02 00 00 00 02 00 00 00  |..G.............|
> 00000420  00 80 00 00 00 80 00 00  00 20 00 00 00 00 00 00  |......... ......|
> 00000430  d9 45 8b 4c 00 00 27 00  53 ef 01 00 01 00 00 00  |.E.L..'.S.......|
> 00000440  2e 44 8b 4c 00 4e ed 00  00 00 00 00 01 00 00 00  |.D.L.N..........|

Also a false positive.

> /dev/sdd1
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
> *
> 00000400  00 20 47 07 90 0e 1c 1d  54 9a 74 01 75 aa c1 08  |. G.....T.t.u...|
> 00000410  35 47 33 07 00 00 00 00  02 00 00 00 02 00 00 00  |5G3.............|
> 00000420  00 80 00 00 00 80 00 00  00 20 00 00 0b b2 ca 4d  |......... .....M|
> 00000430  7b f0 ca 4d 15 00 21 00  53 ef 01 00 01 00 00 00  |{..M..!.S.......|
> 00000440  9d 7f 3b 4d 00 4e ed 00  00 00 00 00 01 00 00 00  |..;M.N..........|
> 00000450  00 00 00 00 0b 00 00 00  00 01 00 00 3c 00 00 00  |............<...|
> 00000460  42 02 00 00 7b 00 00 00  4f a7 6a c3 f8 12 42 1c  |B...{...O.j...B.|
> 00000470  8f d2 5b ec 1a e9 fb 4c  64 61 74 61 00 00 00 00  |..[....Ldata....|
> 00000480  00 00 00 00 00 00 00 00  2f 6d 65 64 69 61 2f 64  |......../media/d|
> 00000490  61 74 61 00 d8 dc 83 00  02 88 ff ff 38 da 59 4c  |ata.........8.YL|
> 000004a0  00 00 00 00 00 c6 70 c3  01 88 ff ff 00 96 56 23  |......p.......V#|
> 000004b0  02 88 ff ff 40 7b 21 1d  02 88 ff ff dd 75 18 81  |....@{!......u..|

This really is an ext4 signature.  Volume label 'data' last
mounted at '/media/data' on Wed, 11 May 2011 15:58:03 GMT.

Given that date and your certainty that your array was v1.2,
this must be garbage left over from a previous filesystem.

> /dev/sde1
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
> *
> 00000400  00 20 47 07 16 10 1c 1d  67 9a 74 01 8b 5d a6 1c  |. G.....g.t..]..|
> 00000410  f5 1f 47 07 00 00 00 00  02 00 00 00 02 00 00 00  |..G.............|
> 00000420  00 80 00 00 00 80 00 00  00 20 00 00 0a 88 62 51  |......... ....bQ|
> 00000430  d5 cb 62 51 05 00 24 00  53 ef 01 00 01 00 00 00  |..bQ..$.S.......|
> 00000440  54 d4 81 50 00 4e ed 00  00 00 00 00 01 00 00 00  |T..P.N..........|
> 00000450  00 00 00 00 0b 00 00 00  00 01 00 00 3c 00 00 00  |............<...|

Another false positive.

>> Also show the following more picky signature search for each drive:
>> dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep -E '[048c]30  .+  53 ef'
> 
> I will provide this as soon as the processes are finished, seems to take
> some more hours.

We'll just have to wait for this.  Your ext4 superblock should be in
there somewhere.

> there is also one more thing I remember, maybe it's helpful: when I was
> starting the reshape, there was this message:
> 
> mdadm: Need to backup 8192K of critical section..

If this was an actual error message, it may not have proceeded.  We just
don't know yet.

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

sadly, my ssh-session died while hexdumping over night, but I post what 
I got (restarted the processes anyway):

> dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep -E '[048c]30  .+  53 
> ef'

sdb1:
http://pastebin.com/ERehrYYR

sdc1:
http://pastebin.com/WMpRtPs5

sdd1:
http://pastebin.com/5QxGuGGY

sde1:
http://pastebin.com/CAC8Pkt9


Best regards

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

Hi Mathias,

On 11/10/2015 03:37 AM, Mathias Mueller wrote:
> Hi Phil,
> 
> sadly, my ssh-session died while hexdumping over night, but I post what
> I got (restarted the processes anyway):

'screen' is your friend for long-running server tasks.  It's installed in many distros by default.

> 
>> dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep -E '[048c]30  .+  53 ef'

I should have put "|head -n 50" on the end of that. :-)

> sdb1:
> http://pastebin.com/ERehrYYR

Pastebin expires.  The list accepts up to ~100k.  I'm pasting this
for the record:

> 00000430  d9 45 8b 4c 00 00 27 00  53 ef 01 00 01 00 00 00  |.E.L..'.S.......|
> 010d8030  9c a9 4c 27 20 97 fe 4c  53 ef 40 e8 dd 2c 1a d6  |..L' ..LS.@..,..|
> 04db5030  4b 5c ea 17 1d 0b b3 1c  53 ef 7c 3f 74 90 97 10  |K\......S.|?t...|
> 07ed8030  73 96 e8 8d 3a d5 9d c4  53 ef dc 1c d8 a5 1d 2b  |s...:...S......+|
> 131af430  c1 77 bd d9 84 c3 25 8f  53 ef 0c e6 17 3e ea 06  |.w....%.S....>..|
> 1ea2b830  87 83 16 e7 b6 ac 24 61  53 ef 70 53 fa 10 dc 27  |......$aS.pS...'|
> 1f79a430  db b2 90 27 87 1c bb bf  53 ef 26 7c 6f 76 23 34  |...'....S.&|ov#4|
> 2a21e830  a7 63 f4 cc 85 1c 7a 8c  53 ef 69 b1 a8 5d ae b0  |.c....z.S.i..]..|
> 3562a830  9d f4 99 e3 62 ae 6b 3f  53 ef 65 04 12 0d 3c 77  |....b.k?S.e...<w|
> 3562c430  15 8e ab 0d 2c cf a2 77  53 ef 03 73 06 6f ab cd  |....,..wS..s.o..|
> 3d254830  3a f2 3f 34 82 1f 8f 98  53 ef 9c 68 c3 81 dc f4  |:.?4....S..h....|
> 4040b030  01 01 10 06 5a 8b 26 43  53 ef ba e2 ce 33 ec 5c  |....Z.&CS....3.\|
> 410dc430  f4 40 bf 4d 8a f1 43 06  53 ef d8 c2 a4 69 ab 0a  |.@.M..C.S....i..|
> 4d8b2c30  32 ed b7 f6 15 bb ab d5  53 ef 1c 6e d6 4c 36 99  |2.......S..n.L6.|
> 4e100030  85 54 ac 57 46 8f a6 57  53 ef 9d 4c 50 60 cc f5  |.T.WF..WS..LP`..|
> 4f23fc30  a8 77 c6 f1 87 80 ca c1  53 ef c7 41 93 db 8b e6  |.w......S..A....|
> 52c41c30  d2 3e d6 ce 01 6d d4 f3  53 ef 3d 7a a2 db 95 5e  |.>...m..S.=z...^|
> 597e1c30  7b 4a 38 58 ff 4c 38 f1  53 ef 77 b1 6c 6a 0a d4  |{J8X.L8.S.w.lj..|
> 5e634430  b7 f2 21 f2 f6 06 0c ff  53 ef 1b a7 bb 78 76 a0  |..!.....S....xv.|
> 6546d430  e7 63 6a 5b 93 69 9e 8b  53 ef 50 99 76 4f 1a 61  |.cj[.i..S.P.vO.a|
> 65d2c430  45 27 46 b4 56 8a 91 ab  53 ef af fa 08 26 a4 99  |E'F.V...S....&..|
> 6625ec30  cd 99 a9 ae 65 08 b3 97  53 ef e3 7e 83 9b 6d a2  |....e...S..~..m.|
> 6b3cc030  cd 6c f0 fd f2 b1 f7 b6  53 ef 8f 9d a3 2a 3f 76  |.l......S....*?v|
> 6d7f2430  07 9b 5a 4f 01 f0 d3 d8  53 ef ed ef b8 26 38 5f  |..ZO....S....&8_|
> 70547030  74 8d 3b 05 2a 99 b4 4f  53 ef 69 20 2d ad 5f 2b  |t.;.*..OS.i -._+|
> 75a74830  f5 c8 3f 03 a9 df 9f 59  53 ef 68 e1 d4 51 9d c5  |..?....YS.h..Q..|
> 83393430  39 54 23 d3 5e b3 7a ef  53 ef 31 90 7f fb ff f3  |9T#.^.z.S.1.....|
> 8442d830  d7 32 76 ca c3 f8 bd a1  53 ef f2 5a 81 ff 27 f2  |.2v.....S..Z..'.|
> 8b4d2c30  b8 97 0c 36 a7 1c a0 29  53 ef 1c dc 21 6d f2 21  |...6...)S...!m.!|
> 9d1f4830  9e 65 ff fe fc 2e 58 e8  53 ef ca 99 9c ad b1 19  |.e....X.S.......|
> a1806030  1d 8e 2b 24 e2 ae 30 54  53 ef 8b 96 16 0b d3 93  |..+$..0TS.......|
> a7c4c030  ef aa f0 35 2d 59 2b d1  53 ef 24 d4 fe e9 4f 35  |...5-Y+.S.$...O5|
> a9597430  75 3a 95 b9 7b 70 c7 9f  53 ef 5c f8 8b 69 20 f0  |u:..{p..S.\..i .|
> ad5ab830  bd 10 15 57 80 89 fd fd  53 ef 2f 04 63 e1 f1 95  |...W....S./.c...|
> b1863830  45 bc 30 8c 0b 9f 2f aa  53 ef bd d1 6e a8 a9 42  |E.0.../.S...n..B|
> b213ec30  a3 3d ba 3d 23 4f 6b fa  53 ef 38 b6 b3 96 37 aa  |.=.=#Ok.S.8...7.|
> bb04a030  85 3b 32 5f 95 fb 42 f1  53 ef 63 7b f9 30 07 81  |.;2_..B.S.c{.0..|
> c7cf2830  66 51 61 82 c5 5a ec c4  53 ef 81 66 df f1 c6 00  |fQa..Z..S..f....|
> c8926c30  29 74 85 31 27 6d 49 1c  53 ef af cd 02 df 4c 2c  |)t.1'mI.S.....L,|
> ca006030  1d 8e 2b 24 e2 ae 30 54  53 ef 8b 96 16 0b d3 93  |..+$..0TS.......|
> cadfbc30  2c c1 b5 c5 72 88 3d bb  53 ef bf de b8 ae b8 a5  |,...r.=.S.......|
> cee41430  a0 51 cf d2 f0 28 78 e2  53 ef 3b c4 f8 aa 95 6c  |.Q...(x.S.;....l|
> d91a3c30  8e 28 30 d6 6f ed de 6e  53 ef 52 5e 27 67 cf f8  |.(0.o..nS.R^'g..|
> d9e76030  14 b0 31 80 64 52 f7 d4  53 ef 4a 48 e5 83 1e 55  |..1.dR..S.JH...U|
> da9ea830  d0 e5 18 b8 ed 03 90 be  53 ef d0 47 90 2f 01 e2  |........S..G./..|
> e6625c30  ca e9 df 4f 39 4d 6d 42  53 ef f1 3c 07 1b 58 de  |...O9MmBS..<..X.|
> f7284830  e7 fa 23 37 40 bc e5 9e  53 ef 01 3e 7e 91 35 5a  |..#7@...S..>~.5Z|
> f981ac30  20 17 eb ed 1e 8f e9 0e  53 ef ac f5 63 ee 9b 08  | .......S...c...|
> fa3cc030  b4 f6 f3 5f a7 16 6a 5c  53 ef 9c 6c 36 53 2c da  |..._..j\S..l6S,.|
> fd574030  db 77 39 c7 8d fe bb 7e  53 ef 5a 85 ea 36 41 8e  |.w9....~S.Z..6A.|

None of these have a write timestamp in the recent past. {Look for 0x55 or
0x56 in the 4th byte on each line.}

> 
> sdc1:
> http://pastebin.com/WMpRtPs5

> 00000430  d9 45 8b 4c 00 00 27 00  53 ef 01 00 01 00 00 00  |.E.L..'.S.......|
> 00100430  28 ab 3f 56 06 00 19 00  53 ef 01 00 01 00 00 00  |(.?V....S.......|                                                                                                                                                               

This ^^ is encouraging: Sun, 08 Nov 2015 20:06:00 GMT
It's in the right spot for data offset == 2048 if this
is the first device.

> 02b80030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|                                                                                                                                                               
> 08100030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|                                                                                                                                                               

These ^^ look like backup ext4 superblocks, written Thu, 07 May 2015
13:41:16 GMT.  If you had uptime from May all the way to November, this
is believable.  And there's more of them scattered below.

> 0a7dc030  45 ae b5 ff 5b 76 73 5b  53 ef 28 ca 3c 11 0c 0f  |E...[vs[S.(.<...|                                                                                                                                                               
> 0d600030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|                                                                                                                                                               
> 0d7a9030  0b f3 69 7e c6 3f 43 15  53 ef 91 f2 36 8f 89 74  |..i~.?C.S...6..t|                                                                                                                                                               
> 11eb9430  73 e9 26 d8 d5 22 07 cd  53 ef 90 13 35 29 92 db  |s.&.."..S...5)..|                                                                                                                                                               
> 12b80030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|                                                                                                                                                               
> 15af5830  60 dc 27 cc 15 97 82 6d  53 ef ab b0 77 a3 84 15  |`.'....mS...w...|                                                                                                                                                               
> 18100030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|                                                                                                                                                               
> 1e8e2030  c1 7a d1 05 cd a8 e0 c2  53 ef f6 7a 9c 33 50 1f  |.z......S..z.3P.|                                                                                                                                                               
> 1f60ac30  48 3c 15 58 ad ef d9 15  53 ef 5b 18 c7 0c 81 1d  |H<.X....S.[.....|                                                                                                                                                               
> 1fccf430  eb 93 ad 0e dc 75 56 3b  53 ef 6e d4 14 ab 2a 6d  |.....uV;S.n...*m|                                                                                                                                                               
> 267ccc30  3c 24 c2 5b 10 a4 9e f5  53 ef bf e1 7f 24 c6 a0  |<$.[....S....$..|                                                                                                                                                               
> 27e15c30  03 07 98 da d5 86 d6 4a  53 ef 20 93 89 b0 3f de  |.......JS. ...?.|                                                                                                                                                               
> 287bdc30  c1 6c 79 2f 8b e0 72 65  53 ef ed f8 92 5d 56 e6  |.ly/..reS....]V.|                                                                                                                                                               
> 2b843830  ab ad 5c 9a 3b e1 c5 22  53 ef ef 8d 1f a8 0f ff  |..\.;.."S.......|                                                                                                                                                               
> 2cac8030  7b 6d 61 51 b9 45 58 87  53 ef 30 13 39 6e 2d 9b  |{maQ.EX.S.0.9n-.|                                                                                                                                                               
> 35b55030  38 e4 b9 f2 9f 54 b2 2b  53 ef a0 b8 a6 4d f4 4b  |8....T.+S....M.K|                                                                                                                                                               
> 362b7030  ec 54 b8 8d 9e b0 73 88  53 ef 45 22 58 6e 58 a5  |.T....s.S.E"XnX.|                                                                                                                                                               
> 383f0830  0b 36 cd 48 d3 a0 5c 5a  53 ef 28 8a e3 7e 28 e2  |.6.H..\ZS.(..~(.|                                                                                                                                                               
> 40db5830  99 22 6a bd 33 f2 39 99  53 ef 86 e4 70 e8 bd d6  |."j.3.9.S...p...|                                                                                                                                                               
> 42b80030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|                                                                                                                                                               
> 4681d430  38 7b 67 9b a5 2e 96 44  53 ef 35 fd c9 e2 a9 71  |8{g....DS.5....q|                                                                                                                                                               
> 48100030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|                                                                                                                                                               
> 4a403c30  b1 25 08 7b 88 0a 69 3f  53 ef fc 93 c8 46 93 f3  |.%.{..i?S....F..|                                                                                                                                                               
> 4bcad030  f4 bf 6f fc 48 34 d6 72  53 ef bb c5 69 d0 d7 27  |..o.H4.rS...i..'|                                                                                                                                                               
> 4eed7830  68 32 c6 73 f4 f0 57 68  53 ef d5 5c 3a 3b ca 49  |h2.s..WhS..\:;.I|                                                                                                                                                               
> 51807430  46 f8 72 b4 ac 58 46 b9  53 ef ad aa d3 de 8c 94  |F.r..XF.S.......|                                                                                                                                                               
> 51ad4430  28 16 ed f1 f2 00 98 b8  53 ef e3 04 2c ad e9 70  |(.......S...,..p|                                                                                                                                                               
> 55491430  d7 3f 87 00 15 70 40 bc  53 ef 7d 18 a1 6e 67 e8  |.?...p@.S.}..ng.|                                                                                                                                                               
> 576ed030  2a f0 e4 4c ad 4a 19 a4  53 ef d8 3c bd 00 c7 7f  |*..L.J..S..<....|                                                                                                                                                               
> 62e40c30  6d 1e 1e 64 c6 29 fc 6c  53 ef 57 bd 2d 99 ca 87  |m..d.).lS.W.-...|                                                                                                                                                               
> 63d2d030  ab bc c3 76 a9 3e af 4c  53 ef 6e 72 11 6a cc 34  |...v.>.LS.nr.j.4|
> 6acc0c30  37 db ad 49 4c 9a 72 4a  53 ef 3b 6e 8f bb 18 6e  |7..IL.rJS.;n...n|
> 75359830  7c d8 e0 62 51 e1 70 fa  53 ef 5f a2 ea c1 92 0f  ||..bQ.p.S._.....|
> 799fb430  77 b2 62 cb 73 8c b5 48  53 ef 76 49 59 fc 97 ad  |w.b.s..HS.vIY...|
> 7badc430  3d 3c 7c 72 b5 0f 6c f1  53 ef cb 6d 6e 69 d4 e5  |=<|r..l.S..mni..|
> 81413c30  a8 58 4a be a2 f6 36 3a  53 ef 0c 7b 31 3c bf 8a  |.XJ...6:S..{1<..|
> 82b80030  7c 6b 4b 55 12 00 19 00  53 ef 00 00 01 00 00 00  ||kKU....S.......|
> 85182430  ef 21 64 5f 3b c4 46 7f  53 ef 9d fc 88 72 e8 d1  |.!d_;.F.S....r..|
> 86245030  86 ca b9 f4 cd ff a9 cd  53 ef ad 62 a8 e8 fb 0b  |........S..b....|
> 864d1430  6c a0 5b 34 9d e6 4b 97  53 ef 87 9b ea 8e 4f ac  |l.[4..K.S.....O.|
> 8cea9030  53 43 cc c0 93 a9 69 9b  53 ef 65 9f ee 82 75 ca  |SC....i.S.e...u.|
> 940c0030  32 8d 87 1f 79 f2 f5 13  53 ef bf 0f 9b a6 e8 6a  |2...y...S......j|
> 9bcd4430  e3 da 47 63 bc eb 10 54  53 ef 54 29 4d 82 07 d2  |..Gc...TS.T)M...|
> aa4e9430  b8 09 b8 5a ea 6e ab ad  53 ef 6f 0e 10 01 5f aa  |...Z.n..S.o..._.|
> b5343830  37 3a 8b f3 c9 bc dd 77  53 ef 38 9a f1 08 13 0e  |7:.....wS.8.....|
> b7c0b030  f6 cc 47 b4 b5 a7 cc 1b  53 ef 78 f8 9e 1e de 7d  |..G.....S.x....}|

> 
> sdd1:
> http://pastebin.com/5QxGuGGY

> 00000430  7b f0 ca 4d 15 00 21 00  53 ef 01 00 01 00 00 00  |{..M..!.S.......|
> 0171c430  b2 44 ab 17 a0 1e b0 43  53 ef c7 5c fb 83 95 47  |.D.....CS..\...G|
> 01734830  a7 81 e8 1d 83 04 88 1c  53 ef b4 e2 11 18 a9 90  |........S.......|
> 06715c30  16 cd a6 93 42 5a f1 21  53 ef 06 3c ea ac 96 2f  |....BZ.!S..<.../|
> 0942cc30  4b 9d 1a a7 04 46 2d 55  53 ef 3f 8c 1e 25 26 94  |K....F-US.?..%&.|
> 0a46b430  c4 43 bc ab 49 30 46 93  53 ef 63 8e ff c9 0e c8  |.C..I0F.S.c.....|
> 0cd7d030  8c 26 bd 38 15 80 e9 d6  53 ef 2a 24 4f 4f a8 af  |.&.8....S.*$OO..|
> 11faec30  7a 7d 11 5e bc b0 c7 4c  53 ef e7 81 e5 30 f2 7a  |z}.^...LS....0.z|
> 187cac30  d4 ce cf bc a5 fc c5 fa  53 ef 8d a6 0e ef 87 7d  |........S......}|
> 1df88030  96 4a 06 8f 63 89 90 9c  53 ef de 53 04 17 ea a6  |.J..c...S..S....|
> 1ec74830  95 08 9e 18 78 27 3c d2  53 ef 4f a8 d8 9b 5c 6a  |....x'<.S.O...\j|
> 1f7ef830  96 37 d6 93 a2 cb ff 1f  53 ef 6d e8 fc 5d 7d 19  |.7......S.m..]}.|
> 22c9a830  6d 89 a4 e7 66 d6 0f 57  53 ef 2c 39 47 c8 a7 cf  |m...f..WS.,9G...|
> 241d3c30  d6 67 39 79 e0 ef 6e a1  53 ef 28 00 fe 1d 38 89  |.g9y..n.S.(...8.|
> 243a8830  50 76 ae 32 e1 ae 73 c4  53 ef b8 30 2a ce cc 2e  |Pv.2..s.S..0*...|
> 278ed430  4b a0 32 0f e8 06 41 ce  53 ef da c0 a4 6b 57 31  |K.2...A.S....kW1|
> 2c1fac30  84 52 60 2c 5e 93 52 45  53 ef 30 37 98 ec c2 56  |.R`,^.RES.07...V|
> 2e503030  22 b6 6b 9b c1 4b 20 7f  53 ef ed 5a 35 34 a0 a5  |".k..K .S..Z54..|
> 390cd830  66 08 74 9b b7 e9 24 16  53 ef 77 71 11 4e b3 28  |f.t...$.S.wq.N.(|
> 3a85ac30  4b 5c 7d fc c0 99 29 7c  53 ef 58 cf ff 1a cf ff  |K\}...)|S.X.....|
> 66c15430  c1 29 7f aa 5e c2 9f ec  53 ef b5 32 ee 3d de ee  |.)..^...S..2.=..|
> 76755830  bb 69 6b 92 cb ba bd 6d  53 ef d6 65 4b 31 ad 03  |.ik....mS..eK1..|
> 851a4830  5c 81 d7 1b d3 2a 5e 83  53 ef 98 d4 5b 2e 4b 2b  |\....*^.S...[.K+|
> 8fda7030  4d f6 ca ba a9 34 f7 87  53 ef 7f 59 7b 70 cb e5  |M....4..S..Y{p..|
> 9cabe030  32 ee 5f 43 4d 11 0e 79  53 ef a5 60 1f c8 cb ac  |2._CM..yS..`....|
> a1293c30  3d 51 81 9a 08 a2 e0 c7  53 ef b0 ec 68 cd 27 50  |=Q......S...h.'P|
> a595bc30  40 70 9e 35 40 fe b7 af  53 ef 52 10 a9 d7 ca 81  |@p.5@...S.R.....|
> a8d12c30  2c c1 b5 c5 72 88 3d bb  53 ef bf de b8 ae b8 a5  |,...r.=.S.......|
> aa15dc30  db b5 ed c3 ee a7 6b c2  53 ef f1 1d 7f 59 69 95  |......k.S....Yi.|
> b963d830  30 a6 25 18 e4 55 b7 3b  53 ef 67 5b 25 59 76 31  |0.%..U.;S.g[%Yv1|
> baefc830  07 88 0f 60 34 a7 8d f1  53 ef 0b 5e 68 21 7f 4d  |...`4...S..^h!.M|
> bbb7b430  51 00 91 ad b6 c3 89 5b  53 ef cf 10 5c 54 ef ff  |Q......[S...\T..|
> bc850c30  e1 d6 6f 4b 52 26 f3 5c  53 ef 1f fd e7 3e 0e f5  |..oKR&.\S....>..|
> c046cc30  4b ad 54 a6 92 54 91 f7  53 ef b5 1f ea 1d 99 68  |K.T..T..S......h|
> c17c0830  37 63 0e 04 5f 09 9a 9d  53 ef 13 74 94 c0 0f eb  |7c.._...S..t....|
> c8080030  98 fd 13 ca db d8 86 59  53 ef 08 ce 0e 98 ec 76  |.......YS......v|
> cd36cc30  4b ad 54 a6 92 54 91 f7  53 ef b5 1f ea 1d 99 68  |K.T..T..S......h|
> d03e7830  f8 18 1d e9 70 61 5a e7  53 ef 3c d6 71 90 c2 23  |....paZ.S.<.q..#|
> d25fa430  a2 00 05 d1 6c e6 23 6d  53 ef 39 55 49 06 64 48  |....l.#mS.9UI.dH|
> d3e86430  b8 c8 8c 86 4b 03 1c 42  53 ef e1 16 79 c6 30 48  |....K..BS...y.0H|
> d7d0e030  64 dd c2 20 da e9 fe c1  53 ef e0 b7 04 71 be ed  |d.. ....S....q..|
> d7d79430  e0 3f 7f f3 16 48 3d ac  53 ef 4a 63 25 69 5f 6f  |.?...H=.S.Jc%i_o|
> d84e6c30  f2 de 6a 35 5d 41 94 d7  53 ef e9 39 2c f1 fb cd  |..j5]A..S..9,...|
> ddd98c30  ca 84 3a 09 8b b0 fb ee  53 ef 45 0f 07 04 b6 04  |..:.....S.E.....|
> ea2e1030  50 be a2 62 3c f0 db 68  53 ef ff a6 85 32 77 e5  |P..b<..hS....2w.|
> ef0b9830  84 5e c3 0d 52 48 76 53  53 ef e8 24 96 73 12 ac  |.^..RHvSS..$.s..|
> f8fe9830  f6 e8 24 f7 c2 9f 45 9a  53 ef 98 74 4b 5f 5d 80  |..$...E.S..tK_].|
> fac0e830  c2 56 65 0c e1 c0 cc ef  53 ef 12 18 b3 89 c4 2d  |.Ve.....S......-|
> 1034b7c30  ea 4b 6a 1b 9a 63 fc 9d  53 ef aa 53 5b e7 43 5f  |.Kj..c..S..S[.C_|
> 10a378430  ff d3 d8 4a 5f 7d 52 c0  53 ef 5b 50 66 f1 38 f5  |...J_}R.S.[Pf.8.|

Nothing here ^^.

> 
> sde1:
> http://pastebin.com/CAC8Pkt9

> 00000430  d5 cb 62 51 05 00 24 00  53 ef 01 00 01 00 00 00  |..bQ..$.S.......|
> 00100430  28 ab 3f 56 06 00 19 00  53 ef 01 00 01 00 ef 1a  |(.?V....S.......|

Another candidate for a first disk, with offset 2048.  Also written
Sun, 08 Nov 2015 20:06:00 GMT.  Either sdc or sde is your first disk.
The other is parity, which is the last disk on the first stripe, with
zeros in the matching offsets of the other two disks.

> 04975030  22 e0 84 41 68 0b 2a 3e  53 ef fa 4d ab cf 92 78  |"..Ah.*>S..M...x|
> 04d09830  09 96 1e 05 dd 53 70 02  53 ef c8 99 81 bb db 08  |.....Sp.S.......|
> 07554c30  ad fb d5 76 b9 04 8b ab  53 ef 6d 9e 7d 92 61 2b  |...v....S.m.}.a+|
> 08100030  7c 6b 49 55 12 00 19 00  53 ef 00 00 01 20 76 ed  ||kIU....S.... v.|

Another possible backup superblock, written Wed, 06 May 2015 01:16:44 GMT.
With more like it below.

> 0f3e5830  1b 2b 97 41 57 15 59 12  53 ef 5f f9 72 45 d5 67  |.+.AW.Y.S._.rE.g|
> 10e04430  32 95 f4 68 40 8b e0 b4  53 ef c8 fe 68 e4 05 3f  |2..h@...S...h..?|
> 17fac430  cb de dc 31 a5 e4 56 b9  53 ef 76 e5 b9 6c cd 58  |...1..V.S.v..l.X|
> 18100030  7c 6b 49 55 12 00 19 00  53 ef 00 00 01 20 76 ed  ||kIU....S.... v.|
> 1f9a1030  f3 95 67 eb f4 fb cb 82  53 ef a4 47 b5 aa 8f ab  |..g.....S..G....|
> 21ecf830  af 55 bc 3f 03 10 fa c6  53 ef 27 50 7a a1 0b 7d  |.U.?....S.'Pz..}|
> 25a48030  f0 f4 9e 58 8e a8 dc e8  53 ef 8f ee f7 5f f2 96  |...X....S...._..|
> 2693e830  14 01 94 72 71 13 73 fd  53 ef 59 75 aa 11 9d 0a  |...rq.s.S.Yu....|
> 26daf430  a9 96 ae a0 5d 4a 71 ad  53 ef e2 64 ee d6 bc ec  |....]Jq.S..d....|
> 2c3e2830  b5 77 fe 66 e0 9a 65 94  53 ef 2e 6e 4e dc 3d 4c  |.w.f..e.S..nN.=L|
> 2d091030  a5 49 93 09 a1 21 16 11  53 ef a9 48 0a 30 43 9c  |.I...!..S..H.0C.|
> 2dc5a030  b7 fd b1 20 3e 74 8b d6  53 ef 4f 78 06 30 88 69  |... >t..S.Ox.0.i|
> 2e604430  89 32 cb a1 9e 8e 87 37  53 ef a8 74 92 be 8f 57  |.2.....7S..t...W|
> 2ea74430  86 8d 2e 0e 11 8f 26 74  53 ef a1 8b 25 ff fb e2  |......&tS...%...|
> 35fdb430  ff 6d 5c 78 cd 3a 8c c3  53 ef 0a ac 87 9f a4 97  |.m\x.:..S.......|
> 399efc30  63 35 60 c7 52 2d 42 8d  53 ef 13 bd 8b 89 df bd  |c5`.R-B.S.......|
> 43eaa030  5b 05 c1 f4 ae b1 b9 b3  53 ef 14 a5 8e 4a ce 1e  |[.......S....J..|
> 48100030  7c 6b 49 55 12 00 19 00  53 ef 00 00 01 20 76 ed  ||kIU....S.... v.|
> 4c538830  67 7e 80 bd 68 d0 d8 d4  53 ef 4d 53 20 e6 40 51  |g~..h...S.MS .@Q|
> 4f810c30  46 0f 08 65 ab e1 fe 6a  53 ef 9c 7b 26 64 bf ff  |F..e...jS..{&d..|
> 59cf1430  df 83 79 60 ff 96 08 6d  53 ef 37 ef 98 a6 b6 29  |..y`...mS.7....)|
> 5f865030  59 fc ab 10 94 83 66 55  53 ef 75 da 43 d0 e8 c1  |Y.....fUS.u.C...|
> 6d536030  59 84 da 0c d1 7b 07 47  53 ef 10 cf 28 ea 73 6d  |Y....{.GS...(.sm|
> 6df76430  d1 3e 2e 1f 15 0e b6 73  53 ef 44 16 7a e7 14 fa  |.>.....sS.D.z...|
> 715f0430  bc bf 2c 95 a3 0d cb 9d  53 ef 4a 32 2d 40 ff f3  |..,.....S.J2-@..|
> 721f5c30  cb 64 cb ea 0a 89 90 a7  53 ef 5e 8f ff fb b0 6c  |.d......S.^....l|
> 7a702c30  2b 84 11 5a 9a 9e 14 42  53 ef 0e ec d6 b5 02 c4  |+..Z...BS.......|
> 7fa5d430  93 0a 2d e1 20 b5 d6 b4  53 ef 07 46 95 9f cb 1c  |..-. ...S..F....|
> 889fcc30  f0 d1 61 a4 73 79 ab b3  53 ef 06 90 38 a1 2a 25  |..a.sy..S...8.*%|
> 89611030  8e bd c1 20 bc 77 91 e9  53 ef b3 73 89 dc 6c 4a  |... .w..S..s..lJ|
> 8e67cc30  43 11 37 a8 86 37 b2 66  53 ef 14 a4 a8 a3 91 15  |C.7..7.fS.......|
> a8029030  ae 74 9a f0 03 c7 66 80  53 ef 3f 30 4e 83 a4 c2  |.t....f.S.?0N...|
> a82ec030  ca 76 cc 64 79 6d 57 25  53 ef 46 65 e0 b7 08 7a  |.v.dymW%S.Fe...z|
> addb3430  ff e7 ee 6b 67 9d 9e d7  53 ef e2 da b9 cf 5b f6  |...kg...S.....[.|
> b9411430  4a 01 00 3f c9 e2 22 15  53 ef dd 6f 45 ad fb f7  |J..?..".S..oE...|
> baf98030  ff fe e7 41 03 36 f7 11  53 ef ed fc 14 02 34 f0  |...A.6..S.....4.|
> bb6f5830  84 75 62 8d 11 48 20 f4  53 ef 04 36 62 71 20 d2  |.ub..H .S..6bq .|
> bcc59830  b5 e4 fd f7 b2 fe a3 30  53 ef 7a 0a b1 2e a2 9a  |.......0S.z.....|
> c0e42030  6e d6 35 c8 53 64 55 a3  53 ef b1 39 d4 78 99 c2  |n.5.SdU.S..9.x..|
> c57a3c30  2f 51 ce a9 9a 5f c9 4d  53 ef 04 d4 eb 8f 52 e6  |/Q..._.MS.....R.|
> ccb2e030  12 df 4a ff a6 c6 37 96  53 ef a7 f6 1c d6 ad d1  |..J...7.S.......|
> cfef5030  59 5c 47 9d e3 5b c7 18  53 ef 6e 1a 61 99 95 fe  |Y\G..[..S.n.a...|
> d63c1030  6c 4a f5 0f 8f 97 17 91  53 ef d1 cf 87 65 42 74  |lJ......S....eBt|
> d8100030  7c 6b 49 55 12 00 19 00  53 ef 00 00 01 20 76 ed  ||kIU....S.... v.|

So, we now know your data offset == 2048, with pretty high confidence.
We know the possible order combinations:
  sdc1 sdb1 sdd1 sde1
  sdc1 sdd1 sdb1 sde1
  sde1 sdb1 sdd1 sdc1
  sde1 sdd1 sdb1 sdc1

We don't know the chunk size for sure.  I don't remember when the default
changed from 64 to 512.  I'd try 64 first.

It's possible you could map the standard locations of ext4 superblocks
versus the locations noted above, and work out the order and chunk
size.  I'll leave that as an exercise for you, if you don't want to
just try the various combinations.  :-)

Remember, "fsck -n" after each --create --assume-clean to see if it
works well.

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

> 'screen' is your friend for long-running server tasks.  It's installed
> in many distros by default.

> I should have put "|head -n 50" on the end of that. :-)

I already restarted the process via screen, I just did not expect, that 
it will take so much time :)


> This ^^ is encouraging: Sun, 08 Nov 2015 20:06:00 GMT
> It's in the right spot for data offset == 2048 if this
> is the first device.

yes, this is about the last time I mounted the array


> These ^^ look like backup ext4 superblocks, written Thu, 07 May 2015
> 13:41:16 GMT.  If you had uptime from May all the way to November, this
> is believable.  And there's more of them scattered below.

nope, I moved the array to another box last evening sunday

> Another candidate for a first disk, with offset 2048.  Also written
> Sun, 08 Nov 2015 20:06:00 GMT.  Either sdc or sde is your first disk.
> The other is parity, which is the last disk on the first stripe, with
> zeros in the matching offsets of the other two disks.


> So, we now know your data offset == 2048, with pretty high confidence.
> We know the possible order combinations:
>   sdc1 sdb1 sdd1 sde1
>   sdc1 sdd1 sdb1 sde1
>   sde1 sdb1 sdd1 sdc1
>   sde1 sdd1 sdb1 sdc1
> 
> We don't know the chunk size for sure.  I don't remember when the 
> default
> changed from 64 to 512.  I'd try 64 first.
> 
> It's possible you could map the standard locations of ext4 superblocks
> versus the locations noted above, and work out the order and chunk
> size.  I'll leave that as an exercise for you, if you don't want to
> just try the various combinations.  :-)
> 
> Remember, "fsck -n" after each --create --assume-clean to see if it
> works well.

You are great, I'll give it a try and try the combinations, thx a lot!

Mathias

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

Is it correct to use --data-offset=1024 when I want to set an offset of 
2048? At least a mdadm --examine on one of the devices shows the offset 
is set to 2048 when using --data-offset=1024.

With --data-offset=1024 and both chunk sizes (64 & 512) I have something 
like a success when using this two orders:

sde sdd sdb sdc
sde sdb sdd sdc

I'm getting:

e2fsck 1.41.12 (17-May-2010)
/dev/md0 has unsupported feature(s): FEATURE_C16 FEATURE_C17 FEATURE_C18 
FEATURE_C19 FEATURE_C21 FEATURE_C22 FEATURE_C23 FEATURE_C25 FEATURE_C27 
FEATURE_C28 FEATURE_I29 FEATURE_R29
e2fsck: Get a newer version of e2fsck!

So I guess I should try some newer fsck version (I'm back on the old 
machine with centos 6, the failed reshaping was done with the new 
machine with centos 7).

Thanks

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 11/10/2015 10:20 AM, Mathias Mueller wrote:
> Hi Phil,
> 
> Is it correct to use --data-offset=1024 when I want to set an offset of
> 2048? At least a mdadm --examine on one of the devices shows the offset
> is set to 2048 when using --data-offset=1024.

This must be a bug in an old mdadm.  Don't use it.

> With --data-offset=1024 and both chunk sizes (64 & 512) I have something
> like a success when using this two orders:
> 
> sde sdd sdb sdc
> sde sdb sdd sdc
> 
> I'm getting:
> 
> e2fsck 1.41.12 (17-May-2010)
> /dev/md0 has unsupported feature(s): FEATURE_C16 FEATURE_C17 FEATURE_C18
> FEATURE_C19 FEATURE_C21 FEATURE_C22 FEATURE_C23 FEATURE_C25 FEATURE_C27
> FEATURE_C28 FEATURE_I29 FEATURE_R29
> e2fsck: Get a newer version of e2fsck!
> 
> So I guess I should try some newer fsck version (I'm back on the old
> machine with centos 6, the failed reshaping was done with the new
> machine with centos 7).

Whoa!  Don't do that.  Upgrade to CentOS 7, or go back to the other
machine.  You didn't mention switching machines, btw.  That would
explain the different drive names vs. roles.  You need to document the
device names vs. drive serial numbers on the current setup before moving
or upgrading.  Then adjust the names worked out above based on the
serial numbers in the new environment.

In general, device names cannot be considered reliable -- that's why
labels and uuids are the preferred way to identify things in today's
systems.  And why mdadm has superblocks to keep track of what's what.

And when you have your system fully running again, document the entire
layout with lsdrv [1].

Phil

[1] https://github.com/pturmel/lsdrv

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil

> Whoa!  Don't do that.  Upgrade to CentOS 7, or go back to the other
> machine.  You didn't mention switching machines, btw.  That would
> explain the different drive names vs. roles.  You need to document the
> device names vs. drive serial numbers on the current setup before 
> moving
> or upgrading.  Then adjust the names worked out above based on the
> serial numbers in the new environment.

oh I'm sorry, I should have make more clear, that I went back to the old 
system after the raid reshape failed. All outputs I posted were from the 
same (old one) machine.

Now I booted the latest centos 7 rescue, but fsck still tells the same:

/dev/md0 has unsupported feature(s): FEATURE_C16 FEATURE_C17 FEATURE_C18 
FEATURE_C19 FEATURE_C21 FEATURE_C22 FEATURE_C23 FEATURE_C25 FEATURE_C27 
FEATURE_C28 FEATURE_I29 FEATURE_R29
e2fsck: Get a newer version of e2fsck!

I created the array with this command:

mdadm --create --assume-clean -l5 -n4 --chunk 64 --data-offset 1024 
--metadata 1.2 /dev/md0 ......

This both combinations

The devices names are now different, but I adjusted the names by serial 
numbers as you told.

Also mdadm --examine on on of the drives tells "Data offset: 2048 
Sectors" when mdadm --create --data-offset 1024 is used. Is this normal? 
It's confusing me (using --data-offset 2048 on creation gives "Data 
offset: 2048 Sectors").

Thanks,

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 11/10/2015 04:02 PM, Mathias Mueller wrote:
> /dev/md0 has unsupported feature(s): FEATURE_C16 FEATURE_C17 FEATURE_C18
> FEATURE_C19 FEATURE_C21 FEATURE_C22 FEATURE_C23 FEATURE_C25 FEATURE_C27
> FEATURE_C28 FEATURE_I29 FEATURE_R29
> e2fsck: Get a newer version of e2fsck!

None of the feature bits are documented to exist.  The choice of first
drive must be wrong.

> I created the array with this command:
> 
> mdadm --create --assume-clean -l5 -n4 --chunk 64 --data-offset 1024
> --metadata 1.2 /dev/md0 ......
> 
> This both combinations

Well, I gave you *four* combinations for order, and two suggestions for
chunk size.  Eight combinations to try.

> The devices names are now different, but I adjusted the names by serial
> numbers as you told.

Ok.

> Also mdadm --examine on on of the drives tells "Data offset: 2048
> Sectors" when mdadm --create --data-offset 1024 is used. Is this normal?
> It's confusing me (using --data-offset 2048 on creation gives "Data
> offset: 2048 Sectors").

mdadm must be enforcing a minimum offset.

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

>> This both combinations
> 
> Well, I gave you *four* combinations for order, and two suggestions for
> chunk size.  Eight combinations to try.

Sorry, this two words were just the beginning of a sentence, which I 
didn't finish an forgot to remove :)

I tried 16 combinations at all (eight with --data-offset=1024 and eight 
with --data-offset=2048), the following four combinations gave the 
"/dev/md0 has unsupported feature(s)" message occurs:

--data-offset=1024 --chunk 64: sde sdd sdb sdc
--data-offset=1024 --chunk 512: sde sdd sdb sdc
--data-offset=1024 --chunk 64: sde sdb sdd sdc
--data-offset=1024 --chunk 512: sde sdb sdd sdc

the other four combinations with --data-offset=1024 and all eight 
combinations with --data-offset=2048 gave

fsck.ext2: Superblock invalid, trying backup superblocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0


>> /dev/md0 has unsupported feature(s): FEATURE_C16 FEATURE_C17 
>> FEATURE_C18
>> FEATURE_C19 FEATURE_C21 FEATURE_C22 FEATURE_C23 FEATURE_C25 
>> FEATURE_C27
>> FEATURE_C28 FEATURE_I29 FEATURE_R29
>> e2fsck: Get a newer version of e2fsck!
> 
> None of the feature bits are documented to exist.  The choice of first
> drive must be wrong.

oh that's good to know

>> Also mdadm --examine on on of the drives tells "Data offset: 2048
>> Sectors" when mdadm --create --data-offset 1024 is used. Is this 
>> normal?
>> It's confusing me (using --data-offset 2048 on creation gives "Data
>> offset: 2048 Sectors").
> 
> mdadm must be enforcing a minimum offset.

is it possible that the data-offset differs from device to device?

Thanks

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 11/10/2015 04:33 PM, Mathias Mueller wrote:
> Hi Phil,
> 
>>> This both combinations
>>
>> Well, I gave you *four* combinations for order, and two suggestions for
>> chunk size.  Eight combinations to try.
> 
> Sorry, this two words were just the beginning of a sentence, which I
> didn't finish an forgot to remove :)
> 
> I tried 16 combinations at all (eight with --data-offset=1024 and eight
> with --data-offset=2048), the following four combinations gave the
> "/dev/md0 has unsupported feature(s)" message occurs:
> 
> --data-offset=1024 --chunk 64: sde sdd sdb sdc
> --data-offset=1024 --chunk 512: sde sdd sdb sdc
> --data-offset=1024 --chunk 64: sde sdb sdd sdc
> --data-offset=1024 --chunk 512: sde sdb sdd sdc
> 
> the other four combinations with --data-offset=1024 and all eight
> combinations with --data-offset=2048 gave
> 
> fsck.ext2: Superblock invalid, trying backup superblocks...
> fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

You're not providing enough information.  Show the new vs. old
name/serial combinations, then show the --create and fsck
command/output, one by one.

For now, use data offset 2048 for all of them.

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil

> You're not providing enough information.  Show the new vs. old
> name/serial combinations, then show the --create and fsck
> command/output, one by one.

sorry, here you go:

sdb -> sde  Serial Number: JK1170YBHYV6MD
sdc -> sdf  Serial Number: JK1100YAG64A1T
sdd -> sdg  Serial Number: JK1121YAG7YDLS
sde -> sdh  Serial Number: ML0220F30PZUVD


> We know the possible order combinations:
>  sdc1 sdb1 sdd1 sde1
>  sdc1 sdd1 sdb1 sde1
>  sde1 sdb1 sdd1 sdc1
>  sde1 sdd1 sdb1 sdc1

so I used this combinations:

sdf1 sde1 sdg1 sdh1
sdf1 sdg1 sde1 sdh1
sdh1 sde1 sdg1 sdf1
sdh1 sdg1 sde1 sdf1


> For now, use data offset 2048 for all of them.

output of all eight combinations:


mdadm --create --assume-clean -l5 -n4 --chunk=64 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdf1 /dev/sde1 /dev/sdg1 /dev/sdh1
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Tue Nov 10 22:24:54 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Tue Nov 10 22:24:54 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Tue Nov 10 22:24:54 2015
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Tue Nov 10 22:24:54 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>

=======================

mdadm --create --assume-clean -l5 -n4 --chunk=512 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdf1 /dev/sde1 /dev/sdg1 /dev/sdh1
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:29:57 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:29:57 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:29:57 2015
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:29:57 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>


================================
================================

mdadm --create --assume-clean -l5 -n4 --chunk=64 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdf1 /dev/sdg1 /dev/sde1 /dev/sdh1
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:31:56 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:31:56 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:31:56 2015
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:31:56 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>

=====================

mdadm --create --assume-clean -l5 -n4 --chunk=512 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdf1 /dev/sdg1 /dev/sde1 /dev/sdh1
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:05 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:05 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:05 2015
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:05 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>


=====================
=====================

mdadm --create --assume-clean -l5 -n4 --chunk=64 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdh1 /dev/sde1 /dev/sdg1 /dev/sdf1
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:37:01 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:37:01 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:37:01 2015
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:37:01 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>

=====================

mdadm --create --assume-clean -l5 -n4 --chunk=512 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdh1 /dev/sde1 /dev/sdg1 /dev/sdf1
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:59 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:59 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:59 2015
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:34:59 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>


=====================
=====================

mdadm --create --assume-clean -l5 -n4 --chunk=64 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdh1 /dev/sdg1 /dev/sde1 /dev/sdf1
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:38:17 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:38:17 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:38:17 2015
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:38:17 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>

=====================

mdadm --create --assume-clean -l5 -n4 --chunk=512 --data-offset=2048 
--metadata=1.2 /dev/md0 /dev/sdh1 /dev/sdg1 /dev/sde1 /dev/sdf1
mdadm: /dev/sdh1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Mon Apr  8 11:04:10 2013
mdadm: /dev/sdh1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:39:16 2015
mdadm: /dev/sdg1 appears to contain an ext2fs file system
        size=1953512000K  mtime=Wed May 11 17:58:03 2011
mdadm: /dev/sdg1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:39:16 2015
mdadm: /dev/sde1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:39:16 2015
mdadm: /dev/sdf1 appears to contain an ext2fs file system
        size=1953513560K  mtime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sdf1 appears to be part of a raid array:
        level=raid5 devices=4 ctime=Wed Nov 11 00:39:16 2015
Continue creating array? y
mdadm: array /dev/md0 started.

fsck -n /dev/md0
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate 
superblock:
     e2fsck -b 8193 <device>


Thanks and best regards!

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 11/10/2015 06:47 PM, Mathias Mueller wrote:
> Hi Phil

> sorry, here you go:
> 
> sdb -> sde  Serial Number: JK1170YBHYV6MD
> sdc -> sdf  Serial Number: JK1100YAG64A1T
> sdd -> sdg  Serial Number: JK1121YAG7YDLS
> sde -> sdh  Serial Number: ML0220F30PZUVD
> 
> 
>> We know the possible order combinations:
>>  sdc1 sdb1 sdd1 sde1
>>  sdc1 sdd1 sdb1 sde1
>>  sde1 sdb1 sdd1 sdc1
>>  sde1 sdd1 sdb1 sdc1
> 
> so I used this combinations:
> 
> sdf1 sde1 sdg1 sdh1
> sdf1 sdg1 sde1 sdh1
> sdh1 sde1 sdg1 sdf1
> sdh1 sdg1 sde1 sdf1
> 
> 
>> For now, use data offset 2048 for all of them.
> 
> output of all eight combinations:

Hmmm.  Very strange.  Let's see the dump of the start of the data area of all four partitions:

for x in /dev/sd[efgh]1 ; do echo -e "\n$x\n" ; dd if=$x skip=2048 count=16 2>/dev/null |hexdump -C ; done

Phil

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 11/10/2015 07:06 PM, Mathias Mueller wrote:
> Hi Phil
> 
>> Hmmm.  Very strange.  Let's see the dump of the start of the data area
>> of all four partitions:
>>
>> for x in /dev/sd[efgh]1 ; do echo -e "\n$x\n" ; dd if=$x skip=2048
>> count=16 2>/dev/null |hexdump -C ; done

Very good.

sdf1 is definitely first.  sdh1 is definitely last.  I don't see
anything obviously wrong with the superblock.  The handful of fields
I've looked at are appropriate, including the magic number, write
timestamp, and last mount point.  (/srv).

So, I fully expected one of those first four combinations to work.  I
don't know why they didn't.

The fsck version dates to 2013, so it might be something related to
that.  My next step would be to try all of this after booting into a
modern environment, like the rescue cd from sysrescuecd.org.  Recheck
serial numbers and device names again, if you do that.

If that fails the same way, I'm at a loss.  You might want to point the
ext4 mailing list at this thread's archive and ask for advice.  There's
some possibility of trying various backup superblocks if they can be
located -- I'm not confident enough to advise you on that.

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

> So, I fully expected one of those first four combinations to work.  I
> don't know why they didn't.
> 
> The fsck version dates to 2013, so it might be something related to
> that.  My next step would be to try all of this after booting into a
> modern environment, like the rescue cd from sysrescuecd.org.  Recheck
> serial numbers and device names again, if you do that.

I tried the first four combinations with latest sysrescuecd, but the 
results are the same.

Mathias

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

first thanks a lot for all your help and time you already spent on me 
and my problem. I gathered some more information and hope you can help 
me a bit more.

I found this on serverfault.com and tried to figure out my raid layout 
using some image file that is stored on my raid for sure:
http://serverfault.com/questions/347606/recover-raid-5-data-after-created-new-array-instead-of-re-using

this is my current device mapping:

JK1100YAG64A1T -> sdc
ML0220F30PZUVD -> sdd (this is the disk I added last)
JK1121YAG7YDLS -> sde
JK1170YBHYV6MD -> sdf


First I  tried to figure out the chunk size. I used a jpg file that was 
stored on the raid for sure and read 8x32 Bytes with an offset of 64k 
from that file. Than I used bgrep to search for the bytestrings on my 
physical devices. I found five sequent byte strings on one physical 
device, so I think the chunk sice must be bigger than 256k.

Because I was pretty sure, that my chunk size is 512k, I was reading 
5x32Byte with an offset of 512k from my jpg file, to determine the order 
of the devices. After bgrep was done, I got the following results:

- Every bytestring was found 3x on my physical devices, I think it's 
because the file is existing 3x in different folders on my raid device 
(which is pretty certain, I know that the file is existing 2x at least)
- sdd1 seems to have a different offset as the other physical devices, 
it's the disk I added last after switching from debian to centos some 
years ago
- the following tables show at which offsets and on which physical 
devices the five byte strings have been found:


offset          sdc1      sdd1      sde1      sdf1
---------------------------------------------------
5400bd3000                 2
5400cc3000                           3         1
5400d43000       4                             5



offset          sdc1      sdd1      sde1      sdf1
---------------------------------------------------
87b4e52000                 3
87b4f42000       1                             2
87b4fc2000       5                   4



offset          sdc1      sdd1      sde1      sdf1
---------------------------------------------------
cb2d87f000                 2
cb2d96f000                                     1
cb2d9ef000       4                   3         5




So I think using the last table is the easiest one to figure out the 
partition order and the data offset between sdd1 and the other devices?


offset          sdc1      sdd1      sde1      sdf1
---------------------------------------------------
cb2d96f000                (2)        P         1
cb2d9ef000       4         P         3         5

If I am right, it would mean, that the order of my physical devices is 
sdf1 sdd1 sde1 sdc1 (without knowing which one is the first partition) 
and the offset between sdd1 and the other devices is f0000. Are my 
assumptions correct? How can I go on with this information?

Thanks a lot in advance

Best regards

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

Hi Mathias,

On 01/18/2016 10:33 AM, Mathias Mueller wrote:
> Hi Phil,
> 
> first thanks a lot for all your help and time you already spent on me
> and my problem. I gathered some more information and hope you can help
> me a bit more.

You're welcome.

> First I  tried to figure out the chunk size. I used a jpg file that was
> stored on the raid for sure and read 8x32 Bytes with an offset of 64k
> from that file. Than I used bgrep to search for the bytestrings on my
> physical devices. I found five sequent byte strings on one physical
> device, so I think the chunk sice must be bigger than 256k.

Very interesting.  You could go one step further and compare the jpeg
file contents in the first 1M against the locations found to determine
where the chunks actually start and end on each device.  The final
offset will be a chunk multiple before these boundaries.  Or do md5 sums
of 4k blocks to reduce the amount to inspect.

> If I am right, it would mean, that the order of my physical devices is
> sdf1 sdd1 sde1 sdc1 (without knowing which one is the first partition)
> and the offset between sdd1 and the other devices is f0000. Are my
> assumptions correct? How can I go on with this information?

This will help (and looks correct), but I don't remember what we did to
try to find the beginning of your filesystem.  Did we search for a
filesystem signature?  Please re-summarize that.

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

I forgot to add some information: when I was creating the bytestrings 
from my jpg file, I did not start from 0k but from 100k of the jpg file 
(to skip the jpg header).

> Very interesting.  You could go one step further and compare the jpeg
> file contents in the first 1M against the locations found to determine
> where the chunks actually start and end on each device.  The final
> offset will be a chunk multiple before these boundaries.  Or do md5 
> sums
> of 4k blocks to reduce the amount to inspect.

How exactly can I do this? Should I create more Bytestrings and do more 
brep with them on my physical devices? I have already results from 
searching bytestrings with an offset of 64k (starting from 100k to 612k 
of my jpeg file, so 9 bytestrings at all). Should I provide a table of 
the results?

> This will help (and looks correct), but I don't remember what we did to
> try to find the beginning of your filesystem.  Did we search for a
> filesystem signature?  Please re-summarize that.

Yes, we were searching for an ext4 signature, there were some fragments 
of old file systems, but you also found two promising signatures, both 
with this timestamp: Sun, 08 Nov 2015 20:06:00 GMT (which is 
approximately the last time I mounted the filesystem)

here is a link to the archived message:

http://www.spinics.net/lists/raid/msg50433.html

the device names have changed from the archived post to now:

sdb -> sdf  Serial Number: JK1170YBHYV6MD
sdc -> sdc  Serial Number: JK1100YAG64A1T
sdd -> sde  Serial Number: JK1121YAG7YDLS
sde -> sdd  Serial Number: ML0220F30PZUVD

Thanks and best regards

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

Hi Mathias,

On 01/19/2016 09:35 AM, Mathias Mueller wrote:
> Hi Phil,
> 
> I forgot to add some information: when I was creating the bytestrings
> from my jpg file, I did not start from 0k but from 100k of the jpg file
> (to skip the jpg header).

Ok. But I'm still not confident of chunk boundaries.

>> Very interesting.  You could go one step further and compare the jpeg
>> file contents in the first 1M against the locations found to determine
>> where the chunks actually start and end on each device.  The final
>> offset will be a chunk multiple before these boundaries.  Or do md5 sums
>> of 4k blocks to reduce the amount to inspect.
> 
> How exactly can I do this? Should I create more Bytestrings and do more
> brep with them on my physical devices? I have already results from
> searching bytestrings with an offset of 64k (starting from 100k to 612k
> of my jpeg file, so 9 bytestrings at all). Should I provide a table of
> the results?

Sigh.  I couldn't help myself.  New utility attached.  Curse you Mathias
for an interesting problem! ;-)

Call it with your jpeg and the devices to search, like so:

findHash.py /path/to/picture.jpeg /dev/sd[bcde]

It'll make a map of hashes of each 4k block in the jpeg and then search
the listed devices for those hashes, building a map of the file
fragments.  This will clearly show chunk boundaries.

Please show the output.

Phil

[-- Attachment #2: findHash.py --]
[-- Type: text/x-python, Size: 2243 bytes --]

#! /usr/bin/python2
#
# Locate 4k fragments of a subject file in one or more other files or
# devices.  Only reports two or more consecutive matches.
#
# Usage:
#   findHash.py /path/to/subject/file /dev/sdx|/path/to/image/file [/dev/sdy ...]

import hashlib, sys, datetime

# Read the known file 4k at a time, building a dictionary of
# md5 hashes vs. offset.  Use a large buffer for speed.
# Drops any partial block at the end of the file.
d = {}
pos = long(0)
f = open(sys.argv[1], 'r', 1<<20)
b = f.read(4096)
while len(b)==4096:
	md5 = hashlib.md5()
	md5.update(b)
	h = md5.digest()
	hlist = d.get(h)
	if not hlist:
		hlist = []
		d[h] = hlist
#		print "New hash %s at %8.8x" % (h.encode('hex'), pos)
	hlist.append(pos)
	pos += 4096
	b = f.read(4096)
f.close()

print "%d Unique hashes in %s" % (len(d), sys.argv[1])

def checkAndPrint(match):
	if match[2]>4096:
		print "%20s @ %12.12x:%12.12x ~= %8.8x:%8.8x" % (fname, match[1], match[1]+match[2]-1, match[0], match[0]+match[2]-1)

# Read the candidate files/devices, looking for possible matches.  Match
# entries are vectors of known file offset, candidate file offset, and
# length.
for fname in sys.argv[2:]:
	print "\nSearching for pieces of %s in %s:..." % (sys.argv[1], fname)
	pos = long(0)
	f = open(fname, 'r', 1<<24)
	matches = []
	b = f.read(4096)
	lastts = None
	while len(b)==4096:
		if not (pos & 0x7ffffff):
			ts = datetime.datetime.now()
			if lastts:
				print "@ %12.12x %.1fMB/s   \r" % (pos, 128.0/((ts-lastts).total_seconds())),
			else:
				print "@ %12.12x...\r" % pos,
			sys.stdout.flush()
			lastts = ts
		md5 = hashlib.md5()
		md5.update(b)
		h = md5.digest()
		if h in d:
			i = 0
			while i<len(matches):
				match = matches[i]
				target = match[0]+match[2]
				continuations = [x for x in d[h] if x==target]
				if continuations:
					match[2] += 4096
					i += 1
				else:
					del matches[i]
					checkAndPrint(match)
			if not matches:
				matches = [[x, pos, 4096] for x in d[h]]
		else:
			for match in matches:
				checkAndPrint(match)
			matches = []
		pos += 4096
		b = f.read(4096)
	print "End of %s at %12.12x" % (fname, pos)
	# show matches that continue to the end of the candidate file/device.
	for match in matches:
		checkAndPrint(match)

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 01/19/2016 12:51 PM, Phil Turmel wrote:

> Sigh.  I couldn't help myself.  New utility attached.  Curse you Mathias
> for an interesting problem! ;-)

I decided that this might be useful to others, particularly if someone
wants to tweak it for more parallelization, so I prettied it up for
github with a readme and a proper copyright statement.

https://github.com/pturmel/findHash

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

thanks a lot for your patience and sorry for driving you crazy. I will 
post results as soon as your nice tool has finished :)

Mathias

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

it's done:

658 Unique hashes in picture.jpg

Searching for pieces of picture.jpg in /dev/sdc1:...
          /dev/sdc1 @ 005400d00000:005400d7ffff ~= 00156000:001d5fff
          /dev/sdc1 @ 0087b4f29000:0087b4f7ffff ~= 00000000:00056fff
          /dev/sdc1 @ 0087b4f80000:0087b4ffffff ~= 001d7000:00256fff
          /dev/sdc1 @ 00cb2d980000:00cb2d9fffff ~= 0012a000:001a9fff
End of /dev/sdc1 at 01d1c1016000

Searching for pieces of picture.jpg in /dev/sdd1:...
          /dev/sdd1 @ 005400c80000:005400cfffff ~= 000d6000:00155fff
          /dev/sdd1 @ 0087b4f80000:0087b4ffffff ~= 00157000:001d6fff
          /dev/sdd1 @ 00cb2d980000:00cb2d9fffff ~= 000aa000:00129fff
End of /dev/sdd1 at 01d1c1016000

Searching for pieces of picture.jpg in /dev/sde1:...
           /dev/sde1 @ 005400b90000:005400c0ffff ~= 00056000:000d5fff
          /dev/sde1 @ 005400c10000:005400c4bfff ~= 00256000:00291fff
          /dev/sde1 @ 0087b4e10000:0087b4e8ffff ~= 000d7000:00156fff
          /dev/sde1 @ 00cb2d810000:00cb2d88ffff ~= 0002a000:000a9fff
          /dev/sde1 @ 00cb2d910000:00cb2d977fff ~= 0022a000:00291fff
End of /dev/sde1 at 01d1c0e90000

Searching for pieces of picture.jpg in /dev/sdf1:...
          /dev/sdf1 @ 005400caa000:005400cfffff ~= 00000000:00055fff
          /dev/sdf1 @ 005400d00000:005400d7ffff ~= 001d6000:00255fff
          /dev/sdf1 @ 0087b4f00000:0087b4f7ffff ~= 00057000:000d6fff
          /dev/sdf1 @ 0087b4f80000:0087b4fbafff ~= 00257000:00291fff
          /dev/sdf1 @ 00cb2d956000:00cb2d97ffff ~= 00000000:00029fff
          /dev/sdf1 @ 00cb2d980000:00cb2d9fffff ~= 001aa000:00229fff
End of /dev/sdf1 at 01d1c110e000


Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: broken raid level 5 array caused by user error

[Phil Turmel]

Hi Mathias,

On 01/22/2016 04:30 AM, Mathias Mueller wrote:
> Hi Phil,
> 
> it's done:
> 
> 658 Unique hashes in picture.jpg

Looks good.  Rearranging a bit:

>           /dev/sdf1 @ 005400caa000:005400cfffff ~= 00000000:00055fff
>           /dev/sde1 @ 005400b90000:005400c0ffff ~= 00056000:000d5fff
>           /dev/sdd1 @ 005400c80000:005400cfffff ~= 000d6000:00155fff
>           /dev/sdc1 @ 005400d00000:005400d7ffff ~= 00156000:001d5fff
>           /dev/sdf1 @ 005400d00000:005400d7ffff ~= 001d6000:00255fff
>           /dev/sde1 @ 005400c10000:005400c4bfff ~= 00256000:00291fff

>           /dev/sdc1 @ 0087b4f29000:0087b4f7ffff ~= 00000000:00056fff
>           /dev/sdf1 @ 0087b4f00000:0087b4f7ffff ~= 00057000:000d6fff
>           /dev/sde1 @ 0087b4e10000:0087b4e8ffff ~= 000d7000:00156fff
>           /dev/sdd1 @ 0087b4f80000:0087b4ffffff ~= 00157000:001d6fff
>           /dev/sdc1 @ 0087b4f80000:0087b4ffffff ~= 001d7000:00256fff
>           /dev/sdf1 @ 0087b4f80000:0087b4fbafff ~= 00257000:00291fff

>           /dev/sdf1 @ 00cb2d956000:00cb2d97ffff ~= 00000000:00029fff
>           /dev/sde1 @ 00cb2d810000:00cb2d88ffff ~= 0002a000:000a9fff
>           /dev/sdd1 @ 00cb2d980000:00cb2d9fffff ~= 000aa000:00129fff
>           /dev/sdc1 @ 00cb2d980000:00cb2d9fffff ~= 0012a000:001a9fff
>           /dev/sdf1 @ 00cb2d980000:00cb2d9fffff ~= 001aa000:00229fff
>           /dev/sde1 @ 00cb2d910000:00cb2d977fff ~= 0022a000:00291fff

Device order is definitely sdf1, sde1, sdd1, sdc1, sdf1 ....

> End of /dev/sdc1 at 01d1c1016000
> End of /dev/sdd1 at 01d1c1016000
> End of /dev/sde1 at 01d1c0e90000
> End of /dev/sdf1 at 01d1c110e000

sdc1 and sdd1 are the same size and have the same chunk alignment
offsets.  sdf1 is bigger but has the same chunk offsets as sdc1 and
sdd1.  sde1 is smaller than sd[cd]1 by 0x186000 and has chunk offsets
0xf0000 earlier than the others for a given stripe.

sde1 has a data offset that ends in 0x10000 or 0x90000.  0x10000 is 64k,
an offset used by default early in mdadm's history.

The others have data offsets that end in 0x00000 or 0x80000.  0x80000 is
512k.  0x100000 is 1M, another default offset in mdadm history, and
happily the correct increment (0xf0000) bigger than the 64k suspicion I
have for sde1.

Here are the four --create operations I would try:

(This first one is most likely it based on the signature analysis in
November.)

mdadm --create --assume-clean --data-offset=variable \
  --raid-devices=4 --chunk=512 --level=5 /dev/mdX \
  /dev/sdc1:1024 /dev/sdf1:1024 /dev/sde1:64 /dev/sdd1:1024

mdadm --create --assume-clean --data-offset=variable \
  --raid-devices=4 --chunk=512 --level=5 /dev/mdX \
  /dev/sdf1:1024 /dev/sde1:64 /dev/sdd1:1024 /dev/sdc1:1024

mdadm --create --assume-clean --data-offset=variable \
  --raid-devices=4 --chunk=512 --level=5 /dev/mdX \
  /dev/sde1:64 /dev/sdd1:1024 /dev/sdc1:1024 /dev/sdf1:1024

mdadm --create --assume-clean --data-offset=variable \
  --raid-devices=4 --chunk=512 --level=5 /dev/mdX \
  /dev/sdd1:1024 /dev/sdc1:1024 /dev/sdf1:1024 /dev/sde1:64

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

you are the man! The first one is working!

So please tell me, where does the beer delivery goes to? :)

Thank you so much, especially for so much patience with the dumbest user 
on earth.

Mathias

PS: I'm serious with the beer or whatever you prefer

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 01/22/2016 12:39 PM, Mathias Mueller wrote:
> Hi Phil,
> 
> you are the man! The first one is working!

Ah, that's good.

> So please tell me, where does the beer delivery goes to? :)

Sadly, it's hard to ship beer here in the US without government involvement.

> Thank you so much, especially for so much patience with the dumbest user
> on earth.

You're welcome.  But don't put yourself down -- it was your idea to
search for parts of a known file, after all.

> Mathias
> 
> PS: I'm serious with the beer or whatever you prefer

A modest tip via paypal would be fine, if you insist.  Or something from
my public wishlist:

http://smile.amazon.com/registry/wishlist/2EE91JWNIYSC3

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

thanks a lot once more, I would never managed it without your help. It's 
very nice, that you spend so much time in supporting everyone and 
writing very helpful tools. I learned a lot, technically and also to 
have more patience and thinking twice _before_ doing things :)

> Sadly, it's hard to ship beer here in the US without government 
> involvement.

That's a very sad fact! I send you something via paypal, so you might 
get some german beer from some US store, if you like. Tell me, if you 
need consulting ;)

Keep on going and best regards

Mathias

Re: broken raid level 5 array caused by user error

[Phil Turmel]

On 11/10/2015 06:59 PM, Phil Turmel wrote:
>>> For now, use data offset 2048 for all of them.
>>
>> output of all eight combinations:
> 
> Hmmm.  Very strange.  Let's see the dump of the start of the data area of all four partitions:
> 
> for x in /dev/sd[efgh]1 ; do echo -e "\n$x\n" ; dd if=$x skip=2048 count=16 2>/dev/null |hexdump -C ; done

I just noticed your response to this didn't make it to the list.  It was
108k.  If you do it again with count=8 -- that'll be enough for everyone
to understand my other reply.

Phil

Re: broken raid level 5 array caused by user error

[Mathias Mueller]

Hi Phil,

> I just noticed your response to this didn't make it to the list.  It 
> was
> 108k.  If you do it again with count=8 -- that'll be enough for 
> everyone
> to understand my other reply.

no problem, here you go:


/dev/sde1

00000000  00 00 c0 1f 10 00 c0 1f  20 00 c0 1f 00 00 00 20  |........ 
...... |
00000010  00 00 05 00 00 00 00 00  00 00 00 00 00 20 df 2d  
|............. .-|
00000020  01 00 c0 1f 11 00 c0 1f  20 02 c0 1f 00 00 00 20  |........ 
...... |
00000030  00 00 05 00 00 00 00 00  00 00 00 00 00 20 04 8d  
|............. ..|
00000040  02 00 c0 1f 12 00 c0 1f  20 04 c0 1f 00 00 00 20  |........ 
...... |
00000050  00 00 05 00 00 00 00 00  00 00 00 00 00 20 6a 2c  
|............. j,|
00000060  03 00 c0 1f 13 00 c0 1f  20 06 c0 1f 00 00 00 20  |........ 
...... |
00000070  00 00 05 00 00 00 00 00  00 00 00 00 00 20 b1 8c  
|............. ..|
00000080  04 00 c0 1f 14 00 c0 1f  20 08 c0 1f 00 00 00 20  |........ 
...... |
00000090  00 00 05 00 00 00 00 00  00 00 00 00 00 20 b5 2e  
|............. ..|
000000a0  05 00 c0 1f 15 00 c0 1f  20 0a c0 1f 00 00 00 20  |........ 
...... |
000000b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 6e 8e  
|............. n.|
000000c0  06 00 c0 1f 16 00 c0 1f  20 0c c0 1f 00 00 00 20  |........ 
...... |
000000d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 00 2f  
|............. ./|
000000e0  07 00 c0 1f 17 00 c0 1f  20 0e c0 1f 00 00 00 20  |........ 
...... |
000000f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 db 8f  
|............. ..|
00000100  08 00 c0 1f 18 00 c0 1f  20 10 c0 1f 00 00 00 20  |........ 
...... |
00000110  00 00 05 00 00 00 00 00  00 00 00 00 00 20 0b 2b  
|............. .+|
00000120  09 00 c0 1f 19 00 c0 1f  20 12 c0 1f 00 00 00 20  |........ 
...... |
00000130  00 00 05 00 00 00 00 00  00 00 00 00 00 20 d0 8b  
|............. ..|
00000140  0a 00 c0 1f 1a 00 c0 1f  20 14 c0 1f 00 00 00 20  |........ 
...... |
00000150  00 00 05 00 00 00 00 00  00 00 00 00 00 20 be 2a  
|............. .*|
00000160  0b 00 c0 1f 1b 00 c0 1f  20 16 c0 1f 00 00 00 20  |........ 
...... |
00000170  00 00 05 00 00 00 00 00  00 00 00 00 00 20 65 8a  
|............. e.|
00000180  0c 00 c0 1f 1c 00 c0 1f  20 18 c0 1f 00 00 00 20  |........ 
...... |
00000190  00 00 05 00 00 00 00 00  00 00 00 00 00 20 61 28  
|............. a(|
000001a0  0d 00 c0 1f 1d 00 c0 1f  20 1a c0 1f 00 00 00 20  |........ 
...... |
000001b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ba 88  
|............. ..|
000001c0  0e 00 c0 1f 1e 00 c0 1f  20 1c c0 1f 00 00 00 20  |........ 
...... |
000001d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 d4 29  
|............. .)|
000001e0  0f 00 c0 1f 1f 00 c0 1f  20 1e c0 1f 00 00 00 20  |........ 
...... |
000001f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 0f 89  
|............. ..|
00000200  00 00 c8 1f 10 00 c8 1f  20 00 c8 1f 00 00 00 20  |........ 
...... |
00000210  00 00 05 00 00 00 00 00  00 00 00 00 00 20 c9 5e  
|............. .^|
00000220  01 00 c8 1f 11 00 c8 1f  20 02 c8 1f 00 00 00 20  |........ 
...... |
00000230  00 00 05 00 00 00 00 00  00 00 00 00 00 20 12 fe  
|............. ..|
00000240  02 00 c8 1f 12 00 c8 1f  20 04 c8 1f 00 00 00 20  |........ 
...... |
00000250  00 00 05 00 00 00 00 00  00 00 00 00 00 20 7c 5f  
|............. |_|
00000260  03 00 c8 1f 13 00 c8 1f  20 06 c8 1f 00 00 00 20  |........ 
...... |
00000270  00 00 05 00 00 00 00 00  00 00 00 00 00 20 a7 ff  
|............. ..|
00000280  04 00 c8 1f 14 00 c8 1f  20 08 c8 1f 00 00 00 20  |........ 
...... |
00000290  00 00 05 00 00 00 00 00  00 00 00 00 00 20 a3 5d  
|............. .]|
000002a0  05 00 c8 1f 15 00 c8 1f  20 0a c8 1f 00 00 00 20  |........ 
...... |
000002b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 78 fd  
|............. x.|
000002c0  06 00 c8 1f 16 00 c8 1f  20 0c c8 1f 00 00 00 20  |........ 
...... |
000002d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 16 5c  
|............. .\|
000002e0  07 00 c8 1f 17 00 c8 1f  20 0e c8 1f 00 00 00 20  |........ 
...... |
000002f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 cd fc  
|............. ..|
00000300  08 00 c8 1f 18 00 c8 1f  20 10 c8 1f 00 00 00 20  |........ 
...... |
00000310  00 00 05 00 00 00 00 00  00 00 00 00 00 20 1d 58  
|............. .X|
00000320  09 00 c8 1f 19 00 c8 1f  20 12 c8 1f 00 00 00 20  |........ 
...... |
00000330  00 00 05 00 00 00 00 00  00 00 00 00 00 20 c6 f8  
|............. ..|
00000340  0a 00 c8 1f 1a 00 c8 1f  20 14 c8 1f 00 00 00 20  |........ 
...... |
00000350  00 00 05 00 00 00 00 00  00 00 00 00 00 20 a8 59  
|............. .Y|
00000360  0b 00 c8 1f 1b 00 c8 1f  20 16 c8 1f 00 00 00 20  |........ 
...... |
00000370  00 00 05 00 00 00 00 00  00 00 00 00 00 20 73 f9  
|............. s.|
00000380  0c 00 c8 1f 1c 00 c8 1f  20 18 c8 1f 00 00 00 20  |........ 
...... |
00000390  00 00 05 00 00 00 00 00  00 00 00 00 00 20 77 5b  
|............. w[|
000003a0  0d 00 c8 1f 1d 00 c8 1f  20 1a c8 1f 00 00 00 20  |........ 
...... |
000003b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ac fb  
|............. ..|
000003c0  0e 00 c8 1f 1e 00 c8 1f  20 1c c8 1f 00 00 00 20  |........ 
...... |
000003d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 c2 5a  
|............. .Z|
000003e0  0f 00 c8 1f 1f 00 c8 1f  20 1e c8 1f 00 00 00 20  |........ 
...... |
000003f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 19 fa  
|............. ..|
00000400  00 00 d0 1f 10 00 d0 1f  20 00 d0 1f 00 00 00 20  |........ 
...... |
00000410  00 00 05 00 00 00 00 00  00 00 00 00 00 20 f3 cb  
|............. ..|
00000420  01 00 d0 1f 11 00 d0 1f  20 02 d0 1f 00 00 00 20  |........ 
...... |
00000430  00 00 05 00 00 00 00 00  00 00 00 00 00 20 28 6b  
|............. (k|
00000440  02 00 d0 1f 12 00 d0 1f  20 04 d0 1f 00 00 00 20  |........ 
...... |
00000450  00 00 05 00 00 00 00 00  00 00 00 00 00 20 46 ca  
|............. F.|
00000460  03 00 d0 1f 13 00 d0 1f  20 06 d0 1f 00 00 00 20  |........ 
...... |
00000470  00 00 05 00 00 00 00 00  00 00 00 00 00 20 9d 6a  
|............. .j|
00000480  04 00 d0 1f 14 00 d0 1f  20 08 d0 1f 00 00 00 20  |........ 
...... |
00000490  00 00 05 00 00 00 00 00  00 00 00 00 00 20 99 c8  
|............. ..|
000004a0  05 00 d0 1f 15 00 d0 1f  20 0a d0 1f 00 00 00 20  |........ 
...... |
000004b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 42 68  
|............. Bh|
000004c0  06 00 d0 1f 16 00 d0 1f  20 0c d0 1f 00 00 00 20  |........ 
...... |
000004d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 2c c9  
|............. ,.|
000004e0  07 00 d0 1f 17 00 d0 1f  20 0e d0 1f 00 00 00 20  |........ 
...... |
000004f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 f7 69  
|............. .i|
00000500  08 00 d0 1f 18 00 d0 1f  20 10 d0 1f 00 00 00 20  |........ 
...... |
00000510  00 00 05 00 00 00 00 00  00 00 00 00 00 20 27 cd  
|............. '.|
00000520  09 00 d0 1f 19 00 d0 1f  20 12 d0 1f 00 00 00 20  |........ 
...... |
00000530  00 00 05 00 00 00 00 00  00 00 00 00 00 20 fc 6d  
|............. .m|
00000540  0a 00 d0 1f 1a 00 d0 1f  20 14 d0 1f 00 00 00 20  |........ 
...... |
00000550  00 00 05 00 00 00 00 00  00 00 00 00 00 20 92 cc  
|............. ..|
00000560  0b 00 d0 1f 1b 00 d0 1f  20 16 d0 1f 00 00 00 20  |........ 
...... |
00000570  00 00 05 00 00 00 00 00  00 00 00 00 00 20 49 6c  
|............. Il|
00000580  0c 00 d0 1f 1c 00 d0 1f  20 18 d0 1f 00 00 00 20  |........ 
...... |
00000590  00 00 05 00 00 00 00 00  00 00 00 00 00 20 4d ce  
|............. M.|
000005a0  0d 00 d0 1f 1d 00 d0 1f  20 1a d0 1f 00 00 00 20  |........ 
...... |
000005b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 96 6e  
|............. .n|
000005c0  0e 00 d0 1f 1e 00 d0 1f  20 1c d0 1f 00 00 00 20  |........ 
...... |
000005d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 f8 cf  
|............. ..|
000005e0  0f 00 d0 1f 1f 00 d0 1f  20 1e d0 1f 00 00 00 20  |........ 
...... |
000005f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 23 6f  
|............. #o|
00000600  00 00 d8 1f 10 00 d8 1f  20 00 d8 1f 00 00 00 20  |........ 
...... |
00000610  00 00 05 00 00 00 00 00  00 00 00 00 00 20 e5 b8  
|............. ..|
00000620  01 00 d8 1f 11 00 d8 1f  20 02 d8 1f 00 00 00 20  |........ 
...... |
00000630  00 00 05 00 00 00 00 00  00 00 00 00 00 20 3e 18  
|............. >.|
00000640  02 00 d8 1f 12 00 d8 1f  20 04 d8 1f 00 00 00 20  |........ 
...... |
00000650  00 00 05 00 00 00 00 00  00 00 00 00 00 20 50 b9  
|............. P.|
00000660  03 00 d8 1f 13 00 d8 1f  20 06 d8 1f 00 00 00 20  |........ 
...... |
00000670  00 00 05 00 00 00 00 00  00 00 00 00 00 20 8b 19  
|............. ..|
00000680  04 00 d8 1f 14 00 d8 1f  20 08 d8 1f 00 00 00 20  |........ 
...... |
00000690  00 00 05 00 00 00 00 00  00 00 00 00 00 20 8f bb  
|............. ..|
000006a0  05 00 d8 1f 15 00 d8 1f  20 0a d8 1f 00 00 00 20  |........ 
...... |
000006b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 54 1b  
|............. T.|
000006c0  06 00 d8 1f 16 00 d8 1f  20 0c d8 1f 00 00 00 20  |........ 
...... |
000006d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 3a ba  
|............. :.|
000006e0  07 00 d8 1f 17 00 d8 1f  20 0e d8 1f 00 00 00 20  |........ 
...... |
000006f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 e1 1a  
|............. ..|
00000700  08 00 d8 1f 18 00 d8 1f  20 10 d8 1f 00 00 00 20  |........ 
...... |
00000710  00 00 05 00 00 00 00 00  00 00 00 00 00 20 31 be  
|............. 1.|
00000720  09 00 d8 1f 19 00 d8 1f  20 12 d8 1f 00 00 00 20  |........ 
...... |
00000730  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ea 1e  
|............. ..|
00000740  0a 00 d8 1f 1a 00 d8 1f  20 14 d8 1f 00 00 00 20  |........ 
...... |
00000750  00 00 05 00 00 00 00 00  00 00 00 00 00 20 84 bf  
|............. ..|
00000760  0b 00 d8 1f 1b 00 d8 1f  20 16 d8 1f 00 00 00 20  |........ 
...... |
00000770  00 00 05 00 00 00 00 00  00 00 00 00 00 20 5f 1f  
|............. _.|
00000780  0c 00 d8 1f 1c 00 d8 1f  20 18 d8 1f 00 00 00 20  |........ 
...... |
00000790  00 00 05 00 00 00 00 00  00 00 00 00 00 20 5b bd  
|............. [.|
000007a0  0d 00 d8 1f 1d 00 d8 1f  20 1a d8 1f 00 00 00 20  |........ 
...... |
000007b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 80 1d  
|............. ..|
000007c0  0e 00 d8 1f 1e 00 d8 1f  20 1c d8 1f 00 00 00 20  |........ 
...... |
000007d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ee bc  
|............. ..|
000007e0  0f 00 d8 1f 1f 00 d8 1f  20 1e d8 1f 00 00 00 20  |........ 
...... |
000007f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 35 1c  
|............. 5.|
00000800  00 00 e0 1f 10 00 e0 1f  20 00 e0 1f 00 00 00 20  |........ 
...... |
00000810  00 00 05 00 00 00 00 00  00 00 00 00 00 20 84 a1  
|............. ..|
00000820  01 00 e0 1f 11 00 e0 1f  20 02 e0 1f 00 00 00 20  |........ 
...... |
00000830  00 00 05 00 00 00 00 00  00 00 00 00 00 20 5f 01  
|............. _.|
00000840  02 00 e0 1f 12 00 e0 1f  20 04 e0 1f 00 00 00 20  |........ 
...... |
00000850  00 00 05 00 00 00 00 00  00 00 00 00 00 20 31 a0  
|............. 1.|
00000860  03 00 e0 1f 13 00 e0 1f  20 06 e0 1f 00 00 00 20  |........ 
...... |
00000870  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ea 00  
|............. ..|
00000880  04 00 e0 1f 14 00 e0 1f  20 08 e0 1f 00 00 00 20  |........ 
...... |
00000890  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ee a2  
|............. ..|
000008a0  05 00 e0 1f 15 00 e0 1f  20 0a e0 1f 00 00 00 20  |........ 
...... |
000008b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 35 02  
|............. 5.|
000008c0  06 00 e0 1f 16 00 e0 1f  20 0c e0 1f 00 00 00 20  |........ 
...... |
000008d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 5b a3  
|............. [.|
000008e0  07 00 e0 1f 17 00 e0 1f  20 0e e0 1f 00 00 00 20  |........ 
...... |
000008f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 80 03  
|............. ..|
00000900  08 00 e0 1f 18 00 e0 1f  20 10 e0 1f 00 00 00 20  |........ 
...... |
00000910  00 00 05 00 00 00 00 00  00 00 00 00 00 20 50 a7  
|............. P.|
00000920  09 00 e0 1f 19 00 e0 1f  20 12 e0 1f 00 00 00 20  |........ 
...... |
00000930  00 00 05 00 00 00 00 00  00 00 00 00 00 20 8b 07  
|............. ..|
00000940  0a 00 e0 1f 1a 00 e0 1f  20 14 e0 1f 00 00 00 20  |........ 
...... |
00000950  00 00 05 00 00 00 00 00  00 00 00 00 00 20 e5 a6  
|............. ..|
00000960  0b 00 e0 1f 1b 00 e0 1f  20 16 e0 1f 00 00 00 20  |........ 
...... |
00000970  00 00 05 00 00 00 00 00  00 00 00 00 00 20 3e 06  
|............. >.|
00000980  0c 00 e0 1f 1c 00 e0 1f  20 18 e0 1f 00 00 00 20  |........ 
...... |
00000990  00 00 05 00 00 00 00 00  00 00 00 00 00 20 3a a4  
|............. :.|
000009a0  0d 00 e0 1f 1d 00 e0 1f  20 1a e0 1f 00 00 00 20  |........ 
...... |
000009b0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 e1 04  
|............. ..|
000009c0  0e 00 e0 1f 1e 00 e0 1f  20 1c e0 1f 00 00 00 20  |........ 
...... |
000009d0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 8f a5  
|............. ..|
000009e0  0f 00 e0 1f 1f 00 e0 1f  20 1e e0 1f 00 00 00 20  |........ 
...... |
000009f0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 54 05  
|............. T.|
00000a00  00 00 e8 1f 10 00 e8 1f  20 00 e8 1f 00 00 00 20  |........ 
...... |
00000a10  00 00 05 00 00 00 00 00  00 00 00 00 00 20 92 d2  
|............. ..|
00000a20  01 00 e8 1f 11 00 e8 1f  20 02 e8 1f 00 00 00 20  |........ 
...... |
00000a30  00 00 05 00 00 00 00 00  00 00 00 00 00 20 49 72  
|............. Ir|
00000a40  02 00 e8 1f 12 00 e8 1f  20 04 e8 1f 00 00 00 20  |........ 
...... |
00000a50  00 00 05 00 00 00 00 00  00 00 00 00 00 20 27 d3  
|............. '.|
00000a60  03 00 e8 1f 13 00 e8 1f  20 06 e8 1f 00 00 00 20  |........ 
...... |
00000a70  00 00 05 00 00 00 00 00  00 00 00 00 00 20 fc 73  
|............. .s|
00000a80  04 00 e8 1f 14 00 e8 1f  20 08 e8 1f 00 00 00 20  |........ 
...... |
00000a90  00 00 05 00 00 00 00 00  00 00 00 00 00 20 f8 d1  
|............. ..|
00000aa0  05 00 e8 1f 15 00 e8 1f  20 0a e8 1f 00 00 00 20  |........ 
...... |
00000ab0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 23 71  
|............. #q|
00000ac0  06 00 e8 1f 16 00 e8 1f  20 0c e8 1f 00 00 00 20  |........ 
...... |
00000ad0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 4d d0  
|............. M.|
00000ae0  07 00 e8 1f 17 00 e8 1f  20 0e e8 1f 00 00 00 20  |........ 
...... |
00000af0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 96 70  
|............. .p|
00000b00  08 00 e8 1f 18 00 e8 1f  20 10 e8 1f 00 00 00 20  |........ 
...... |
00000b10  00 00 05 00 00 00 00 00  00 00 00 00 00 20 46 d4  
|............. F.|
00000b20  09 00 e8 1f 19 00 e8 1f  20 12 e8 1f 00 00 00 20  |........ 
...... |
00000b30  00 00 05 00 00 00 00 00  00 00 00 00 00 20 9d 74  
|............. .t|
00000b40  0a 00 e8 1f 1a 00 e8 1f  20 14 e8 1f 00 00 00 20  |........ 
...... |
00000b50  00 00 05 00 00 00 00 00  00 00 00 00 00 20 f3 d5  
|............. ..|
00000b60  0b 00 e8 1f 1b 00 e8 1f  20 16 e8 1f 00 00 00 20  |........ 
...... |
00000b70  00 00 05 00 00 00 00 00  00 00 00 00 00 20 28 75  
|............. (u|
00000b80  0c 00 e8 1f 1c 00 e8 1f  20 18 e8 1f 00 00 00 20  |........ 
...... |
00000b90  00 00 05 00 00 00 00 00  00 00 00 00 00 20 2c d7  
|............. ,.|
00000ba0  0d 00 e8 1f 1d 00 e8 1f  20 1a e8 1f 00 00 00 20  |........ 
...... |
00000bb0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 f7 77  
|............. .w|
00000bc0  0e 00 e8 1f 1e 00 e8 1f  20 1c e8 1f 00 00 00 20  |........ 
...... |
00000bd0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 99 d6  
|............. ..|
00000be0  0f 00 e8 1f 1f 00 e8 1f  20 1e e8 1f 00 00 00 20  |........ 
...... |
00000bf0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 42 76  
|............. Bv|
00000c00  00 00 f0 1f 10 00 f0 1f  20 00 f0 1f 00 00 00 20  |........ 
...... |
00000c10  00 00 05 00 00 00 00 00  00 00 00 00 00 20 a8 47  
|............. .G|
00000c20  01 00 f0 1f 11 00 f0 1f  20 02 f0 1f 00 00 00 20  |........ 
...... |
00000c30  00 00 05 00 00 00 00 00  00 00 00 00 00 20 73 e7  
|............. s.|
00000c40  02 00 f0 1f 12 00 f0 1f  20 04 f0 1f 00 00 00 20  |........ 
...... |
00000c50  00 00 05 00 00 00 00 00  00 00 00 00 00 20 1d 46  
|............. .F|
00000c60  03 00 f0 1f 13 00 f0 1f  20 06 f0 1f 00 00 00 20  |........ 
...... |
00000c70  00 00 05 00 00 00 00 00  00 00 00 00 00 20 c6 e6  
|............. ..|
00000c80  04 00 f0 1f 14 00 f0 1f  20 08 f0 1f 00 00 00 20  |........ 
...... |
00000c90  00 00 05 00 00 00 00 00  00 00 00 00 00 20 c2 44  
|............. .D|
00000ca0  05 00 f0 1f 15 00 f0 1f  20 0a f0 1f 00 00 00 20  |........ 
...... |
00000cb0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 19 e4  
|............. ..|
00000cc0  06 00 f0 1f 16 00 f0 1f  20 0c f0 1f 00 00 00 20  |........ 
...... |
00000cd0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 77 45  
|............. wE|
00000ce0  07 00 f0 1f 17 00 f0 1f  20 0e f0 1f 00 00 00 20  |........ 
...... |
00000cf0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ac e5  
|............. ..|
00000d00  08 00 f0 1f 18 00 f0 1f  20 10 f0 1f 00 00 00 20  |........ 
...... |
00000d10  00 00 05 00 00 00 00 00  00 00 00 00 00 20 7c 41  
|............. |A|
00000d20  09 00 f0 1f 19 00 f0 1f  20 12 f0 1f 00 00 00 20  |........ 
...... |
00000d30  00 00 05 00 00 00 00 00  00 00 00 00 00 20 a7 e1  
|............. ..|
00000d40  0a 00 f0 1f 1a 00 f0 1f  20 14 f0 1f 00 00 00 20  |........ 
...... |
00000d50  00 00 05 00 00 00 00 00  00 00 00 00 00 20 c9 40  
|............. .@|
00000d60  0b 00 f0 1f 1b 00 f0 1f  20 16 f0 1f 00 00 00 20  |........ 
...... |
00000d70  00 00 05 00 00 00 00 00  00 00 00 00 00 20 12 e0  
|............. ..|
00000d80  0c 00 f0 1f 1c 00 f0 1f  20 18 f0 1f 00 00 00 20  |........ 
...... |
00000d90  00 00 05 00 00 00 00 00  00 00 00 00 00 20 16 42  
|............. .B|
00000da0  0d 00 f0 1f 1d 00 f0 1f  20 1a f0 1f 00 00 00 20  |........ 
...... |
00000db0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 cd e2  
|............. ..|
00000dc0  0e 00 f0 1f 1e 00 f0 1f  20 1c f0 1f 00 00 00 20  |........ 
...... |
00000dd0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 a3 43  
|............. .C|
00000de0  0f 00 f0 1f 1f 00 f0 1f  20 1e f0 1f 00 00 00 20  |........ 
...... |
00000df0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 78 e3  
|............. x.|
00000e00  00 00 f8 1f 10 00 f8 1f  20 00 f8 1f 00 00 00 20  |........ 
...... |
00000e10  00 00 05 00 00 00 00 00  00 00 00 00 00 20 be 34  
|............. .4|
00000e20  01 00 f8 1f 11 00 f8 1f  20 02 f8 1f 00 00 00 20  |........ 
...... |
00000e30  00 00 05 00 00 00 00 00  00 00 00 00 00 20 65 94  
|............. e.|
00000e40  02 00 f8 1f 12 00 f8 1f  20 04 f8 1f 00 00 00 20  |........ 
...... |
00000e50  00 00 05 00 00 00 00 00  00 00 00 00 00 20 0b 35  
|............. .5|
00000e60  03 00 f8 1f 13 00 f8 1f  20 06 f8 1f 00 00 00 20  |........ 
...... |
00000e70  00 00 05 00 00 00 00 00  00 00 00 00 00 20 d0 95  
|............. ..|
00000e80  04 00 f8 1f 14 00 f8 1f  20 08 f8 1f 00 00 00 20  |........ 
...... |
00000e90  00 00 05 00 00 00 00 00  00 00 00 00 00 20 d4 37  
|............. .7|
00000ea0  05 00 f8 1f 15 00 f8 1f  20 0a f8 1f 00 00 00 20  |........ 
...... |
00000eb0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 0f 97  
|............. ..|
00000ec0  06 00 f8 1f 16 00 f8 1f  20 0c f8 1f 00 00 00 20  |........ 
...... |
00000ed0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 61 36  
|............. a6|
00000ee0  07 00 f8 1f 17 00 f8 1f  20 0e f8 1f 00 00 00 20  |........ 
...... |
00000ef0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 ba 96  
|............. ..|
00000f00  08 00 f8 1f 18 00 f8 1f  20 10 f8 1f 00 00 00 20  |........ 
...... |
00000f10  00 00 05 00 00 00 00 00  00 00 00 00 00 20 6a 32  
|............. j2|
00000f20  09 00 f8 1f 19 00 f8 1f  20 12 f8 1f 00 00 00 20  |........ 
...... |
00000f30  00 00 05 00 00 00 00 00  00 00 00 00 00 20 b1 92  
|............. ..|
00000f40  0a 00 f8 1f 1a 00 f8 1f  20 14 f8 1f 00 00 00 20  |........ 
...... |
00000f50  00 00 05 00 00 00 00 00  00 00 00 00 00 20 df 33  
|............. .3|
00000f60  0b 00 f8 1f 1b 00 f8 1f  20 16 f8 1f 00 00 00 20  |........ 
...... |
00000f70  00 00 05 00 00 00 00 00  00 00 00 00 00 20 04 93  
|............. ..|
00000f80  0c 00 f8 1f 1c 00 f8 1f  20 18 f8 1f 00 00 00 20  |........ 
...... |
00000f90  00 00 05 00 00 00 00 00  00 00 00 00 00 20 00 31  
|............. .1|
00000fa0  0d 00 f8 1f 1d 00 f8 1f  20 1a f8 1f 00 00 00 20  |........ 
...... |
00000fb0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 db 91  
|............. ..|
00000fc0  0e 00 f8 1f 1e 00 f8 1f  20 1c f8 1f 00 00 00 20  |........ 
...... |
00000fd0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 b5 30  
|............. .0|
00000fe0  0f 00 f8 1f 1f 00 f8 1f  20 1e f8 1f 00 00 00 20  |........ 
...... |
00000ff0  00 00 05 00 00 00 00 00  00 00 00 00 00 20 6e 90  
|............. n.|
00001000

/dev/sdf1

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  
|................|
*
00000400  00 20 d5 15 80 2b 54 57  00 00 00 00 c8 99 09 00  |. 
...+TW........|
00000410  5f 91 bc 15 00 00 00 00  02 00 00 00 02 00 00 00  
|_...............|
00000420  00 80 00 00 00 80 00 00  00 20 00 00 f8 aa 3f 56  |......... 
....?V|
00000430  28 ab 3f 56 06 00 19 00  53 ef 01 00 01 00 00 00  
|(.?V....S.......|
00000440  ac 31 3c 56 00 a7 76 00  00 00 00 00 01 00 00 00  
|.1<V..v.........|
00000450  00 00 00 00 0b 00 00 00  00 01 00 00 3c 00 00 00  
|............<...|
00000460  42 02 00 00 7b 00 00 00  73 6f 67 e3 a3 03 45 44  
|B...{...sog...ED|
00000470  9c 0a 6a 32 29 00 5e 39  00 00 00 00 00 00 00 00  
|..j2).^9........|
00000480  00 00 00 00 00 00 00 00  2f 73 72 76 00 18 fd e7  
|......../srv....|
00000490  1e 01 88 ff ff 5b 29 7f  71 00 00 00 00 00 f9 fe  
|.....[).q.......|
000004a0  3a 01 88 ff ff 00 f9 fe  3a 01 88 ff ff 80 65 4d  
|:.......:.....eM|
000004b0  3d 01 88 ff ff 00 75 40  19 01 88 ff ff 10 ab 0b  
|=.....u@........|
000004c0  a0 ff ff ff ff b8 44 5f  00 00 00 00 00 00 a2 02  
|......D_........|
000004d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  
|................|
000004e0  08 00 00 00 00 00 00 00  00 00 00 00 72 a4 a9 9f  
|............r...|
000004f0  f9 41 49 5d 88 a2 b1 6a  c5 02 4d 5b 01 01 00 00  
|.AI]...j..M[....|
00000500  00 00 00 00 00 00 00 00  d4 2c 8e 4c 0a f3 02 00  
|.........,.L....|
00000510  04 00 00 00 00 00 00 00  00 00 00 00 ff 7f 00 00  
|................|
00000520  00 80 88 0e ff 7f 00 00  01 00 00 00 ff ff 88 0e  
|................|
00000530  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  
|................|
00000540  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 08  
|................|
00000550  00 00 00 00 00 00 00 00  00 00 00 00 1c 00 1c 00  
|................|
00000560  02 00 00 00 ef 7f 00 00  00 00 00 00 00 00 00 00  
|................|
00000570  00 00 00 00 04 00 00 00  01 d3 b5 ba 01 00 00 00  
|................|
00000580  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  
|................|
*
00001000

/dev/sdg1

00000000  70 81 00 00 70 81 01 00  70 81 02 00 70 81 03 00  
|p...p...p...p...|
00000010  70 81 04 00 70 81 0c 00  70 81 0d 00 70 81 18 00  
|p...p...p...p...|
00000020  70 81 28 00 70 81 3e 00  70 81 79 00 70 81 ab 00  
|p.(.p.>.p.y.p...|
00000030  70 81 38 01 70 81 6c 01  70 81 45 04 70 81 b0 04  
|p.8.p.l.p.E.p...|
00000040  70 81 1a 06 70 81 d0 0c  70 81 84 1e 70 81 d3 20  
|p...p...p...p.. |
00000050  70 81 71 26 00 00 00 00  00 00 00 00 00 00 00 00  
|p.q&............|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  
|................|
*
00001000

/dev/sdh1

00000000  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000020  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000040  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000060  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000080  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000000a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000000c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000000e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000100  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000120  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000140  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000160  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000180  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000001a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000001c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000001e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000200  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000220  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000230  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000240  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000260  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000270  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000280  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000002a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000002c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000002d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000002e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000002f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000300  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000310  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000320  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000330  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000340  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000350  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000360  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000370  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000380  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000390  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000003a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000003b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000003c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000003d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000003e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000400  00 20 d5 35 80 2b 54 77  00 00 00 20 c8 99 09 00  |. .5.+Tw... 
....|
00000410  5f 91 bc 15 00 00 00 00  02 00 00 00 02 00 ef 1a  
|_...............|
00000420  00 80 00 20 00 80 00 20  00 20 00 20 f8 aa 3f 56  |... ... . . 
..?V|
00000430  28 ab 3f 56 06 00 19 00  53 ef 01 00 01 00 ef 1a  
|(.?V....S.......|
00000440  ac 31 3c 76 00 a7 76 20  00 00 00 20 01 00 00 00  |.1<v..v ... 
....|
00000450  00 00 00 00 0b 00 00 00  00 01 00 00 3c 00 ef 1a  
|............<...|
00000460  42 02 00 20 7b 00 00 20  73 6f 67 c3 a3 03 45 44  |B.. {.. 
sog...ED|
00000470  9c 0a 6a 32 29 00 5e 39  00 00 00 00 00 00 ef 1a  
|..j2).^9........|
00000480  00 00 00 20 00 00 00 20  2f 73 72 56 00 18 fd e7  |... ... 
/srV....|
00000490  1e 01 88 ff ff 5b 29 7f  71 00 00 00 00 00 16 e4  
|.....[).q.......|
000004a0  3a 01 88 df ff 00 f9 de  3a 01 88 df ff 80 65 4d  
|:.......:.....eM|
000004b0  3d 01 88 ff ff 00 75 40  19 01 88 ff ff 10 44 11  
|=.....u@......D.|
000004c0  a0 ff ff df ff b8 44 7f  00 00 00 20 00 00 a2 02  |......D.... 
....|
000004d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000004e0  08 00 00 20 00 00 00 20  00 00 00 20 72 a4 a9 9f  |... ... ... 
r...|
000004f0  f9 41 49 5d 88 a2 b1 6a  c5 02 4d 5b 01 01 ef 1a  
|.AI]...j..M[....|
00000500  00 00 00 20 00 00 00 20  d4 2c 8e 6c 0a f3 02 00  |... ... 
.,.l....|
00000510  04 00 00 00 00 00 00 00  00 00 00 00 ff 7f ef 1a  
|................|
00000520  00 80 88 2e ff 7f 00 20  01 00 00 20 ff ff 88 0e  |....... ... 
....|
00000530  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000540  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 08  |... ... ... 
....|
00000550  00 00 00 00 00 00 00 00  00 00 00 00 1c 00 f3 1a  
|................|
00000560  02 00 00 20 ef 7f 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000570  00 00 00 00 04 00 00 00  01 d3 b5 ba 01 00 ef 1a  
|................|
00000580  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000590  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000005a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000005b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000005c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000005d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000005e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000005f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000600  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000610  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000620  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000630  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000640  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000650  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000660  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000670  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000680  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000690  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000006a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000006b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000006c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000006d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000006e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000006f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000700  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000710  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000720  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000730  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000740  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000750  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000760  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000770  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000780  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000790  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000007a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000007b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000007c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000007d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000007e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000007f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000800  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000810  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000820  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000830  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000840  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000850  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000860  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000870  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000880  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000890  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000008a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000008b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000008c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000008d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000008e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000008f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000900  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000910  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000920  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000930  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000940  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000950  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000960  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000970  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000980  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000990  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000009a0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000009b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000009c0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000009d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
000009e0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
000009f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000a00  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000a10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000a20  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000a30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000a40  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000a50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000a60  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000a70  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000a80  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000a90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000aa0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000ab0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000ac0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000ad0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000ae0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000af0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000b00  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000b10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000b20  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000b30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000b40  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000b50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000b60  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000b70  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000b80  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000b90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000ba0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000bb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000bc0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000bd0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000be0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000bf0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000c00  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000c10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000c20  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000c30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000c40  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000c50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000c60  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000c70  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000c80  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000c90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000ca0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000cb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000cc0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000cd0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000ce0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000cf0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000d00  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000d10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000d20  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000d30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000d40  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000d50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000d60  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000d70  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000d80  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000d90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000da0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000db0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000dc0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000dd0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000de0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000df0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000e00  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000e10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000e20  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000e30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000e40  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000e50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000e60  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000e70  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000e80  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000e90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000ea0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000eb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000ec0  00 00 00 20 00 00 00 20  00 00 00 20 eb 07 00 00  |... ... ... 
....|
00000ed0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 18 fb  
|................|
00000ee0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000ef0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000f00  00 00 00 20 00 00 00 20  00 00 00 20 ab 04 00 00  |... ... ... 
....|
00000f10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 fd df  
|................|
00000f20  00 00 00 20 00 00 00 20  00 00 00 20 ff 07 00 00  |... ... ... 
....|
00000f30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 5d 0f  
|..............].|
00000f40  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000f50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000f60  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000f70  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000f80  00 00 00 20 00 00 00 20  00 00 00 20 f5 00 00 00  |... ... ... 
....|
00000f90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 cc e4  
|................|
00000fa0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000fb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000fc0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000fd0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00000fe0  00 00 00 20 00 00 00 20  00 00 00 20 00 00 00 00  |... ... ... 
....|
00000ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ef 1a  
|................|
00001000