All of lore.kernel.org
 help / color / mirror / Atom feed
* RAIS5 Rebuild Help - Possible Data Offset
@ 2014-01-30 14:20 Yorik
  2014-01-31 10:25 ` Robin Hill
  0 siblings, 1 reply; 9+ messages in thread
From: Yorik @ 2014-01-30 14:20 UTC (permalink / raw)
  To: linux-raid

Hello

Sorry if double post had to remove some HTML.

I've been using "mdadm" for about 8-9 years with a lot of success in
different Array configurations. I am a huge fan of Software RAID. I
recently suffered a failure that was very unfortunate and I now have
an issue with differing Data Offset and other issues preventing my
Array from rebuilding. A brief history follows:

Array originally created as RAID5, 3x 3TB Drives in May 2011 on a
(then) latest Fedora build.
Added more drives in 2012 and 2013 with upgraded Fedora builds.
Recently added another drive and changed to RAID6, 6x6TB Drives.
Suffered a failure shortly after 2 more drives that kicked the last 2
drives off and one of the 6 other that were previously running.
During the RAID rebuild, my computer locked up and then suffered a
Power Failure (having a bad day) that stopped my OS Disk (SSD) from
allowing me to reboot. Eventually rebuilt the OS with Fedora 20,
wiping the majority of the SSD and config files with it.

The RAID6 does not start, stating that it only has 5 of the 8 drives
required. Looking into this, the drives seem to be clean and check
okay, but there is even differences (expected) and Data Offset
difference between drives that I read (Neil Brown Blog date
-20120615073245) was not necessarily good for rebuilding and that I
should ask for a version of mdadm that can handle this as it's
possible that creating with an older version, then adding new drives
is causing an issue. Data Offsets of 2048 and 4096 exist.

Also of note is that I have some drives that are on the Array without
partitions as they would not add to the Array with partitions on them
due to size when they were added. I think that I had version issue
with fdisk, but it seemed to work okay until now.

When I run the following command:

mdadm --create --assume-clean --level=6 --raid-devices=8
--size=2930263552 /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd
/dev/sdi /dev/sde /dev/sdg /dev/sdf

I get:

mdadm: /dev/sda1 is smaller than given size. 2930134016K < 2930263552K
+ metadata
mdadm: /dev/sdb1 is smaller than given size. 2930134016K < 2930263552K
+ metadata
mdadm: /dev/sdc1 is smaller than given size. 2930134016K < 2930263552K
+ metadata
mdadm: /dev/sdd is smaller than given size. 2930135040K < 2930263552K + metadata
mdadm: /dev/sdi is smaller than given size. 2930135040K < 2930263552K + metadata
mdadm: /dev/sde is smaller than given size. 2930135040K < 2930263552K + metadata
mdadm: /dev/sdg is smaller than given size. 2930135040K < 2930263552K + metadata
mdadm: /dev/sdf is smaller than given size. 2930135040K < 2930263552K + metadata
mdadm: create aborted

Which makes me think that there are further issues. Should I be
looking for an older version of mdadm or a different version? Can
anyone offer some advice please or where to look next.

Regards
Derek

Information follows:

--Drive information--
mdadm --examine <Appropriate partitions and drives>
/dev/sda1:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 0 sectors
   Unused Space : before=1976 sectors, after=1024 sectors
          State : clean
    Device UUID : e27f0cdd:f9ade1a9:069a8d4d:a4e7d767

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:35:32 2014
       Checksum : 944933 - correct
         Events : 3118496

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 0 sectors
   Unused Space : before=1976 sectors, after=1024 sectors
          State : active
    Device UUID : ff0ddbdd:5d3123ae:3b813125:97a96017

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:35:32 2014
       Checksum : 8cd103dc - correct
         Events : 3118496

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 0 sectors
   Unused Space : before=1976 sectors, after=1024 sectors
          State : active
    Device UUID : f4106804:6552a7b0:2a960408:b7ca41ff

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:35:32 2014
       Checksum : 80965de9 - correct
         Events : 3118496

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860529072 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 4096 sectors
   Super Offset : 0 sectors
   Unused Space : before=4024 sectors, after=1968 sectors
          State : active
    Device UUID : d9d26a74:100488a7:8841e37a:f84e07b7

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:35:32 2014
       Checksum : 121e0ccb - correct
         Events : 3118496

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860529072 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 4096 sectors
   Super Offset : 0 sectors
   Unused Space : before=4024 sectors, after=1968 sectors
          State : active
    Device UUID : a73e6b7b:4791f3d6:b60e1025:0dda98c1

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:35:32 2014
       Checksum : fd485e13 - correct
         Events : 3118496

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860529072 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 4096 sectors
   Super Offset : 0 sectors
   Unused Space : before=4024 sectors, after=1968 sectors
          State : clean
    Device UUID : 25acca8d:45ae78df:77d834cb:b97cfadd

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:26:20 2014
       Checksum : daba52df - correct
         Events : 3118491

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 7
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860529072 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 4096 sectors
   Super Offset : 0 sectors
   Unused Space : before=4024 sectors, after=1968 sectors
          State : clean
    Device UUID : c4fca7c8:450ddd64:41aae795:334c9512

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:26:20 2014
       Checksum : 9a49a3c2 - correct
         Events : 3118491

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x5
     Array UUID : 80009fa3:a96743bd:f5dbfd87:18a1c52d
           Name : desktop:0
  Creation Time : Tue May 31 01:55:34 2011
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860529072 (2794.52 GiB 3000.59 GB)
     Array Size : 17581581312 (16767.10 GiB 18003.54 GB)
  Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
    Data Offset : 4096 sectors
   Super Offset : 0 sectors
   Unused Space : before=4024 sectors, after=1968 sectors
          State : clean
    Device UUID : 09f19ee9:414e5ea8:b755a246:a0a2c2f7

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 30978048 (29.54 GiB 31.72 GB)
  Delta Devices : 2 (6->8)

    Update Time : Tue Jan 21 23:26:20 2014
       Checksum : 94a9dae5 - correct
         Events : 3118491

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

--Device ID's vs Device Name--
lrwxrwxrwx 1 root root  9 Jan 27 23:03
ata-HL-DT-ST_DVDRAM_GH22NS70_K0028Q64636 -> ../../sr0
lrwxrwxrwx 1 root root  9 Jan 27 23:03
ata-OCZ-AGILITY3_OCZ-8D0T42404K8H12P6 -> ../../sdh
lrwxrwxrwx 1 root root 10 Jan 27 23:03
ata-OCZ-AGILITY3_OCZ-8D0T42404K8H12P6-part1 -> ../../sdh1
lrwxrwxrwx 1 root root 10 Jan 27 23:03
ata-OCZ-AGILITY3_OCZ-8D0T42404K8H12P6-part2 -> ../../sdh2
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST3000VX000-1CU166_Z1F47ATC
-> ../../sdg
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST3000VX000-1CU166_Z1F49MSQ
-> ../../sdf
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST3000VX000-1CU166_Z1F49MYJ
-> ../../sdi
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST3000VX000-9YW166_Z1F13TGY
-> ../../sde
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST33000651AS_9XK0B9RF -> ../../sdb
lrwxrwxrwx 1 root root 10 Jan 27 23:03 ata-ST33000651AS_9XK0B9RF-part1
-> ../../sdb1
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST33000651AS_9XK0BR4Q -> ../../sda
lrwxrwxrwx 1 root root 10 Jan 27 23:03 ata-ST33000651AS_9XK0BR4Q-part1
-> ../../sda1
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST33000651AS_9XK0F087 -> ../../sdc
lrwxrwxrwx 1 root root 10 Jan 27 23:03 ata-ST33000651AS_9XK0F087-part1
-> ../../sdc1
lrwxrwxrwx 1 root root  9 Jan 27 23:03 ata-ST33000651AS_9XK0FEHD -> ../../sdd
lrwxrwxrwx 1 root root 10 Jan 27 23:03 dm-name-vg_desktop-lv_home -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jan 27 23:03 dm-name-vg_desktop-lv_root -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jan 27 23:03
dm-uuid-LVM-gPmDONxd0J5Xzhng4nMgLyJywSvcsrFB1bQOUpSoE0SzQ6kHuvqiMzxpkiH0ZgWJ
-> ../../dm-1
lrwxrwxrwx 1 root root 10 Jan 27 23:03
dm-uuid-LVM-gPmDONxd0J5Xzhng4nMgLyJywSvcsrFBsTxjvh7YlYcmxUryNXlejEsR55sjfh9w
-> ../../dm-0
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c5002dae3e0a -> ../../sdb
lrwxrwxrwx 1 root root 10 Jan 27 23:03 wwn-0x5000c5002dae3e0a-part1 ->
../../sdb1
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c5002dfdfece -> ../../sdc
lrwxrwxrwx 1 root root 10 Jan 27 23:03 wwn-0x5000c5002dfdfece-part1 ->
../../sdc1
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c5002e0ff1ce -> ../../sda
lrwxrwxrwx 1 root root 10 Jan 27 23:03 wwn-0x5000c5002e0ff1ce-part1 ->
../../sda1
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c5002e10998c -> ../../sdd
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c5004e34e5a2 -> ../../sde
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c500657e23a6 -> ../../sdi
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c500657e27c4 -> ../../sdf
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5000c500657e2888 -> ../../sdg
lrwxrwxrwx 1 root root  9 Jan 27 23:03 wwn-0x5e83a97f6722d623 -> ../../sdh
lrwxrwxrwx 1 root root 10 Jan 27 23:03 wwn-0x5e83a97f6722d623-part1 ->
../../sdh1
lrwxrwxrwx 1 root root 10 Jan 27 23:03 wwn-0x5e83a97f6722d623-part2 ->
../../sdh2

cat /etc/mdadm.conf
ARRAY /dev/md0 devices=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3,/dev/sde3,/dev/sdf3,/dev/sdg3,/dev/sdh3

--Other--
# uname -a
Linux desktop.yorik 3.12.8-300.fc20.x86_64 #1 SMP Thu Jan 16 01:07:50
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

# mdadm -V
mdadm - v3.3 - 3rd September 2013

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-01-30 14:20 RAIS5 Rebuild Help - Possible Data Offset Yorik
@ 2014-01-31 10:25 ` Robin Hill
  2014-02-01  7:56   ` Yorik
  0 siblings, 1 reply; 9+ messages in thread
From: Robin Hill @ 2014-01-31 10:25 UTC (permalink / raw)
  To: Yorik; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 3518 bytes --]

On Thu Jan 30, 2014 at 10:20:45pm +0800, Yorik wrote:

> Hello
> 
> Sorry if double post had to remove some HTML.
> 
> I've been using "mdadm" for about 8-9 years with a lot of success in
> different Array configurations. I am a huge fan of Software RAID. I
> recently suffered a failure that was very unfortunate and I now have
> an issue with differing Data Offset and other issues preventing my
> Array from rebuilding. A brief history follows:
> 
> Array originally created as RAID5, 3x 3TB Drives in May 2011 on a
> (then) latest Fedora build.
> Added more drives in 2012 and 2013 with upgraded Fedora builds.
> Recently added another drive and changed to RAID6, 6x6TB Drives.
> Suffered a failure shortly after 2 more drives that kicked the last 2
> drives off and one of the 6 other that were previously running.
> During the RAID rebuild, my computer locked up and then suffered a
> Power Failure (having a bad day) that stopped my OS Disk (SSD) from
> allowing me to reboot. Eventually rebuilt the OS with Fedora 20,
> wiping the majority of the SSD and config files with it.
> 
> The RAID6 does not start, stating that it only has 5 of the 8 drives
> required. Looking into this, the drives seem to be clean and check
> okay, but there is even differences (expected) and Data Offset
> difference between drives that I read (Neil Brown Blog date
> -20120615073245) was not necessarily good for rebuilding and that I
> should ask for a version of mdadm that can handle this as it's
> possible that creating with an older version, then adding new drives
> is causing an issue. Data Offsets of 2048 and 4096 exist.
> 
> Also of note is that I have some drives that are on the Array without
> partitions as they would not add to the Array with partitions on them
> due to size when they were added. I think that I had version issue
> with fdisk, but it seemed to work okay until now.
> 
> When I run the following command:
> 
> mdadm --create --assume-clean --level=6 --raid-devices=8
> --size=2930263552 /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd
> /dev/sdi /dev/sde /dev/sdg /dev/sdf
> 
> I get:
> 
> mdadm: /dev/sda1 is smaller than given size. 2930134016K < 2930263552K
> + metadata
> mdadm: /dev/sdb1 is smaller than given size. 2930134016K < 2930263552K
> + metadata
> mdadm: /dev/sdc1 is smaller than given size. 2930134016K < 2930263552K
> + metadata
> mdadm: /dev/sdd is smaller than given size. 2930135040K < 2930263552K + metadata
> mdadm: /dev/sdi is smaller than given size. 2930135040K < 2930263552K + metadata
> mdadm: /dev/sde is smaller than given size. 2930135040K < 2930263552K + metadata
> mdadm: /dev/sdg is smaller than given size. 2930135040K < 2930263552K + metadata
> mdadm: /dev/sdf is smaller than given size. 2930135040K < 2930263552K + metadata
> mdadm: create aborted
> 
> Which makes me think that there are further issues. Should I be
> looking for an older version of mdadm or a different version? Can
> anyone offer some advice please or where to look next.
> 
You don't mention having tried a force assemble, which should have been
done before resorting to re-creating the array. What's the output from
running "mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]"?

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-01-31 10:25 ` Robin Hill
@ 2014-02-01  7:56   ` Yorik
  2014-03-31  6:29     ` NeilBrown
  0 siblings, 1 reply; 9+ messages in thread
From: Yorik @ 2014-02-01  7:56 UTC (permalink / raw)
  To: Yorik, linux-raid

Thank you for your help. Sorry for my omission, I thought I had put
that in. It was one of the first things that I tried and have numerous
times since.

#mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
mdadm: /dev/md0 assembled from 5 drives - not enough to start the array.

Then if I try the command again, I get the following

# mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
mdadm: /dev/sda1 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdd is busy - skipping
mdadm: /dev/sde is busy - skipping
mdadm: /dev/sdf is busy - skipping
mdadm: /dev/sdg is busy - skipping
mdadm: /dev/sdi is busy - skipping

And my /proc/mdstat looks like this, with all drives marked as spare.

# cat /proc/mdstat
Personalities :
md0 : inactive sda1[0](S) sdf[7](S) sdg[8](S) sde[5](S) sdi[6](S)
sdd[4](S) sdc1[2](S) sdb1[1](S)
      23442114872 blocks super 1.1

unused devices: <none>

If I stop the Array and try again, the following messages are written
into /var/log/messages when the command is run.

 Feb  1 15:53:52 desktop kernel: [405874.418851] md: bind<sdg>
Feb  1 15:53:52 desktop kernel: [405874.419043] md: bind<sdf>
Feb  1 15:53:52 desktop kernel: [405874.419189] md: bind<sda1>
Feb  1 15:53:52 desktop udisksd[1715]: Error creating watch for file
/sys/devices/virtual/block/md0/md/sync_action: No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error creating watch for file
/sys/devices/virtual/block/md0/md/degraded: No such file or directory
(g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/degraded': Failed to open file
'/sys/devices/virtual/block/md0/md/degraded': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_action': Failed to open file
'/sys/devices/virtual/block/md0/md/sync_action': No such file or
directory (g-file-error-quark, 4)
Feb  1 15:53:52 desktop udisksd[1715]: Error reading sysfs attr
`/sys/devices/virtual/block/md0/md/sync_completed': Failed to open
file '/sys/devices/virtual/block/md0/md/sync_completed': No such file
or directory (g-file-error-quark, 4)

Any other ideas appreciated..

On 31 January 2014 18:25, Robin Hill <robin@robinhill.me.uk> wrote:
> On Thu Jan 30, 2014 at 10:20:45pm +0800, Yorik wrote:
>
>> Hello
>>
>> Sorry if double post had to remove some HTML.
>>
>> I've been using "mdadm" for about 8-9 years with a lot of success in
>> different Array configurations. I am a huge fan of Software RAID. I
>> recently suffered a failure that was very unfortunate and I now have
>> an issue with differing Data Offset and other issues preventing my
>> Array from rebuilding. A brief history follows:
>>
>> Array originally created as RAID5, 3x 3TB Drives in May 2011 on a
>> (then) latest Fedora build.
>> Added more drives in 2012 and 2013 with upgraded Fedora builds.
>> Recently added another drive and changed to RAID6, 6x6TB Drives.
>> Suffered a failure shortly after 2 more drives that kicked the last 2
>> drives off and one of the 6 other that were previously running.
>> During the RAID rebuild, my computer locked up and then suffered a
>> Power Failure (having a bad day) that stopped my OS Disk (SSD) from
>> allowing me to reboot. Eventually rebuilt the OS with Fedora 20,
>> wiping the majority of the SSD and config files with it.
>>
>> The RAID6 does not start, stating that it only has 5 of the 8 drives
>> required. Looking into this, the drives seem to be clean and check
>> okay, but there is even differences (expected) and Data Offset
>> difference between drives that I read (Neil Brown Blog date
>> -20120615073245) was not necessarily good for rebuilding and that I
>> should ask for a version of mdadm that can handle this as it's
>> possible that creating with an older version, then adding new drives
>> is causing an issue. Data Offsets of 2048 and 4096 exist.
>>
>> Also of note is that I have some drives that are on the Array without
>> partitions as they would not add to the Array with partitions on them
>> due to size when they were added. I think that I had version issue
>> with fdisk, but it seemed to work okay until now.
>>
>> When I run the following command:
>>
>> mdadm --create --assume-clean --level=6 --raid-devices=8
>> --size=2930263552 /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd
>> /dev/sdi /dev/sde /dev/sdg /dev/sdf
>>
>> I get:
>>
>> mdadm: /dev/sda1 is smaller than given size. 2930134016K < 2930263552K
>> + metadata
>> mdadm: /dev/sdb1 is smaller than given size. 2930134016K < 2930263552K
>> + metadata
>> mdadm: /dev/sdc1 is smaller than given size. 2930134016K < 2930263552K
>> + metadata
>> mdadm: /dev/sdd is smaller than given size. 2930135040K < 2930263552K + metadata
>> mdadm: /dev/sdi is smaller than given size. 2930135040K < 2930263552K + metadata
>> mdadm: /dev/sde is smaller than given size. 2930135040K < 2930263552K + metadata
>> mdadm: /dev/sdg is smaller than given size. 2930135040K < 2930263552K + metadata
>> mdadm: /dev/sdf is smaller than given size. 2930135040K < 2930263552K + metadata
>> mdadm: create aborted
>>
>> Which makes me think that there are further issues. Should I be
>> looking for an older version of mdadm or a different version? Can
>> anyone offer some advice please or where to look next.
>>
> You don't mention having tried a force assemble, which should have been
> done before resorting to re-creating the array. What's the output from
> running "mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]"?
>
> Cheers,
>     Robin
> --
>      ___
>     ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>    / / )      | Little Jim says ....                            |
>   // !!       |      "He fallen in de water !!"                 |

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-02-01  7:56   ` Yorik
@ 2014-03-31  6:29     ` NeilBrown
  2014-03-31  6:54       ` Yorik
  0 siblings, 1 reply; 9+ messages in thread
From: NeilBrown @ 2014-03-31  6:29 UTC (permalink / raw)
  To: Yorik; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

On Sat, 1 Feb 2014 15:56:14 +0800 Yorik <yoriks13@westnet.com.au> wrote:

> Thank you for your help. Sorry for my omission, I thought I had put
> that in. It was one of the first things that I tried and have numerous
> times since.
> 
> #mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
> mdadm: /dev/md0 assembled from 5 drives - not enough to start the array.
> 
> Then if I try the command again, I get the following
> 
> # mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
> mdadm: /dev/sda1 is busy - skipping
> mdadm: /dev/sdb1 is busy - skipping
> mdadm: /dev/sdc1 is busy - skipping
> mdadm: /dev/sdd is busy - skipping
> mdadm: /dev/sde is busy - skipping
> mdadm: /dev/sdf is busy - skipping
> mdadm: /dev/sdg is busy - skipping
> mdadm: /dev/sdi is busy - skipping
> 
> And my /proc/mdstat looks like this, with all drives marked as spare.
> 
> # cat /proc/mdstat
> Personalities :
> md0 : inactive sda1[0](S) sdf[7](S) sdg[8](S) sde[5](S) sdi[6](S)
> sdd[4](S) sdc1[2](S) sdb1[1](S)
>       23442114872 blocks super 1.1
> 
> unused devices: <none>

If this is still a problem then please:
1/ get the latest mdadm:
     cd /tmp
     git clone git://neil.brown.name/mdadm
     cd mdadm
     make
2/ stop the array
     ./mdadm --stop /dev/md0
3/ force-assemble the array with --verbose:

     ./mdadm --assemble --force --verbose /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]

and report all the output of the last command.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-03-31  6:29     ` NeilBrown
@ 2014-03-31  6:54       ` Yorik
  2014-03-31  7:12         ` NeilBrown
  0 siblings, 1 reply; 9+ messages in thread
From: Yorik @ 2014-03-31  6:54 UTC (permalink / raw)
  To: NeilBrown; +Cc: linux-raid

Hi Neil

Thanks very much for the help and advice.. Output as follows.

[root@desktop mdadm]# ./mdadm --assemble --force --verbose /dev/md0
/dev/sd[abc]1 /dev/sd[defgi]
mdadm: looking for devices for /dev/md0
mdadm: superblock on /dev/sdf doesn't match others - assembly aborted


Maybe there has been some differences in Data Offset and superblock
sizes with different versions of mdadm. I am not really sure what to
do now, I just got back from a long project overseas and was about to
start looking deeper into my options.

Appreciate any further help or advice that you have.

Regards
Derek

On 31 March 2014 14:29, NeilBrown <neilb@suse.de> wrote:
> On Sat, 1 Feb 2014 15:56:14 +0800 Yorik <yoriks13@westnet.com.au> wrote:
>
>> Thank you for your help. Sorry for my omission, I thought I had put
>> that in. It was one of the first things that I tried and have numerous
>> times since.
>>
>> #mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
>> mdadm: /dev/md0 assembled from 5 drives - not enough to start the array.
>>
>> Then if I try the command again, I get the following
>>
>> # mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
>> mdadm: /dev/sda1 is busy - skipping
>> mdadm: /dev/sdb1 is busy - skipping
>> mdadm: /dev/sdc1 is busy - skipping
>> mdadm: /dev/sdd is busy - skipping
>> mdadm: /dev/sde is busy - skipping
>> mdadm: /dev/sdf is busy - skipping
>> mdadm: /dev/sdg is busy - skipping
>> mdadm: /dev/sdi is busy - skipping
>>
>> And my /proc/mdstat looks like this, with all drives marked as spare.
>>
>> # cat /proc/mdstat
>> Personalities :
>> md0 : inactive sda1[0](S) sdf[7](S) sdg[8](S) sde[5](S) sdi[6](S)
>> sdd[4](S) sdc1[2](S) sdb1[1](S)
>>       23442114872 blocks super 1.1
>>
>> unused devices: <none>
>
> If this is still a problem then please:
> 1/ get the latest mdadm:
>      cd /tmp
>      git clone git://neil.brown.name/mdadm
>      cd mdadm
>      make
> 2/ stop the array
>      ./mdadm --stop /dev/md0
> 3/ force-assemble the array with --verbose:
>
>      ./mdadm --assemble --force --verbose /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
>
> and report all the output of the last command.
>
> NeilBrown
>

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-03-31  6:54       ` Yorik
@ 2014-03-31  7:12         ` NeilBrown
  2014-03-31  7:24           ` Yorik
  0 siblings, 1 reply; 9+ messages in thread
From: NeilBrown @ 2014-03-31  7:12 UTC (permalink / raw)
  To: Yorik; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

On Mon, 31 Mar 2014 14:54:31 +0800 Yorik <yoriks13@westnet.com.au> wrote:

> Hi Neil
> 
> Thanks very much for the help and advice.. Output as follows.
> 
> [root@desktop mdadm]# ./mdadm --assemble --force --verbose /dev/md0
> /dev/sd[abc]1 /dev/sd[defgi]
> mdadm: looking for devices for /dev/md0
> mdadm: superblock on /dev/sdf doesn't match others - assembly aborted
> 
> 
> Maybe there has been some differences in Data Offset and superblock
> sizes with different versions of mdadm. I am not really sure what to
> do now, I just got back from a long project overseas and was about to
> start looking deeper into my options.
> 
> Appreciate any further help or advice that you have.


Could you post the full output of 
  mdadm --examine /dev/[abc]1 /dev/sd[defgi]

Some is clearly different at least with sdf.

NeilBrown

> 
> Regards
> Derek
> 
> On 31 March 2014 14:29, NeilBrown <neilb@suse.de> wrote:
> > On Sat, 1 Feb 2014 15:56:14 +0800 Yorik <yoriks13@westnet.com.au> wrote:
> >
> >> Thank you for your help. Sorry for my omission, I thought I had put
> >> that in. It was one of the first things that I tried and have numerous
> >> times since.
> >>
> >> #mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
> >> mdadm: /dev/md0 assembled from 5 drives - not enough to start the array.
> >>
> >> Then if I try the command again, I get the following
> >>
> >> # mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
> >> mdadm: /dev/sda1 is busy - skipping
> >> mdadm: /dev/sdb1 is busy - skipping
> >> mdadm: /dev/sdc1 is busy - skipping
> >> mdadm: /dev/sdd is busy - skipping
> >> mdadm: /dev/sde is busy - skipping
> >> mdadm: /dev/sdf is busy - skipping
> >> mdadm: /dev/sdg is busy - skipping
> >> mdadm: /dev/sdi is busy - skipping
> >>
> >> And my /proc/mdstat looks like this, with all drives marked as spare.
> >>
> >> # cat /proc/mdstat
> >> Personalities :
> >> md0 : inactive sda1[0](S) sdf[7](S) sdg[8](S) sde[5](S) sdi[6](S)
> >> sdd[4](S) sdc1[2](S) sdb1[1](S)
> >>       23442114872 blocks super 1.1
> >>
> >> unused devices: <none>
> >
> > If this is still a problem then please:
> > 1/ get the latest mdadm:
> >      cd /tmp
> >      git clone git://neil.brown.name/mdadm
> >      cd mdadm
> >      make
> > 2/ stop the array
> >      ./mdadm --stop /dev/md0
> > 3/ force-assemble the array with --verbose:
> >
> >      ./mdadm --assemble --force --verbose /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
> >
> > and report all the output of the last command.
> >
> > NeilBrown
> >


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-03-31  7:12         ` NeilBrown
@ 2014-03-31  7:24           ` Yorik
  2014-03-31  8:37             ` NeilBrown
  0 siblings, 1 reply; 9+ messages in thread
From: Yorik @ 2014-03-31  7:24 UTC (permalink / raw)
  To: NeilBrown; +Cc: linux-raid

Hi Neil

Thanks for your quick reply. :) Output as follows. and yes I agree.
sdf is strange, also some devices think the Array has different number
of devices Active.. It all got quite muddled up at the time of
failure.

[root@desktop mdadm]# ./mdadm --examine /dev/sd[abc]1 /dev/sd[defgi]
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e32ad2c6:5f99dd8f:2572c3a8:c485db00
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:38:19 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 90e00065:90a8aa11:a636fb20:7b381843

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:38:19 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : e84e430c - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e32ad2c6:5f99dd8f:2572c3a8:c485db00
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:38:19 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 364b1f3d:060ba7b0:e1023e93:4dd320b8

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:38:19 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 46b47738 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e32ad2c6:5f99dd8f:2572c3a8:c485db00
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:38:19 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : c90f3c3d:c0656476:2219a286:1f03efba

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:38:19 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 2c0dc99 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e32ad2c6:5f99dd8f:2572c3a8:c485db00
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:38:19 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=2992 sectors
          State : clean
    Device UUID : 8554f422:ec64f7bc:5c34f9b6:208c4c47

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:38:19 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : ebc0d06c - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e32ad2c6:5f99dd8f:2572c3a8:c485db00
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:38:19 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=2992 sectors
          State : clean
    Device UUID : 16f97e63:6c3ec636:2612a1ec:abce713a

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:38:19 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : cee76ed4 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 26124817:0056ac1b:d1906468:a8784872
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:33:56 2014
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=2992 sectors
          State : clean
    Device UUID : 17feeac8:451539a0:c0ecdef8:1319398f

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:33:56 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c1d231a - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : a5cb0ced:fee37e80:bb291f1a:c5b2f600
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:28:54 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=2992 sectors
          State : clean
    Device UUID : 052e90fb:b4ed4385:d7cf3dec:ad9e7004

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:28:54 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 76aac59 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : a5cb0ced:fee37e80:bb291f1a:c5b2f600
           Name : desktop.yorik:0  (local to host desktop.yorik)
  Creation Time : Tue Feb  4 00:28:54 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=2992 sectors
          State : clean
    Device UUID : 96d39460:5658d235:111d5f75:33e188d3

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Feb  4 00:28:54 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 75374c49 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

Regards
Derek

On 31 March 2014 15:12, NeilBrown <neilb@suse.de> wrote:
> On Mon, 31 Mar 2014 14:54:31 +0800 Yorik <yoriks13@westnet.com.au> wrote:
>
>> Hi Neil
>>
>> Thanks very much for the help and advice.. Output as follows.
>>
>> [root@desktop mdadm]# ./mdadm --assemble --force --verbose /dev/md0
>> /dev/sd[abc]1 /dev/sd[defgi]
>> mdadm: looking for devices for /dev/md0
>> mdadm: superblock on /dev/sdf doesn't match others - assembly aborted
>>
>>
>> Maybe there has been some differences in Data Offset and superblock
>> sizes with different versions of mdadm. I am not really sure what to
>> do now, I just got back from a long project overseas and was about to
>> start looking deeper into my options.
>>
>> Appreciate any further help or advice that you have.
>
>
> Could you post the full output of
>   mdadm --examine /dev/[abc]1 /dev/sd[defgi]
>
> Some is clearly different at least with sdf.
>
> NeilBrown
>
>>
>> Regards
>> Derek
>>
>> On 31 March 2014 14:29, NeilBrown <neilb@suse.de> wrote:
>> > On Sat, 1 Feb 2014 15:56:14 +0800 Yorik <yoriks13@westnet.com.au> wrote:
>> >
>> >> Thank you for your help. Sorry for my omission, I thought I had put
>> >> that in. It was one of the first things that I tried and have numerous
>> >> times since.
>> >>
>> >> #mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
>> >> mdadm: /dev/md0 assembled from 5 drives - not enough to start the array.
>> >>
>> >> Then if I try the command again, I get the following
>> >>
>> >> # mdadm -Af /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
>> >> mdadm: /dev/sda1 is busy - skipping
>> >> mdadm: /dev/sdb1 is busy - skipping
>> >> mdadm: /dev/sdc1 is busy - skipping
>> >> mdadm: /dev/sdd is busy - skipping
>> >> mdadm: /dev/sde is busy - skipping
>> >> mdadm: /dev/sdf is busy - skipping
>> >> mdadm: /dev/sdg is busy - skipping
>> >> mdadm: /dev/sdi is busy - skipping
>> >>
>> >> And my /proc/mdstat looks like this, with all drives marked as spare.
>> >>
>> >> # cat /proc/mdstat
>> >> Personalities :
>> >> md0 : inactive sda1[0](S) sdf[7](S) sdg[8](S) sde[5](S) sdi[6](S)
>> >> sdd[4](S) sdc1[2](S) sdb1[1](S)
>> >>       23442114872 blocks super 1.1
>> >>
>> >> unused devices: <none>
>> >
>> > If this is still a problem then please:
>> > 1/ get the latest mdadm:
>> >      cd /tmp
>> >      git clone git://neil.brown.name/mdadm
>> >      cd mdadm
>> >      make
>> > 2/ stop the array
>> >      ./mdadm --stop /dev/md0
>> > 3/ force-assemble the array with --verbose:
>> >
>> >      ./mdadm --assemble --force --verbose /dev/md0 /dev/sd[abc]1 /dev/sd[defgi]
>> >
>> > and report all the output of the last command.
>> >
>> > NeilBrown
>> >
>

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-03-31  7:24           ` Yorik
@ 2014-03-31  8:37             ` NeilBrown
  2014-04-01  3:49               ` Yorik
  0 siblings, 1 reply; 9+ messages in thread
From: NeilBrown @ 2014-03-31  8:37 UTC (permalink / raw)
  To: Yorik; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 1315 bytes --]

On Mon, 31 Mar 2014 15:24:07 +0800 Yorik <yoriks13@westnet.com.au> wrote:

> Hi Neil
> 
> Thanks for your quick reply. :) Output as follows. and yes I agree.
> sdf is strange, also some devices think the Array has different number
> of devices Active.. It all got quite muddled up at the time of
> failure.

So.....
sdf thinks
   Raid Devices : 6
others think
   Raid Devices : 8

So that's not going to work.
However if we just ignore sdf, then sdg and sdi thinkg

     Array UUID : a5cb0ced:fee37e80:bb291f1a:c5b2f600

while the rest think:

     Array UUID : e32ad2c6:5f99dd8f:2572c3a8:c485db00

So you have at most 5 device which think they are the same array,
which isn't enough for an 8-device RAID6.

It looks like you have tried to --create the array several times,
at

  Creation Time : Tue Feb  4 00:28:54 2014
  Creation Time : Tue Feb  4 00:33:56 2014
  Creation Time : Tue Feb  4 00:38:19 2014

to be precise.

Your only real chance is to find out the real geometry of the
array before all the problem happened, and create it properly...
It might be worth checking sda sdb and sdc (with out partitions)
as well as sdd1 sde1 sdf1 sdg1 sdi1 to see if there is anything there.

But if you have backups or some other way to regenerate the data ......

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: RAIS5 Rebuild Help - Possible Data Offset
  2014-03-31  8:37             ` NeilBrown
@ 2014-04-01  3:49               ` Yorik
  0 siblings, 0 replies; 9+ messages in thread
From: Yorik @ 2014-04-01  3:49 UTC (permalink / raw)
  To: NeilBrown; +Cc: linux-raid

Hi Neil

Thanks very much for your help and pointers. This is what I was
thinking a while ago when I was looking at this. Suffered a few very
quick failures in quick succession and probably got too trigger happy
on my rebuilding in between.
I will look deeper and report back if I come up with anything useful.

Thanks again.

Cheers
Derek

On 31 March 2014 16:37, NeilBrown <neilb@suse.de> wrote:
> On Mon, 31 Mar 2014 15:24:07 +0800 Yorik <yoriks13@westnet.com.au> wrote:
>
>> Hi Neil
>>
>> Thanks for your quick reply. :) Output as follows. and yes I agree.
>> sdf is strange, also some devices think the Array has different number
>> of devices Active.. It all got quite muddled up at the time of
>> failure.
>
> So.....
> sdf thinks
>    Raid Devices : 6
> others think
>    Raid Devices : 8
>
> So that's not going to work.
> However if we just ignore sdf, then sdg and sdi thinkg
>
>      Array UUID : a5cb0ced:fee37e80:bb291f1a:c5b2f600
>
> while the rest think:
>
>      Array UUID : e32ad2c6:5f99dd8f:2572c3a8:c485db00
>
> So you have at most 5 device which think they are the same array,
> which isn't enough for an 8-device RAID6.
>
> It looks like you have tried to --create the array several times,
> at
>
>   Creation Time : Tue Feb  4 00:28:54 2014
>   Creation Time : Tue Feb  4 00:33:56 2014
>   Creation Time : Tue Feb  4 00:38:19 2014
>
> to be precise.
>
> Your only real chance is to find out the real geometry of the
> array before all the problem happened, and create it properly...
> It might be worth checking sda sdb and sdc (with out partitions)
> as well as sdd1 sde1 sdf1 sdg1 sdi1 to see if there is anything there.
>
> But if you have backups or some other way to regenerate the data ......
>
> NeilBrown

^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-04-01  3:49 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-30 14:20 RAIS5 Rebuild Help - Possible Data Offset Yorik
2014-01-31 10:25 ` Robin Hill
2014-02-01  7:56   ` Yorik
2014-03-31  6:29     ` NeilBrown
2014-03-31  6:54       ` Yorik
2014-03-31  7:12         ` NeilBrown
2014-03-31  7:24           ` Yorik
2014-03-31  8:37             ` NeilBrown
2014-04-01  3:49               ` Yorik

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.