RAID MIA. Again. (Kinda.)

RAID MIA. Again. (Kinda.)

[Ken D'Ambrosio]

Hey, all.  I had a problem a month or so ago where I had three mirrors,
and one of them went AWOL after a system migration.  Finally got as much
as I could back, created a RAID-5 of four of the 1.5 TB disks, put
everything on it, recovered more from backup, and was feeling pretty good.

Then the power went out.

Now, now RAID-5.  Or, rather, mdadm sees that there are RAID partitions,
and starts; fdisk shows me an unpartitioned disk.  Which is, essentially,
the exact same thing I had happen last time.  (Except that, this time, I
*know* my mdadm.conf is correct, because it hasn't changed -- but, for the
hell of it, I did a "mdadm --detail --scan", and it matches.)

WTF?  Am I too stupid for software RAID?  Is there something really
obvious I'm missing?  Any help at all would be most graciously accepted.

Thanks,

-Ken


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: RAID MIA. Again. (Kinda.)

[Neil Brown]

On Thu, 4 Mar 2010 11:12:29 -0500 (EST)
"Ken D'Ambrosio" <ken@jots.org> wrote:

> Hey, all.  I had a problem a month or so ago where I had three mirrors,
> and one of them went AWOL after a system migration.  Finally got as much
> as I could back, created a RAID-5 of four of the 1.5 TB disks, put
> everything on it, recovered more from backup, and was feeling pretty good.
> 
> Then the power went out.
> 
> Now, now RAID-5.  Or, rather, mdadm sees that there are RAID partitions,
> and starts; fdisk shows me an unpartitioned disk.  Which is, essentially,
> the exact same thing I had happen last time.  (Except that, this time, I
> *know* my mdadm.conf is correct, because it hasn't changed -- but, for the
> hell of it, I did a "mdadm --detail --scan", and it matches.)
> 
> WTF?  Am I too stupid for software RAID?  Is there something really
> obvious I'm missing?  Any help at all would be most graciously accepted.

You missed the bit where you provide concrete information rather than
vaguaries.

I'm guessing that you created the array over whole-devices, and then
partitioned the array - is that correct?
If fdisk shows you an unpartitioned array, maybe just the partition table is
corrupt.   Seems strange.

To so that I/we don't have to guess, please give exact commands that you run
and the exact output so we have access to the same information as you.

Too much data is much much better than not enough.

NeilBrown

Re: RAID MIA. Again. (Kinda.)

[Ken D'Ambrosio]

On Thu, March 4, 2010 3:50 pm, Neil Brown wrote:
> On Thu, 4 Mar 2010 11:12:29 -0500 (EST)
> "Ken D'Ambrosio" <ken@jots.org> wrote:

[bad things happened to Ken's RAID-5 here]

>
> You missed the bit where you provide concrete information rather than
> vaguaries.

Humble apologies; I'm not well-enough versed with the intricacies of Linux
RAID to know what's appropriate and not.

> I'm guessing that you created the array over whole-devices, and then
> partitioned the array - is that correct? If fdisk shows you an
> unpartitioned array, maybe just the partition table is corrupt.   Seems
> strange.

Actually, no.  These were created using /dev/sd[abcd]2 -- I saved off
space on sd[abcd]1 for swap, /tmp, etc.  Done via the Ubuntu installer, if
that makes a difference.  For the record, all the /dev/sd[abcd]1 non-RAID
partitions look fine.

> To so that I/we don't have to guess, please give exact commands that you
> run and the exact output so we have access to the same information as you.

Well, I rebooted, and was surprised that nothing RAID-esque came up. 
Since my OS is on one of the afore-mentioned non-RAIDed partitions, the
OS, itself, booted, but none of the RAID partitions mounted.  I tried to
mount, and failed.  That's when I checked the RAID device, /dev/md0. 
fdisk showed it lacking any partitions, but the mdadm.conf file hadn't
been touched for a couple of weeks, so I was pretty sure nothing there had
changed.  On the off chance that the SCSI drives had re-ordered
themselves, I went through all 24 permutations of
mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
since I wasn't sure if the drive order was significant.  All of them
"worked," inasmuch as they created /dev/md0, but in all cases it was
partitionless.  I also tried mdadm --detail --scan, to verify that it
matched UUIDs with those in the /etc/mdadm.conf file, and it did (the
array line looks thusly:
ARRAY /dev/md0 level=raid5 num-devices=4
UUID=1e89645a:7e24dcef:9e77d54f:077a6a6f  )


> Too much data is much much better than not enough.

Granted... but, sometimes -- especially when learning -- there can be a
bad signal:noise ratio.  Tends to make me hesitant when I'm a (relative)
newbie to a given topic.  That being said, I *think* I've figured out what
I should be doing, but I also think I did it.  Did I miss something?

Thanks!

-Ken

> NeilBrown
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is believed to be clean.
>
>



-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: RAID MIA. Again. (Kinda.)

[Michael Evans]

On Thu, Mar 4, 2010 at 2:21 PM, Michael Evans <mjevans1983@gmail.com> wrote:
> Try providing the output of;
> for ii in /dev/[sh]d[a-z] ; do parted $ii print ; done
>
> If you have any kind of partition table this should tell us what it
> is.  If you don't, something odd is happening.
>

Also, you'll need to run that as the root (unix term for absolute
administrator account) user.  Since you mentioned installing and lack
partitions you probably need a linux on CD or usb stick distribution;
http://www.sysresccd.org/Main_Page works well for me and might help
you enough to get you in to user mode; as well as non-destructively
auto-detecting most storage setups and having very recent software.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID MIA. Again. (Kinda.)

[Ken D'Ambrosio]

On Thu, March 4, 2010 5:21 pm, Michael Evans wrote:
> Try providing the output of;
> for ii in /dev/[sh]d[a-z] ; do parted $ii print ; done

Mea culpa; I'd said:

 I went through all 24 permutations of
 mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
 since I wasn't sure if the drive order was significant.  All of them
 "worked," inasmuch as they created /dev/md0, but in all cases it was
 partitionless.

Which I assumed implied that /dev/sd[a-d]2 was valid on all disks, though,
in hindsight, I wasn't explicit.  So: /dev/sd[a-d]2 exists on all drives
as partition id "fd" (Linux raid autodetect).  It's /dev/md0 that shows as
a valid, 4.4 TB disk... with no partition.

-Ken


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: RAID MIA. Again. (Kinda.)

[Michael Evans]

On Thu, Mar 4, 2010 at 3:41 PM, Ken D'Ambrosio <ken@jots.org> wrote:
> On Thu, March 4, 2010 5:21 pm, Michael Evans wrote:
>> Try providing the output of;
>> for ii in /dev/[sh]d[a-z] ; do parted $ii print ; done
>
> Mea culpa; I'd said:
>
>  I went through all 24 permutations of
>  mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
>  since I wasn't sure if the drive order was significant.  All of them
>  "worked," inasmuch as they created /dev/md0, but in all cases it was
>  partitionless.
>
> Which I assumed implied that /dev/sd[a-d]2 was valid on all disks, though,
> in hindsight, I wasn't explicit.  So: /dev/sd[a-d]2 exists on all drives
> as partition id "fd" (Linux raid autodetect).  It's /dev/md0 that shows as
> a valid, 4.4 TB disk... with no partition.
>
> -Ken

It sounds like you might be interested in this script:

http://www.linuxfoundation.org/collaborate/workgroups/linux-raid/raid_recovery

It's untested, but it may produce a sequence that shows you valid data.

Also if you 'partitioned' the resulting raid device what you most
likely did was use it as an LVM physical volume, and then create
logical volumes from it.  At least that's the way I'd do it.

After you have /dev/md0 (or whatever) running try running this;

vgscan ; vgdisplay

You might see your 'partitions' listed, at which point you can do a
read-only fsck, then read-only mount and determine if they are in fact
whole, or if they are instead corrupt in that configuration.

If you manually let mdadm guess where to put the devices based on the
stored metadata then it will probably determine the correct order for
you; presuming you haven't already over-written that with invalid
data.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID MIA. Again. (Kinda.)

[Neil Brown]

On Thu, 4 Mar 2010 17:00:10 -0500 (EST)
"Ken D'Ambrosio" <ken@jots.org> wrote:

> On Thu, March 4, 2010 3:50 pm, Neil Brown wrote:
> > On Thu, 4 Mar 2010 11:12:29 -0500 (EST)
> > "Ken D'Ambrosio" <ken@jots.org> wrote:
> 
> [bad things happened to Ken's RAID-5 here]
> 
> >
> > You missed the bit where you provide concrete information rather than
> > vaguaries.
> 
> Humble apologies; I'm not well-enough versed with the intricacies of Linux
> RAID to know what's appropriate and not.
> 
> > I'm guessing that you created the array over whole-devices, and then
> > partitioned the array - is that correct? If fdisk shows you an
> > unpartitioned array, maybe just the partition table is corrupt.   Seems
> > strange.
> 
> Actually, no.  These were created using /dev/sd[abcd]2 -- I saved off
> space on sd[abcd]1 for swap, /tmp, etc.  Done via the Ubuntu installer, if
> that makes a difference.  For the record, all the /dev/sd[abcd]1 non-RAID
> partitions look fine.
> 
> > To so that I/we don't have to guess, please give exact commands that you
> > run and the exact output so we have access to the same information as you.
> 
> Well, I rebooted, and was surprised that nothing RAID-esque came up. 
> Since my OS is on one of the afore-mentioned non-RAIDed partitions, the
> OS, itself, booted, but none of the RAID partitions mounted.  I tried to
> mount, and failed.  That's when I checked the RAID device, /dev/md0. 
> fdisk showed it lacking any partitions, but the mdadm.conf file hadn't
> been touched for a couple of weeks, so I was pretty sure nothing there had
> changed.  On the off chance that the SCSI drives had re-ordered
> themselves, I went through all 24 permutations of
> mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
> since I wasn't sure if the drive order was significant.  All of them
> "worked," inasmuch as they created /dev/md0, but in all cases it was
> partitionless.  I also tried mdadm --detail --scan, to verify that it
> matched UUIDs with those in the /etc/mdadm.conf file, and it did (the
> array line looks thusly:
> ARRAY /dev/md0 level=raid5 num-devices=4
> UUID=1e89645a:7e24dcef:9e77d54f:077a6a6f  )
> 
> 
> > Too much data is much much better than not enough.
> 
> Granted... but, sometimes -- especially when learning -- there can be a
> bad signal:noise ratio.  Tends to make me hesitant when I'm a (relative)
> newbie to a given topic.  That being said, I *think* I've figured out what
> I should be doing, but I also think I did it.  Did I miss something?

Which part of "please give exact commands that you
run and the exact output"
Did you have trouble with?

Try this:

cat /etc/fstab
cat /etc/mdadm.conf
mdadm -Esvv
mdadm -Asv
blkid -p /dev/md*

And include all the output.

(sorry if I seem grumpy, but I'm a bit tired which makes it harder to appear
polite).

NeilBrown

Re: RAID MIA. Again. (Kinda.)

[Ken D'Ambrosio]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

On Thu, March 4, 2010 8:30 pm, Neil Brown wrote:
> Try this:
>
>
> cat /etc/fstab cat /etc/mdadm.conf mdadm -Esvv mdadm -Asv blkid -p /dev/md*
>
> And include all the output.

See attached.  Note that fstab is essentially irrelevant; I'd mounted by
hand, and was going to add the entry after I got the server back in the
server room.

> (sorry if I seem grumpy, but I'm a bit tired which makes it harder to
> appear polite).

Know what you mean; got four hours', myself, due to a virtualization
migration.

For the record, I've got my backup tapes ordered (again) -- if you can
tell me what I did wrong, that would be almost as satisfying as getting
the data back.

Thanks,

-Ken
-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


[-- Attachment #2: fdisk.txt --]
[-- Type: text/plain, Size: 218 bytes --]

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md0: 4410.9 GB, 4410904805376 bytes
2 heads, 4 sectors/track, 1076881056 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

[-- Attachment #3: mdadm.conf --]
[-- Type: application/octet-stream, Size: 780 bytes --]

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

ARRAY /dev/md0 level=raid5 num-devices=4 UUID=1e89645a:7e24dcef:9e77d54f:077a6a6f

#ARRAY /dev/md0 level=raid5 num-devices=4 devices=sda2,sdb2,sdd2,sdc2

# This file was auto-generated on Thu, 11 Feb 2010 21:56:16 +0000
# by mkconf $Id$

[-- Attachment #4: mdadm-Asv.txt --]
[-- Type: text/plain, Size: 2642 bytes --]

mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: no RAID superblock on /dev/sdd1
mdadm: /dev/sdd1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: cannot open device /dev/loop1: Device or resource busy
mdadm: /dev/loop1 has wrong uuid.
mdadm: cannot open device /dev/loop0: Device or resource busy
mdadm: /dev/loop0 has wrong uuid.
mdadm: /dev/sdd2 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdc2 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdb2 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sda2 is identified as a member of /dev/md0, slot 2.
mdadm: added /dev/sdc2 to /dev/md0 as 1
mdadm: added /dev/sda2 to /dev/md0 as 2
mdadm: added /dev/sdd2 to /dev/md0 as 3
mdadm: added /dev/sdb2 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 4 drives.
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/block/9:0
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: cannot open device /dev/sdd2: Device or resource busy
mdadm: no recogniseable superblock on /dev/sdd1
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sdc2: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb2: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: cannot open device /dev/loop1: Device or resource busy
mdadm: cannot open device /dev/loop0: Device or resource busy

[-- Attachment #5: mdadm-Esvv.txt --]
[-- Type: text/plain, Size: 4608 bytes --]

mdadm: No md superblock detected on /dev/block/9:0.
mdadm: No md superblock detected on /dev/sde1.
mdadm: No md superblock detected on /dev/sde.
mdadm: No md superblock detected on /dev/sdd1.
mdadm: No md superblock detected on /dev/sdd.
mdadm: No md superblock detected on /dev/sdc1.
mdadm: No md superblock detected on /dev/sdc.
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdb.
mdadm: No md superblock detected on /dev/sda1.
mdadm: No md superblock detected on /dev/sda.
mdadm: No md superblock detected on /dev/loop1.
mdadm: No md superblock detected on /dev/loop0.
/dev/sdd2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 1e89645a:7e24dcef:9e77d54f:077a6a6f
  Creation Time : Thu Feb  4 20:54:58 2010
     Raid Level : raid5
  Used Dev Size : 1435841408 (1369.33 GiB 1470.30 GB)
     Array Size : 4307524224 (4107.98 GiB 4410.90 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Fri Mar  5 07:04:06 2010
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d85ef7b3 - correct
         Events : 35041

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       50        3      active sync   /dev/sdd2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8        2        2      active sync   /dev/sda2
   3     3       8       50        3      active sync   /dev/sdd2
/dev/sdc2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 1e89645a:7e24dcef:9e77d54f:077a6a6f
  Creation Time : Thu Feb  4 20:54:58 2010
     Raid Level : raid5
  Used Dev Size : 1435841408 (1369.33 GiB 1470.30 GB)
     Array Size : 4307524224 (4107.98 GiB 4410.90 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Fri Mar  5 07:04:06 2010
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d85ef79f - correct
         Events : 35041

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       34        1      active sync   /dev/sdc2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8        2        2      active sync   /dev/sda2
   3     3       8       50        3      active sync   /dev/sdd2
/dev/sdb2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 1e89645a:7e24dcef:9e77d54f:077a6a6f
  Creation Time : Thu Feb  4 20:54:58 2010
     Raid Level : raid5
  Used Dev Size : 1435841408 (1369.33 GiB 1470.30 GB)
     Array Size : 4307524224 (4107.98 GiB 4410.90 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Fri Mar  5 07:04:06 2010
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d85ef78d - correct
         Events : 35041

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       18        0      active sync   /dev/sdb2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8        2        2      active sync   /dev/sda2
   3     3       8       50        3      active sync   /dev/sdd2
/dev/sda2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 1e89645a:7e24dcef:9e77d54f:077a6a6f
  Creation Time : Thu Feb  4 20:54:58 2010
     Raid Level : raid5
  Used Dev Size : 1435841408 (1369.33 GiB 1470.30 GB)
     Array Size : 4307524224 (4107.98 GiB 4410.90 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Fri Mar  5 07:04:06 2010
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d85ef781 - correct
         Events : 35041

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8        2        2      active sync   /dev/sda2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8        2        2      active sync   /dev/sda2
   3     3       8       50        3      active sync   /dev/sdd2

[-- Attachment #6: blkid.txt --]
[-- Type: text/plain, Size: 84 bytes --]

/dev/md0: UUID="d6158445-e38e-4942-9395-e1bd29935d22" TYPE="xfs" USAGE="filesystem" 

Re: RAID MIA. Again. (Kinda.)

[Michael Evans]

On Fri, Mar 5, 2010 at 7:24 AM, Ken D'Ambrosio <ken@jots.org> wrote:
> On Thu, March 4, 2010 8:30 pm, Neil Brown wrote:
>> Try this:
>>
>>
>> cat /etc/fstab cat /etc/mdadm.conf mdadm -Esvv mdadm -Asv blkid -p /dev/md*
>>
>> And include all the output.
>
> See attached.  Note that fstab is essentially irrelevant; I'd mounted by
> hand, and was going to add the entry after I got the server back in the
> server room.
>
>> (sorry if I seem grumpy, but I'm a bit tired which makes it harder to
>> appear polite).
>
> Know what you mean; got four hours', myself, due to a virtualization
> migration.
>
> For the record, I've got my backup tapes ordered (again) -- if you can
> tell me what I did wrong, that would be almost as satisfying as getting
> the data back.
>
> Thanks,
>
> -Ken
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>

Here's a stab at it. I've never tried to do it like this and haven't
glanced at the code so I can't be sure, but this seems to be the most
likely suspect:

#ARRAY /dev/md0 level=raid5 num-devices=4 devices=sda2,sdb2,sdd2,sdc2

Your commented out version seems to specify this mapping
0,sda2
1,sdb2
2,sdd2
3,sdc2

However the array is detected like this by uuid alone (I re-ordered
the entries so they're easier to read):

mdadm: /dev/sdb2 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc2 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sda2 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd2 is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdb2 to /dev/md0 as 0
mdadm: added /dev/sdc2 to /dev/md0 as 1
mdadm: added /dev/sda2 to /dev/md0 as 2
mdadm: added /dev/sdd2 to /dev/md0 as 3
mdadm: /dev/md0 has been started with 4 drives.

This corresponds to:

0,sdb2
1,sdc2
2,sda2
3,sdd2

Note how the drives are out of order?  Thus the reason for  only using
UUID and NOT explicitly specifying an ordering?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID MIA. Again. (Kinda.)

[Neil Brown]

On Fri, 5 Mar 2010 10:24:55 -0500 (EST)
"Ken D'Ambrosio" <ken@jots.org> wrote:

> On Thu, March 4, 2010 8:30 pm, Neil Brown wrote:
> > Try this:
> >
> >
> > cat /etc/fstab cat /etc/mdadm.conf mdadm -Esvv mdadm -Asv blkid -p /dev/md*
> >
> > And include all the output.
> 
> See attached.  Note that fstab is essentially irrelevant; I'd mounted by
> hand, and was going to add the entry after I got the server back in the
> server room.

So fdisk says there is no partition table on /dev/md0, but
but blkid said there is an XFS filesystem there.
So maybe you never partitioned /dev/md0?  
What happens if you

   mount /dev/md0 /somewhere

??

NeilBrown

Re: RAID MIA. Again. (Kinda.)

[Neil Brown]

On Fri, 5 Mar 2010 12:05:32 -0800
Michael Evans <mjevans1983@gmail.com> wrote:


> Here's a stab at it. I've never tried to do it like this and haven't
> glanced at the code so I can't be sure, but this seems to be the most
> likely suspect:
> 
> #ARRAY /dev/md0 level=raid5 num-devices=4 devices=sda2,sdb2,sdd2,sdc2
> 
> Your commented out version seems to specify this mapping
> 0,sda2
> 1,sdb2
> 2,sdd2
> 3,sdc2
> 

No it doesn't.  The 'devices=' lists is an unordered list (as set?).  It just
says "only try to include these devices in the array".  They still have to
have valid metadata, and it is the content of the metadata the defines the
role of the device in the array.

NeilBrown

Re: RAID MIA. Again. (Kinda.)

[Michael Evans]

On Fri, Mar 5, 2010 at 12:38 PM, Neil Brown <neilb@suse.de> wrote:
> On Fri, 5 Mar 2010 12:05:32 -0800
> Michael Evans <mjevans1983@gmail.com> wrote:
>
>
>> Here's a stab at it. I've never tried to do it like this and haven't
>> glanced at the code so I can't be sure, but this seems to be the most
>> likely suspect:
>>
>> #ARRAY /dev/md0 level=raid5 num-devices=4 devices=sda2,sdb2,sdd2,sdc2
>>
>> Your commented out version seems to specify this mapping
>> 0,sda2
>> 1,sdb2
>> 2,sdd2
>> 3,sdc2
>>
>
> No it doesn't.  The 'devices=' lists is an unordered list (as set?).  It just
> says "only try to include these devices in the array".  They still have to
> have valid metadata, and it is the content of the metadata the defines the
> role of the device in the array.
>
> NeilBrown

Then if mounting the filesystem used to work but now doesn't I have no
idea; based on the currently provided information I saw no other
suspect.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID MIA. Again. (Kinda.)

[Ken D'Ambrosio]

On Fri, March 5, 2010 3:37 pm, Neil Brown wrote:
> mount /dev/md0 /somewhere
>
> ??

Sorry this took so long; I've been busy dying from Martian Death Flu. 
So... I tried what you suggested.

And it worked.

WTF?!

How can you mount something that doesn't show a partition in fdisk?  I'm
-very-, -very- confused here.  Grateful, yes, but confused.  Is this
something I'm just not "getting?"

Thanks much, regardless, for the pointer -- don't know if I would ever
have tried mounting it without a partition showing.

*confused but happy*

-Ken


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: RAID MIA. Again. (Kinda.)

[John Robinson]

On 09/03/2010 14:43, Ken D'Ambrosio wrote:
> On Fri, March 5, 2010 3:37 pm, Neil Brown wrote:
>> mount /dev/md0 /somewhere
>>
>> ??
> 
> Sorry this took so long; I've been busy dying from Martian Death Flu. 
> So... I tried what you suggested.
> 
> And it worked.
> 
> WTF?!
> 
> How can you mount something that doesn't show a partition in fdisk?  I'm
> -very-, -very- confused here.  Grateful, yes, but confused.  Is this
> something I'm just not "getting?"

Yes, I think so. You can potentially make and mount a filesystem on any 
block device. That could be a whole disc or a partition, or it could be 
something else. There are various tools for making new block devices 
over the top of others, for example you can use LVM, md or dm. Then 
there are yet more tools for making block devices out of other stuff, 
like loopback and nbd. All these new block devices may in turn be 
partitioned, but they don't have to be. In fact partitioning is just a 
simple way of splitting a block device up into other smaller block devices.

Cheers,

John.

Re: RAID MIA. Again. (Kinda.)

[Neil Brown]

On Tue, 9 Mar 2010 09:43:48 -0500 (EST)
"Ken D'Ambrosio" <ken@jots.org> wrote:

> On Fri, March 5, 2010 3:37 pm, Neil Brown wrote:
> > mount /dev/md0 /somewhere
> >
> > ??
> 
> Sorry this took so long; I've been busy dying from Martian Death Flu. 
> So... I tried what you suggested.
> 
> And it worked.
> 
> WTF?!
> 
> How can you mount something that doesn't show a partition in fdisk?  I'm
> -very-, -very- confused here.  Grateful, yes, but confused.  Is this
> something I'm just not "getting?"

Yes, this something you are just "not getting", but it is very simple so
won't take a moment to explain.

A partition table is simply a description of how to divide a disk drive up
into smaller units.  In stead of 1 big device, you appear to have several
small devices.
You can create a filesystem on any of these things - the big device, or any
of the smaller devices.

You can use fdisk to divide /dev/sda up into /dev/sda1, /dev/sda2, etc, but
you don't have to.  You can "mkfs /dev/sda" if you like.
Similarly, you can devide /dev/md0 up into /dev/md0p1, /dev/md0p2, with
fdisk, but you don't have to.
Commonly, people use /dev/sda1 rather than /dev/sda, and
  /dev/md0 rather than /dev/md0p1.
But this is just common practice, not enforced (that it wasn't long ago
when /dev/md0p1 could not be created, but that isn't important here).

A partition table does add one byte of extra information - a partition type.
This is a hint as to what sort of thing is stored in the partition.  However
that hint is not widely used in Linux.  I think some installers use it to
automagically find 'swap' and maybe 'root', but that is about it.  For many
purposes the partition type can be ignored.

When you created your filesystem, you must have created it on /dev/md0, not
on a partition.

Hope it is all clear now.

NeilBrown


> 
> Thanks much, regardless, for the pointer -- don't know if I would ever
> have tried mounting it without a partition showing.
> 
> *confused but happy*
> 
> -Ken
> 
>