RAID1 assembled broken array

RAID1 assembled broken array

[Matthias Urlichs]

I had a somewhat strange error today.

One of my servers has a RAID1 array. Two partitions at the end of the 
disk; the RAID superblocks are at the end of the partition.

After a hard reboot today, one of the disks managed to not have its 
partition table scanned correctly, most probably because the disk was 
hung and the ("intelligent") controller got confused about it. After the 
initial scan, however, it came up correctly.

This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting 
volume was somewhat unuseable. To say the least.

My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that 
replacing these with a pattern that explicitly only matches partitions, 
not disks, would make the problem go away, and that the lesson from 
today's disaster recovery effort is to always explicitly list the allowed 
partition names, instead of being lazy and using 'DEVICE=partitions'.

-- 

Re: RAID1 assembled broken array

[Majed B.]

Specifying partition names may cause problems if the disk names
changed. Specify the UUID instead.

On Tue, Sep 15, 2009 at 10:22 PM, Matthias Urlichs <matthias@urlichs.de> wrote:
> I had a somewhat strange error today.
>
> One of my servers has a RAID1 array. Two partitions at the end of the
> disk; the RAID superblocks are at the end of the partition.
>
> After a hard reboot today, one of the disks managed to not have its
> partition table scanned correctly, most probably because the disk was
> hung and the ("intelligent") controller got confused about it. After the
> initial scan, however, it came up correctly.
>
> This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
> and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting
> volume was somewhat unuseable. To say the least.
>
> My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that
> replacing these with a pattern that explicitly only matches partitions,
> not disks, would make the problem go away, and that the lesson from
> today's disaster recovery effort is to always explicitly list the allowed
> partition names, instead of being lazy and using 'DEVICE=partitions'.
>
> --
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID1 assembled broken array

[Matthias Urlichs]

On Tue, 2009-09-15 at 22:24 +0300, Majed B. wrote:
> Specifying partition names may cause problems if the disk names
> changed. Specify the UUID instead.
> 
What I was trying to say is "use a pattern that matches all of your
partitions and none of your actual disks". That should be safe from
changing disk file names.

Besides, specifying UUIDs does not work because it runs into the exact
problem we're trying to solve in the first place. Behold the "blkid"
output from the server in question:

/dev/sdb2: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid" 
/dev/sdb: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid" 

Interestingly, for some reason it doesn't do that with /dev/sda -- most
probably because the disks in question use different head/sector values
and thus have slightly different partition layout.

Re: RAID1 assembled broken array

[Majed B.]

Weird. I haven't seen that before on my disks/partitions.

user@Adam:~$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 2009-09-14 18:05
cf70cb2b-8e1e-44f3-b361-4a27e169eef2 -> ../../sdi1
lrwxrwxrwx 1 root root 10 2009-09-14 18:05
f54698f9-72a0-4226-9ada-34af2699a25c -> ../../sdi5

My array is offline at the moment, so I can't print the disks' info,
but as you can see, each partition has its own UUID, and if each
partition belonged to a different array then it would have a different
UUID.

Sorry for your loss :/

On Tue, Sep 15, 2009 at 10:33 PM, Matthias Urlichs <matthias@urlichs.de> wrote:
> On Tue, 2009-09-15 at 22:24 +0300, Majed B. wrote:
>> Specifying partition names may cause problems if the disk names
>> changed. Specify the UUID instead.
>>
> What I was trying to say is "use a pattern that matches all of your
> partitions and none of your actual disks". That should be safe from
> changing disk file names.
>
> Besides, specifying UUIDs does not work because it runs into the exact
> problem we're trying to solve in the first place. Behold the "blkid"
> output from the server in question:
>
> /dev/sdb2: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
> /dev/sdb: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
>
> Interestingly, for some reason it doesn't do that with /dev/sda -- most
> probably because the disks in question use different head/sector values
> and thus have slightly different partition layout.
>
>



-- 
       Majed B.

Re: RAID1 assembled broken array

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

On Sep 15, 2009, at 3:22 PM, Matthias Urlichs wrote:
> I had a somewhat strange error today.
>
> One of my servers has a RAID1 array. Two partitions at the end of the
> disk; the RAID superblocks are at the end of the partition.
>
> After a hard reboot today, one of the disks managed to not have its
> partition table scanned correctly, most probably because the disk was
> hung and the ("intelligent") controller got confused about it. After  
> the
> initial scan, however, it came up correctly.
>
> This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
> and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting
> volume was somewhat unuseable. To say the least.
>
> My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that
> replacing these with a pattern that explicitly only matches  
> partitions,
> not disks, would make the problem go away, and that the lesson from
> today's disaster recovery effort is to always explicitly list the  
> allowed
> partition names, instead of being lazy and using 'DEVICE=partitions'.


Wrong lesson.  The correct lesson to gather from this is to prefer  
version 1.1 or 1.2 superblocks wherever possible.  Superblocks at the  
beginning of the device disappear when there is no partition table,  
superblocks at the end can be confused for superblocks belonging to  
the whole device when there is no partition table.

--

Doug Ledford <dledford@redhat.com>

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: RAID1 assembled broken array

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2564 bytes --]

On Sep 15, 2009, at 3:43 PM, Majed B. wrote:
> Weird. I haven't seen that before on my disks/partitions.
>
> user@Adam:~$ ls -l /dev/disk/by-uuid/
> total 0
> lrwxrwxrwx 1 root root 10 2009-09-14 18:05
> cf70cb2b-8e1e-44f3-b361-4a27e169eef2 -> ../../sdi1
> lrwxrwxrwx 1 root root 10 2009-09-14 18:05
> f54698f9-72a0-4226-9ada-34af2699a25c -> ../../sdi5
>
> My array is offline at the moment, so I can't print the disks' info,
> but as you can see, each partition has its own UUID, and if each
> partition belonged to a different array then it would have a different
> UUID.

As did Matthias' setup.  However, if you fail to read the partition  
table on a device, *and* the superblock on any given partition is in  
the exact same location as it would be if you put a superblock on the  
entire device, then mdadm can get confused and think the superblock  
belongs to the entire device and not just the partition that the  
superblock is at the end of.  When that happens, it will think the  
data starts at block 0 of the drive, instead of block 0 of the  
partition and the data read will be inconsistent.  That's what  
happened to Matthias.

> Sorry for your loss :/
>
> On Tue, Sep 15, 2009 at 10:33 PM, Matthias Urlichs <matthias@urlichs.de 
> > wrote:
>> On Tue, 2009-09-15 at 22:24 +0300, Majed B. wrote:
>>> Specifying partition names may cause problems if the disk names
>>> changed. Specify the UUID instead.
>>>
>> What I was trying to say is "use a pattern that matches all of your
>> partitions and none of your actual disks". That should be safe from
>> changing disk file names.
>>
>> Besides, specifying UUIDs does not work because it runs into the  
>> exact
>> problem we're trying to solve in the first place. Behold the "blkid"
>> output from the server in question:
>>
>> /dev/sdb2: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
>> /dev/sdb: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
>>
>> Interestingly, for some reason it doesn't do that with /dev/sda --  
>> most
>> probably because the disks in question use different head/sector  
>> values
>> and thus have slightly different partition layout.
>>
>>
>
>
>
> -- 
>       Majed B.
> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--

Doug Ledford <dledford@redhat.com>

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: RAID1 assembled broken array

[Matthias Urlichs]

On Tue, 2009-09-15 at 16:00 -0400, Doug Ledford wrote:
> Wrong lesson.  The correct lesson to gather from this is to prefer  
> version 1.1 or 1.2 superblocks wherever possible.

Thanks for reminding me. I _knew_ there was something I should have
added to that conclusion. :-/

Re: RAID1 assembled broken array

[NeilBrown]

On Wed, September 16, 2009 6:00 am, Doug Ledford wrote:
> On Sep 15, 2009, at 3:22 PM, Matthias Urlichs wrote:
>> I had a somewhat strange error today.
>>
>> One of my servers has a RAID1 array. Two partitions at the end of the
>> disk; the RAID superblocks are at the end of the partition.
>>
>> After a hard reboot today, one of the disks managed to not have its
>> partition table scanned correctly, most probably because the disk was
>> hung and the ("intelligent") controller got confused about it. After
>> the
>> initial scan, however, it came up correctly.
>>
>> This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
>> and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting
>> volume was somewhat unuseable. To say the least.
>>
>> My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that
>> replacing these with a pattern that explicitly only matches
>> partitions,
>> not disks, would make the problem go away, and that the lesson from
>> today's disaster recovery effort is to always explicitly list the
>> allowed
>> partition names, instead of being lazy and using 'DEVICE=partitions'.
>
>
> Wrong lesson.  The correct lesson to gather from this is to prefer
> version 1.1 or 1.2 superblocks wherever possible.  Superblocks at the
> beginning of the device disappear when there is no partition table,
> superblocks at the end can be confused for superblocks belonging to
> the whole device when there is no partition table.

1.0 also protects from this problem.  The 1.x metadata stores the
offset of the superblock from the start of the device.  That will appear
to be wrong if you find a partition-superblock when reading from a
whole-device, so mdadm will reject the device as not having a valid
superblock.

NeilBrown

Re: RAID1 assembled broken array

[Matthias Urlichs]

On Wed, 2009-09-16 at 07:01 +1000, NeilBrown wrote:
> 1.0 also protects from this problem.

So what's the best way to update a 0.9 superblock to 1.0 format?

mdadm --assemble --metadata=1.0 --update=summaries ... ?

Re: RAID1 assembled broken array

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

On 07:01, NeilBrown wrote:
> > Wrong lesson.  The correct lesson to gather from this is to prefer
> > version 1.1 or 1.2 superblocks wherever possible.  Superblocks at the
> > beginning of the device disappear when there is no partition table,
> > superblocks at the end can be confused for superblocks belonging to
> > the whole device when there is no partition table.
> 
> 1.0 also protects from this problem.  The 1.x metadata stores the
> offset of the superblock from the start of the device.  That will appear
> to be wrong if you find a partition-superblock when reading from a
> whole-device, so mdadm will reject the device as not having a valid
> superblock.

BTW: Why are new arrays still created with 0.90 metadata format by
default?

Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: RAID1 assembled broken array

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

On Sep 16, 2009, at 12:50 AM, Matthias Urlichs wrote:
> On Wed, 2009-09-16 at 07:01 +1000, NeilBrown wrote:
>> 1.0 also protects from this problem.
>
> So what's the best way to update a 0.9 superblock to 1.0 format?
>
> mdadm --assemble --metadata=1.0 --update=summaries ... ?

I don't think you can.  You just have to create a new array with the  
new superblock.

--

Doug Ledford <dledford@redhat.com>

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: RAID1 assembled broken array

[Matthias Urlichs]

On Thu, 2009-09-17 at 12:44 -0400, Doug Ledford wrote:
> >> 1.0 also protects from this problem.
> >
> > So what's the best way to update a 0.9 superblock to 1.0 format?
> >
> > mdadm --assemble --metadata=1.0 --update=summaries ... ?
> 
> I don't think you can.  You just have to create a new array with the  
> new superblock.
> 
Hmm. Looks like you're right.

Consider this to be a feature request, then. If 1.0 is more reasonable
than 0.9, there should be a "safe" upgrade path.

Re: RAID1 assembled broken array

[Majed B.]

0.9 shouldn't be the default if 1.0 or 1.x is better anyway :/

On Thu, Sep 17, 2009 at 8:09 PM, Matthias Urlichs <matthias@urlichs.de> wrote:
> On Thu, 2009-09-17 at 12:44 -0400, Doug Ledford wrote:
>> >> 1.0 also protects from this problem.
>> >
>> > So what's the best way to update a 0.9 superblock to 1.0 format?
>> >
>> > mdadm --assemble --metadata=1.0 --update=summaries ... ?
>>
>> I don't think you can.  You just have to create a new array with the
>> new superblock.
>>
> Hmm. Looks like you're right.
>
> Consider this to be a feature request, then. If 1.0 is more reasonable
> than 0.9, there should be a "safe" upgrade path.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID1 assembled broken array

[Neil Brown]

On Thursday September 17, matthias@urlichs.de wrote:
> On Thu, 2009-09-17 at 12:44 -0400, Doug Ledford wrote:
> > >> 1.0 also protects from this problem.
> > >
> > > So what's the best way to update a 0.9 superblock to 1.0 format?
> > >
> > > mdadm --assemble --metadata=1.0 --update=summaries ... ?
> > 
> > I don't think you can.  You just have to create a new array with the  
> > new superblock.
> > 
> Hmm. Looks like you're right.
> 
> Consider this to be a feature request, then. If 1.0 is more reasonable
> than 0.9, there should be a "safe" upgrade path.

Yes, definitely.  There should be.

Now if only there were more hours in the day.... :-)

NeilBrown

Re: RAID1 assembled broken array

[Neil Brown]

On Thursday September 17, maan@systemlinux.org wrote:
> On 07:01, NeilBrown wrote:
> > > Wrong lesson.  The correct lesson to gather from this is to prefer
> > > version 1.1 or 1.2 superblocks wherever possible.  Superblocks at the
> > > beginning of the device disappear when there is no partition table,
> > > superblocks at the end can be confused for superblocks belonging to
> > > the whole device when there is no partition table.
> > 
> > 1.0 also protects from this problem.  The 1.x metadata stores the
> > offset of the superblock from the start of the device.  That will appear
> > to be wrong if you find a partition-superblock when reading from a
> > whole-device, so mdadm will reject the device as not having a valid
> > superblock.
> 
> BTW: Why are new arrays still created with 0.90 metadata format by
> default?

Because I'm a chicken....

I guess it probably is time ... but to we make the default 1.0, which
is compatible with people's expectations, to 1.1 which is generally a
safer approach (you cannot mount a bare device by mistake).

Of course a distro can import their own default using the CREATE line
in mdadm.conf.


NeilBrown

Re: RAID1 assembled broken array

[Chris Webb]

Neil Brown <neilb@suse.de> writes:

> On Thursday September 17, maan@systemlinux.org wrote:
>
> > BTW: Why are new arrays still created with 0.90 metadata format by
> > default?
> 
> Because I'm a chicken....
> 
> I guess it probably is time ... but to we make the default 1.0, which
> is compatible with people's expectations, to 1.1 which is generally a
> safer approach (you cannot mount a bare device by mistake).

One problem with 1.x metadata formats is that you can't assemble them at
boot time without building an initrd/initramfs containing userspace to put
them together. An md=0,xxx kernel argument only appears to work for 0.90
superblocks. (Would you accept a patch to fix this; it doesn't feel like it
should be particularly hard?)

Another issue with 1.1 and 1.2 metadata is that you can't make a mirror out
of sd*, install a bootloader (e.g. extlinux) on the md, and then boot the
corresponding system. That said, I guess it wouldn't be hard to make a
little 'mbr' that understands the data offset in the md 1.2 container, so if
I'm honest, it's just laziness that keeps me using 0.90 where it works out
of the box without the need to hack any bootloader code or create untidy
partitions just for booting.

Best wishes,

Chris.

Re: RAID1 assembled broken array

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

On 18:12, Neil Brown wrote:
> > BTW: Why are new arrays still created with 0.90 metadata format by
> > default?
> 
> Because I'm a chicken....
> 
> I guess it probably is time ... but to we make the default 1.0, which
> is compatible with people's expectations, to 1.1 which is generally a
> safer approach (you cannot mount a bare device by mistake).

People will soon have to use 1.x anyway as drives are getting bigger
than 2T. I'd vote for making 1.1 or 1.2 metadata the default for the
reasons Doug pointed out. Failing to mount the bare device if there
is a partition table is IMHO more important than meeting (broken)
expectations.

Maybe we should depricate kernel-level autodetection at the same time
so that using an initramfs becomes mandatory for setups with / on md.

Best
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: RAID1 assembled broken array

[John Robinson]

On 26/09/2009 10:58, Andre Noll wrote:
> On 18:12, Neil Brown wrote:
>>> BTW: Why are new arrays still created with 0.90 metadata format by
>>> default?
>> Because I'm a chicken....
>>
>> I guess it probably is time ... but to we make the default 1.0, which
>> is compatible with people's expectations, to 1.1 which is generally a
>> safer approach (you cannot mount a bare device by mistake).
> 
> People will soon have to use 1.x anyway as drives are getting bigger
> than 2T. I'd vote for making 1.1 or 1.2 metadata the default for the
> reasons Doug pointed out. Failing to mount the bare device if there
> is a partition table is IMHO more important than meeting (broken)
> expectations.
> 
> Maybe we should depricate kernel-level autodetection at the same time
> so that using an initramfs becomes mandatory for setups with / on md.

It already is deprecated; afaik all the recent distros already use an 
initramfs, even if they still use 0.90 metadata.

I second making this a priority to get into the next round of distro 
updates, for all the reasons given. As soon as metadata version upgrade 
facilities go in to mdadm, 0.90 metadata can also be deprecated and 
perhaps mdadm can issue warnings, or at least the kernel autoassembly 
code can start issuing warnings (if it doesn't already).

Cheers,

John.

Re: RAID1 assembled broken array

[Clinton Lee Taylor]

>>>> BTW: Why are new arrays still created with 0.90 metadata format by
>>>> default?
>>>
>>> Because I'm a chicken....
>>>
>>> I guess it probably is time ... but to we make the default 1.0, which
>>> is compatible with people's expectations, to 1.1 which is generally a
>>> safer approach (you cannot mount a bare device by mistake).
>>
>> People will soon have to use 1.x anyway as drives are getting bigger
>> than 2T. I'd vote for making 1.1 or 1.2 metadata the default for the
>> reasons Doug pointed out. Failing to mount the bare device if there
>> is a partition table is IMHO more important than meeting (broken)
>> expectations.
>>
>> Maybe we should depricate kernel-level autodetection at the same time
>> so that using an initramfs becomes mandatory for setups with / on md.
>
> It already is deprecated; afaik all the recent distros already use an
> initramfs, even if they still use 0.90 metadata.
>
> I second making this a priority to get into the next round of distro
> updates, for all the reasons given. As soon as metadata version upgrade
> facilities go in to mdadm, 0.90 metadata can also be deprecated and perhaps
> mdadm can issue warnings, or at least the kernel autoassembly code can start
> issuing warnings (if it doesn't already).

 All this talk of metadata 1.x, what about all the metadata 0.90 RAID
array's out there?  What is needed for an upgrade or conversion?
Would a conversion program need to understand the under lying
filesystem?

Thanks
Mailed
LeeT

Re: RAID1 assembled broken array

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

On 21:21, Clinton Lee Taylor wrote:
> >> Maybe we should depricate kernel-level autodetection at the same time
> >> so that using an initramfs becomes mandatory for setups with / on md.
> >
> > It already is deprecated; afaik all the recent distros already use an
> > initramfs, even if they still use 0.90 metadata.
> >
> > I second making this a priority to get into the next round of distro
> > updates, for all the reasons given. As soon as metadata version upgrade
> > facilities go in to mdadm, 0.90 metadata can also be deprecated and perhaps
> > mdadm can issue warnings, or at least the kernel autoassembly code can start
> > issuing warnings (if it doesn't already).
> 
>  All this talk of metadata 1.x, what about all the metadata 0.90 RAID
> array's out there?  What is needed for an upgrade or conversion?

We're just talking about making 1.x the default for creating _new_
arrays. The old v0.90 metadata format will still be supported for
years. However, if one could easily upgrade from v0.90 to v1.x
metadata, we could stop supporting the old format a bit earlier.

It would be good enough to teach mdadm to convert the superblocks of
inactive arrays only.

> Would a conversion program need to understand the under lying
> filesystem?

No.

Regards
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]