RAID10 - Ensure mirrors aren't on same JBOD

RAID10 - Ensure mirrors aren't on same JBOD

[Andrew Klaassen]

I've got 4 fibre channel JBODs that I want to use with mdadm RAID 10.  They have 14 drives each, for a total of 56 drives.

I'd like to create a RAID-10 device where no copies end up on the same JBOD.  IOW, if copy 1 goes to the first drive on the first JBOD, copy 2 should go to the first drive on the third JBOD.  If copy 1 goes to the second drive on the second JBOD, copy 2 should go to the second drive on the fourth JBOD.

How should I order the drives on my mdadm --create command line?

Here's more-or-less what I want in pictures, using 3 drives per JBOD for simplicity, with A,B,a,b,1,2 meaning the same thing they do in the RAID FAQ:

jbod-1-1 Aa1 Ba1 Ca1
jbod-1-2 Ab1 Bb1 Cb1
jbod-1-3 Ac1 Bc1 Cc1

jbod-2-1 Da1 Ea1 Fa1
jbod-2-2 Db1 Eb1 Fb1
jbod-2-3 Dc1 Ec1 Fc1

jbod-3-1 Aa2 Ba2 Ca2
jbod-3-2 Ab2 Bb2 Cb2
jbod-3-3 Ac2 Bc2 Cc2

jbod-4-1 Da2 Ea2 Fa2
jbod-4-2 Db2 Eb2 Fb2
jbod-4-3 Dc2 Ec2 Fc2

The data doesn't have to look exactly like that, as long as the principle is maintained: The copies of the data should be on disks in separate JBODs.

Thanks.

Andrew

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Michael Evans]

On Mon, Apr 26, 2010 at 9:15 AM, Andrew Klaassen <clawsoon@yahoo.com> wrote:
> I've got 4 fibre channel JBODs that I want to use with mdadm RAID 10.  They have 14 drives each, for a total of 56 drives.
>
> I'd like to create a RAID-10 device where no copies end up on the same JBOD.  IOW, if copy 1 goes to the first drive on the first JBOD, copy 2 should go to the first drive on the third JBOD.  If copy 1 goes to the second drive on the second JBOD, copy 2 should go to the second drive on the fourth JBOD.
>
> How should I order the drives on my mdadm --create command line?
>
> Here's more-or-less what I want in pictures, using 3 drives per JBOD for simplicity, with A,B,a,b,1,2 meaning the same thing they do in the RAID FAQ:
>
> jbod-1-1 Aa1 Ba1 Ca1
> jbod-1-2 Ab1 Bb1 Cb1
> jbod-1-3 Ac1 Bc1 Cc1
>
> jbod-2-1 Da1 Ea1 Fa1
> jbod-2-2 Db1 Eb1 Fb1
> jbod-2-3 Dc1 Ec1 Fc1
>
> jbod-3-1 Aa2 Ba2 Ca2
> jbod-3-2 Ab2 Bb2 Cb2
> jbod-3-3 Ac2 Bc2 Cc2
>
> jbod-4-1 Da2 Ea2 Fa2
> jbod-4-2 Db2 Eb2 Fb2
> jbod-4-3 Dc2 Ec2 Fc2
>
> The data doesn't have to look exactly like that, as long as the principle is maintained: The copies of the data should be on disks in separate JBODs.
>
> Thanks.
>
> Andrew
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

I don't think I've read the RAID FAQ you speak of, a link would have
been helpful.

However as long as you remember that currently raid10 cannot be grown
(*may have very very recently changed) you can use any of the layouts
you like (I tend to prefer the 'far' layout).

mdadm --create ... --layout=(man mdadm) blockdev0 blockdev1 blockdev2 blockdev3

n2 == Creates almost the behavior you described; each stripe consists
of b0c0 b0c1 b1c0 b1c1 (blockXcopyY)
o2 == Data backup is in the next stripe: that is mirrored and rotated
stripes: b0c0 b1c0 b2c0 b3c0 // b3c1 b0c1 b1c1 b2c1
f2 == The first half is like raid0; the second half is like the o2
above, but over the entire first half.

More info on far from man 4 md
       When 'far' replicas are chosen, the multiple copies of a given
chunk are laid out quite distant from each other.  The  first  copy
of all  data  blocks  will  be striped across the early part of all
drives in RAID0 fashion, and then the next copy of all blocks will be
striped across a later section of all drives, always ensuring that all
copies of any given block are on different drives.

       The 'far' arrangement can give sequential read performance
equal to that of a RAID0 array, but at the cost of  reduced  write
performance.

To elaborate: a 'drive' being a given block device you have specified
via --create or --add.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Andrew Klaassen]

--- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com>  wrote:

> I don't think I've read the RAID FAQ you speak of, a link
> would have been helpful.

Sorry... this one, question #7:

http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=blob_plain;f=debian/FAQ;hb=HEAD

> mdadm --create ... --layout=(man mdadm) blockdev0 blockdev1
> blockdev2 blockdev3
>
> n2 == Creates almost the behavior you described; each
> stripe consists
> of b0c0 b0c1 b1c0 b1c1 (blockXcopyY)
> o2 == Data backup is in the next stripe: that is mirrored
> and rotated
> stripes: b0c0 b1c0 b2c0 b3c0 // b3c1 b0c1 b1c1 b2c1
> f2 == The first half is like raid0; the second half is like
> the o2
> above, but over the entire first half.
>
> More info on far from man 4 md
>        When 'far' replicas are
> chosen, the multiple copies of a given
> chunk are laid out quite distant from each other. 
> The  first  copy
> of all  data  blocks  will  be striped
> across the early part of all
> drives in RAID0 fashion, and then the next copy of all
> blocks will be
> striped across a later section of all drives, always
> ensuring that all
> copies of any given block are on different drives.
>
>        The 'far' arrangement can
> give sequential read performance
> equal to that of a RAID0 array, but at the cost of 
> reduced  write
> performance.

Yep, I've read all that, and it tells me what happens in the trivial case when 4 drives are specified.

However, it doesn't say what happens with a larger number of drives.  I'm looking for the kind of control I can get with RAID 1+0, where I can specify exactly which drives are mirrors of each other.  Is that possible with RAID10?

Thanks.

Andrew

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Keld Simonsen]

On Mon, Apr 26, 2010 at 11:18:50AM -0700, Michael Evans wrote:
> 
> However as long as you remember that currently raid10 cannot be grown
> (*may have very very recently changed) you can use any of the layouts
> you like (I tend to prefer the 'far' layout).
> 
> mdadm --create ... --layout=(man mdadm) blockdev0 blockdev1 blockdev2 blockdev3
> 
> n2 == Creates almost the behavior you described; each stripe consists
> of b0c0 b0c1 b1c0 b1c1 (blockXcopyY)
> o2 == Data backup is in the next stripe: that is mirrored and rotated
> stripes: b0c0 b1c0 b2c0 b3c0 // b3c1 b0c1 b1c1 b2c1
> f2 == The first half is like raid0; the second half is like the o2
> above, but over the entire first half.

mnjae, the second part of a f2 layout is also laid out in a raid0 fashion.

> More info on far from man 4 md
>        When 'far' replicas are chosen, the multiple copies of a given
> chunk are laid out quite distant from each other.  The  first  copy
> of all  data  blocks  will  be striped across the early part of all
> drives in RAID0 fashion, and then the next copy of all blocks will be
> striped across a later section of all drives, always ensuring that all
> copies of any given block are on different drives.
> 
>        The 'far' arrangement can give sequential read performance
> equal to that of a RAID0 array, but at the cost of  reduced  write
> performance.

If you employ a file system (like ext3), then the elevator algorithm
tends to eliminate the write performance hit.

best regards
keld

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Keld Simonsen]

On Mon, Apr 26, 2010 at 11:57:09AM -0700, Andrew Klaassen wrote:
> --- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com> wrote:
> 
> > > --- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com>
> > wrote:
> > > However, it doesn't say what happens with a larger
> > > number of drives.  I'm looking for the kind of control I
> > > can get with RAID 1+0, where I can specify which drives are
> > > mirrors of each other.  Is that possible with RAID10?
> > 
> > No, not predictably; the only provided guarantee is that
> > the data will
> > /not/ be duplicated on the same block-device (usually
> > drive).  In
> > practice you will quickly be able to determine where a
> > particular
> > version with a given range of input causes data to be
> > stored, but
> > there is no requirement that future versions produce the
> > precise same
> > alignment and offsets.
> 
> Ah.  So if I want this level of control, I have to sacrifice the advantages of RAID 10 and go with RAID 1+0.
> 
> Good to know.  Thanks.


I think you can control the exact layout of copies in raid10 at creation time for
the raid. How exactly it will play I am not sure - maybe Neil can tell us.

Anyway you should be able to use raid10 as one of the components in a raid 1+0 like scenario, eg as
the raid1 part of the raid 1+0.

best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Michael Evans]

On Mon, Apr 26, 2010 at 1:20 PM, Andrew Klaassen <clawsoon@yahoo.com> wrote:
> --- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com> wrote:
>
>> Or write your own program to generate mdadm headers on
>> uninitialized devices.  Or if it's just a one time thing,
>> write the headers, stop the array, make sure it says what
>> you want, and then force a resync before writing any data.
>
> So the headers determine which disks mirror which disks?
>
> Does that mean that if I ordered my disks correctly on the --create commandline, I could depend on all of the data being ordered the way that some of the data is?  I.e. if I figured out which two disks were mirroring each other, I would know that they'd always mirror each other as long as I didn't update the array metadata in any way?
>
> Thanks.
>
> Andrew
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Logically that is a valid assumption.  The meta-data identifies the
devices and describes the device storage layout.  If you pop in any
older metadata devices with a newer kernel and/or version of mdadm the
old layout must still be supported so that the data can be accessed.
Why would anyone want to bother moving already written data without a
reshape command of some kind?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Andrew Klaassen]

--- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com> wrote:

> Or write your own program to generate mdadm headers on
> uninitialized devices.  Or if it's just a one time thing,
> write the headers, stop the array, make sure it says what
> you want, and then force a resync before writing any data.

So the headers determine which disks mirror which disks?

Does that mean that if I ordered my disks correctly on the --create commandline, I could depend on all of the data being ordered the way that some of the data is?  I.e. if I figured out which two disks were mirroring each other, I would know that they'd always mirror each other as long as I didn't update the array metadata in any way?

Thanks.

Andrew





--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Michael Evans]

On Mon, Apr 26, 2010 at 11:57 AM, Andrew Klaassen <clawsoon@yahoo.com> wrote:
> --- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com> wrote:
>
>> > --- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com>
>> wrote:
>> > However, it doesn't say what happens with a larger
>> > number of drives.  I'm looking for the kind of control I
>> > can get with RAID 1+0, where I can specify which drives are
>> > mirrors of each other.  Is that possible with RAID10?
>>
>> No, not predictably; the only provided guarantee is that
>> the data will
>> /not/ be duplicated on the same block-device (usually
>> drive).  In
>> practice you will quickly be able to determine where a
>> particular
>> version with a given range of input causes data to be
>> stored, but
>> there is no requirement that future versions produce the
>> precise same
>> alignment and offsets.
>
> Ah.  So if I want this level of control, I have to sacrifice the advantages of RAID 10 and go with RAID 1+0.
>
> Good to know.  Thanks.
>
> Andrew
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Or write your own program to generate mdadm headers on uninitialized
devices.  Or if it's just a one time thing, write the headers, stop
the array, make sure it says what you want, and then force a resync
before writing any data.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 - Ensure mirrors aren't on same JBOD

[Andrew Klaassen]

--- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com> wrote:

> > --- On Mon, 4/26/10, Michael Evans <mjevans1983@gmail.com>
> wrote:
> > However, it doesn't say what happens with a larger
> > number of drives.  I'm looking for the kind of control I
> > can get with RAID 1+0, where I can specify which drives are
> > mirrors of each other.  Is that possible with RAID10?
> 
> No, not predictably; the only provided guarantee is that
> the data will
> /not/ be duplicated on the same block-device (usually
> drive).  In
> practice you will quickly be able to determine where a
> particular
> version with a given range of input causes data to be
> stored, but
> there is no requirement that future versions produce the
> precise same
> alignment and offsets.

Ah.  So if I want this level of control, I have to sacrifice the advantages of RAID 10 and go with RAID 1+0.

Good to know.  Thanks.

Andrew





--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html