[linux-lvm] RFE? Really power of 2? extents, chunks and raid alignment

[linux-lvm] RFE? Really power of 2? extents, chunks and raid alignment

[Linda Walsh]

I was trying to figure out why I didn't get performance out of my RAID,
except when operating on really large files where it's many stripes v. 1 
stripe.

I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested
stripesize of 64k, so a stripe-width of 768k.

Some issues that have been nagging me though are getting my allocations
lined up on 768k boundaries.  xfs is no prob -- tell it 64k and 12 and 
it does it.

But I just realized that lvm doesn't really tell me where it is aligning 
things
and worse, only lets me align both chunksizes for lv's and extents for vg's
in powers of 2.   Um... Not a multiple of 4k? or 64k? 

Am I wrong in thinking this would tend to give me both pv's and lv's that
are very likely NOT to be stripe-width aligned, but, worse, not stripewidth
alignable, at all.

This would mean that a high performance file system aware of RAID stripe 
with
that tries to allocate chunks starting on a 768k boundary are likely to 
just
get it completely wrong?  No?

Or what am I missing?

Thanks...
-linda

Re: [linux-lvm] RFE? Really power of 2? extents, chunks and raid alignment

[Lars Ellenberg]

On Sun, Jul 22, 2012 at 11:22:39AM -0700, Linda Walsh wrote:
> 
> 
> 
> I was trying to figure out why I didn't get performance out of my RAID,
> except when operating on really large files where it's many stripes
> v. 1 stripe.
> 
> I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested
> stripesize of 64k, so a stripe-width of 768k.
> 
> Some issues that have been nagging me though are getting my allocations
> lined up on 768k boundaries.  xfs is no prob -- tell it 64k and 12
> and it does it.
> 
> But I just realized that lvm doesn't really tell me where it is
> aligning things

pvs --units s -o pv_name,vg_name,pe_start

pvcreate: --metadatasize, --dataalignment, --dataalignmentoffset
vgcreate: --physicalextentsize

> and worse, only lets me align both chunksizes for lv's and extents for vg's
> in powers of 2.   Um... Not a multiple of 4k? or 64k?
> 
> Am I wrong in thinking this would tend to give me both pv's and lv's that
> are very likely NOT to be stripe-width aligned, but, worse, not stripewidth
> alignable, at all.
> 
> This would mean that a high performance file system aware of RAID
> stripe with
> that tries to allocate chunks starting on a 768k boundary are likely
> to just
> get it completely wrong?  No?
> 
> Or what am I missing?

If you get the pvcreate --dataalignment right,
Any PE size 256k and larger will be fine,
just make sure you allocate your LV sizes in multiple of 3 PEs.

lvcreate -n lv_bla -l $(( (lv_size_kb / pe_size_kb + 2) / 3 * 3 )) vg_foo


-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

Re: [linux-lvm] RFE? Really power of 2? extents, chunks and raid alignment

[Mike Snitzer]

On Sun, Jul 22 2012 at  2:22pm -0400,
Linda Walsh <lvm@tlinx.org> wrote:

> 
> 
> 
> I was trying to figure out why I didn't get performance out of my RAID,
> except when operating on really large files where it's many stripes
> v. 1 stripe.
> 
> I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested
> stripesize of 64k, so a stripe-width of 768k.
> 
> Some issues that have been nagging me though are getting my allocations
> lined up on 768k boundaries.  xfs is no prob -- tell it 64k and 12
> and it does it.
> 
> But I just realized that lvm doesn't really tell me where it is
> aligning things
> and worse, only lets me align both chunksizes for lv's and extents for vg's
> in powers of 2.   Um... Not a multiple of 4k? or 64k?
> 
> Am I wrong in thinking this would tend to give me both pv's and lv's that
> are very likely NOT to be stripe-width aligned, but, worse, not stripewidth
> alignable, at all.
> 
> This would mean that a high performance file system aware of RAID
> stripe with
> that tries to allocate chunks starting on a 768k boundary are likely
> to just
> get it completely wrong?  No?
> 
> Or what am I missing?

The DM update for the 3.6 merge window adds non power of 2 support in
the kernel (for the stripe and thin-pool targets).

So the lvm2 constraints that require a power of 2 chunksize will be
relaxed very shortly.

Re: [linux-lvm] RFE? Really power of 2? extents, chunks and raid alignment

[Linda Walsh]

[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]





Mike Snitzer wrote:
>
>> I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested
>> stripesize of 64k, so a stripe-width of 768k.
>>     
I must state this correction -- to the above -- I was thinking I had to
use the least common multiplier -- but in reality, I _think_ (subject to
later revision, both -- reality and what I think, ;-)), I should be able to
use the greatest common denominator  and that should work
equally well.

Nevertheless, this:
>
> The DM update for the 3.6 merge window adds non power of 2 support in
> the kernel (for the stripe and thin-pool targets).
>
> So the lvm2 constraints that require a power of 2 chunksize will be
> relaxed very shortly.
>   
is great news, as I'm sure that being able to synchronize stripes to
chunksize AND extent size, would allow better performance for RAID aware
file systems...

I.e. As it is now, I need to have my extents line up, my chunks line up
AND my file system block allocations line up....*ouch*....

If any of those start at 'fixed offsets' from the beginnning of their
space and that fixed size isn't a multiple of the extent size (for VG's)
OR chunksize for LV's, then all of the VG's/LV's will won't line up --
this is only an issue if you have multiple VG's per PV (i.e. the PV's
will use the same physical disks, and the LV's would also use the same
disks, but what you thought was an aligned database mirrored in
1 partition might be completely non-aligned if anything under it is
    misaligned.

I very well do NOT know, the full impact of such -- only that it would
cause unpredictable or variable performance, which as an engineer,
bothers me.




[-- Attachment #2: Type: text/html, Size: 5612 bytes --]

Re: [linux-lvm] RFE? Really power of 2? extents, chunks and raid alignment

[Linda A. Walsh]

Lars Ellenberg wrote:
> pvs --units s -o pv_name,vg_name,pe_start
>
> pvcreate: --metadatasize, --dataalignment, --dataalignmentoffset
> vgcreate: --physicalextentsize
>   
---
    BTW, thanks for this ... I missed it earlier -- and have been sure 
to save away a
copy for future reference...

    now if only disk prices would come back down (or rather (*sardonic 
laugh*), the dollar
value would go back up)......