[linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Dale Stephenson]

Distribution: centos-release-7-3.1611.el7.centos.x86_64
Kernel: Linux 3.10.0-514.26.2.el7.x86_64
LVM: 2.02.166(2)-RHEL7 (2016-11-16)

Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size.  The array had 64 k stripes.
Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G.  data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
Virtual volume “thin” was 300 GB.  I also filled it with dd so that it would be fully provisioned before the test.
Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.

Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write.  Single thread, 4k blocksize, 90s run time.

The thin volume should much worse performance in this test:

Thin:
Random read : io=51607MB, bw=587168KB/s, iops=146792, runt= 90001msec
Random write: io=46530MB, bw=529406KB/s, iops=132351, runt= 90001msec
Sequential read : io=176561MB, bw=1961.7MB/s, iops=10462, runt= 90006msec
Sequential write: io=162451MB, bw=1804.1MB/s, iops=9626, runt= 90006msec

Thick:
Random read : io=88350MB, bw=981.68MB/s, iops=251303, runt= 90001msec
Random write: io=77905MB, bw=886372KB/s, iops=221592, runt= 90001msec
Sequential read : io=89095MB, bw=989.96MB/s, iops=253421, runt= 90001msec
Sequential write: io=77742MB, bw=884520KB/s, iops=221130, runt= 90001msec

As you can see, the number of iops for the thin-provisioned volume is dramatically less than for thick volumes.  With the default 64k chunk size this is also true.

Running the same fio test with 512k request size shows similar performance for thick and thin volumes, so the maximum potential throughput of the thin volume seems broadly similar to thin volumes.  But I’d like the maximum number of iops to at least be within shouting distance of thick volumes.  Is there anything that can be done to improve that with either this version of lvm/device-mapper or later versions?  I typically use xfs on top of volumes, and xfs is very fond of 4k requests.

Dale Stephenson

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Zdenek Kabelac]

Dne 13.9.2017 v 17:33 Dale Stephenson napsal(a):
> Distribution: centos-release-7-3.1611.el7.centos.x86_64
> Kernel: Linux 3.10.0-514.26.2.el7.x86_64
> LVM: 2.02.166(2)-RHEL7 (2016-11-16)
> 
> Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size.  The array had 64 k stripes.
> Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G.  data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
> Virtual volume “thin” was 300 GB.  I also filled it with dd so that it would be fully provisioned before the test.
> Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.
> 
> Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write.  Single thread, 4k blocksize, 90s run time.

Hi

Can you please provide output of:

lvs -a -o+stripes,stripesize,seg_pe_ranges

so we can see how is your stripe placed on devices ?

SSD typically do needs ideally write 512K chunks.
(something like  'lvcreate -LXXX -i8 -I512k vgname' )

Wouldn't be 'faster' to just concatenate 8 disks together instead of striping 
- or stripe only across 2 disk - and then you concatenate 4 such striped areas...

64k stripes do not seem to look like ideal match in this case of 3 disk with 
512K blocks

Regards

Zdenek

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Dale Stephenson]

> On Sep 13, 2017, at 4:19 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
> 
> Dne 13.9.2017 v 17:33 Dale Stephenson napsal(a):
>> Distribution: centos-release-7-3.1611.el7.centos.x86_64
>> Kernel: Linux 3.10.0-514.26.2.el7.x86_64
>> LVM: 2.02.166(2)-RHEL7 (2016-11-16)
>> Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size.  The array had 64 k stripes.
>> Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G.  data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
>> Virtual volume “thin” was 300 GB.  I also filled it with dd so that it would be fully provisioned before the test.
>> Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.
>> Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write.  Single thread, 4k blocksize, 90s run time.
> 
> Hi
> 
> Can you please provide output of:
> 
> lvs -a -o+stripes,stripesize,seg_pe_ranges
> 
> so we can see how is your stripe placed on devices ?

Sure, thank you for your help:
# lvs -a -o+stripes,stripesize,seg_pe_ranges
  LV               VG     Attr       LSize   Pool     Origin Data%  Meta%  Move Log Cpy%Sync Convert #Str Stripe PE Ranges               
  [lvol0_pmspare]  volgr0 ewi-------  16.00g                                                            1     0  /dev/md127:867328-871423
  thick            volgr0 -wi-a----- 300.00g                                                            1     0  /dev/md127:790528-867327
  thin             volgr0 Vwi-a-t--- 300.00g thinpool        100.00                                     0     0                          
  thinpool         volgr0 twi-aot---   3.00t                 9.77   0.13                                1     0  thinpool_tdata:0-786431 
  [thinpool_tdata] volgr0 Twi-ao----   3.00t                                                            1     0  /dev/md127:0-786431     
  [thinpool_tmeta] volgr0 ewi-ao----  16.00g                                                            1     0  /dev/sdb4:0-4095        

md127 is an 8-drive RAID 0

As you can see, there’s no lvm striping; I rely on the software RAID underneath for that.  Both thick and thin lvols are on the same PV.
> 
> SSD typically do needs ideally write 512K chunks.

I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect that to have any impact on a single threaded test using 4k request size.  Is there a hidden relationship that I’m unaware of?

> (something like  'lvcreate -LXXX -i8 -I512k vgname’)
> 
Would making lvm stripe on top of an md that already stripes confer any performance benefit in general, or for small (4k) requests in particular?

> Wouldn't be 'faster' to just concatenate 8 disks together instead of striping - or stripe only across 2 disk - and then you concatenate 4 such striped areas…
> 
For sustained throughput I would expect striping of 8 disks to blow away concatenation — however, for small requests I wouldn’t expect any advantage.  On a non-redundant array, I would expect a single threaded test using 4k requests is going to end up reading/writing data from exactly one disk regardless of whether the underlying drives are concatenated or stripes.

> 64k stripes do not seem to look like ideal match in this case of 3 disk with 512K blocks

My mistake, I was sloppy with my terminology.  My RAID 0 had a 64k *chunksize*, so it was a stripe of 64k chunks, not a 64k stripe.  The stripe size was 512K, matching the thinpool chunk size.  My understanding is that having the thin pool chunksize match the full stripe size of the underlying array is the best performing choice (at least for throughput).

What is the best choice for handling 4k request sizes?

Thank you,
Dale Stephenson

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Zdenek Kabelac]

Dne 14.9.2017 v 00:39 Dale Stephenson napsal(a):
> 
>> On Sep 13, 2017, at 4:19 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
>>
>> Dne 13.9.2017 v 17:33 Dale Stephenson napsal(a):
>>> Distribution: centos-release-7-3.1611.el7.centos.x86_64
>>> Kernel: Linux 3.10.0-514.26.2.el7.x86_64
>>> LVM: 2.02.166(2)-RHEL7 (2016-11-16)
>>> Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size.  The array had 64 k stripes.
>>> Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G.  data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
>>> Virtual volume “thin” was 300 GB.  I also filled it with dd so that it would be fully provisioned before the test.
>>> Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.
>>> Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write.  Single thread, 4k blocksize, 90s run time.
>>
>> Hi
>>
>> Can you please provide output of:
>>
>> lvs -a -o+stripes,stripesize,seg_pe_ranges
>>
>> so we can see how is your stripe placed on devices ?
> 
> Sure, thank you for your help:
> # lvs -a -o+stripes,stripesize,seg_pe_ranges
>    LV               VG     Attr       LSize   Pool     Origin Data%  Meta%  Move Log Cpy%Sync Convert #Str Stripe PE Ranges
>    [lvol0_pmspare]  volgr0 ewi-------  16.00g                                                            1     0  /dev/md127:867328-871423
>    thick            volgr0 -wi-a----- 300.00g                                                            1     0  /dev/md127:790528-867327
>    thin             volgr0 Vwi-a-t--- 300.00g thinpool        100.00                                     0     0
>    thinpool         volgr0 twi-aot---   3.00t                 9.77   0.13                                1     0  thinpool_tdata:0-786431
>    [thinpool_tdata] volgr0 Twi-ao----   3.00t                                                            1     0  /dev/md127:0-786431
>    [thinpool_tmeta] volgr0 ewi-ao----  16.00g                                                            1     0  /dev/sdb4:0-4095
> 
> md127 is an 8-drive RAID 0
> 
> As you can see, there’s no lvm striping; I rely on the software RAID underneath for that.  Both thick and thin lvols are on the same PV.
>>
>> SSD typically do needs ideally write 512K chunks.
> 
> I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect that to have any impact on a single threaded test using 4k request size.  Is there a hidden relationship that I’m unaware of?


Yep - it seems the setup in this case is the best fit.

If you can reevaluate different setups you may possibly get much higher 
throughput.

My guess would be - the best targeting layout should be probably striping no 
more then 2-3 disks and use bigger striping block.

And then just 'join' 'smaller' arrays together in lvm2 in 1 big LV.


> 
>> (something like  'lvcreate -LXXX -i8 -I512k vgname’)
>>
> Would making lvm stripe on top of an md that already stripes confer any performance benefit in general, or for small (4k) requests in particular?

Rule #1 - try to avoid 'over-combining' things together.
  - measure performance from 'bottom'  upward in your device stack.
If the underlying devices gives poor speed - you can't make it better by any 
super0smart disk-layout on top of it.


> 
>> Wouldn't be 'faster' to just concatenate 8 disks together instead of striping - or stripe only across 2 disk - and then you concatenate 4 such striped areas…
>>
> For sustained throughput I would expect striping of 8 disks to blow away concatenation — however, for small requests I wouldn’t expect any advantage.  On a non-redundant array, I would expect a single threaded test using 4k requests is going to end up reading/writing data from exactly one disk regardless of whether the underlying drives are concatenated or stripes.
It always depends which kind of load you expect the most.

I suspect spreading 4K blocks across 8 SSD is likely very far away from ideal 
layout.

Any SSD is typically very bad with 4K blocks -  it you want to 'spread' the 
load on mores SSDs  do not use less the 64K stripe chunks per SSD - this gives 
you (8*64)  512K stripe size.

As for thin-pool chunksize -  if you plan to use lots of snapshots - keep the 
value lowest possible - 64K  or 128K thin-pool chunksize.

But I'd still suggest to reevaluate/benchmark setup where you will use much 
lower number of SSD for load spreading - and use bigger strip chunks per each 
device.  This should nicely improve performance in case of 'bigger' writes
and not that much slow things down with  4K loads....


> What is the best choice for handling 4k request sizes?

Possibly NVMe can do a better job here.

Regards

Zdenek

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Zdenek Kabelac]

Dne 14.9.2017 v 11:00 Zdenek Kabelac napsal(a):
> Dne 14.9.2017 v 00:39 Dale Stephenson napsal(a):
>>
>>> On Sep 13, 2017, at 4:19 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
>>>
>>
>> md127 is an 8-drive RAID 0
>>
>> As you can see, there’s no lvm striping; I rely on the software RAID 
>> underneath for that.  Both thick and thin lvols are on the same PV.
>>>
>>> SSD typically do needs ideally write 512K chunks.
>>
>> I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect 
>> that to have any impact on a single threaded test using 4k request size.  Is 
>> there a hidden relationship that I’m unaware of?
> 
> 
> Yep - it seems the setup in this case is the best fit.

Sorry my typo here - is NOT ;)


Zdenek

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Gionatan Danti]

On 14/09/2017 11:37, Zdenek Kabelac wrote:
> Sorry my typo here - is NOT ;)
> 
> 
> Zdenek

Hi Zdenek,
as the only variable is the LVM volume type (fat/thick vs thin), why the 
thin volume is slower than the thick one?

I mean: all other things being equal, what is holding back the thin volume?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Gionatan Danti]

On 14/09/2017 11:37, Zdenek Kabelac wrote:
> Sorry my typo here - is NOT ;)
> 
> 
> Zdenek

Hi Zdenek,
as the only variable is the LVM volume type (fat/thick vs thin), why the 
thin volume is slower than the thick one?

I mean: all other things being equal, what is holding back the thin volume?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Zdenek Kabelac]

Dne 14.9.2017 v 12:57 Gionatan Danti napsal(a):
> On 14/09/2017 11:37, Zdenek Kabelac wrote:
>> Sorry my typo here - is NOT ;)
>>
>>
>> Zdenek
> 
> Hi Zdenek,
> as the only variable is the LVM volume type (fat/thick vs thin), why the thin 
> volume is slower than the thick one?
> 
> I mean: all other things being equal, what is holding back the thin volume?



So few more question:

What is '/dev/sdb4'  ? - is it also some fast SSD ?

([thinpool_tmeta] volgr0 ewi-ao----  16.00g  1     0  /dev/sdb4:0-4095
  - just checking to be sure your metadata device is not placed on rotational 
storage device)...

What is your thin-pool chunk size - is it 64K ?
- if your raise thin-pool chunk size up - is it getting any better ?


Regards

Zdenek

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Dale Stephenson]

> On Sep 14, 2017, at 7:13 AM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
> 
> Dne 14.9.2017 v 12:57 Gionatan Danti napsal(a):
>> On 14/09/2017 11:37, Zdenek Kabelac wrote:
>>> Sorry my typo here - is NOT ;)
>>> 
>>> 
>>> Zdenek
>> Hi Zdenek,
>> as the only variable is the LVM volume type (fat/thick vs thin), why the thin volume is slower than the thick one?
>> I mean: all other things being equal, what is holding back the thin volume?
> 
Gionatan has hit on the heart of my concern.  I ordinarily expect minimal performance hit from remapping, and that’s clearly the case with the thick volume.  But it’s not the case with thin, even though I’ve already fully provisioned it.  Why is thin so much slower, and what can I do to make it faster?

> So few more question:
> 
> What is '/dev/sdb4'  ? - is it also some fast SSD ?
> 
Correct, it’s an SSD identical to the ones used in the array.

> - just checking to be sure your metadata device is not placed on rotational storage device)…
> 
It is not in this case.

> What is your thin-pool chunk size - is it 64K ?

In this case it is 512k, because I believed that to be the optimal chunk size for a 512k raid stripe.

> - if your raise thin-pool chunk size up - is it getting any better ?
> 

I have only tested direct to device at 64k and 512k, 512k performs better.  It is not obvious to me why 512k *should* perform better when all the requests are only 4k, except that a given size of metadata would obviously map to a larger size.  (That is, if no contiguous chunk indicators are used — in this particular case I would expect the thin provisioning to be completely contiguous.)

Thank you for your help.
Dale Stephenson

Re: [linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Dale Stephenson]

> On Sep 14, 2017, at 5:00 AM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
> 
> Dne 14.9.2017 v 00:39 Dale Stephenson napsal(a):
>> I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect that to have any impact on a single threaded test using 4k request size.  Is there a hidden relationship that I’m unaware of?
> 
> If you can reevaluate different setups you may possibly get much higher throughput.
> 
To be clear, while I would be interested in hearing about differing setups that would improve the 4k iops performance overall; I am most interested in getting the thin performance comparable to the thick performance.  I did not expect such a large differential between the two types.

> My guess would be - the best targeting layout should be probably striping no more then 2-3 disks and use bigger striping block.
> 
> And then just 'join' 'smaller' arrays together in lvm2 in 1 big LV.
> 
Is this advice given because the drives involved are SSD?  With rotational drives I would never expect concatenated small RAID 0 arrays to provide superior throughput to a single RAID 0.  The best chunk size for sustained throughput depends on the typical transaction, theoretical best case would have typical request size equal to a full stripe.

I would not expect the raid arrangement (at least for RAID 0 or concatenated drives) to have a significant impact on 4k request performance, however.  Too small to take advantage of striping.  Is there a reason why the arrangement would make a difference for this size?
> 
>>> (something like  'lvcreate -LXXX -i8 -I512k vgname’)
>>> 
>> Would making lvm stripe on top of an md that already stripes confer any performance benefit in general, or for small (4k) requests in particular?
> 
> Rule #1 - try to avoid 'over-combining' things together.
> - measure performance from 'bottom'  upward in your device stack.
> If the underlying devices gives poor speed - you can't make it better by any super0smart disk-layout on top of it.
> 
Fair enough.  I have done exactly that — the performance of the underlying md RAID and thick volumes is virtually identical, as I expected.  The thin performance is much, much lower on the same RAID.  So it’s not the underlying layer that is causing the poor speed, it’s the thin volume.

> 
>>> Wouldn't be 'faster' to just concatenate 8 disks together instead of striping - or stripe only across 2 disk - and then you concatenate 4 such striped areas…
>>> 
>> For sustained throughput I would expect striping of 8 disks to blow away concatenation — however, for small requests I wouldn’t expect any advantage. On a non-redundant array, I would expect a single threaded test using 4k requests is going to end up reading/writing data from exactly one disk regardless of whether the underlying drives are concatenated or stripes.
> It always depends which kind of load you expect the most.
> 
> I suspect spreading 4K blocks across 8 SSD is likely very far away from ideal layout.

Very true.  Trying to read/write a single 4k request across 8 drives, whether they are SSD or not, would require a 512-byte chunksize.  Not only is that not possible with md, the overhead associated with generating 8 i/o requests would more than eat up any tiny gain from parallel i/o.  (For rotational drives it’d be even worse, since you’d ensure the response time is governed by the slowest drive).

Which is why for a test like this, which is just using 4k requests in a single thread, I don’t really expect the arrangement of the RAID to have any significant effect.  No matter what kind of drive you are using, and however your drives are arranged, I’d expect the 4k requests only to read or write to a single disk.  So why is it so much slower on thin volumes than thick?

> Any SSD is typically very bad with 4K blocks -  it you want to 'spread' the load on mores SSDs  do not use less the 64K stripe chunks per SSD - this gives you (8*64) 512K stripe size.
> 
I am not using 4k “blocks” anywhere in the remapping layers.  I am using a 4k *request* size.  I want the highest possible number of iops for this request size; I know the throughput will stink compared to larger request sizes, but that’s to be expected.

> As for thin-pool chunksize -  if you plan to use lots of snapshots - keep the value lowest possible - 64K  or 128K thin-pool chunksize.
> 
Snapshots are the main attraction of thin provisioning to me, so 64k would be my preferred thin-pool chunksize for that reason.  However, neither a 64k thin-pool chunksize nor a 512k thin-pool chunksize eliminates the large performance difference between thin and thick volumes for 4k requests.

> But I'd still suggest to reevaluate/benchmark setup where you will use much lower number of SSD for load spreading - and use bigger strip chunks per each device. This should nicely improve performance in case of 'bigger' writes
> and not that much slow things down with  4K loads….
> 
Is the transactional latency so high compared to the data transfer itself that 3-drive SSD RAID 0 should outperform 8-drive SSD RAID 0 for a constant stripe size?  Sounds like an interesting test to run, but I wouldn’t expect it to affect 4k requests.

> 
>> What is the best choice for handling 4k request sizes?
> 
> Possibly NVMe can do a better job here.


What can be done to make *thin provisioning* handle 4k request sizes better?  There may be hardware that performs better for 4k requests than the ones I’m using, but I’m not bothered by the lvm thick volume performance on it.  What can I do to make the lvm thin volume perform as close to thick as possible?

Thank you,
Dale Stephenson