[linux-lvm] Thin pool performance when allocating lots of blocks

[linux-lvm] Thin pool performance when allocating lots of blocks

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 750 bytes --]

Are thin volumes (which start as snapshots of a blank volume) efficient
for building virtual machine images?  Given the nature of this workload
(writing to lots of new, possibly-small files, then copying data from
them to a huge disk image), I expect that this will cause sharing to be
broken many, many times, and the kernel code that breaks sharing appears
to be rather heavyweight.  Furthermore, since zeroing is enabled, this
might cause substantial write amplification.  Turning zeroing off is not
an option for security reasons.

Is there a way to determine if breaking sharing is the cause of
performance problems?  If it is, are there any better solutions?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Thin pool performance when allocating lots of blocks

[Zdenek Kabelac]

Dne 08. 02. 22 v 20:00 Demi Marie Obenour napsal(a):
> Are thin volumes (which start as snapshots of a blank volume) efficient
> for building virtual machine images?  Given the nature of this workload
> (writing to lots of new, possibly-small files, then copying data from
> them to a huge disk image), I expect that this will cause sharing to be
> broken many, many times, and the kernel code that breaks sharing appears
> to be rather heavyweight.  Furthermore, since zeroing is enabled, this
> might cause substantial write amplification.  Turning zeroing off is not
> an option for security reasons.
> 
> Is there a way to determine if breaking sharing is the cause of
> performance problems?  If it is, are there any better solutions?

Hi

Usually the smaller the thin chunks size is the smaller the problem gets.
With current released version of thin-provisioning minimal chunk size is 
64KiB. So you can't use smaller value to further reduce this impact.

Note - even if you do a lot of tiny 4KiB writes  - only the 'first' such write 
into 64K area breaks sharing all following writes to same location no longer 
have this penalty (also zeroing with 64K is less impactful...)

But it's clear thin-provisioning comes with some price - so if it's not good 
enough from time constrains some other solutions might need to be explored.
(i.e. caching, better hw, splitting  FS into multiple partitions with 
'read-only sections,....)

For analysis of device hot points you could check 'dmstats' tool for DM devices.

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Thin pool performance when allocating lots of blocks

[Demi Marie Obenour]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 1770 bytes --]

On 2/8/22 15:37, Zdenek Kabelac wrote:
> Dne 08. 02. 22 v 20:00 Demi Marie Obenour napsal(a):
>> Are thin volumes (which start as snapshots of a blank volume) efficient
>> for building virtual machine images?  Given the nature of this workload
>> (writing to lots of new, possibly-small files, then copying data from
>> them to a huge disk image), I expect that this will cause sharing to be
>> broken many, many times, and the kernel code that breaks sharing appears
>> to be rather heavyweight.  Furthermore, since zeroing is enabled, this
>> might cause substantial write amplification.  Turning zeroing off is not
>> an option for security reasons.
>>
>> Is there a way to determine if breaking sharing is the cause of
>> performance problems?  If it is, are there any better solutions?
> 
> Hi
> 
> Usually the smaller the thin chunks size is the smaller the problem gets.
> With current released version of thin-provisioning minimal chunk size is 
> 64KiB. So you can't use smaller value to further reduce this impact.
> 
> Note - even if you do a lot of tiny 4KiB writes  - only the 'first' such write 
> into 64K area breaks sharing all following writes to same location no longer 
> have this penalty (also zeroing with 64K is less impactful...)
> 
> But it's clear thin-provisioning comes with some price - so if it's not good 
> enough from time constrains some other solutions might need to be explored.
> (i.e. caching, better hw, splitting  FS into multiple partitions with 
> 'read-only sections,....)

Are the code paths that break sharing as heavyweight as I was worried
about?  Would a hypothetical dm-thin2 that used dm-bio-prison-v2 be
faster?

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #1.1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 4963 bytes --]

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Thin pool performance when allocating lots of blocks

[Zdenek Kabelac]

Dne 08. 02. 22 v 22:02 Demi Marie Obenour napsal(a):
> On 2/8/22 15:37, Zdenek Kabelac wrote:
>> Dne 08. 02. 22 v 20:00 Demi Marie Obenour napsal(a):
>>> Are thin volumes (which start as snapshots of a blank volume) efficient
>>> for building virtual machine images?  Given the nature of this workload
>>> (writing to lots of new, possibly-small files, then copying data from
>>> them to a huge disk image), I expect that this will cause sharing to be
>>> broken many, many times, and the kernel code that breaks sharing appears
>>> to be rather heavyweight.  Furthermore, since zeroing is enabled, this
>>> might cause substantial write amplification.  Turning zeroing off is not
>>> an option for security reasons.
>>>
>>> Is there a way to determine if breaking sharing is the cause of
>>> performance problems?  If it is, are there any better solutions?
>>
>> Hi
>>
>> Usually the smaller the thin chunks size is the smaller the problem gets.
>> With current released version of thin-provisioning minimal chunk size is
>> 64KiB. So you can't use smaller value to further reduce this impact.
>>
>> Note - even if you do a lot of tiny 4KiB writes  - only the 'first' such write
>> into 64K area breaks sharing all following writes to same location no longer
>> have this penalty (also zeroing with 64K is less impactful...)
>>
>> But it's clear thin-provisioning comes with some price - so if it's not good
>> enough from time constrains some other solutions might need to be explored.
>> (i.e. caching, better hw, splitting  FS into multiple partitions with
>> 'read-only sections,....)
> 
> Are the code paths that break sharing as heavyweight as I was worried
> about?  Would a hypothetical dm-thin2 that used dm-bio-prison-v2 be
> faster?
> 

Biggest problem is the size of chunks - the smaller chunk you could use,
the less amplification you get. On the other hand the amount of metadata 
handling is increasing. Then there is a lot about parallelization, locking and 
disk synchronization.

If you are more interested in this topic, dive into kernel code.
Also I'd suggest to make some good benchmarking.

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/