Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot

From: Zdenek Kabelac <zdenek.kabelac@gmail.com>
To: Mitta Sai Chaithanya <mittas@microsoft.com>,
	LVM2 development <lvm-devel@redhat.com>,
	Pawan Sharma <sharmapawan@microsoft.com>,
	"linux-lvm@redhat.com" <linux-lvm@redhat.com>
Cc: Kapil Upadhayay <kupadhayay@microsoft.com>
Subject: Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
Date: Mon, 17 Oct 2022 15:10:46 +0200	[thread overview]
Message-ID: <0e032336-8658-5686-3dfb-0e50cdcb7d91@gmail.com> (raw)
In-Reply-To: <SI2P153MB0444EE48C7C6E92FB448262ED7249@SI2P153MB0444.APCP153.PROD.OUTLOOK.COM>

Dne 14. 10. 22 v 21:31 Mitta Sai Chaithanya napsal(a):
> Hi Zdenek Kabelac,
>            Thanks for your quick reply and suggestions.
> 
> We conducted couple of tests on Ubuntu 22.04 and observed similar performance 
> behavior post thin snapshot deletion without writing any data anywhere.
> 
> *Commands used to create Thin LVM volume*:
> - lvcreate  -L 480G --poolmetadataspare n --poolmetadatasize 16G 
> --chunksize=64K --thinpool  ThinDataLV ThinVolGrp
> - lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp

Hi

So now it's clear you are talking about thin snapshots - this is a very 
different story going on here (as we normally use term "COW" volumes for thick 
old snapshots)

I'll consult more with thinp author - however it does look to me you are using 
same device to store  data & metadata.

This is always a highly sub-optimal solution - the metadata device is likely 
best to be stored on fast (low latency) devices.

So my wild guess - you are possibly using rotational device backend to store 
your  thin-pools metadata volume and then your setups gets very sensitive on 
the metadata fragmentation.

Thin-pool was designed to be used with SSD/NVMe for metadata which is way less 
sensitive on seeking.

So when you 'create' snapshot - metadata gets updated - when you remove thin 
snapshot - metadata gets again a lots of changes (especially when your origin 
volume is already populated) - and fragmentation is inevitable and you are 
getting high penalty of holding metadata device on the same drive as your data 
device.

So while there are some plans to improve some metadata logistic - I'd not 
expect miracles on you particular setup - I'd highly recommend to plug-in some 
  SSD/NVMe storage for storing your thinpool metadata - this is the way to go 
to get better 'benchmarking' numbers here.

For an improvement on your setup - try to seek larger chunk size values where 
your data 'sharing' is still reasonably valuable - this depends on data-type 
usage - but chunk size 256K might be possibly a good compromise (with disabled 
zeroing - if you hunt for the best performance).

Regards

Zdenek

PS: later mails suggest you are using some 'MS Azure' devices?? - so please 
redo your testing with your local hardware/storage - where you have precise 
guarantees of storage drive performance - testing in the Cloud is random by 
design....

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/