* [linux-lvm] LVM2 : performance drop even after deleting the snapshot
@ 2022-10-12 17:12 Pawan Sharma
2022-10-13 6:53 ` Pawan Sharma
0 siblings, 1 reply; 23+ messages in thread
From: Pawan Sharma @ 2022-10-12 17:12 UTC (permalink / raw)
To: linux-lvm; +Cc: Kapil Upadhayay, Mitta Sai Chaithanya
[-- Attachment #1.1: Type: text/plain, Size: 1338 bytes --]
Hi Everyone,
We are evaluating lvm2 snapshots and doing performance testing on it. This is what we are doing :
1. dump some data to lvm2 volume (using fio)
2. take the snapshot
3. delete the snapshot (no IOs anywhere after creating the snapshot)
4. run the fio on lvm2 volume
Here as you can see, we are just creating the snapshot and immediately deleting it. There are no IOs to the main volume or anywhere. When we run the fio after this (step 4) and we see around 50% drop in performance with reference to the number we get in step 1.
It is expected to see a performance drop if there is a snapshot because of the COW. But here we deleted the snapshot, and it is not referring to any data also. We should not see any performance drop here.
Could someone please help me understand this behavior. Why are we seeing the performance drop in this case? It seems like we deleted the snapshot but still it is not deleted, and we are paying the COW penalty.
System Info:
OS : ubuntu 18.04
Kernel : 5.4.0
# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.41.0
We also tried on latest ubuntu with newer version of LVM. We got the same behavior.
Any help/pointers would be appreciated. Thanks in advance.
Regards,
Pawan
[-- Attachment #1.2: Type: text/html, Size: 5746 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
2022-10-12 17:12 [linux-lvm] LVM2 : performance drop even after deleting the snapshot Pawan Sharma
@ 2022-10-13 6:53 ` Pawan Sharma
0 siblings, 0 replies; 23+ messages in thread
From: Pawan Sharma @ 2022-10-13 6:53 UTC (permalink / raw)
To: linux-lvm, lvm-devel; +Cc: Kapil Upadhayay, Mitta Sai Chaithanya
[-- Attachment #1.1: Type: text/plain, Size: 1711 bytes --]
adding this to lvm-devel mailing list also.
Regards,
Pawan
________________________________
From: Pawan Sharma
Sent: Wednesday, October 12, 2022 10:42 PM
To: linux-lvm@redhat.com <linux-lvm@redhat.com>
Cc: Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay <kupadhayay@microsoft.com>
Subject: LVM2 : performance drop even after deleting the snapshot
Hi Everyone,
We are evaluating lvm2 snapshots and doing performance testing on it. This is what we are doing :
1. dump some data to lvm2 volume (using fio)
2. take the snapshot
3. delete the snapshot (no IOs anywhere after creating the snapshot)
4. run the fio on lvm2 volume
Here as you can see, we are just creating the snapshot and immediately deleting it. There are no IOs to the main volume or anywhere. When we run the fio after this (step 4) and we see around 50% drop in performance with reference to the number we get in step 1.
It is expected to see a performance drop if there is a snapshot because of the COW. But here we deleted the snapshot, and it is not referring to any data also. We should not see any performance drop here.
Could someone please help me understand this behavior. Why are we seeing the performance drop in this case? It seems like we deleted the snapshot but still it is not deleted, and we are paying the COW penalty.
System Info:
OS : ubuntu 18.04
Kernel : 5.4.0
# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.41.0
We also tried on latest ubuntu with newer version of LVM. We got the same behavior.
Any help/pointers would be appreciated. Thanks in advance.
Regards,
Pawan
[-- Attachment #1.2: Type: text/html, Size: 5620 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* LVM2 : performance drop even after deleting the snapshot
@ 2022-10-13 6:53 ` Pawan Sharma
0 siblings, 0 replies; 23+ messages in thread
From: Pawan Sharma @ 2022-10-13 6:53 UTC (permalink / raw)
To: lvm-devel
adding this to lvm-devel mailing list also.
Regards,
Pawan
________________________________
From: Pawan Sharma
Sent: Wednesday, October 12, 2022 10:42 PM
To: linux-lvm@redhat.com <linux-lvm@redhat.com>
Cc: Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay <kupadhayay@microsoft.com>
Subject: LVM2 : performance drop even after deleting the snapshot
Hi Everyone,
We are evaluating lvm2 snapshots and doing performance testing on it. This is what we are doing :
1. dump some data to lvm2 volume (using fio)
2. take the snapshot
3. delete the snapshot (no IOs anywhere after creating the snapshot)
4. run the fio on lvm2 volume
Here as you can see, we are just creating the snapshot and immediately deleting it. There are no IOs to the main volume or anywhere. When we run the fio after this (step 4) and we see around 50% drop in performance with reference to the number we get in step 1.
It is expected to see a performance drop if there is a snapshot because of the COW. But here we deleted the snapshot, and it is not referring to any data also. We should not see any performance drop here.
Could someone please help me understand this behavior. Why are we seeing the performance drop in this case? It seems like we deleted the snapshot but still it is not deleted, and we are paying the COW penalty.
System Info:
OS : ubuntu 18.04
Kernel : 5.4.0
# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.41.0
We also tried on latest ubuntu with newer version of LVM. We got the same behavior.
Any help/pointers would be appreciated. Thanks in advance.
Regards,
Pawan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20221013/3a0c62f5/attachment.htm>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
2022-10-13 6:53 ` Pawan Sharma
@ 2022-10-13 10:50 ` Zdenek Kabelac
-1 siblings, 0 replies; 23+ messages in thread
From: Zdenek Kabelac @ 2022-10-13 10:50 UTC (permalink / raw)
To: LVM2 development, Pawan Sharma, linux-lvm
Cc: Kapil Upadhayay, Mitta Sai Chaithanya
Dne 13. 10. 22 v 8:53 Pawan Sharma napsal(a):
> adding this to lvm-devel mailing list also.
>
> Regards,
> Pawan
> ------------------------------------------------------------------------------
> *From:* Pawan Sharma
> *Sent:* Wednesday, October 12, 2022 10:42 PM
> *To:* linux-lvm@redhat.com <linux-lvm@redhat.com>
> *Cc:* Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay
> <kupadhayay@microsoft.com>
> *Subject:* LVM2 : performance drop even after deleting the snapshot
> Hi Everyone,
>
>
> We are evaluating lvm2 snapshots and doing performance testing on it. This is
> what we are doing :
>
> 1. dump some data to lvm2 volume (using fio)
> 2. take the snapshot
> 3. delete the snapshot (no IOs anywhere after creating the snapshot)
> 4. run the fio on lvm2 volume
>
> Here as you can see, we are just creating the snapshot and immediately
> deleting it. There are no IOs to the main volume or anywhere. When we run the
> fio after this (step 4) and we see around 50% drop in performance with
> reference to the number we get in step 1.
>
> It is expected to see a performance drop if there is a snapshot because of the
> COW. But here we deleted the snapshot, and it is not referring to any data
> also. We should not see any performance drop here.
>
> Could someone please help me understand this behavior. Why are we seeing the
> performance drop in this case? It seems like we deleted the snapshot but still
> it is not deleted, and we are paying the COW penalty.
>
> System Info:
>
> OS : ubuntu 18.04
> Kernel : 5.4.0
>
> # lvm version
> LVM version:2.02.176(2) (2017-11-03)
> Library version: 1.02.145 (2017-11-03)
> Driver version:4.41.0
>
> We also tried on latest ubuntu with newer version of LVM. We got the same
> behavior.
>
>
Hi
Debugging 5 year old software is likely not going to get lot of attention
from upstream.
So please:
a) reproduce the issue with some recent kernel & lvm2
b) take 'dmsetup table && dmsetup status' before you run every 'fio' test
and present here your result in some form - otherwise we can hardly see what
is the problem.
What should be expected - if you use old/thick snapshots - when you 'drop'
snapshot - you have your original intact LV - so results should mostly match
results before you take the snapshot - but you clearly have to take into
account if you use some 'SSD/NVMe' discarding and other things - so always run
series of tests and average your results.
If you use thin snapshot - that you can get various results depending on your
settings of thin chunks, discard usage.
Also maybe try your benchmark with different filesystems...
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* LVM2 : performance drop even after deleting the snapshot
@ 2022-10-13 10:50 ` Zdenek Kabelac
0 siblings, 0 replies; 23+ messages in thread
From: Zdenek Kabelac @ 2022-10-13 10:50 UTC (permalink / raw)
To: lvm-devel
Dne 13. 10. 22 v 8:53 Pawan Sharma napsal(a):
> adding this to lvm-devel mailing list also.
>
> Regards,
> Pawan
> ------------------------------------------------------------------------------
> *From:* Pawan Sharma
> *Sent:* Wednesday, October 12, 2022 10:42 PM
> *To:* linux-lvm at redhat.com <linux-lvm@redhat.com>
> *Cc:* Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay
> <kupadhayay@microsoft.com>
> *Subject:* LVM2 : performance drop even after deleting the snapshot
> Hi Everyone,
>
>
> We are evaluating lvm2 snapshots and doing performance testing on it. This is
> what we are doing :
>
> 1. dump some data to lvm2 volume (using fio)
> 2. take the snapshot
> 3. delete the snapshot (no IOs anywhere after creating the snapshot)
> 4. run the fio on lvm2 volume
>
> Here as you can see, we are just creating the snapshot and immediately
> deleting it. There are no IOs to the main volume or anywhere. When we run the
> fio after this (step 4) and we see around 50% drop in performance with
> reference to the number we get in step 1.
>
> It is expected to see a performance drop if there is a snapshot because of the
> COW. But here we deleted the snapshot, and it is not referring to any data
> also. We should not see any performance drop here.
>
> Could?someone please help me understand this behavior. Why are we seeing the
> performance drop in this case? It seems like we deleted the snapshot but still
> it is not deleted, and we are paying the COW penalty.
>
> System Info:
>
> OS : ubuntu 18.04
> Kernel : 5.4.0
>
> # lvm version
> LVM version:2.02.176(2) (2017-11-03)
> Library version: 1.02.145 (2017-11-03)
> Driver version:4.41.0
>
> We also tried on latest ubuntu with newer version of LVM. We got the same
> behavior.
>
>
Hi
Debugging 5 year old software is likely not going to get lot of attention
from upstream.
So please:
a) reproduce the issue with some recent kernel & lvm2
b) take 'dmsetup table && dmsetup status' before you run every 'fio' test
and present here your result in some form - otherwise we can hardly see what
is the problem.
What should be expected - if you use old/thick snapshots - when you 'drop'
snapshot - you have your original intact LV - so results should mostly match
results before you take the snapshot - but you clearly have to take into
account if you use some 'SSD/NVMe' discarding and other things - so always run
series of tests and average your results.
If you use thin snapshot - that you can get various results depending on your
settings of thin chunks, discard usage.
Also maybe try your benchmark with different filesystems...
Regards
Zdenek
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
2022-10-13 10:50 ` Zdenek Kabelac
@ 2022-10-14 19:31 ` Mitta Sai Chaithanya
-1 siblings, 0 replies; 23+ messages in thread
From: Mitta Sai Chaithanya @ 2022-10-14 19:31 UTC (permalink / raw)
To: Zdenek Kabelac, LVM2 development, Pawan Sharma, linux-lvm; +Cc: Kapil Upadhayay
[-- Attachment #1.1: Type: text/plain, Size: 6181 bytes --]
Hi Zdenek Kabelac,
Thanks for your quick reply and suggestions.
We conducted couple of tests on Ubuntu 22.04 and observed similar performance behavior post thin snapshot deletion without writing any data anywhere.
Commands used to create Thin LVM volume:
- lvcreate -L 480G --poolmetadataspare n --poolmetadatasize 16G --chunksize=64K --thinpool ThinDataLV ThinVolGrp
- lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp
ThinLV display:
--- Logical volume ---
LV Path /dev/ThinVolGrp/ext4.ThinLV
LV Name ext4.ThinLV
VG Name ThinVolGrp
LV UUID sRcj9L-Ili4-dR3I-MeJI-3KLv-xPUP-VMd1LC
LV Write Access read/write
LV Creation host, time kapil-upstream, 2022-10-14 18:05:22 +0000
LV Pool name ThinDataLV
LV Status available
# open 1
LV Size 100.00 GiB
Mapped size 21.51%
Current LE 25600
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
dmsetup table output after creation of thin lvm volume:
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status after creation of thin lvm volume:
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4878/4145152 8325/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 1065600 209715199
FIO Command to dump some data:
fio --filename=/dev/ThinVolGrp/ext4.ThinLV --name=<name> --time_based --group_reporting --ioengine=libaio --direct=1 --size=20G --runtime=120 --rw=randrw --rwmixread=50 --bs=16K --iodepth=8 --numjobs=3 --randrepeat=0 --randseed=$(date +%s)
For detailed information of fio and dmsetup results click here<https://gist.github.com/mittachaitu/17006a1e3f08ad0c17add7ea007e63b6#file-lvm_snapshot_performance>.
Note: All the tests are conducted using thin lvm volume and thin snapshot
Environment details:
Kernel Version: 5.15.0-1021-azure
LVM Version: LVM version: 2.03.11(2) (2021-01-08)
Library version: 1.02.175 (2021-01-08)
Driver version: 4.45.0
FIO Version: 3.28
Please let us know if you required more information and tests that needs to be run.
Thanks & Regards
mittachaitu (Sai)
From: Zdenek Kabelac<mailto:zdenek.kabelac@gmail.com>
Sent: Thursday, October 13, 2022 4:20 PM
To: LVM2 development<mailto:lvm-devel@redhat.com>; Pawan Sharma<mailto:sharmapawan@microsoft.com>; linux-lvm@redhat.com<mailto:linux-lvm@redhat.com>
Cc: Kapil Upadhayay<mailto:kupadhayay@microsoft.com>; Mitta Sai Chaithanya<mailto:mittas@microsoft.com>
Subject: [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
[Some people who received this message don't often get email from zdenek.kabelac@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Dne 13. 10. 22 v 8:53 Pawan Sharma napsal(a):
> adding this to lvm-devel mailing list also.
>
> Regards,
> Pawan
> ------------------------------------------------------------------------------
> *From:* Pawan Sharma
> *Sent:* Wednesday, October 12, 2022 10:42 PM
> *To:* linux-lvm@redhat.com <linux-lvm@redhat.com>
> *Cc:* Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay
> <kupadhayay@microsoft.com>
> *Subject:* LVM2 : performance drop even after deleting the snapshot
> Hi Everyone,
>
>
> We are evaluating lvm2 snapshots and doing performance testing on it. This is
> what we are doing :
>
> 1. dump some data to lvm2 volume (using fio)
> 2. take the snapshot
> 3. delete the snapshot (no IOs anywhere after creating the snapshot)
> 4. run the fio on lvm2 volume
>
> Here as you can see, we are just creating the snapshot and immediately
> deleting it. There are no IOs to the main volume or anywhere. When we run the
> fio after this (step 4) and we see around 50% drop in performance with
> reference to the number we get in step 1.
>
> It is expected to see a performance drop if there is a snapshot because of the
> COW. But here we deleted the snapshot, and it is not referring to any data
> also. We should not see any performance drop here.
>
> Could someone please help me understand this behavior. Why are we seeing the
> performance drop in this case? It seems like we deleted the snapshot but still
> it is not deleted, and we are paying the COW penalty.
>
> System Info:
>
> OS : ubuntu 18.04
> Kernel : 5.4.0
>
> # lvm version
> LVM version:2.02.176(2) (2017-11-03)
> Library version: 1.02.145 (2017-11-03)
> Driver version:4.41.0
>
> We also tried on latest ubuntu with newer version of LVM. We got the same
> behavior.
>
>
Hi
Debugging 5 year old software is likely not going to get lot of attention
from upstream.
So please:
a) reproduce the issue with some recent kernel & lvm2
b) take 'dmsetup table && dmsetup status' before you run every 'fio' test
and present here your result in some form - otherwise we can hardly see what
is the problem.
What should be expected - if you use old/thick snapshots - when you 'drop'
snapshot - you have your original intact LV - so results should mostly match
results before you take the snapshot - but you clearly have to take into
account if you use some 'SSD/NVMe' discarding and other things - so always run
series of tests and average your results.
If you use thin snapshot - that you can get various results depending on your
settings of thin chunks, discard usage.
Also maybe try your benchmark with different filesystems...
Regards
Zdenek
[-- Attachment #1.2: Type: text/html, Size: 15763 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
@ 2022-10-14 19:31 ` Mitta Sai Chaithanya
0 siblings, 0 replies; 23+ messages in thread
From: Mitta Sai Chaithanya @ 2022-10-14 19:31 UTC (permalink / raw)
To: lvm-devel
Hi Zdenek Kabelac,
Thanks for your quick reply and suggestions.
We conducted couple of tests on Ubuntu 22.04 and observed similar performance behavior post thin snapshot deletion without writing any data anywhere.
Commands used to create Thin LVM volume:
- lvcreate -L 480G --poolmetadataspare n --poolmetadatasize 16G --chunksize=64K --thinpool ThinDataLV ThinVolGrp
- lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp
ThinLV display:
--- Logical volume ---
LV Path /dev/ThinVolGrp/ext4.ThinLV
LV Name ext4.ThinLV
VG Name ThinVolGrp
LV UUID sRcj9L-Ili4-dR3I-MeJI-3KLv-xPUP-VMd1LC
LV Write Access read/write
LV Creation host, time kapil-upstream, 2022-10-14 18:05:22 +0000
LV Pool name ThinDataLV
LV Status available
# open 1
LV Size 100.00 GiB
Mapped size 21.51%
Current LE 25600
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
dmsetup table output after creation of thin lvm volume:
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status after creation of thin lvm volume:
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4878/4145152 8325/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 1065600 209715199
FIO Command to dump some data:
fio --filename=/dev/ThinVolGrp/ext4.ThinLV --name=<name> --time_based --group_reporting --ioengine=libaio --direct=1 --size=20G --runtime=120 --rw=randrw --rwmixread=50 --bs=16K --iodepth=8 --numjobs=3 --randrepeat=0 --randseed=$(date +%s)
For detailed information of fio and dmsetup results click here<https://gist.github.com/mittachaitu/17006a1e3f08ad0c17add7ea007e63b6#file-lvm_snapshot_performance>.
Note: All the tests are conducted using thin lvm volume and thin snapshot
Environment details:
Kernel Version: 5.15.0-1021-azure
LVM Version: LVM version: 2.03.11(2) (2021-01-08)
Library version: 1.02.175 (2021-01-08)
Driver version: 4.45.0
FIO Version: 3.28
Please let us know if you required more information and tests that needs to be run.
Thanks & Regards
mittachaitu (Sai)
From: Zdenek Kabelac<mailto:zdenek.kabelac@gmail.com>
Sent: Thursday, October 13, 2022 4:20 PM
To: LVM2 development<mailto:lvm-devel@redhat.com>; Pawan Sharma<mailto:sharmapawan@microsoft.com>; linux-lvm at redhat.com<mailto:linux-lvm@redhat.com>
Cc: Kapil Upadhayay<mailto:kupadhayay@microsoft.com>; Mitta Sai Chaithanya<mailto:mittas@microsoft.com>
Subject: [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
[Some people who received this message don't often get email from zdenek.kabelac at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Dne 13. 10. 22 v 8:53 Pawan Sharma napsal(a):
> adding this to lvm-devel mailing list also.
>
> Regards,
> Pawan
> ------------------------------------------------------------------------------
> *From:* Pawan Sharma
> *Sent:* Wednesday, October 12, 2022 10:42 PM
> *To:* linux-lvm at redhat.com <linux-lvm@redhat.com>
> *Cc:* Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay
> <kupadhayay@microsoft.com>
> *Subject:* LVM2 : performance drop even after deleting the snapshot
> Hi Everyone,
>
>
> We are evaluating lvm2 snapshots and doing performance testing on it. This is
> what we are doing :
>
> 1. dump some data to lvm2 volume (using fio)
> 2. take the snapshot
> 3. delete the snapshot (no IOs anywhere after creating the snapshot)
> 4. run the fio on lvm2 volume
>
> Here as you can see, we are just creating the snapshot and immediately
> deleting it. There are no IOs to the main volume or anywhere. When we run the
> fio after this (step 4) and we see around 50% drop in performance with
> reference to the number we get in step 1.
>
> It is expected to see a performance drop if there is a snapshot because of the
> COW. But here we deleted the snapshot, and it is not referring to any data
> also. We should not see any performance drop here.
>
> Could someone please help me understand this behavior. Why are we seeing the
> performance drop in this case? It seems like we deleted the snapshot but still
> it is not deleted, and we are paying the COW penalty.
>
> System Info:
>
> OS : ubuntu 18.04
> Kernel : 5.4.0
>
> # lvm version
> LVM version:2.02.176(2) (2017-11-03)
> Library version: 1.02.145 (2017-11-03)
> Driver version:4.41.0
>
> We also tried on latest ubuntu with newer version of LVM. We got the same
> behavior.
>
>
Hi
Debugging 5 year old software is likely not going to get lot of attention
from upstream.
So please:
a) reproduce the issue with some recent kernel & lvm2
b) take 'dmsetup table && dmsetup status' before you run every 'fio' test
and present here your result in some form - otherwise we can hardly see what
is the problem.
What should be expected - if you use old/thick snapshots - when you 'drop'
snapshot - you have your original intact LV - so results should mostly match
results before you take the snapshot - but you clearly have to take into
account if you use some 'SSD/NVMe' discarding and other things - so always run
series of tests and average your results.
If you use thin snapshot - that you can get various results depending on your
settings of thin chunks, discard usage.
Also maybe try your benchmark with different filesystems...
Regards
Zdenek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20221014/11cbf81b/attachment-0001.htm>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
2022-10-13 6:53 ` Pawan Sharma
@ 2022-10-14 19:50 ` Roger Heflin
-1 siblings, 0 replies; 23+ messages in thread
From: Roger Heflin @ 2022-10-14 19:50 UTC (permalink / raw)
To: LVM general discussion and development
Cc: Kapil Upadhayay, Mitta Sai Chaithanya, lvm-devel
What is the underlying disk hardware you are running this on?
virtual, spinning, ssd, nvme?
On Thu, Oct 13, 2022 at 2:01 AM Pawan Sharma <sharmapawan@microsoft.com> wrote:
>
> adding this to lvm-devel mailing list also.
>
> Regards,
> Pawan
> ________________________________
> From: Pawan Sharma
> Sent: Wednesday, October 12, 2022 10:42 PM
> To: linux-lvm@redhat.com <linux-lvm@redhat.com>
> Cc: Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay <kupadhayay@microsoft.com>
> Subject: LVM2 : performance drop even after deleting the snapshot
>
> Hi Everyone,
>
>
> We are evaluating lvm2 snapshots and doing performance testing on it. This is what we are doing :
>
> dump some data to lvm2 volume (using fio)
> take the snapshot
> delete the snapshot (no IOs anywhere after creating the snapshot)
> run the fio on lvm2 volume
>
> Here as you can see, we are just creating the snapshot and immediately deleting it. There are no IOs to the main volume or anywhere. When we run the fio after this (step 4) and we see around 50% drop in performance with reference to the number we get in step 1.
>
> It is expected to see a performance drop if there is a snapshot because of the COW. But here we deleted the snapshot, and it is not referring to any data also. We should not see any performance drop here.
>
> Could someone please help me understand this behavior. Why are we seeing the performance drop in this case? It seems like we deleted the snapshot but still it is not deleted, and we are paying the COW penalty.
>
> System Info:
>
> OS : ubuntu 18.04
> Kernel : 5.4.0
>
> # lvm version
> LVM version: 2.02.176(2) (2017-11-03)
> Library version: 1.02.145 (2017-11-03)
> Driver version: 4.41.0
>
> We also tried on latest ubuntu with newer version of LVM. We got the same behavior.
>
> Any help/pointers would be appreciated. Thanks in advance.
>
> Regards,
> Pawan
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [linux-lvm] LVM2 : performance drop even after deleting the snapshot
@ 2022-10-14 19:50 ` Roger Heflin
0 siblings, 0 replies; 23+ messages in thread
From: Roger Heflin @ 2022-10-14 19:50 UTC (permalink / raw)
To: lvm-devel
What is the underlying disk hardware you are running this on?
virtual, spinning, ssd, nvme?
On Thu, Oct 13, 2022 at 2:01 AM Pawan Sharma <sharmapawan@microsoft.com> wrote:
>
> adding this to lvm-devel mailing list also.
>
> Regards,
> Pawan
> ________________________________
> From: Pawan Sharma
> Sent: Wednesday, October 12, 2022 10:42 PM
> To: linux-lvm at redhat.com <linux-lvm@redhat.com>
> Cc: Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay <kupadhayay@microsoft.com>
> Subject: LVM2 : performance drop even after deleting the snapshot
>
> Hi Everyone,
>
>
> We are evaluating lvm2 snapshots and doing performance testing on it. This is what we are doing :
>
> dump some data to lvm2 volume (using fio)
> take the snapshot
> delete the snapshot (no IOs anywhere after creating the snapshot)
> run the fio on lvm2 volume
>
> Here as you can see, we are just creating the snapshot and immediately deleting it. There are no IOs to the main volume or anywhere. When we run the fio after this (step 4) and we see around 50% drop in performance with reference to the number we get in step 1.
>
> It is expected to see a performance drop if there is a snapshot because of the COW. But here we deleted the snapshot, and it is not referring to any data also. We should not see any performance drop here.
>
> Could someone please help me understand this behavior. Why are we seeing the performance drop in this case? It seems like we deleted the snapshot but still it is not deleted, and we are paying the COW penalty.
>
> System Info:
>
> OS : ubuntu 18.04
> Kernel : 5.4.0
>
> # lvm version
> LVM version: 2.02.176(2) (2017-11-03)
> Library version: 1.02.145 (2017-11-03)
> Driver version: 4.41.0
>
> We also tried on latest ubuntu with newer version of LVM. We got the same behavior.
>
> Any help/pointers would be appreciated. Thanks in advance.
>
> Regards,
> Pawan
> _______________________________________________
> linux-lvm mailing list
> linux-lvm at redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
2022-10-14 19:50 ` Roger Heflin
@ 2022-10-14 20:28 ` Roberto Fastec
-1 siblings, 0 replies; 23+ messages in thread
From: Roberto Fastec @ 2022-10-14 20:28 UTC (permalink / raw)
To: LVM general discussion and development
Cc: Kapil Upadhayay, lvm-devel, Mitta Sai Chaithanya
[-- Attachment #1.1: Type: text/plain, Size: 2920 bytes --]
TIP and HINT
forget SSDs with LVM unless of enterprise level
especially if you are going to use/implement the thin provisioning
How to identify an SSD of enterprise level:
it costs from 1,00 euro per gigabyte up to 1,50 euro per gigabyte
Kind regards
Roberto Gini
Technical Manager @ www.RecuperoDatiRAIDFAsTec.it
Ottieni BlueMail per Android
Il giorno 14 ott 2022, 21:50, alle ore 21:50, Roger Heflin <rogerheflin@gmail.com> ha scritto:
>What is the underlying disk hardware you are running this on?
>virtual, spinning, ssd, nvme?
>
>On Thu, Oct 13, 2022 at 2:01 AM Pawan Sharma
><sharmapawan@microsoft.com> wrote:
>>
>> adding this to lvm-devel mailing list also.
>>
>> Regards,
>> Pawan
>> ________________________________
>> From: Pawan Sharma
>> Sent: Wednesday, October 12, 2022 10:42 PM
>> To: linux-lvm@redhat.com <linux-lvm@redhat.com>
>> Cc: Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay
><kupadhayay@microsoft.com>
>> Subject: LVM2 : performance drop even after deleting the snapshot
>>
>> Hi Everyone,
>>
>>
>> We are evaluating lvm2 snapshots and doing performance testing on it.
>This is what we are doing :
>>
>> dump some data to lvm2 volume (using fio)
>> take the snapshot
>> delete the snapshot (no IOs anywhere after creating the snapshot)
>> run the fio on lvm2 volume
>>
>> Here as you can see, we are just creating the snapshot and
>immediately deleting it. There are no IOs to the main volume or
>anywhere. When we run the fio after this (step 4) and we see around 50%
>drop in performance with reference to the number we get in step 1.
>>
>> It is expected to see a performance drop if there is a snapshot
>because of the COW. But here we deleted the snapshot, and it is not
>referring to any data also. We should not see any performance drop
>here.
>>
>> Could someone please help me understand this behavior. Why are we
>seeing the performance drop in this case? It seems like we deleted the
>snapshot but still it is not deleted, and we are paying the COW
>penalty.
>>
>> System Info:
>>
>> OS : ubuntu 18.04
>> Kernel : 5.4.0
>>
>> # lvm version
>> LVM version: 2.02.176(2) (2017-11-03)
>> Library version: 1.02.145 (2017-11-03)
>> Driver version: 4.41.0
>>
>> We also tried on latest ubuntu with newer version of LVM. We got the
>same behavior.
>>
>> Any help/pointers would be appreciated. Thanks in advance.
>>
>> Regards,
>> Pawan
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@redhat.com
>https://listman.redhat.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
[-- Attachment #1.2: Type: text/html, Size: 3937 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [linux-lvm] LVM2 : performance drop even after deleting the snapshot
@ 2022-10-14 20:28 ` Roberto Fastec
0 siblings, 0 replies; 23+ messages in thread
From: Roberto Fastec @ 2022-10-14 20:28 UTC (permalink / raw)
To: lvm-devel
TIP and HINT
forget SSDs with LVM unless of enterprise level
especially if you are going to use/implement the thin provisioning
How to identify an SSD of enterprise level:
it costs from 1,00 euro per gigabyte up to 1,50 euro per gigabyte
Kind regards
Roberto Gini
Technical Manager @ www.RecuperoDatiRAIDFAsTec.it
?Ottieni BlueMail per Android ?
Il giorno 14 ott 2022, 21:50, alle ore 21:50, Roger Heflin <rogerheflin@gmail.com> ha scritto:
>What is the underlying disk hardware you are running this on?
>virtual, spinning, ssd, nvme?
>
>On Thu, Oct 13, 2022 at 2:01 AM Pawan Sharma
><sharmapawan@microsoft.com> wrote:
>>
>> adding this to lvm-devel mailing list also.
>>
>> Regards,
>> Pawan
>> ________________________________
>> From: Pawan Sharma
>> Sent: Wednesday, October 12, 2022 10:42 PM
>> To: linux-lvm at redhat.com <linux-lvm@redhat.com>
>> Cc: Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay
><kupadhayay@microsoft.com>
>> Subject: LVM2 : performance drop even after deleting the snapshot
>>
>> Hi Everyone,
>>
>>
>> We are evaluating lvm2 snapshots and doing performance testing on it.
>This is what we are doing :
>>
>> dump some data to lvm2 volume (using fio)
>> take the snapshot
>> delete the snapshot (no IOs anywhere after creating the snapshot)
>> run the fio on lvm2 volume
>>
>> Here as you can see, we are just creating the snapshot and
>immediately deleting it. There are no IOs to the main volume or
>anywhere. When we run the fio after this (step 4) and we see around 50%
>drop in performance with reference to the number we get in step 1.
>>
>> It is expected to see a performance drop if there is a snapshot
>because of the COW. But here we deleted the snapshot, and it is not
>referring to any data also. We should not see any performance drop
>here.
>>
>> Could someone please help me understand this behavior. Why are we
>seeing the performance drop in this case? It seems like we deleted the
>snapshot but still it is not deleted, and we are paying the COW
>penalty.
>>
>> System Info:
>>
>> OS : ubuntu 18.04
>> Kernel : 5.4.0
>>
>> # lvm version
>> LVM version: 2.02.176(2) (2017-11-03)
>> Library version: 1.02.145 (2017-11-03)
>> Driver version: 4.41.0
>>
>> We also tried on latest ubuntu with newer version of LVM. We got the
>same behavior.
>>
>> Any help/pointers would be appreciated. Thanks in advance.
>>
>> Regards,
>> Pawan
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm at redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm at redhat.com
>https://listman.redhat.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20221014/3bcff7ed/attachment.htm>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
2022-10-14 20:28 ` Roberto Fastec
@ 2022-10-17 5:01 ` Kapil Upadhayay
-1 siblings, 0 replies; 23+ messages in thread
From: Kapil Upadhayay @ 2022-10-17 5:01 UTC (permalink / raw)
To: Roberto Fastec, LVM general discussion and development
Cc: lvm-devel, Mitta Sai Chaithanya
[-- Attachment #1.1: Type: text/plain, Size: 6295 bytes --]
The thin volume was created on a block disk provided by Microsoft Azure Managed disk provisioned on premium SSD which is ready to be used in production environment. The disk can provide sustained Max IOPs at 5000.
Thanks,
Kapil Upadhayay.
From: Roberto Fastec <roberto.fastec@gmail.com>
Sent: Saturday, October 15, 2022 1:58 AM
To: LVM general discussion and development <linux-lvm@redhat.com>
Cc: Kapil Upadhayay <kupadhayay@microsoft.com>; Mitta Sai Chaithanya <mittas@microsoft.com>; lvm-devel@redhat.com
Subject: [EXTERNAL] Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
Some people who received this message don't often get email from roberto.fastec@gmail.com<mailto:roberto.fastec@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
TIP and HINT
forget SSDs with LVM unless of enterprise level
especially if you are going to use/implement the thin provisioning
How to identify an SSD of enterprise level:
it costs from 1,00 euro per gigabyte up to 1,50 euro per gigabyte
Kind regards
Roberto Gini
Technical Manager @ www.RecuperoDatiRAIDFAsTec.it<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.recuperodatiraidfastec.it%2F&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9i73rMTiPlvD6jlcxLti7Pvrku7jE%2BLciLP57umpwpA%3D&reserved=0>
Ottieni BlueMail per Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbluemail.me%2F&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hGhews%2FSF8d7zCJvRzIOs24CE0xLHH1NqyWjL3IqthI%3D&reserved=0>
Il giorno 14 ott 2022, alle ore 21:50, Roger Heflin <rogerheflin@gmail.com<mailto:rogerheflin@gmail.com>> ha scritto:
What is the underlying disk hardware you are running this on?
virtual, spinning, ssd, nvme?
On Thu, Oct 13, 2022 at 2:01 AM Pawan Sharma <sharmapawan@microsoft.com<mailto:sharmapawan@microsoft.com>> wrote:
adding this to lvm-devel mailing list also.
Regards,
Pawan
________________________________
From: Pawan Sharma
Sent: Wednesday, October 12, 2022 10:42 PM
To: linux-lvm@redhat.com<mailto:linux-lvm@redhat.com> <linux-lvm@redhat.com<mailto:linux-lvm@redhat.com>>
Cc: Mitta Sai Chaithanya <mittas@microsoft.com<mailto:mittas@microsoft.com>>; Kapil Upadhayay <kupadhayay@microsoft.com<mailto:kupadhayay@microsoft.com>>
Subject: LVM2 : performance drop even after deleting the snapshot
Hi Everyone,
We are evaluating lvm2 snapshots and doing performance testing on it. This is what we are doing :
dump some data to lvm2 volume (using fio)
take the snapshot
delete the snapshot (no IOs anywhere after creating the snapshot)
run the fio on lvm2 volume
Here as you can see, we are just creating the snapshot and immediately deleting it. There are no IOs to the main volume or anywhere. When we run the fio after this (step 4) and we see around 50% drop in performance with reference to the number we get in step 1.
It is expected to see a performance drop if there is a snapshot because of the COW. But here we deleted the snapshot, and it is not referring to any data also. We should not see any performance drop here.
Could someone please help me understand this behavior. Why are we seeing the performance drop in this case? It seems like we deleted the snapshot but still it is not deleted, and we are paying the COW penalty.
System Info:
OS : ubuntu 18.04
Kernel : 5.4.0
# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.41.0
We also tried on latest ubuntu with newer version of LVM. We got the same behavior.
Any help/pointers would be appreciated. Thanks in advance.
Regards,
Pawan
________________________________
linux-lvm mailing list
linux-lvm@redhat.com<mailto:linux-lvm@redhat.com>
https://listman.redhat.com/mailman/listinfo/linux-lvm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistman.redhat.com%2Fmailman%2Flistinfo%2Flinux-lvm&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2HZwpc4MH0w7bCVwNuohJ7jG5naZaCNGdAJHwsIzsZg%3D&reserved=0>
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftldp.org%2FHOWTO%2FLVM-HOWTO&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nMH03VV84Qe%2FkZ5g0hoBsZi53QKOYs%2B9CiTnPb6xu4s%3D&reserved=0>/
________________________________
linux-lvm mailing list
linux-lvm@redhat.com<mailto:linux-lvm@redhat.com>
https://listman.redhat.com/mailman/listinfo/linux-lvm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistman.redhat.com%2Fmailman%2Flistinfo%2Flinux-lvm&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2HZwpc4MH0w7bCVwNuohJ7jG5naZaCNGdAJHwsIzsZg%3D&reserved=0>
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftldp.org%2FHOWTO%2FLVM-HOWTO&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nMH03VV84Qe%2FkZ5g0hoBsZi53QKOYs%2B9CiTnPb6xu4s%3D&reserved=0>/
[-- Attachment #1.2: Type: text/html, Size: 11859 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [linux-lvm] LVM2 : performance drop even after deleting the snapshot
@ 2022-10-17 5:01 ` Kapil Upadhayay
0 siblings, 0 replies; 23+ messages in thread
From: Kapil Upadhayay @ 2022-10-17 5:01 UTC (permalink / raw)
To: lvm-devel
The thin volume was created on a block disk provided by Microsoft Azure Managed disk provisioned on premium SSD which is ready to be used in production environment. The disk can provide sustained Max IOPs at 5000.
Thanks,
Kapil Upadhayay.
From: Roberto Fastec <roberto.fastec@gmail.com>
Sent: Saturday, October 15, 2022 1:58 AM
To: LVM general discussion and development <linux-lvm@redhat.com>
Cc: Kapil Upadhayay <kupadhayay@microsoft.com>; Mitta Sai Chaithanya <mittas@microsoft.com>; lvm-devel at redhat.com
Subject: [EXTERNAL] Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
Some people who received this message don't often get email from roberto.fastec at gmail.com<mailto:roberto.fastec@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
TIP and HINT
forget SSDs with LVM unless of enterprise level
especially if you are going to use/implement the thin provisioning
How to identify an SSD of enterprise level:
it costs from 1,00 euro per gigabyte up to 1,50 euro per gigabyte
Kind regards
Roberto Gini
Technical Manager @ www.RecuperoDatiRAIDFAsTec.it<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.recuperodatiraidfastec.it%2F&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9i73rMTiPlvD6jlcxLti7Pvrku7jE%2BLciLP57umpwpA%3D&reserved=0>
Ottieni BlueMail per Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbluemail.me%2F&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hGhews%2FSF8d7zCJvRzIOs24CE0xLHH1NqyWjL3IqthI%3D&reserved=0>
Il giorno 14 ott 2022, alle ore 21:50, Roger Heflin <rogerheflin at gmail.com<mailto:rogerheflin@gmail.com>> ha scritto:
What is the underlying disk hardware you are running this on?
virtual, spinning, ssd, nvme?
On Thu, Oct 13, 2022 at 2:01 AM Pawan Sharma <sharmapawan at microsoft.com<mailto:sharmapawan@microsoft.com>> wrote:
adding this to lvm-devel mailing list also.
Regards,
Pawan
________________________________
From: Pawan Sharma
Sent: Wednesday, October 12, 2022 10:42 PM
To: linux-lvm at redhat.com<mailto:linux-lvm@redhat.com> <linux-lvm at redhat.com<mailto:linux-lvm@redhat.com>>
Cc: Mitta Sai Chaithanya <mittas at microsoft.com<mailto:mittas@microsoft.com>>; Kapil Upadhayay <kupadhayay at microsoft.com<mailto:kupadhayay@microsoft.com>>
Subject: LVM2 : performance drop even after deleting the snapshot
Hi Everyone,
We are evaluating lvm2 snapshots and doing performance testing on it. This is what we are doing :
dump some data to lvm2 volume (using fio)
take the snapshot
delete the snapshot (no IOs anywhere after creating the snapshot)
run the fio on lvm2 volume
Here as you can see, we are just creating the snapshot and immediately deleting it. There are no IOs to the main volume or anywhere. When we run the fio after this (step 4) and we see around 50% drop in performance with reference to the number we get in step 1.
It is expected to see a performance drop if there is a snapshot because of the COW. But here we deleted the snapshot, and it is not referring to any data also. We should not see any performance drop here.
Could someone please help me understand this behavior. Why are we seeing the performance drop in this case? It seems like we deleted the snapshot but still it is not deleted, and we are paying the COW penalty.
System Info:
OS : ubuntu 18.04
Kernel : 5.4.0
# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.41.0
We also tried on latest ubuntu with newer version of LVM. We got the same behavior.
Any help/pointers would be appreciated. Thanks in advance.
Regards,
Pawan
________________________________
linux-lvm mailing list
linux-lvm at redhat.com<mailto:linux-lvm@redhat.com>
https://listman.redhat.com/mailman/listinfo/linux-lvm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistman.redhat.com%2Fmailman%2Flistinfo%2Flinux-lvm&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2HZwpc4MH0w7bCVwNuohJ7jG5naZaCNGdAJHwsIzsZg%3D&reserved=0>
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftldp.org%2FHOWTO%2FLVM-HOWTO&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nMH03VV84Qe%2FkZ5g0hoBsZi53QKOYs%2B9CiTnPb6xu4s%3D&reserved=0>/
________________________________
linux-lvm mailing list
linux-lvm at redhat.com<mailto:linux-lvm@redhat.com>
https://listman.redhat.com/mailman/listinfo/linux-lvm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistman.redhat.com%2Fmailman%2Flistinfo%2Flinux-lvm&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2HZwpc4MH0w7bCVwNuohJ7jG5naZaCNGdAJHwsIzsZg%3D&reserved=0>
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftldp.org%2FHOWTO%2FLVM-HOWTO&data=05%7C01%7Ckupadhayay%40microsoft.com%7Ccd9e45aadb0c4dd0a03508daae22aa91%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638013761207553033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nMH03VV84Qe%2FkZ5g0hoBsZi53QKOYs%2B9CiTnPb6xu4s%3D&reserved=0>/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20221017/87b29702/attachment-0001.htm>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
2022-10-14 19:31 ` Mitta Sai Chaithanya
@ 2022-10-17 13:10 ` Zdenek Kabelac
-1 siblings, 0 replies; 23+ messages in thread
From: Zdenek Kabelac @ 2022-10-17 13:10 UTC (permalink / raw)
To: Mitta Sai Chaithanya, LVM2 development, Pawan Sharma, linux-lvm
Cc: Kapil Upadhayay
Dne 14. 10. 22 v 21:31 Mitta Sai Chaithanya napsal(a):
> Hi Zdenek Kabelac,
> Thanks for your quick reply and suggestions.
>
> We conducted couple of tests on Ubuntu 22.04 and observed similar performance
> behavior post thin snapshot deletion without writing any data anywhere.
>
> *Commands used to create Thin LVM volume*:
> - lvcreate -L 480G --poolmetadataspare n --poolmetadatasize 16G
> --chunksize=64K --thinpool ThinDataLV ThinVolGrp
> - lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp
Hi
So now it's clear you are talking about thin snapshots - this is a very
different story going on here (as we normally use term "COW" volumes for thick
old snapshots)
I'll consult more with thinp author - however it does look to me you are using
same device to store data & metadata.
This is always a highly sub-optimal solution - the metadata device is likely
best to be stored on fast (low latency) devices.
So my wild guess - you are possibly using rotational device backend to store
your thin-pools metadata volume and then your setups gets very sensitive on
the metadata fragmentation.
Thin-pool was designed to be used with SSD/NVMe for metadata which is way less
sensitive on seeking.
So when you 'create' snapshot - metadata gets updated - when you remove thin
snapshot - metadata gets again a lots of changes (especially when your origin
volume is already populated) - and fragmentation is inevitable and you are
getting high penalty of holding metadata device on the same drive as your data
device.
So while there are some plans to improve some metadata logistic - I'd not
expect miracles on you particular setup - I'd highly recommend to plug-in some
SSD/NVMe storage for storing your thinpool metadata - this is the way to go
to get better 'benchmarking' numbers here.
For an improvement on your setup - try to seek larger chunk size values where
your data 'sharing' is still reasonably valuable - this depends on data-type
usage - but chunk size 256K might be possibly a good compromise (with disabled
zeroing - if you hunt for the best performance).
Regards
Zdenek
PS: later mails suggest you are using some 'MS Azure' devices?? - so please
redo your testing with your local hardware/storage - where you have precise
guarantees of storage drive performance - testing in the Cloud is random by
design....
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
@ 2022-10-17 13:10 ` Zdenek Kabelac
0 siblings, 0 replies; 23+ messages in thread
From: Zdenek Kabelac @ 2022-10-17 13:10 UTC (permalink / raw)
To: lvm-devel
Dne 14. 10. 22 v 21:31 Mitta Sai Chaithanya napsal(a):
> Hi Zdenek Kabelac,
> ????????? Thanks for your quick reply and suggestions.
>
> We conducted couple of tests on Ubuntu 22.04 and observed similar performance
> behavior post thin snapshot deletion without writing any data anywhere.
>
> *Commands used to create Thin LVM volume*:
> - lvcreate??-L 480G --poolmetadataspare n --poolmetadatasize 16G
> --chunksize=64K --thinpool??ThinDataLV ThinVolGrp
> - lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp
Hi
So now it's clear you are talking about thin snapshots - this is a very
different story going on here (as we normally use term "COW" volumes for thick
old snapshots)
I'll consult more with thinp author - however it does look to me you are using
same device to store data & metadata.
This is always a highly sub-optimal solution - the metadata device is likely
best to be stored on fast (low latency) devices.
So my wild guess - you are possibly using rotational device backend to store
your thin-pools metadata volume and then your setups gets very sensitive on
the metadata fragmentation.
Thin-pool was designed to be used with SSD/NVMe for metadata which is way less
sensitive on seeking.
So when you 'create' snapshot - metadata gets updated - when you remove thin
snapshot - metadata gets again a lots of changes (especially when your origin
volume is already populated) - and fragmentation is inevitable and you are
getting high penalty of holding metadata device on the same drive as your data
device.
So while there are some plans to improve some metadata logistic - I'd not
expect miracles on you particular setup - I'd highly recommend to plug-in some
SSD/NVMe storage for storing your thinpool metadata - this is the way to go
to get better 'benchmarking' numbers here.
For an improvement on your setup - try to seek larger chunk size values where
your data 'sharing' is still reasonably valuable - this depends on data-type
usage - but chunk size 256K might be possibly a good compromise (with disabled
zeroing - if you hunt for the best performance).
Regards
Zdenek
PS: later mails suggest you are using some 'MS Azure' devices?? - so please
redo your testing with your local hardware/storage - where you have precise
guarantees of storage drive performance - testing in the Cloud is random by
design....
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
2022-10-17 13:10 ` Zdenek Kabelac
(?)
@ 2022-10-17 13:41 ` Erwin van Londen
2022-10-20 18:19 ` Zdenek Kabelac
-1 siblings, 1 reply; 23+ messages in thread
From: Erwin van Londen @ 2022-10-17 13:41 UTC (permalink / raw)
To: linux-lvm
[-- Attachment #1.1.1: Type: text/plain, Size: 4220 bytes --]
From the looks of it the disk, as provisioned out of an Azure pool, is
likely backed by an enterprise raid array. When you provision the pools
with discard_passdown the removal of the snapshot will also be pushed
down to the underlying hypervisor or disk array. You would need to wait
till that process is completed in order to make any comparisons.
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4878/4145152
8325/7864320 - rw discard_passdown queue_if_no_space - 1024
As per man page
--discards passdown|nopassdown|ignore
Specifies how the device-mapper thin pool layer in the kernel should
handle discards. ignore causes the thin pool to ignore discards.
nopassdown causes the
thin pool to process discards itself to allow reuse of unneeded extents
in the thin pool. passdown causes the thin pool to process discards
itself (like
nopassdown) and pass the discards to the underlying device.
Try the same operation after changing the thin volume
lvchange --discards nopassdown VG/ThinPoolLV
--
Kind regards,
Erwin van LondenEvL Consulting
ABN 43 560 744 507
Mobile+61-434-325795Phone+61-7-
53213176Webhttp://erwinvanlonden.netConferencehttps://iene.3cx.com.au/meet/erwinvlwebmeet
Web Talkhttps://iene.3cx.com.au/callus/#erwinvlwebphone
On Mon, 2022-10-17 at 15:10 +0200, Zdenek Kabelac wrote:
> Dne 14. 10. 22 v 21:31 Mitta Sai Chaithanya napsal(a):
> > Hi Zdenek Kabelac,
> > Thanks for your quick reply and suggestions.
> >
> > We conducted couple of tests on Ubuntu 22.04 and observed similar
> > performance
> > behavior post thin snapshot deletion without writing any data
> > anywhere.
> >
> > *Commands used to create Thin LVM volume*:
> > - lvcreate -L 480G --poolmetadataspare n --poolmetadatasize 16G
> > --chunksize=64K --thinpool ThinDataLV ThinVolGrp
> > - lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp
>
>
> Hi
>
> So now it's clear you are talking about thin snapshots - this is a
> very
> different story going on here (as we normally use term "COW" volumes
> for thick
> old snapshots)
>
> I'll consult more with thinp author - however it does look to me you
> are using
> same device to store data & metadata.
>
> This is always a highly sub-optimal solution - the metadata device is
> likely
> best to be stored on fast (low latency) devices.
>
> So my wild guess - you are possibly using rotational device backend
> to store
> your thin-pools metadata volume and then your setups gets very
> sensitive on
> the metadata fragmentation.
>
> Thin-pool was designed to be used with SSD/NVMe for metadata which is
> way less
> sensitive on seeking.
>
> So when you 'create' snapshot - metadata gets updated - when you
> remove thin
> snapshot - metadata gets again a lots of changes (especially when
> your origin
> volume is already populated) - and fragmentation is inevitable and
> you are
> getting high penalty of holding metadata device on the same drive as
> your data
> device.
>
> So while there are some plans to improve some metadata logistic - I'd
> not
> expect miracles on you particular setup - I'd highly recommend to
> plug-in some
> SSD/NVMe storage for storing your thinpool metadata - this is the
> way to go
> to get better 'benchmarking' numbers here.
>
> For an improvement on your setup - try to seek larger chunk size
> values where
> your data 'sharing' is still reasonably valuable - this depends on
> data-type
> usage - but chunk size 256K might be possibly a good compromise (with
> disabled
> zeroing - if you hunt for the best performance).
>
>
> Regards
>
> Zdenek
>
> PS: later mails suggest you are using some 'MS Azure' devices?? - so
> please
> redo your testing with your local hardware/storage - where you have
> precise
> guarantees of storage drive performance - testing in the Cloud is
> random by
> design....
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
[-- Attachment #1.1.2: Type: text/html, Size: 6218 bytes --]
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] LVM2 : performance drop even after deleting the snapshot
2022-10-14 20:28 ` Roberto Fastec
@ 2022-10-17 15:16 ` Demi Marie Obenour
-1 siblings, 0 replies; 23+ messages in thread
From: Demi Marie Obenour @ 2022-10-17 15:16 UTC (permalink / raw)
To: LVM general discussion and development
Cc: Kapil Upadhayay, Mitta Sai Chaithanya, lvm-devel
[-- Attachment #1.1: Type: text/plain, Size: 437 bytes --]
On Fri, Oct 14, 2022 at 10:28:28PM +0200, Roberto Fastec wrote:
> TIP and HINT
> forget SSDs with LVM unless of enterprise level
> especially if you are going to use/implement the thin provisioning
As a user and developer of Qubes OS, this makes me nervous. Qubes OS
uses LVM heavily in the default configuration, and is designed for
end-user systems.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [linux-lvm] LVM2 : performance drop even after deleting the snapshot
@ 2022-10-17 15:16 ` Demi Marie Obenour
0 siblings, 0 replies; 23+ messages in thread
From: Demi Marie Obenour @ 2022-10-17 15:16 UTC (permalink / raw)
To: lvm-devel
On Fri, Oct 14, 2022 at 10:28:28PM +0200, Roberto Fastec wrote:
> TIP and HINT
> forget SSDs with LVM unless of enterprise level
> especially if you are going to use/implement the thin provisioning
As a user and developer of Qubes OS, this makes me nervous. Qubes OS
uses LVM heavily in the default configuration, and is designed for
end-user systems.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20221017/94f74bd8/attachment.sig>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
2022-10-17 13:10 ` Zdenek Kabelac
@ 2022-10-18 3:33 ` Pawan Sharma
-1 siblings, 0 replies; 23+ messages in thread
From: Pawan Sharma @ 2022-10-18 3:33 UTC (permalink / raw)
To: Zdenek Kabelac, LVM2 development, linux-lvm
Cc: Kapil Upadhayay, Mitta Sai Chaithanya
[-- Attachment #1.1: Type: text/plain, Size: 3445 bytes --]
Hi Zdenek,
I would like to highlight one point here is that we are creating and then deleting the snapshot immediately without writing anything anywhere. In this case, we are expecting the performance to go back to what it was before taking the thin snapshot. Here we are not getting the original performance after deleting the snapshot. Do you know any reason why that would be happening.
Regards,
Pawan
________________________________
From: Zdenek Kabelac <zdenek.kabelac@gmail.com>
Sent: Monday, October 17, 2022 6:40 PM
To: Mitta Sai Chaithanya <mittas@microsoft.com>; LVM2 development <lvm-devel@redhat.com>; Pawan Sharma <sharmapawan@microsoft.com>; linux-lvm@redhat.com <linux-lvm@redhat.com>
Cc: Kapil Upadhayay <kupadhayay@microsoft.com>
Subject: Re: [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
[Some people who received this message don't often get email from zdenek.kabelac@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Dne 14. 10. 22 v 21:31 Mitta Sai Chaithanya napsal(a):
> Hi Zdenek Kabelac,
> Thanks for your quick reply and suggestions.
>
> We conducted couple of tests on Ubuntu 22.04 and observed similar performance
> behavior post thin snapshot deletion without writing any data anywhere.
>
> *Commands used to create Thin LVM volume*:
> - lvcreate -L 480G --poolmetadataspare n --poolmetadatasize 16G
> --chunksize=64K --thinpool ThinDataLV ThinVolGrp
> - lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp
Hi
So now it's clear you are talking about thin snapshots - this is a very
different story going on here (as we normally use term "COW" volumes for thick
old snapshots)
I'll consult more with thinp author - however it does look to me you are using
same device to store data & metadata.
This is always a highly sub-optimal solution - the metadata device is likely
best to be stored on fast (low latency) devices.
So my wild guess - you are possibly using rotational device backend to store
your thin-pools metadata volume and then your setups gets very sensitive on
the metadata fragmentation.
Thin-pool was designed to be used with SSD/NVMe for metadata which is way less
sensitive on seeking.
So when you 'create' snapshot - metadata gets updated - when you remove thin
snapshot - metadata gets again a lots of changes (especially when your origin
volume is already populated) - and fragmentation is inevitable and you are
getting high penalty of holding metadata device on the same drive as your data
device.
So while there are some plans to improve some metadata logistic - I'd not
expect miracles on you particular setup - I'd highly recommend to plug-in some
SSD/NVMe storage for storing your thinpool metadata - this is the way to go
to get better 'benchmarking' numbers here.
For an improvement on your setup - try to seek larger chunk size values where
your data 'sharing' is still reasonably valuable - this depends on data-type
usage - but chunk size 256K might be possibly a good compromise (with disabled
zeroing - if you hunt for the best performance).
Regards
Zdenek
PS: later mails suggest you are using some 'MS Azure' devices?? - so please
redo your testing with your local hardware/storage - where you have precise
guarantees of storage drive performance - testing in the Cloud is random by
design....
[-- Attachment #1.2: Type: text/html, Size: 7527 bytes --]
[-- Attachment #2: Type: text/plain, Size: 202 bytes --]
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
@ 2022-10-18 3:33 ` Pawan Sharma
0 siblings, 0 replies; 23+ messages in thread
From: Pawan Sharma @ 2022-10-18 3:33 UTC (permalink / raw)
To: lvm-devel
Hi Zdenek,
I would like to highlight one point here is that we are creating and then deleting the snapshot immediately without writing anything anywhere. In this case, we are expecting the performance to go back to what it was before taking the thin snapshot. Here we are not getting the original performance after deleting the snapshot. Do you know any reason why that would be happening.
Regards,
Pawan
________________________________
From: Zdenek Kabelac <zdenek.kabelac@gmail.com>
Sent: Monday, October 17, 2022 6:40 PM
To: Mitta Sai Chaithanya <mittas@microsoft.com>; LVM2 development <lvm-devel@redhat.com>; Pawan Sharma <sharmapawan@microsoft.com>; linux-lvm at redhat.com <linux-lvm@redhat.com>
Cc: Kapil Upadhayay <kupadhayay@microsoft.com>
Subject: Re: [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
[Some people who received this message don't often get email from zdenek.kabelac at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Dne 14. 10. 22 v 21:31 Mitta Sai Chaithanya napsal(a):
> Hi Zdenek Kabelac,
> Thanks for your quick reply and suggestions.
>
> We conducted couple of tests on Ubuntu 22.04 and observed similar performance
> behavior post thin snapshot deletion without writing any data anywhere.
>
> *Commands used to create Thin LVM volume*:
> - lvcreate -L 480G --poolmetadataspare n --poolmetadatasize 16G
> --chunksize=64K --thinpool ThinDataLV ThinVolGrp
> - lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp
Hi
So now it's clear you are talking about thin snapshots - this is a very
different story going on here (as we normally use term "COW" volumes for thick
old snapshots)
I'll consult more with thinp author - however it does look to me you are using
same device to store data & metadata.
This is always a highly sub-optimal solution - the metadata device is likely
best to be stored on fast (low latency) devices.
So my wild guess - you are possibly using rotational device backend to store
your thin-pools metadata volume and then your setups gets very sensitive on
the metadata fragmentation.
Thin-pool was designed to be used with SSD/NVMe for metadata which is way less
sensitive on seeking.
So when you 'create' snapshot - metadata gets updated - when you remove thin
snapshot - metadata gets again a lots of changes (especially when your origin
volume is already populated) - and fragmentation is inevitable and you are
getting high penalty of holding metadata device on the same drive as your data
device.
So while there are some plans to improve some metadata logistic - I'd not
expect miracles on you particular setup - I'd highly recommend to plug-in some
SSD/NVMe storage for storing your thinpool metadata - this is the way to go
to get better 'benchmarking' numbers here.
For an improvement on your setup - try to seek larger chunk size values where
your data 'sharing' is still reasonably valuable - this depends on data-type
usage - but chunk size 256K might be possibly a good compromise (with disabled
zeroing - if you hunt for the best performance).
Regards
Zdenek
PS: later mails suggest you are using some 'MS Azure' devices?? - so please
redo your testing with your local hardware/storage - where you have precise
guarantees of storage drive performance - testing in the Cloud is random by
design....
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20221018/c9f6da66/attachment.htm>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
2022-10-18 3:33 ` Pawan Sharma
@ 2022-10-18 11:15 ` Zdenek Kabelac
-1 siblings, 0 replies; 23+ messages in thread
From: Zdenek Kabelac @ 2022-10-18 11:15 UTC (permalink / raw)
To: Pawan Sharma, LVM2 development, linux-lvm
Cc: Kapil Upadhayay, Mitta Sai Chaithanya
Dne 18. 10. 22 v 5:33 Pawan Sharma napsal(a):
> Hi Zdenek,
>
> I would like to highlight one point here is that we are creating and then
> deleting the snapshot immediately without writing anything anywhere. In this
> case, we are expecting the performance to go back to what it was before taking
> the thin snapshot. Here we are not getting the original performance after
> deleting the snapshot. Do you know any reason why that would be happening.
As explained in my previous post - with thin-provisioning - you are getting
metadata updates for bTrees - thus there is no 'revert' to previous 'metadata
state' - there is rolling update of bTrees which is by design 'seek
unfriendly' - so for the performance hunting users the use of SSD/NVMe type
storage for these metadata volumes is basically a must (and it's been designed
for that).
The old 'thick' snapshot where you allocate explicit COW LV storage is going
to give here your expected behavior - however you will (of course) loose all
the benefits you get with thin-pools.
With thin-pool (as also mentioned in my previous post) - if you can't afford
dedicated low-latency storage - you need to scale-up chunk size - so the
amount of metadata updates is reduced together (lowering seeking). I'm afraid
you can't expect much more in the near future.
FYI there is to be merged in the upcoming kernel this patch set:
https://listman.redhat.com/archives/dm-devel/2022-October/052367.html
which should also help a lot with multithreaded load on thin-pools
There is also some new metadata format being experimented with - but whether
this will also tackle anything in the seek friendlier logic is hard to tell...
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
@ 2022-10-18 11:15 ` Zdenek Kabelac
0 siblings, 0 replies; 23+ messages in thread
From: Zdenek Kabelac @ 2022-10-18 11:15 UTC (permalink / raw)
To: lvm-devel
Dne 18. 10. 22 v 5:33 Pawan Sharma napsal(a):
> Hi Zdenek,
>
> I would like to highlight one point here is that we are creating and then
> deleting the snapshot immediately without writing anything anywhere. In this
> case, we are expecting the performance to go back to what it was before taking
> the thin snapshot. Here we are not getting the original performance after
> deleting the snapshot. Do you know any reason why that would be happening.
As explained in my previous post - with thin-provisioning - you are getting
metadata updates for bTrees - thus there is no 'revert' to previous 'metadata
state' - there is rolling update of bTrees which is by design 'seek
unfriendly' - so for the performance hunting users the use of SSD/NVMe type
storage for these metadata volumes is basically a must (and it's been designed
for that).
The old 'thick' snapshot where you allocate explicit COW LV storage is going
to give here your expected behavior - however you will (of course) loose all
the benefits you get with thin-pools.
With thin-pool (as also mentioned in my previous post) - if you can't afford
dedicated low-latency storage - you need to scale-up chunk size - so the
amount of metadata updates is reduced together (lowering seeking). I'm afraid
you can't expect much more in the near future.
FYI there is to be merged in the upcoming kernel this patch set:
https://listman.redhat.com/archives/dm-devel/2022-October/052367.html
which should also help a lot with multithreaded load on thin-pools
There is also some new metadata format being experimented with - but whether
this will also tackle anything in the seek friendlier logic is hard to tell...
Regards
Zdenek
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [linux-lvm] [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot
2022-10-17 13:41 ` [linux-lvm] " Erwin van Londen
@ 2022-10-20 18:19 ` Zdenek Kabelac
0 siblings, 0 replies; 23+ messages in thread
From: Zdenek Kabelac @ 2022-10-20 18:19 UTC (permalink / raw)
To: LVM general discussion and development, Erwin van Londen
Dne 17. 10. 22 v 15:41 Erwin van Londen napsal(a):
> From the looks of it the disk, as provisioned out of an Azure pool, is likely
> backed by an enterprise raid array. When you provision the pools with
> discard_passdown the removal of the snapshot will also be pushed down to the
> underlying hypervisor or disk array. You would need to wait till that process
> is completed in order to make any comparisons.
>
> ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4878/4145152
> 8325/7864320 - rw discard_passdown queue_if_no_space - 1024
>
> As per man page
>
> --discards passdown|nopassdown|ignore
> Specifies how the device-mapper thin pool layer in the kernel should handle
> discards. ignore causes the thin pool to ignore discards. nopassdown causes the
> thin pool to process discards itself to allow reuse of unneeded extents in the
> thin pool. passdown causes the thin pool to process discards itself (like
> nopassdown) and pass the discards to the underlying device.
>
> Try the same operation after changing the thin volume
>
> lvchange --discards nopassdown VG/ThinPoolLV
Discard here is likely irrelevant - since there will likely no blocks for
discarding.
When the user removes thin LV (which happens to be sharing its block with
some other thin LV (origin -> snapshot)) there is just some metadata update
reducing sharing of blocks with origin thinLV - so nothing to be discard for
data (since snapshot is removed after its creation without any use - only if
the origin would be meanwhile in this short period of time changed
dramatically - then exclusively owned parts of such snapshot may be discarded)
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2022-10-20 18:19 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-10-12 17:12 [linux-lvm] LVM2 : performance drop even after deleting the snapshot Pawan Sharma
2022-10-13 6:53 ` Pawan Sharma
2022-10-13 6:53 ` Pawan Sharma
2022-10-13 10:50 ` [linux-lvm] " Zdenek Kabelac
2022-10-13 10:50 ` Zdenek Kabelac
2022-10-14 19:31 ` [linux-lvm] [EXTERNAL] " Mitta Sai Chaithanya
2022-10-14 19:31 ` Mitta Sai Chaithanya
2022-10-17 13:10 ` [linux-lvm] " Zdenek Kabelac
2022-10-17 13:10 ` Zdenek Kabelac
2022-10-17 13:41 ` [linux-lvm] " Erwin van Londen
2022-10-20 18:19 ` Zdenek Kabelac
2022-10-18 3:33 ` Pawan Sharma
2022-10-18 3:33 ` Pawan Sharma
2022-10-18 11:15 ` [linux-lvm] " Zdenek Kabelac
2022-10-18 11:15 ` Zdenek Kabelac
2022-10-14 19:50 ` [linux-lvm] " Roger Heflin
2022-10-14 19:50 ` Roger Heflin
2022-10-14 20:28 ` Roberto Fastec
2022-10-14 20:28 ` Roberto Fastec
2022-10-17 5:01 ` Kapil Upadhayay
2022-10-17 5:01 ` Kapil Upadhayay
2022-10-17 15:16 ` Demi Marie Obenour
2022-10-17 15:16 ` Demi Marie Obenour
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.