[linux-lvm] faster snapshot creation?

[linux-lvm] faster snapshot creation?

[Eric Toombs]

Snapshot creation is already pretty fast:

> $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template 
>   Logical volume "snap" created.
> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k
> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps

That's about half a second in real time. But I have a scenario that
would benefit from it being even faster. I'm doing many small unit tests
starting from a template filesystem. I do the snapshot, run the unit
test on the snapshot, then delete the snapshot afterwards using
lvremove. Each unit test, though, takes much less than a second to run
(often on the order of 10ms), so most of the time is being spent making
these snapshots.

So, is there a sort of "dumber" way of making these snapshots, maybe by
changing the allocation algorithm or something?

Re: [linux-lvm] faster snapshot creation?

[Gionatan Danti]

Il 2020-02-22 12:58 Eric Toombs ha scritto:
> So, is there a sort of "dumber" way of making these snapshots, maybe by
> changing the allocation algorithm or something?

Hi, I think that total snapshot creation time is dominated by LVM 
flushing its (meta)data to the physical disks. Two things to try:
- use thin lvm snapshot (see man lvmthin);
- create a pv/vg/lv from a memory-backed loop device (ie: truncate 
--size=1G /dev/shm/test.img; losetup -f /dev/shm/test.img; pvcreate 
/dev/loop0 and so on...)

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it [1]
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] faster snapshot creation?

[Zdenek Kabelac]

Dne 22. 02. 20 v 12:58 Eric Toombs napsal(a):
> Snapshot creation is already pretty fast:
> 
>> $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template
>>    Logical volume "snap" created.
>> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k
>> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps
> 
> That's about half a second in real time. But I have a scenario that
> would benefit from it being even faster. I'm doing many small unit tests
> starting from a template filesystem. I do the snapshot, run the unit
> test on the snapshot, then delete the snapshot afterwards using
> lvremove. Each unit test, though, takes much less than a second to run
> (often on the order of 10ms), so most of the time is being spent making
> these snapshots.
> 
> So, is there a sort of "dumber" way of making these snapshots, maybe by
> changing the allocation algorithm or something?

Hi

IMHO - what takes most of the time are these couple things:

Each command has 'non-trivial' time overhead on its startup (scanning you 
system with devices and validating everything)

For old snapshots -  COW are needs to be 'created' & 'zeroed' as separate LV.

Then you need to 'flush' all existing IO on origin device (so it's in the 
consistent states - i.e. the filesystem synchronizes all it's content in its 
metadata) - this all takes some measurable amount of time.

You can 'prepare' empty zeroed LV ahead of time and then use
'lvconvert' to attach snapshot (with -Zn)  - this should speed-up attachment
of snapshot. For the 'second' point you could likely issue 'sync' ahead of 
time so most buffers will be flushed (if there is no big IO traffic).

Saying all this - why are you using old snapshot when you are targeting for 
performance ??

You really should consider usage of thin-pool - where you could chain a long 
series of snapshot without having a dramatic performance degradation of the 
whole IO throughput  - old snapshot are really meant to be used only if you 
want to take i.e. backup of a filesystem and you need some 'consistent' point 
in time - for everything else you should be using thin-pools nowdays...

Regards

Zdenek

Re: [linux-lvm] faster snapshot creation?

[Eric Toombs]

I tried thin pools and got the same snapshot creation time, about .45s.

I also tried creating the entire vg in memory with losetup and that
didn't change the creation time either.

On 2020-02-25 04:38, Zdenek Kabelac wrote:
> Dne 22. 02. 20 v 12:58 Eric Toombs napsal(a):
>> Snapshot creation is already pretty fast:
>>
>>> $ time sudo lvcreate --size 512M --snapshot --name snap
>>> /dev/testdbs/template
>>> �� Logical volume "snap" created.
>>> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata
>>> 28916maxresident)k
>>> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps
>>
>> That's about half a second in real time. But I have a scenario that
>> would benefit from it being even faster. I'm doing many small unit tests
>> starting from a template filesystem. I do the snapshot, run the unit
>> test on the snapshot, then delete the snapshot afterwards using
>> lvremove. Each unit test, though, takes much less than a second to run
>> (often on the order of 10ms), so most of the time is being spent making
>> these snapshots.
>>
>> So, is there a sort of "dumber" way of making these snapshots, maybe by
>> changing the allocation algorithm or something?
> 
> Hi
> 
> IMHO - what takes most of the time are these couple things:
> 
> Each command has 'non-trivial' time overhead on its startup (scanning
> you system with devices and validating everything)
> 
> For old snapshots -� COW are needs to be 'created' & 'zeroed' as
> separate LV.
> 
> Then you need to 'flush' all existing IO on origin device (so it's in
> the consistent states - i.e. the filesystem synchronizes all it's
> content in its metadata) - this all takes some measurable amount of time.
> 
> You can 'prepare' empty zeroed LV ahead of time and then use
> 'lvconvert' to attach snapshot (with -Zn)� - this should speed-up
> attachment
> of snapshot. For the 'second' point you could likely issue 'sync' ahead
> of time so most buffers will be flushed (if there is no big IO traffic).
> 
> Saying all this - why are you using old snapshot when you are targeting
> for performance ??
> 
> You really should consider usage of thin-pool - where you could chain a
> long series of snapshot without having a dramatic performance
> degradation of the whole IO throughput� - old snapshot are really meant
> to be used only if you want to take i.e. backup of a filesystem and you
> need some 'consistent' point in time - for everything else you should be
> using thin-pools nowdays...
> 
> Regards
> 
> Zdenek
> 

Re: [linux-lvm] faster snapshot creation?

[Zdenek Kabelac]

Dne 25. 02. 20 v 22:35 Eric Toombs napsal(a):
> I tried thin pools and got the same snapshot creation time, about .45s.
> 

As it has been said - lvm2  is NOT only sending couple ioctl() to create snapshot.

Every lvm2 command essentially does a full system validation with respect to 
currently configured set of metadata - this exec-time  by faaaaaar exceeds
the actual snapshot creation time.
(Also snapshot monitoring is pretty 'time expensive' operation on its own)


> I also tried creating the entire vg in memory with losetup and that
> didn't change the creation time either.

lvm2 was never optimized towards 'hyper fast' manipulation with a single 
device - after all you are impacting your system resources in a major way - 
thus couple milliseconds here and there for majority of users doesn't really 
matter.

What is the most important in terms of speed is the 'minimal' delay which can 
be observed on origin being in use (aka 'suspend-resume' time-frame of origin 
takes short amount of time to block actual usage of the origin device - user 
should not see big latency...)

If you are aspiring of creating tens of snapshot per second - you probably 
have very unusual workflow/requirement and mostly likely lvm2 is not the right 
tool for such task ATM.

There used to exist 'slight' optimization where lvm2 executed as 'lvm shell' 
(so you basically stream lvm command though a pipe to such lvmshell) was 
'sharing' some cached content between runs of individual commands - but this 
was mostly lost and most likely would not squeeze a lot the time you refer.

So my advice - if you really need to use and create old snaps in very fast way 
is to developed your very own tool working in you restricted environment where 
you  might probably not care about meta/data consistencies, deal with speed of 
udev and gazillion other issues - you can run your small 'ioctl()' stream much 
more efficiently.

On the other hand it's well beyond my imagination what is good for 'very fast' 
creation of a snapshot when the overall performance of the system is then 
degraded by 50% or more...  but user can choice...


Zdenek

Re: [linux-lvm] faster snapshot creation?

[Douglas Paul]

On Wed, Feb 26, 2020 at 11:01:26AM +0100, Zdenek Kabelac wrote:
> If you are aspiring of creating tens of snapshot per second - you probably 
> have very unusual workflow/requirement and mostly likely lvm2 is not the right 
> tool for such task ATM.
> [...] 
> So my advice - if you really need to use and create old snaps in very fast way 
> is to developed your very own tool working in you restricted environment where 
> you  might probably not care about meta/data consistencies, deal with speed of 
> udev and gazillion other issues - you can run your small 'ioctl()' stream much 
> more efficiently.

For this workflow, it seems that using qemu-img in conjunction with
qemu-nbd might be a good option.

In this case, you build your reference image in a qcow image (as might
be used for a VM). For a test, you then create a temporary image based
on this reference image (using it as a backing file). Then you can mount
this new image using qemu-nbd to get a block device for it, and mount
the filesystem.

After your test is done, you can unmount, delete the nbd mapping, and
simply delete the temporary image you created.

It should be quite fast and lightweight. You should even be able to run
multiple tests in parallel.

-- 
Douglas Paul

Re: [linux-lvm] faster snapshot creation?

[Stuart D. Gathman]

On Sat, 22 Feb 2020, Eric Toombs wrote:

> Snapshot creation is already pretty fast:
>
>> $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template
>>   Logical volume "snap" created.
>> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k
>> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps
>
> That's about half a second in real time. But I have a scenario that
> would benefit from it being even faster. I'm doing many small unit tests
>
> So, is there a sort of "dumber" way of making these snapshots, maybe by
> changing the allocation algorithm or something?

How about using a filesystem that supports snapshot, e.g. nilfs, or
(I think) btrfs?  That would be much faster than doing it at the LVM
level, which has to sync metadata and stuff.

a) load your template into work directory
b) tag snapshot
c) run test (possibly in container)
d) restore tagged snapshot
e) goto c