* [linux-lvm] faster snapshot creation?
@ 2020-02-22 11:58 Eric Toombs
2020-02-25 9:31 ` Gionatan Danti
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Eric Toombs @ 2020-02-22 11:58 UTC (permalink / raw)
To: linux-lvm
Snapshot creation is already pretty fast:
> $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template
> Logical volume "snap" created.
> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k
> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps
That's about half a second in real time. But I have a scenario that
would benefit from it being even faster. I'm doing many small unit tests
starting from a template filesystem. I do the snapshot, run the unit
test on the snapshot, then delete the snapshot afterwards using
lvremove. Each unit test, though, takes much less than a second to run
(often on the order of 10ms), so most of the time is being spent making
these snapshots.
So, is there a sort of "dumber" way of making these snapshots, maybe by
changing the allocation algorithm or something?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linux-lvm] faster snapshot creation?
2020-02-22 11:58 [linux-lvm] faster snapshot creation? Eric Toombs
@ 2020-02-25 9:31 ` Gionatan Danti
2020-02-25 9:38 ` Zdenek Kabelac
2020-02-26 0:38 ` Stuart D. Gathman
2 siblings, 0 replies; 7+ messages in thread
From: Gionatan Danti @ 2020-02-25 9:31 UTC (permalink / raw)
To: LVM general discussion and development; +Cc: Eric Toombs
Il 2020-02-22 12:58 Eric Toombs ha scritto:
> So, is there a sort of "dumber" way of making these snapshots, maybe by
> changing the allocation algorithm or something?
Hi, I think that total snapshot creation time is dominated by LVM
flushing its (meta)data to the physical disks. Two things to try:
- use thin lvm snapshot (see man lvmthin);
- create a pv/vg/lv from a memory-backed loop device (ie: truncate
--size=1G /dev/shm/test.img; losetup -f /dev/shm/test.img; pvcreate
/dev/loop0 and so on...)
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it [1]
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linux-lvm] faster snapshot creation?
2020-02-22 11:58 [linux-lvm] faster snapshot creation? Eric Toombs
2020-02-25 9:31 ` Gionatan Danti
@ 2020-02-25 9:38 ` Zdenek Kabelac
2020-02-25 21:35 ` Eric Toombs
2020-02-26 0:38 ` Stuart D. Gathman
2 siblings, 1 reply; 7+ messages in thread
From: Zdenek Kabelac @ 2020-02-25 9:38 UTC (permalink / raw)
To: LVM general discussion and development, Eric Toombs
Dne 22. 02. 20 v 12:58 Eric Toombs napsal(a):
> Snapshot creation is already pretty fast:
>
>> $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template
>> Logical volume "snap" created.
>> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k
>> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps
>
> That's about half a second in real time. But I have a scenario that
> would benefit from it being even faster. I'm doing many small unit tests
> starting from a template filesystem. I do the snapshot, run the unit
> test on the snapshot, then delete the snapshot afterwards using
> lvremove. Each unit test, though, takes much less than a second to run
> (often on the order of 10ms), so most of the time is being spent making
> these snapshots.
>
> So, is there a sort of "dumber" way of making these snapshots, maybe by
> changing the allocation algorithm or something?
Hi
IMHO - what takes most of the time are these couple things:
Each command has 'non-trivial' time overhead on its startup (scanning you
system with devices and validating everything)
For old snapshots - COW are needs to be 'created' & 'zeroed' as separate LV.
Then you need to 'flush' all existing IO on origin device (so it's in the
consistent states - i.e. the filesystem synchronizes all it's content in its
metadata) - this all takes some measurable amount of time.
You can 'prepare' empty zeroed LV ahead of time and then use
'lvconvert' to attach snapshot (with -Zn) - this should speed-up attachment
of snapshot. For the 'second' point you could likely issue 'sync' ahead of
time so most buffers will be flushed (if there is no big IO traffic).
Saying all this - why are you using old snapshot when you are targeting for
performance ??
You really should consider usage of thin-pool - where you could chain a long
series of snapshot without having a dramatic performance degradation of the
whole IO throughput - old snapshot are really meant to be used only if you
want to take i.e. backup of a filesystem and you need some 'consistent' point
in time - for everything else you should be using thin-pools nowdays...
Regards
Zdenek
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linux-lvm] faster snapshot creation?
2020-02-25 9:38 ` Zdenek Kabelac
@ 2020-02-25 21:35 ` Eric Toombs
2020-02-26 10:01 ` Zdenek Kabelac
0 siblings, 1 reply; 7+ messages in thread
From: Eric Toombs @ 2020-02-25 21:35 UTC (permalink / raw)
To: Zdenek Kabelac, LVM general discussion and development
I tried thin pools and got the same snapshot creation time, about .45s.
I also tried creating the entire vg in memory with losetup and that
didn't change the creation time either.
On 2020-02-25 04:38, Zdenek Kabelac wrote:
> Dne 22. 02. 20 v 12:58 Eric Toombs napsal(a):
>> Snapshot creation is already pretty fast:
>>
>>> $ time sudo lvcreate --size 512M --snapshot --name snap
>>> /dev/testdbs/template
>>> �� Logical volume "snap" created.
>>> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata
>>> 28916maxresident)k
>>> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps
>>
>> That's about half a second in real time. But I have a scenario that
>> would benefit from it being even faster. I'm doing many small unit tests
>> starting from a template filesystem. I do the snapshot, run the unit
>> test on the snapshot, then delete the snapshot afterwards using
>> lvremove. Each unit test, though, takes much less than a second to run
>> (often on the order of 10ms), so most of the time is being spent making
>> these snapshots.
>>
>> So, is there a sort of "dumber" way of making these snapshots, maybe by
>> changing the allocation algorithm or something?
>
> Hi
>
> IMHO - what takes most of the time are these couple things:
>
> Each command has 'non-trivial' time overhead on its startup (scanning
> you system with devices and validating everything)
>
> For old snapshots -� COW are needs to be 'created' & 'zeroed' as
> separate LV.
>
> Then you need to 'flush' all existing IO on origin device (so it's in
> the consistent states - i.e. the filesystem synchronizes all it's
> content in its metadata) - this all takes some measurable amount of time.
>
> You can 'prepare' empty zeroed LV ahead of time and then use
> 'lvconvert' to attach snapshot (with -Zn)� - this should speed-up
> attachment
> of snapshot. For the 'second' point you could likely issue 'sync' ahead
> of time so most buffers will be flushed (if there is no big IO traffic).
>
> Saying all this - why are you using old snapshot when you are targeting
> for performance ??
>
> You really should consider usage of thin-pool - where you could chain a
> long series of snapshot without having a dramatic performance
> degradation of the whole IO throughput� - old snapshot are really meant
> to be used only if you want to take i.e. backup of a filesystem and you
> need some 'consistent' point in time - for everything else you should be
> using thin-pools nowdays...
>
> Regards
>
> Zdenek
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linux-lvm] faster snapshot creation?
2020-02-25 21:35 ` Eric Toombs
@ 2020-02-26 10:01 ` Zdenek Kabelac
2020-02-26 14:03 ` Douglas Paul
0 siblings, 1 reply; 7+ messages in thread
From: Zdenek Kabelac @ 2020-02-26 10:01 UTC (permalink / raw)
To: Eric Toombs, LVM general discussion and development
Dne 25. 02. 20 v 22:35 Eric Toombs napsal(a):
> I tried thin pools and got the same snapshot creation time, about .45s.
>
As it has been said - lvm2 is NOT only sending couple ioctl() to create snapshot.
Every lvm2 command essentially does a full system validation with respect to
currently configured set of metadata - this exec-time by faaaaaar exceeds
the actual snapshot creation time.
(Also snapshot monitoring is pretty 'time expensive' operation on its own)
> I also tried creating the entire vg in memory with losetup and that
> didn't change the creation time either.
lvm2 was never optimized towards 'hyper fast' manipulation with a single
device - after all you are impacting your system resources in a major way -
thus couple milliseconds here and there for majority of users doesn't really
matter.
What is the most important in terms of speed is the 'minimal' delay which can
be observed on origin being in use (aka 'suspend-resume' time-frame of origin
takes short amount of time to block actual usage of the origin device - user
should not see big latency...)
If you are aspiring of creating tens of snapshot per second - you probably
have very unusual workflow/requirement and mostly likely lvm2 is not the right
tool for such task ATM.
There used to exist 'slight' optimization where lvm2 executed as 'lvm shell'
(so you basically stream lvm command though a pipe to such lvmshell) was
'sharing' some cached content between runs of individual commands - but this
was mostly lost and most likely would not squeeze a lot the time you refer.
So my advice - if you really need to use and create old snaps in very fast way
is to developed your very own tool working in you restricted environment where
you might probably not care about meta/data consistencies, deal with speed of
udev and gazillion other issues - you can run your small 'ioctl()' stream much
more efficiently.
On the other hand it's well beyond my imagination what is good for 'very fast'
creation of a snapshot when the overall performance of the system is then
degraded by 50% or more... but user can choice...
Zdenek
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linux-lvm] faster snapshot creation?
2020-02-26 10:01 ` Zdenek Kabelac
@ 2020-02-26 14:03 ` Douglas Paul
0 siblings, 0 replies; 7+ messages in thread
From: Douglas Paul @ 2020-02-26 14:03 UTC (permalink / raw)
To: LVM general discussion and development; +Cc: Eric Toombs
On Wed, Feb 26, 2020 at 11:01:26AM +0100, Zdenek Kabelac wrote:
> If you are aspiring of creating tens of snapshot per second - you probably
> have very unusual workflow/requirement and mostly likely lvm2 is not the right
> tool for such task ATM.
> [...]
> So my advice - if you really need to use and create old snaps in very fast way
> is to developed your very own tool working in you restricted environment where
> you might probably not care about meta/data consistencies, deal with speed of
> udev and gazillion other issues - you can run your small 'ioctl()' stream much
> more efficiently.
For this workflow, it seems that using qemu-img in conjunction with
qemu-nbd might be a good option.
In this case, you build your reference image in a qcow image (as might
be used for a VM). For a test, you then create a temporary image based
on this reference image (using it as a backing file). Then you can mount
this new image using qemu-nbd to get a block device for it, and mount
the filesystem.
After your test is done, you can unmount, delete the nbd mapping, and
simply delete the temporary image you created.
It should be quite fast and lightweight. You should even be able to run
multiple tests in parallel.
--
Douglas Paul
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linux-lvm] faster snapshot creation?
2020-02-22 11:58 [linux-lvm] faster snapshot creation? Eric Toombs
2020-02-25 9:31 ` Gionatan Danti
2020-02-25 9:38 ` Zdenek Kabelac
@ 2020-02-26 0:38 ` Stuart D. Gathman
2 siblings, 0 replies; 7+ messages in thread
From: Stuart D. Gathman @ 2020-02-26 0:38 UTC (permalink / raw)
To: LVM general discussion and development
On Sat, 22 Feb 2020, Eric Toombs wrote:
> Snapshot creation is already pretty fast:
>
>> $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template
>> Logical volume "snap" created.
>> 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k
>> 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps
>
> That's about half a second in real time. But I have a scenario that
> would benefit from it being even faster. I'm doing many small unit tests
>
> So, is there a sort of "dumber" way of making these snapshots, maybe by
> changing the allocation algorithm or something?
How about using a filesystem that supports snapshot, e.g. nilfs, or
(I think) btrfs? That would be much faster than doing it at the LVM
level, which has to sync metadata and stuff.
a) load your template into work directory
b) tag snapshot
c) run test (possibly in container)
d) restore tagged snapshot
e) goto c
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-02-26 14:21 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-22 11:58 [linux-lvm] faster snapshot creation? Eric Toombs
2020-02-25 9:31 ` Gionatan Danti
2020-02-25 9:38 ` Zdenek Kabelac
2020-02-25 21:35 ` Eric Toombs
2020-02-26 10:01 ` Zdenek Kabelac
2020-02-26 14:03 ` Douglas Paul
2020-02-26 0:38 ` Stuart D. Gathman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).