[linux-lvm] Looking ahead - tiering with LVM?

[linux-lvm] Looking ahead - tiering with LVM?

[Roy Sigurd Karlsbakk]

Hi all

I just wonder how it could be possible some day, some year, to make lvm use tiering. I guess this has been debated numerous times before and I found this lvmts project, but it hasn't been updated for eight years or so.

To detail, and I apologize if this is known stuff. I just want to underline what I'm asking for here

We have lvmcache today and it works well. Warm data moves to SSDs or similar to cache what's hot. This is nice, but all writes will eventually go to the "slow" storage in the back. With tiering, such as that used by things like Dell Compellent and suchlike, all writes go to the fast storage, given there is room, or the logical volume is flagged for only to be used with slow storage. Then, in the night or whenever there's little traffic, a batch job runs and moves the cold-ish data from the fast(er) levels to a slow(er) level based on atime/mtime/something, per block, not file. I guess this will be analogous to extents in the terms of LVM (IIRC Compellent uses 8MB blocks, but that's not really relevant). This goes on, if some data on lower (slower) levels become popular for some reason, they are elevated to a higher (faster) level at runtime.

As far as I can understand (that is, without having read the code, it's beyond me) it seems like LVM has most of this already. Perhaps not timestamps per block (which will take up a wee bit of space and I/O), but still multi-PV storage in a single VG.

So - what would it really take to move LVM to something like a working tiered storage platform?

Kind regards

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Gionatan Danti]

Il 2020-09-02 20:38 Roy Sigurd Karlsbakk ha scritto:
> Hi all
> 
> I just wonder how it could be possible some day, some year, to make
> lvm use tiering. I guess this has been debated numerous times before
> and I found this lvmts project, but it hasn't been updated for eight
> years or so.

Hi, having developed and supported file-level form of tiered storage in 
response to a specific customer request, I have the feeling that tiered 
storage (both file and block based) is not so useful as it seems. Let me 
explain why I feel so...

The key difference between caching and tiering is that the former does 
not increase total available space, while the latter provides as much 
space as available in each storage tier. For example, 1 TB SSD + 10 TB 
HDD can be combined for a 11 TB tiered volume.

Tiering is useful when the faster volume provides a significant portion 
of the total aggregated sum - which is often not the case. In the 
example above, the SSD only provides a 10% space increase over plain 
caching. You can argue that one can simple enlarge the performane tier, 
for example using a 4 TB SSD + 10 TB HDD, but you are now in the 
ballpark of affording a full-SSD volume - ditching *both* tiering and 
caching.

That said, LVM already provides the basic building block to provide 
tiering as you can pvmove between block devices. The difficult thing is 
how to determine which block to move, and managing them in an automated 
way.

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Roy Sigurd Karlsbakk]

>> I just wonder how it could be possible some day, some year, to make
>> lvm use tiering. I guess this has been debated numerous times before
>> and I found this lvmts project, but it hasn't been updated for eight
>> years or so.
> 
> Hi, having developed and supported file-level form of tiered storage in
> response to a specific customer request, I have the feeling that tiered
> storage (both file and block based) is not so useful as it seems. Let me
> explain why I feel so...

First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal storage in the same areas, but the file size and content will vary over time. With blocklevel tiering, that could work better.

> The key difference between caching and tiering is that the former does
> not increase total available space, while the latter provides as much
> space as available in each storage tier. For example, 1 TB SSD + 10 TB
> HDD can be combined for a 11 TB tiered volume.

This is all known.

> Tiering is useful when the faster volume provides a significant portion
> of the total aggregated sum - which is often not the case. In the
> example above, the SSD only provides a 10% space increase over plain
> caching. You can argue that one can simple enlarge the performane tier,
> for example using a 4 TB SSD + 10 TB HDD, but you are now in the
> ballpark of affording a full-SSD volume - ditching *both* tiering and
> caching.

If you look at IOPS instead of just sequencial speed, you'll see the difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give you 1kIOPS, while a single SSD might give you 50kIOPS or even more. This makes a huge impact.

> That said, LVM already provides the basic building block to provide
> tiering as you can pvmove between block devices. The difficult thing is
> how to determine which block to move, and managing them in an automated
> way.

…which was the reason I asked this question, and which should be quite clear in the original post.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Gionatan Danti]

Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
> First, filelevel is usually useless. Say you have 50 VMs with Windows
> server something. A lot of them are bound to have a ton of equal
> storage in the same areas, but the file size and content will vary
> over time. With blocklevel tiering, that could work better.

It really depends on the use case. I applied it to a fileserver, so 
working at file level was the right choice. For VMs (or big files) it is 
useless, I agree.

> This is all known.

But the only reason to want tiering vs cache is the additional space the 
former provides. If this additional space is so small (compared to the 
combined, total volume space), tiering's advantage shrinks to (almost) 
nothing.

> If you look at IOPS instead of just sequencial speed, you'll see the
> difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give
> you 1kIOPS, while a single SSD might give you 50kIOPS or even more.
> This makes a huge impact.

IOPs are already well server by LVM cache. So, I genuinely ask: what 
would be tiering advantage here? I'll love to ear a reasonable use case.

> …which was the reason I asked this question, and which should be quite
> clear in the original post.

Yeah, but this need a direct reply from a core LVM developer, which I 
wellcome ;)
Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Looking ahead - tiering with LVM?

[John Stoffel]

>>>>> "Gionatan" == Gionatan Danti <g.danti@assyoma.it> writes:

Gionatan> Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
>> First, filelevel is usually useless. Say you have 50 VMs with Windows
>> server something. A lot of them are bound to have a ton of equal
>> storage in the same areas, but the file size and content will vary
>> over time. With blocklevel tiering, that could work better.

Gionatan> It really depends on the use case. I applied it to a
Gionatan> fileserver, so working at file level was the right
Gionatan> choice. For VMs (or big files) it is useless, I agree.

This assumes you're tiering whole files, not at the per-block level
though, right? 

>> This is all known.

Gionatan> But the only reason to want tiering vs cache is the
Gionatan> additional space the former provides. If this additional
Gionatan> space is so small (compared to the combined, total volume
Gionatan> space), tiering's advantage shrinks to (almost) nothing.

Do you have numbers?  I'm using DM_CACHE on my home NAS server box,
and it *does* seem to help, but only in certain cases.   I've got a
750gb home directory LV with an 80gb lv_cache writethrough cache
setup.  So it's not great on write heavy loads, but it's good in read
heavy ones, such as kernel compiles where it does make a difference.

So it's not only the caching being per-file or per-block, but how the
actual cache is done?  writeback is faster, but less reliable if you
crash.  Writethrough is slower, but much more reliable.  

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Zdenek Kabelac]

Dne 09. 09. 20 v 20:47 John Stoffel napsal(a):
>>>>>> "Gionatan" == Gionatan Danti <g.danti@assyoma.it> writes:
> 
> Gionatan> Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
>>> First, filelevel is usually useless. Say you have 50 VMs with Windows
>>> server something. A lot of them are bound to have a ton of equal
>>> storage in the same areas, but the file size and content will vary
>>> over time. With blocklevel tiering, that could work better.
> 
> Gionatan> It really depends on the use case. I applied it to a
> Gionatan> fileserver, so working at file level was the right
> Gionatan> choice. For VMs (or big files) it is useless, I agree.
> 
> This assumes you're tiering whole files, not at the per-block level
> though, right?
> 
>>> This is all known.
> 
> Gionatan> But the only reason to want tiering vs cache is the
> Gionatan> additional space the former provides. If this additional
> Gionatan> space is so small (compared to the combined, total volume
> Gionatan> space), tiering's advantage shrinks to (almost) nothing.
> 
> Do you have numbers?  I'm using DM_CACHE on my home NAS server box,
> and it *does* seem to help, but only in certain cases.   I've got a
> 750gb home directory LV with an 80gb lv_cache writethrough cache
> setup.  So it's not great on write heavy loads, but it's good in read
> heavy ones, such as kernel compiles where it does make a difference.
> 
> So it's not only the caching being per-file or per-block, but how the
> actual cache is done?  writeback is faster, but less reliable if you
> crash.  Writethrough is slower, but much more reliable.

Hi

dm-cache (--type cache) is  hotspot cache (most used areas of device)

dm-writecache (--type writecache) is great with write-extensive load (somewhat 
extends your page cache on your NMVe/SSD/persistent-memory)

We were thinking about layering cached above each other - but so far there
was no big demand and also the complexity of solving problem is rising greatly 
-  aka there is no problem to let users to stack cache on top of another cache
on top of 3rd. cache - but what should have when it starts failing...

AFAIK there is no one yet writing driver for combining i.e. SSD + HDD
into a single drive which would be relocating blocks (so you get total size as 
aproximate sum of both devices) - but there is dm-zoned  which solves somewhat 
similar problem - but I've no experience with that...

Zdenek

Re: [linux-lvm] Looking ahead - tiering with LVM?

[John Stoffel]

>>>>> "Zdenek" == Zdenek Kabelac <zkabelac@redhat.com> writes:

Zdenek> Dne 09. 09. 20 v 20:47 John Stoffel napsal(a):
>>>>>>> "Gionatan" == Gionatan Danti <g.danti@assyoma.it> writes:
>> 
Gionatan> Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
>>>> First, filelevel is usually useless. Say you have 50 VMs with Windows
>>>> server something. A lot of them are bound to have a ton of equal
>>>> storage in the same areas, but the file size and content will vary
>>>> over time. With blocklevel tiering, that could work better.
>> 
Gionatan> It really depends on the use case. I applied it to a
Gionatan> fileserver, so working at file level was the right
Gionatan> choice. For VMs (or big files) it is useless, I agree.
>> 
>> This assumes you're tiering whole files, not at the per-block level
>> though, right?
>> 
>>>> This is all known.
>> 
Gionatan> But the only reason to want tiering vs cache is the
Gionatan> additional space the former provides. If this additional
Gionatan> space is so small (compared to the combined, total volume
Gionatan> space), tiering's advantage shrinks to (almost) nothing.
>> 
>> Do you have numbers?  I'm using DM_CACHE on my home NAS server box,
>> and it *does* seem to help, but only in certain cases.   I've got a
>> 750gb home directory LV with an 80gb lv_cache writethrough cache
>> setup.  So it's not great on write heavy loads, but it's good in read
>> heavy ones, such as kernel compiles where it does make a difference.
>> 
>> So it's not only the caching being per-file or per-block, but how the
>> actual cache is done?  writeback is faster, but less reliable if you
>> crash.  Writethrough is slower, but much more reliable.

Zdenek> dm-cache (--type cache) is  hotspot cache (most used areas of device)

I assume this is what I'm using on Debian Buster (10.5) right now?  I
use the crufty tool 'lvcache' to look at and manage my cache devices.  

Zdenek> dm-writecache (--type writecache) is great with
Zdenek> write-extensive load (somewhat extends your page cache on your
Zdenek> NMVe/SSD/persistent-memory)

I don't think I'm using this at all:

sudo lvcache status data/home
+-----------------------+------------------+
| Field                 | Value            |
+-----------------------+------------------+
| cached                | True             |
| size                  | 806380109824     |
| cache_lv              | home_cache       |
| cache_lv_size         | 85899345920      |
| metadata_lv           | home_cache_cmeta |
| metadata_lv_size      | 83886080         |
| cache_block_size      | 192              |
| cache_utilization     | 873786/873813    |
| cache_utilization_pct | 99.996910094     |
| demotions             | 43213            |
| dirty                 | 0                |
| end                   | 1574961152       |
| features              | 1                |
| md_block_size         | 8                |
| md_utilization        | 2604/20480       |
| md_utilization_pct    | 12.71484375      |
| promotions            | 43208            |
| read_hits             | 138697828        |
| read_misses           | 7874434          |
| segment_type          | cache            |
| start                 | 0                |
| write_hits            | 777455171        |
| write_misses          | 9841866          |
+-----------------------+------------------+


Zdenek> We were thinking about layering cached above each other - but so far there
Zdenek> was no big demand and also the complexity of solving problem is rising greatly 
Zdenek> -  aka there is no problem to let users to stack cache on top of another cache
Zdenek> on top of 3rd. cache - but what should have when it starts failing...

Zdenek> AFAIK there is no one yet writing driver for combining i.e. SSD + HDD
Zdenek> into a single drive which would be relocating blocks (so you get total size as 
Zdenek> aproximate sum of both devices) - but there is dm-zoned  which solves somewhat 
Zdenek> similar problem - but I've no experience with that...

Zdenek> Zdenek

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Gionatan Danti]

Il 2020-09-09 20:47 John Stoffel ha scritto:
> This assumes you're tiering whole files, not at the per-block level
> though, right?

The tiered approach I developed and maintained in the past, yes. For any 
LVM-based tiering, we are speaking about block-level tiering (as LVM 
itself has no "files" concept).

> Do you have numbers?  I'm using DM_CACHE on my home NAS server box,
> and it *does* seem to help, but only in certain cases.   I've got a
> 750gb home directory LV with an 80gb lv_cache writethrough cache
> setup.  So it's not great on write heavy loads, but it's good in read
> heavy ones, such as kernel compiles where it does make a difference.

Numbers for available space for tiering vs cache can vary based on your 
setup. However, storage tiers generally are at least 5-10X apart from 
each other (ie: 1 TB SSD for 10 TB HDD). Hence my gut fealing that 
tiering is not drastically better then  lvm cache. But hey - I reserve 
the right to be totally wrong ;)

> So it's not only the caching being per-file or per-block, but how the
> actual cache is done?  writeback is faster, but less reliable if you
> crash.  Writethrough is slower, but much more reliable.

writeback cache surely is more prone to failure vs writethoug cache. The 
golden rule is that writeback cache should use a mirrored device (with 
device-level powerloss protected writeback cache if sync write speed is 
important).

But this is somewhat ortogonal to the original question: block-level 
tiering itself increases the chances of data loss (ie: losing the SSD 
component will ruin the entire filesystem), so you should used mirrored 
(or parity) device for tiering also.

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Looking ahead - tiering with LVM?

[John Stoffel]

>>>>> "Gionatan" == Gionatan Danti <g.danti@assyoma.it> writes:

Gionatan> Il 2020-09-09 20:47 John Stoffel ha scritto:
>> This assumes you're tiering whole files, not at the per-block level
>> though, right?

Gionatan> The tiered approach I developed and maintained in the past, yes. For any 
Gionatan> LVM-based tiering, we are speaking about block-level tiering (as LVM 
Gionatan> itself has no "files" concept).

>> Do you have numbers?  I'm using DM_CACHE on my home NAS server box,
>> and it *does* seem to help, but only in certain cases.   I've got a
>> 750gb home directory LV with an 80gb lv_cache writethrough cache
>> setup.  So it's not great on write heavy loads, but it's good in read
>> heavy ones, such as kernel compiles where it does make a difference.

Gionatan> Numbers for available space for tiering vs cache can vary
Gionatan> based on your setup. However, storage tiers generally are at
Gionatan> least 5-10X apart from each other (ie: 1 TB SSD for 10 TB
Gionatan> HDD). Hence my gut fealing that tiering is not drastically
Gionatan> better then lvm cache. But hey - I reserve the right to be
Gionatan> totally wrong ;)

Very true, numbers talk, annecdotes walk... 

>> So it's not only the caching being per-file or per-block, but how the
>> actual cache is done?  writeback is faster, but less reliable if you
>> crash.  Writethrough is slower, but much more reliable.

Gionatan> writeback cache surely is more prone to failure vs
Gionatan> writethoug cache. The golden rule is that writeback cache
Gionatan> should use a mirrored device (with device-level powerloss
Gionatan> protected writeback cache if sync write speed is important).

Even in my case I use mirrored SSDs for my cache LVs.  It's the only
sane thing to do IMHO.

Gionatan> But this is somewhat ortogonal to the original question:
Gionatan> block-level tiering itself increases the chances of data
Gionatan> loss (ie: losing the SSD component will ruin the entire
Gionatan> filesystem), so you should used mirrored (or parity) device
Gionatan> for tiering also.

It does, you really need to have a solid setup in terms of hardware,
with known failures modes you can handle, before you start trying to
tier blocks.

Though maybe a write through block cache would be ok, since the cache
would only be for reads, not writes.  Which could help if you have a
bunch of VMs (or containers, or whatevers) with alot of duplicated
data that are all hitting the disk systems at once.

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Gionatan Danti]

Il 2020-09-09 21:53 John Stoffel ha scritto:
> Very true, numbers talk, annecdotes walk...

Sure - lets try to gather some numbers from the data you posted 
before...

> sudo lvcache status data/home
> +-----------------------+------------------+
> | Field                 | Value            |
> +-----------------------+------------------+
> | cached                | True             |
> | size                  | 806380109824     |
> | cache_lv              | home_cache       |
> | cache_lv_size         | 85899345920      |

You cache device is squarely in the 10x ballpark (ie: it is ~9.39x 
smaller than your SSD). Having ~10.64% more space would be nice, but 
hardly game-changer.

> | read_hits             | 138697828        |
> | read_misses           | 7874434          |

You have 94.6% hit rate from your hotspot cache - compare this to an 
ideally managed tiered storage, with its 100% (ideal, not reasonable in 
real world) hit rate. Does it really change anything?

> | write_hits            | 777455171        |
> | write_misses          | 9841866          |

And you have an even better ~98.7% write hit ratio. As you have a 
mirrored cache device, you should be able to set LVM for using a 
writeback cache without risking your data in case of a single cache 
drive failure. I suspect this would do wonder with your hit ratio - but 
again, testing is the only method to be sure.

Thanks for sharing your data!
Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Roy Sigurd Karlsbakk]

>> First, filelevel is usually useless. Say you have 50 VMs with Windows
>> server something. A lot of them are bound to have a ton of equal
>> If you look at IOPS instead of just sequencial speed, you'll see the
>> difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give
>> you 1kIOPS, while a single SSD might give you 50kIOPS or even more.
>> This makes a huge impact.
> 
> IOPs are already well server by LVM cache. So, I genuinely ask: what
> would be tiering advantage here? I'll love to ear a reasonable use case.

LVMcache only helps if the cache is there in the first place and IIRC it's cleared after a reboot. It help won't that much over time with large storage. It also wastes space.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: [linux-lvm] Looking ahead - tiering with LVM?

[Gionatan Danti]

Il 2020-09-09 21:41 Roy Sigurd Karlsbakk ha scritto:
>>> First, filelevel is usually useless. Say you have 50 VMs with Windows
>>> server something. A lot of them are bound to have a ton of equal
>>> If you look at IOPS instead of just sequencial speed, you'll see the
>>> difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give
>>> you 1kIOPS, while a single SSD might give you 50kIOPS or even more.
>>> This makes a huge impact.
>> 
>> IOPs are already well server by LVM cache. So, I genuinely ask: what
>> would be tiering advantage here? I'll love to ear a reasonable use 
>> case.
> 
> LVMcache only helps if the cache is there in the first place and IIRC
> it's cleared after a reboot.

I seem to remember that cache is persistent, but a writeback cache must 
be flushed to the underlying disk in case of unclean shutdown. This, 
however, should not empty the cache itself.

> It help won't that much over time with large storage. It also wastes 
> space.

I tend to disagree: with large storage, you really want an hotspot cache 
vs a tiered approach unless:
a) the storage tiers are comparable in size, which is quite rare;
b) the slow storage does some sort of offline compression/deduplication, 
with the faster layer being a landing zone for newly ingested data.

Can you describe a reasonable real-world setup where plain LVM tiering 
would be useful? Again, this is a genuine question: I am interested in 
different storage setup than mine.

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8