[linux-lvm] Fast thin volume preallocation?

[linux-lvm] Fast thin volume preallocation?

[Gionatan Danti]

Hi all,
doing some tests on a 4-bays, entry-level NAS/SAN system, I discovered 
it is entirely based on lvm thin volumes.

On configuring what it calls "thick volumes" it create a new thin 
logical volume and pre-allocates all space inside the new volume.

What surprised me is the speed at which this allocation happens: a 2 TB 
volume was allocated (ie: all data chunks were touched) in about 2 min. 
This immediately exclude any simple zeroing of the volume, which will 
require much more time.

I tried on a regular CentOS box + lvmthin with lvm zeroing disabled and, 
indeed, allocating all blocks inside a thin volume requires much more 
time. I tried both a very simple "dd if=/dev/zero of=/dev/test/thinvol 
bs=1M oflag=direct" and "blkdiscard -z /dev/test/thinvol".

Being curious, I found that the NAS use a binary [1] that seems to 
create a pattern of null writes with extremely high queue depth [2].

So, my questions:
- for what you know, are commercial NAS using some patched/custom 
lvmthin version which enables fast volume preallocation, or early zero 
rejection?
- does standard lvmthin support something similar? If not, how do you 
see a zero coalesce/compression/trim/whatever feature?
- can I obtain something similar by simply touching (maybe with a 512B 
only write) once each thin chunk?

Thanks.

[1] I am not calling it because I don't know if discovering the binary, 
and so the NAS vendor, is contrary to this mailing list policy.

[2] iostat -x 1 produces the following example output. Please see how 
*no* writes are passed on backing devices:

    extended device statistics
device mgr/s mgw/s    r/s    w/s    kr/s    kw/s   size queue   wait 
svc_t  %b
sda        0     0  105.2    0.0   841.3     0.0    8.0   0.0    0.2 
0.2   2
sdb        0     0   62.9    0.0   503.2     0.0    8.0   0.0    0.2 
0.2   1
sdd        0     0  124.8    0.0   998.5     0.0    8.0   0.0    0.0 
0.0   0
sdc        0     0  119.9    0.0   959.2     0.0    8.0   0.0    0.2 
0.2   2
mtdblock0     0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
  0.0   0
mtdblock1     0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
  0.0   0
mtdblock2     0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
  0.0   0
mtdblock3     0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
  0.0   0
mtdblock4     0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
  0.0   0
mtdblock5     0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
  0.0   0
mtdblock6     0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
  0.0   0
md9        0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
md13       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
md256      0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
md322      0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
md1        0     0  412.8    0.0  3302.2     0.0    8.0   0.0    0.0 
0.0   0
dm-1       0     0  412.8    0.0  3302.2     0.0    8.0   0.0    0.1 
0.1   5
dm-2       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
dm-3       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
dm-4       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
dm-5       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
dm-6       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
dm-7       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0
dm-8       0     0    0.0    0.0     0.0     0.0    0.0   1.0    0.0 
0.0  99
dm-0       0     0    0.0    0.0     0.0     0.0    0.0   1.0    0.0 
0.0  99
dm-9       0     0    0.0    0.0     0.0     0.0    0.0 345966.5    0.0 
  0.0  99
dm-10      0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 
0.0   0

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Fast thin volume preallocation?

[Joe Thornber]

On Fri, May 31, 2019 at 03:13:41PM +0200, Gionatan Danti wrote:

> - does standard lvmthin support something similar? If not, how do you see a
> zero coalesce/compression/trim/whatever feature?

There isn't such a feature as yet.  With the next iteration of thin I'd like to
get away from the fixed allocation block sizes that we're using which should greatly
reduce the number of mappings that we need to create to provision a thick volume and
so speed it up a lot.

Presumably you want a thick volume but inside a thin pool so that you can used snapshots?
If so have you considered the 'external snapshot' feature?

> - can I obtain something similar by simply touching (maybe with a 512B only
> write) once each thin chunk?

Yes.

- Joe

Re: [linux-lvm] Fast thin volume preallocation?

[Gionatan Danti]

Il 03-06-2019 15:23 Joe Thornber ha scritto:
> On Fri, May 31, 2019 at 03:13:41PM +0200, Gionatan Danti wrote:
> 
>> - does standard lvmthin support something similar? If not, how do you 
>> see a
>> zero coalesce/compression/trim/whatever feature?
> 
> There isn't such a feature as yet.

Ok, so the NAS vendor did something custom/proprietary. This is a reason 
why I do *not* like such devices: how much testing and verification are 
done before and after such customization? Not a lot, I would say.

> With the next iteration of thin I'd like to
> get away from the fixed allocation block sizes that we're using which
> should greatly
> reduce the number of mappings that we need to create to provision a
> thick volume and
> so speed it up a lot.

Interesting. Do you plan to have an dynamic chunk size bound to some 
minumum/maximum? If so, how do you plan to tackle situations where a 
single big chunk should be split due to a small CoW write happening 
inside the big chunk? Example: a 128MB chunk-sized thin LV which is 
snapshotted and a small 4K writes happening at offset 32M from chunk 
start.

> Presumably you want a thick volume but inside a thin pool so that you
> can used snapshots?
> If so have you considered the 'external snapshot' feature?

Yes, in some cases they are quite useful. Still, a fast volume 
allocation can be an handy addition.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Fast thin volume preallocation?

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

> 
>> Presumably you want a thick volume but inside a thin pool so that you
>> can used snapshots?
>> If so have you considered the 'external snapshot' feature?
> 
> Yes, in some cases they are quite useful. Still, a fast volume
> allocation can be an handy addition.
> 

Hello.
Can I use external snapshot for fast zero allocation?
 "thpool" - is lvmthin with lvm zeroing disabled

# lvcreate -n ext2T -V 2TiB --thinpool thpool VG
# lvchange --permission r VG/ext2T

# lvcreate -n zeroed_lve -s VG/ext2T --thinpool VG/thpool

Or it will be the same as zeroing enabled?
Thanks.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3599 bytes --]

Re: [linux-lvm] Fast thin volume preallocation?

[Ilia Zykov]

>> Presumably you want a thick volume but inside a thin pool so that you
>> can used snapshots?
>> If so have you considered the 'external snapshot' feature?
> 
> Yes, in some cases they are quite useful. Still, a fast volume
> allocation can be an handy addition.
> 

Hello.
Can I use external snapshot for fast zero allocation?
 "thpool" - is lvmthin with lvm zeroing disabled

# lvcreate -n ext2T -V 2TiB --thinpool thpool VG
# lvchange --permission r VG/ext2T

# lvcreate -n zeroed_lve -s VG/ext2T --thinpool VG/thpool

Or it will be the same as zeroing enabled?
Thanks.

----

Re: [linux-lvm] Fast thin volume preallocation?

[Zdenek Kabelac]

Dne 03. 06. 19 v 23:12 Ilia Zykov napsal(a):
>>
>>> Presumably you want a thick volume but inside a thin pool so that you
>>> can used snapshots?
>>> If so have you considered the 'external snapshot' feature?
>>
>> Yes, in some cases they are quite useful. Still, a fast volume
>> allocation can be an handy addition.
>>
> 
> Hello.
> Can I use external snapshot for fast zero allocation?
>   "thpool" - is lvmthin with lvm zeroing disabled
> 
> # lvcreate -n ext2T -V 2TiB --thinpool thpool VG
> # lvchange --permission r VG/ext2T
> 
> # lvcreate -n zeroed_lve -s VG/ext2T --thinpool VG/thpool
> 
> Or it will be the same as zeroing enabled?


Hi

It's not clear what do you want to achieve.

External origin for thin-volume is always a read-only LV.

External origin can be of any type - i.e. even thinLV from another thin-pool
and single origin can be used for number of thin volumes with external origin.

For now you CANNOT 'merge' such thinLV with external origin - so the only way
to get 'writable' origin back is to drop all thins that are using it as 
external origin.

Now - how the 'zeroing' plays the role here ?

Zeroing is property of thin-pool that serves thinLV.
Unprovisioned blocks always do return 'zeroes'.

When block is provisioned - unwritten portions needs to be zeroed (depending 
on zeroing setting)  - the bigger the block the bigger the chance the zeroing 
will take more time - lvm2 does not recommend using thin-pool with zeroing and 
chunksize bigger then 512KB.

Most modern filesystem keep track of written pieces of a block device 
themselves, so zeroing has almost no practical use.

Regards

Zdenek