Re: [Qemu-devel] [RFC 1/1] qcow2: add ZSTD compression feature

From: "Denis V. Lunev" <den@openvz.org>
To: Kevin Wolf <kwolf@redhat.com>
Cc: Eric Blake <eblake@redhat.com>,
	qemu-devel@nongnu.org, Fam Zheng <famz@redhat.com>,
	Stefan Hajnoczi <stefanha@redhat.com>,
	Max Reitz <mreitz@redhat.com>
Subject: Re: [Qemu-devel] [RFC 1/1] qcow2: add ZSTD compression feature
Date: Fri, 24 Mar 2017 09:24:21 +0300	[thread overview]
Message-ID: <37f96fb3-a132-5fba-80a2-ccb0eb8f81cf@openvz.org> (raw)
In-Reply-To: <20170323212001.GE28118@noname.redhat.com>

On 03/24/2017 12:20 AM, Kevin Wolf wrote:
> Am 23.03.2017 um 16:35 hat Denis V. Lunev geschrieben:
>> On 03/23/2017 06:04 PM, Kevin Wolf wrote:
>>> Am 23.03.2017 um 15:17 hat Eric Blake geschrieben:
>>>> On 03/23/2017 08:28 AM, Denis V. Lunev wrote:
>>>>> ZSDT compression algorithm consumes 3-5 times less CPU power with a
>>>> s/ZSDT/ZSTD/
>>>>
>>>>> comparable comression ratio with zlib. It would be wise to use it for
>>>> s/comression/compression/
>>>>
>>>>> data compression f.e. for backups.
>>> Note that we don't really care that much about fast compression because
>>> that's an one time offline operation. Maybe a better compression ratio
>>> while maintaining decent decompression performance would be the more
>>> important feature?
>>>
>>> Or are you planning to extend the qcow2 driver so that compressed
>>> clusters are used even for writes after the initial conversion? I think
>>> it would be doable, and then I can see that better compression speed
>>> becomes important, too.
>> we should care about backups :) they can be done using compression
>> event right now and this is done in real time when VM is online.
>> Thus any additional CPU overhead counts, even if compressed data is
>> written only once.
> Good point. I have no idea about ZSTD, but maybe compression speed vs.
> ratio can even be configurable?
>
> Anyway, I was mostly trying to get people to discuss the compression
> algorithm. I'm not against this one, but I haven't checked whether it's
> the best option for our case.
>
> So I'd be interested in which algorithms you considered, and what was
> the reason to decide for ZSTD?

Actually I was a bit lazy here and followed result of investigations of
my friends. Anyway, here is a good comparison:

http://fastcompression.blogspot.ru/2015/01/zstd-stronger-compression-algorithm.html

Name            Ratio  C.speed D.speed
                        MB/s    MB/s
zlib 1.2.8 -6   3.099     18     275
*zstd            2.872    201     498*
zlib 1.2.8 -1   2.730     58     250
LZ4 HC r127     2.720     26    1720
QuickLZ 1.5.1b6 2.237    323     373
LZO 2.06        2.106    351     510
Snappy 1.1.0    2.091    238     964
LZ4 r127        2.084    370    1590
LZF 3.6         2.077    220     502

I have validated lines 1 and 2 from this table and obtained same results.
2.87 and 3.01 compression ratios are quite nearby while the speed is
MUCH different.

Den