Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Sage Weil]

On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
> Hello,
> 
> in the lumious release notes is stated that zstd is not supported by
> bluestor due to performance reason. I'm wondering why btrfs instead
> states that zstd is as fast as lz4 but compresses as good as zlib.
> 
> Why is zlib than supported by bluestor? And why does btrfs / facebook
> behave different?
> 
> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
> also supports zstd for RGW compression but zstd is not recommended for
> BlueStore for performance reasons.)"

zstd will work but in our testing the performance wasn't great for 
bluestore in particular.  The problem was that for each compression run 
there is a relatively high start-up cost initializing the zstd 
context/state (IIRC a memset of a huge memory buffer) that dominated the 
execution time... primarily because bluestore is generally compressing 
pretty small chunks of data at a time, not big buffers or streams.

Take a look at unittest_compression timings on compressing 16KB buffers 
(smaller than bluestore needs usually, but illustrated of the problem):

[ RUN      ] Compressor/CompressorTest.compress_16384/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)

It's an order of magnitude slower than zlib or snappy, which probably 
isn't acceptable--even if it is a bit smaller.

We just updated to a newer zstd the other day but I haven't been paying 
attention to the zstd code changes.  When I was working on this the plugin 
was initially also misusing the zstd API, but it was also pointed out 
that the size of the memset is dependent on the compression level.  
Maybe a different (default) choice there woudl help.

https://github.com/facebook/zstd/issues/408#issuecomment-252163241

sage

Re: ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

Hi Sage,

Am 25.10.2017 um 21:54 schrieb Sage Weil:
> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> in the lumious release notes is stated that zstd is not supported by
>> bluestor due to performance reason. I'm wondering why btrfs instead
>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>
>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>> behave different?
>>
>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>> also supports zstd for RGW compression but zstd is not recommended for
>> BlueStore for performance reasons.)"
> 
> zstd will work but in our testing the performance wasn't great for 
> bluestore in particular.  The problem was that for each compression run 
> there is a relatively high start-up cost initializing the zstd 
> context/state (IIRC a memset of a huge memory buffer) that dominated the 
> execution time... primarily because bluestore is generally compressing 
> pretty small chunks of data at a time, not big buffers or streams.
> 
> Take a look at unittest_compression timings on compressing 16KB buffers 
> (smaller than bluestore needs usually, but illustrated of the problem):
> 
> [ RUN      ] Compressor/CompressorTest.compress_16384/0
> [plugin zlib (zlib/isal)]
> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/1
> [plugin zlib (zlib/noisal)]
> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/2
> [plugin snappy (snappy)]
> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/3
> [plugin zstd (zstd)]
> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
> 
> It's an order of magnitude slower than zlib or snappy, which probably 
> isn't acceptable--even if it is a bit smaller.
> 
> We just updated to a newer zstd the other day but I haven't been paying 
> attention to the zstd code changes.  When I was working on this the plugin 
> was initially also misusing the zstd API, but it was also pointed out 
> that the size of the memset is dependent on the compression level.  
> Maybe a different (default) choice there woudl help.
> 
> https://github.com/facebook/zstd/issues/408#issuecomment-252163241

thanks for the fast reply. Btrfs uses a default compression level of 3
but i think this is the default anyway.

Does the zstd plugin of ceph already uses the mentioned
ZSTD_resetCStream instead of creating and initializing a new one every time?

So if performance matters ceph would recommand snappy?

Greets,
Stefan

Re: ceph zstd not for bluestor due to performance reasons

[Haomai Wang]

[-- Attachment #1.1: Type: text/plain, Size: 3287 bytes --]

Stefan Priebe - Profihost AG <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>于2017年10月26日 周四17:06写道：

> Hi Sage,
>
> Am 25.10.2017 um 21:54 schrieb Sage Weil:
> > On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
> >> Hello,
> >>
> >> in the lumious release notes is stated that zstd is not supported by
> >> bluestor due to performance reason. I'm wondering why btrfs instead
> >> states that zstd is as fast as lz4 but compresses as good as zlib.
> >>
> >> Why is zlib than supported by bluestor? And why does btrfs / facebook
> >> behave different?
> >>
> >> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
> >> also supports zstd for RGW compression but zstd is not recommended for
> >> BlueStore for performance reasons.)"
> >
> > zstd will work but in our testing the performance wasn't great for
> > bluestore in particular.  The problem was that for each compression run
> > there is a relatively high start-up cost initializing the zstd
> > context/state (IIRC a memset of a huge memory buffer) that dominated the
> > execution time... primarily because bluestore is generally compressing
> > pretty small chunks of data at a time, not big buffers or streams.
> >
> > Take a look at unittest_compression timings on compressing 16KB buffers
> > (smaller than bluestore needs usually, but illustrated of the problem):
> >
> > [ RUN      ] Compressor/CompressorTest.compress_16384/0
> > [plugin zlib (zlib/isal)]
> > [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
> > [ RUN      ] Compressor/CompressorTest.compress_16384/1
> > [plugin zlib (zlib/noisal)]
> > [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
> > [ RUN      ] Compressor/CompressorTest.compress_16384/2
> > [plugin snappy (sna
> <https://maps.google.com/?q=y+(sna&entry=gmail&source=g>ppy)]
> > [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
> > [ RUN      ] Compressor/CompressorTest.compress_16384/3
> > [plugin zstd (zstd)]
> > [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
> >
> > It's an order of magnitude slower than zlib or snappy, which probably
> > isn't acceptable--even if it is a bit smaller.
> >
> > We just updated to a newer zstd the other day but I haven't been paying
> > attention to the zstd code changes.  When I was working on this the
> plugin
> > was initially also misusing the zstd API, but it was also pointed out
> > that the size of the memset is dependent on the compression level.
> > Maybe a different (default) choice there woudl help.
> >
> > https://github.com/facebook/zstd/issues/408#issuecomment-252163241
>
> thanks for the fast reply. Btrfs uses a default compression level of 3
> but i think this is the default anyway.
>
> Does the zstd plugin of ceph already uses the mentioned
> ZSTD_resetCStream instead of creating and initializing a new one every
> time?
>
> So if performance matters ceph would recommand snappy?
>


in our test, lz4 is better than snappy

>
> Greets,
> Stefan
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 4385 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Sage Weil]

On Thu, 26 Oct 2017, Haomai Wang wrote:
> in our test, lz4 is better than snappy

Let's switch the default then?

sage

Re: ceph zstd not for bluestor due to performance reasons

[Sage Weil]

On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
> Hi Sage,
> 
> Am 25.10.2017 um 21:54 schrieb Sage Weil:
> > On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
> >> Hello,
> >>
> >> in the lumious release notes is stated that zstd is not supported by
> >> bluestor due to performance reason. I'm wondering why btrfs instead
> >> states that zstd is as fast as lz4 but compresses as good as zlib.
> >>
> >> Why is zlib than supported by bluestor? And why does btrfs / facebook
> >> behave different?
> >>
> >> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
> >> also supports zstd for RGW compression but zstd is not recommended for
> >> BlueStore for performance reasons.)"
> > 
> > zstd will work but in our testing the performance wasn't great for 
> > bluestore in particular.  The problem was that for each compression run 
> > there is a relatively high start-up cost initializing the zstd 
> > context/state (IIRC a memset of a huge memory buffer) that dominated the 
> > execution time... primarily because bluestore is generally compressing 
> > pretty small chunks of data at a time, not big buffers or streams.
> > 
> > Take a look at unittest_compression timings on compressing 16KB buffers 
> > (smaller than bluestore needs usually, but illustrated of the problem):
> > 
> > [ RUN      ] Compressor/CompressorTest.compress_16384/0
> > [plugin zlib (zlib/isal)]
> > [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
> > [ RUN      ] Compressor/CompressorTest.compress_16384/1
> > [plugin zlib (zlib/noisal)]
> > [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
> > [ RUN      ] Compressor/CompressorTest.compress_16384/2
> > [plugin snappy (snappy)]
> > [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
> > [ RUN      ] Compressor/CompressorTest.compress_16384/3
> > [plugin zstd (zstd)]
> > [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
> > 
> > It's an order of magnitude slower than zlib or snappy, which probably 
> > isn't acceptable--even if it is a bit smaller.
> > 
> > We just updated to a newer zstd the other day but I haven't been paying 
> > attention to the zstd code changes.  When I was working on this the plugin 
> > was initially also misusing the zstd API, but it was also pointed out 
> > that the size of the memset is dependent on the compression level.  
> > Maybe a different (default) choice there woudl help.
> > 
> > https://github.com/facebook/zstd/issues/408#issuecomment-252163241
> 
> thanks for the fast reply. Btrfs uses a default compression level of 3
> but i think this is the default anyway.
> 
> Does the zstd plugin of ceph already uses the mentioned
> ZSTD_resetCStream instead of creating and initializing a new one every time?

Hmm, it doesn't:

	https://github.com/ceph/ceph/blob/master/src/compressor/zstd/ZstdCompressor.h#L29

but perhaps that was because it didn't make a difference?  Might be worth 
revisiting.

> So if performance matters ceph would recommand snappy?

Yep!

sage

Re: ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

Hi Sage,

Am 26.10.2017 um 13:58 schrieb Sage Weil:
> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hello,
>>>>
>>>> in the lumious release notes is stated that zstd is not supported by
>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>
>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>> behave different?
>>>>
>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>> BlueStore for performance reasons.)"
>>>
>>> zstd will work but in our testing the performance wasn't great for 
>>> bluestore in particular.  The problem was that for each compression run 
>>> there is a relatively high start-up cost initializing the zstd 
>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>> execution time... primarily because bluestore is generally compressing 
>>> pretty small chunks of data at a time, not big buffers or streams.
>>>
>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>
>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>> [plugin zlib (zlib/isal)]
>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>> [plugin zlib (zlib/noisal)]
>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>> [plugin snappy (snappy)]
>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>> [plugin zstd (zstd)]
>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>
>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>> isn't acceptable--even if it is a bit smaller.

i've fixed the zstd compression plugin to use reset stream instead of
initializing new objects.

What's needed to run only / just the unittest_compression test?

Stefan

Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Sage Weil]

On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
> Hi Sage,
> 
> Am 26.10.2017 um 13:58 schrieb Sage Weil:
> > On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
> >> Hi Sage,
> >>
> >> Am 25.10.2017 um 21:54 schrieb Sage Weil:
> >>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
> >>>> Hello,
> >>>>
> >>>> in the lumious release notes is stated that zstd is not supported by
> >>>> bluestor due to performance reason. I'm wondering why btrfs instead
> >>>> states that zstd is as fast as lz4 but compresses as good as zlib.
> >>>>
> >>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
> >>>> behave different?
> >>>>
> >>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
> >>>> also supports zstd for RGW compression but zstd is not recommended for
> >>>> BlueStore for performance reasons.)"
> >>>
> >>> zstd will work but in our testing the performance wasn't great for 
> >>> bluestore in particular.  The problem was that for each compression run 
> >>> there is a relatively high start-up cost initializing the zstd 
> >>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
> >>> execution time... primarily because bluestore is generally compressing 
> >>> pretty small chunks of data at a time, not big buffers or streams.
> >>>
> >>> Take a look at unittest_compression timings on compressing 16KB buffers 
> >>> (smaller than bluestore needs usually, but illustrated of the problem):
> >>>
> >>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
> >>> [plugin zlib (zlib/isal)]
> >>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
> >>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
> >>> [plugin zlib (zlib/noisal)]
> >>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
> >>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
> >>> [plugin snappy (snappy)]
> >>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
> >>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
> >>> [plugin zstd (zstd)]
> >>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
> >>>
> >>> It's an order of magnitude slower than zlib or snappy, which probably 
> >>> isn't acceptable--even if it is a bit smaller.
> 
> i've fixed the zstd compression plugin to use reset stream instead of
> initializing new objects.
> 
> What's needed to run only / just the unittest_compression test?

make unittest_compression && bin/unittest_compression

should do it!

sage

Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

Thanks - not a C++ guy what's wrong with it?

/build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
'virtual int ZstdCompressor::compress(const bufferlist&,
ceph::bufferlist&)':
/build/ceph/src/compressor/zstd/ZstdCompressor.h:48:28: error:
'ZSTD_resetCStream' was not declared in this scope
     ZSTD_resetCStream(cs, 0);
                            ^
/build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
'virtual int ZstdCompressor::decompress(ceph::buffer::list::iterator&,
size_t, ceph::bufferlist&)':
/build/ceph/src/compressor/zstd/ZstdCompressor.h:89:28: error:
'ZSTD_resetDStream' was not declared in this scope
     ZSTD_resetDStream(ds, 0);


zstd.h which is included has the function:

src/zstd/lib/zstd.h:ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream*
zcs, unsigned long long pledgedSrcSize);  /**< re-use compression
parameters from previous init; skip dictionary loading stage; zcs must
be init at least once before */

Stefan

Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for 
>>>>> bluestore in particular.  The problem was that for each compression run 
>>>>> there is a relatively high start-up cost initializing the zstd 
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>>>> execution time... primarily because bluestore is generally compressing 
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
> 
> make unittest_compression && bin/unittest_compression
> 
> should do it!
> 
> sage
> 

Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

OK zstd.h needs:
#define ZSTD_STATIC_LINKING_ONLY

to export that one.

Stefan

Am 04.11.2017 um 21:23 schrieb Stefan Priebe - Profihost AG:
> Thanks - not a C++ guy what's wrong with it?
> 
> /build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
> 'virtual int ZstdCompressor::compress(const bufferlist&,
> ceph::bufferlist&)':
> /build/ceph/src/compressor/zstd/ZstdCompressor.h:48:28: error:
> 'ZSTD_resetCStream' was not declared in this scope
>      ZSTD_resetCStream(cs, 0);
>                             ^
> /build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
> 'virtual int ZstdCompressor::decompress(ceph::buffer::list::iterator&,
> size_t, ceph::bufferlist&)':
> /build/ceph/src/compressor/zstd/ZstdCompressor.h:89:28: error:
> 'ZSTD_resetDStream' was not declared in this scope
>      ZSTD_resetDStream(ds, 0);
> 
> 
> zstd.h which is included has the function:
> 
> src/zstd/lib/zstd.h:ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream*
> zcs, unsigned long long pledgedSrcSize);  /**< re-use compression
> parameters from previous init; skip dictionary loading stage; zcs must
> be init at least once before */
> 
> Stefan
> 
> Am 04.11.2017 um 21:10 schrieb Sage Weil:
>> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>>> Hi Sage,
>>>
>>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>> Hi Sage,
>>>>>
>>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>>
>>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>>> behave different?
>>>>>>>
>>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>>> BlueStore for performance reasons.)"
>>>>>>
>>>>>> zstd will work but in our testing the performance wasn't great for 
>>>>>> bluestore in particular.  The problem was that for each compression run 
>>>>>> there is a relatively high start-up cost initializing the zstd 
>>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>>>>> execution time... primarily because bluestore is generally compressing 
>>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>>
>>>>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>>
>>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>>>>> [plugin zlib (zlib/isal)]
>>>>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>>>>> [plugin zlib (zlib/noisal)]
>>>>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>>>>> [plugin snappy (snappy)]
>>>>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>>>>> [plugin zstd (zstd)]
>>>>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>>
>>>>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>>>>> isn't acceptable--even if it is a bit smaller.
>>>
>>> i've fixed the zstd compression plugin to use reset stream instead of
>>> initializing new objects.
>>>
>>> What's needed to run only / just the unittest_compression test?
>>
>> make unittest_compression && bin/unittest_compression
>>
>> should do it!
>>
>> sage
>>

Re: ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

Hi Sage,

this does not work for me.
>> What's needed to run only / just the unittest_compression test?
>
> make unittest_compression && bin/unittest_compression
>
> should do it!

#:/build/ceph/src# make unittest_compression && bin/unittest_compression
make: *** No rule to make target 'unittest_compression'.  Stop.

Stefan


Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for 
>>>>> bluestore in particular.  The problem was that for each compression run 
>>>>> there is a relatively high start-up cost initializing the zstd 
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>>>> execution time... primarily because bluestore is generally compressing 
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
> 
> make unittest_compression && bin/unittest_compression
> 
> should do it!
> 
> sage
> 

Re: ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

Hi,

arg sorry i got it. Building the test right now. Will report results
shortly.

Greets,
Stefan

Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for 
>>>>> bluestore in particular.  The problem was that for each compression run 
>>>>> there is a relatively high start-up cost initializing the zstd 
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>>>> execution time... primarily because bluestore is generally compressing 
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
> 
> make unittest_compression && bin/unittest_compression
> 
> should do it!
> 
> sage
> 

Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

The compression unit test does not work for me:
 bin/unittest_compression
2017-11-05 08:05:36.734567 7f7e8322c180 -1 did not load config file,
using default settings.
2017-11-05 08:05:36.735744 7f7e8322c180 -1 Errors while parsing config file!
2017-11-05 08:05:36.735760 7f7e8322c180 -1 parse_file: cannot open
/etc/ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.735761 7f7e8322c180 -1 parse_file: cannot open
~/.ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.735762 7f7e8322c180 -1 parse_file: cannot open
ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737096 7f7e8322c180 -1 Errors while parsing config file!
2017-11-05 08:05:36.737109 7f7e8322c180 -1 parse_file: cannot open
/etc/ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737110 7f7e8322c180 -1 parse_file: cannot open
~/.ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737110 7f7e8322c180 -1 parse_file: cannot open
ceph.conf: (2) No such file or directory
[==========] Running 68 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from ZlibCompressor
[ RUN      ] ZlibCompressor.zlib_isal_compatibility
[       OK ] ZlibCompressor.zlib_isal_compatibility (3 ms)
[ RUN      ] ZlibCompressor.isal_compress_zlib_decompress_random
[       OK ] ZlibCompressor.isal_compress_zlib_decompress_random (76 ms)
[ RUN      ] ZlibCompressor.isal_compress_zlib_decompress_walk
[       OK ] ZlibCompressor.isal_compress_zlib_decompress_walk (65 ms)
[----------] 3 tests from ZlibCompressor (144 ms total)

[----------] 1 test from CompressionPlugin
[ RUN      ] CompressionPlugin.all
/build/ceph/src/test/compressor/test_compression.cc:389: Failure
Value of: factory
  Actual: false
Expected: true
2017-11-05 08:05:36.884467 7f7e8322c180 -1 load failed dlopen():
"/usr/lib/ceph/compressor/libceph_invalid.so: cannot open shared object
file: No such file or directory" or "/usr/lib/ceph/libceph_invalid.so:
cannot open shared object file: No such file or directory"
2017-11-05 08:05:36.884528 7f7e8322c180 -1 load failed dlopen():
"/usr/lib/ceph/compressor/libceph_example.so: cannot open shared object
file: No such file or directory" or "/usr/lib/ceph/libceph_example.so:
cannot open shared object file: No such file or directory"
*** Caught signal (Segmentation fault) **

i've no idea why it tries to use libceph_invalid and libceph_example

Stefan

Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for 
>>>>> bluestore in particular.  The problem was that for each compression run 
>>>>> there is a relatively high start-up cost initializing the zstd 
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>>>> execution time... primarily because bluestore is generally compressing 
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
> 
> make unittest_compression && bin/unittest_compression
> 
> should do it!
> 
> sage
> 

Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

Hi,

ok got it running by using:

 bin/unittest_compression --compression-dir=lib --plugin-dir=lib

Results:
[----------] 64 tests from Compressor/CompressorTest
[ RUN      ] Compressor/CompressorTest.load_plugin/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.load_plugin/0 (3 ms)
[ RUN      ] Compressor/CompressorTest.load_plugin/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.load_plugin/1 (2 ms)
[ RUN      ] Compressor/CompressorTest.load_plugin/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.load_plugin/2 (2 ms)
[ RUN      ] Compressor/CompressorTest.load_plugin/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.load_plugin/3 (3 ms)
[ RUN      ] Compressor/CompressorTest.small_round_trip/0
[plugin zlib (zlib/isal)]
orig 77 compressed 168 with zlib/isal
[       OK ] Compressor/CompressorTest.small_round_trip/0 (3 ms)
[ RUN      ] Compressor/CompressorTest.small_round_trip/1
[plugin zlib (zlib/noisal)]
orig 77 compressed 66 with zlib/noisal
[       OK ] Compressor/CompressorTest.small_round_trip/1 (2 ms)
[ RUN      ] Compressor/CompressorTest.small_round_trip/2
[plugin snappy (snappy)]
orig 77 compressed 74 with snappy
[       OK ] Compressor/CompressorTest.small_round_trip/2 (2 ms)
[ RUN      ] Compressor/CompressorTest.small_round_trip/3
[plugin zstd (zstd)]
orig 77 compressed 79 with zstd
[       OK ] Compressor/CompressorTest.small_round_trip/3 (2 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_repeated/0
[plugin zlib (zlib/isal)]
orig 4194344 compressed 54878 with zlib/isal
[       OK ] Compressor/CompressorTest.big_round_trip_repeated/0 (41 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_repeated/1
[plugin zlib (zlib/noisal)]
orig 4194344 compressed 14306 with zlib/noisal
[       OK ] Compressor/CompressorTest.big_round_trip_repeated/1 (64 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_repeated/2
[plugin snappy (snappy)]
orig 4194344 compressed 201194 with snappy
[       OK ] Compressor/CompressorTest.big_round_trip_repeated/2 (22 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_repeated/3
[plugin zstd (zstd)]
orig 4194344 compressed 472 with zstd
[       OK ] Compressor/CompressorTest.big_round_trip_repeated/3 (21 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_randomish/0
[plugin zlib (zlib/isal)]
orig 10485760 compressed 5989211 with zlib/isal
[       OK ] Compressor/CompressorTest.big_round_trip_randomish/0 (415 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_randomish/1
[plugin zlib (zlib/noisal)]
orig 10485760 compressed 4928804 with zlib/noisal
[       OK ] Compressor/CompressorTest.big_round_trip_randomish/1 (1151 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_randomish/2
[plugin snappy (snappy)]
orig 10485760 compressed 8683058 with snappy
[       OK ] Compressor/CompressorTest.big_round_trip_randomish/2 (300 ms)
[ RUN      ] Compressor/CompressorTest.big_round_trip_randomish/3
[plugin zstd (zstd)]
orig 10485760 compressed 4829428 with zstd
[       OK ] Compressor/CompressorTest.big_round_trip_randomish/3 (1444 ms)
[ RUN      ] Compressor/CompressorTest.compress_decompress/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_decompress/0 (3 ms)
[ RUN      ] Compressor/CompressorTest.compress_decompress/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_decompress/1 (2 ms)
[ RUN      ] Compressor/CompressorTest.compress_decompress/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_decompress/2 (1 ms)
[ RUN      ] Compressor/CompressorTest.compress_decompress/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_decompress/3 (1 ms)
[ RUN      ] Compressor/CompressorTest.sharded_input_decompress/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.sharded_input_decompress/0 (3 ms)
[ RUN      ] Compressor/CompressorTest.sharded_input_decompress/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.sharded_input_decompress/1 (3 ms)
[ RUN      ] Compressor/CompressorTest.sharded_input_decompress/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.sharded_input_decompress/2 (2 ms)
[ RUN      ] Compressor/CompressorTest.sharded_input_decompress/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.sharded_input_decompress/3 (1 ms)
[ RUN      ] Compressor/CompressorTest.compress_1024/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_1024/0 (73 ms)
[ RUN      ] Compressor/CompressorTest.compress_1024/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_1024/1 (614 ms)
[ RUN      ] Compressor/CompressorTest.compress_1024/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_1024/2 (15 ms)
[ RUN      ] Compressor/CompressorTest.compress_1024/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_1024/3 (566 ms)
[ RUN      ] Compressor/CompressorTest.compress_2048/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_2048/0 (102 ms)
[ RUN      ] Compressor/CompressorTest.compress_2048/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_2048/1 (819 ms)
[ RUN      ] Compressor/CompressorTest.compress_2048/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_2048/2 (18 ms)
[ RUN      ] Compressor/CompressorTest.compress_2048/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_2048/3 (1721 ms)
[ RUN      ] Compressor/CompressorTest.compress_4096/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_4096/0 (150 ms)
[ RUN      ] Compressor/CompressorTest.compress_4096/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_4096/1 (1103 ms)
[ RUN      ] Compressor/CompressorTest.compress_4096/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_4096/2 (61 ms)
[ RUN      ] Compressor/CompressorTest.compress_4096/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_4096/3 (2206 ms)
[ RUN      ] Compressor/CompressorTest.compress_8192/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_8192/0 (193 ms)
[ RUN      ] Compressor/CompressorTest.compress_8192/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_8192/1 (1457 ms)
[ RUN      ] Compressor/CompressorTest.compress_8192/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_8192/2 (114 ms)
[ RUN      ] Compressor/CompressorTest.compress_8192/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_8192/3 (3121 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_16384/0 (288 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_16384/1 (2090 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_16384/2 (202 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_16384/3 (4561 ms)
[ RUN      ] Compressor/CompressorTest.decompress_1024/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.decompress_1024/0 (176 ms)
[ RUN      ] Compressor/CompressorTest.decompress_1024/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.decompress_1024/1 (108 ms)
[ RUN      ] Compressor/CompressorTest.decompress_1024/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.decompress_1024/2 (7 ms)
[ RUN      ] Compressor/CompressorTest.decompress_1024/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.decompress_1024/3 (12 ms)
[ RUN      ] Compressor/CompressorTest.decompress_2048/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.decompress_2048/0 (220 ms)
[ RUN      ] Compressor/CompressorTest.decompress_2048/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.decompress_2048/1 (145 ms)
[ RUN      ] Compressor/CompressorTest.decompress_2048/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.decompress_2048/2 (9 ms)
[ RUN      ] Compressor/CompressorTest.decompress_2048/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.decompress_2048/3 (133 ms)
[ RUN      ] Compressor/CompressorTest.decompress_4096/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.decompress_4096/0 (287 ms)
[ RUN      ] Compressor/CompressorTest.decompress_4096/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.decompress_4096/1 (195 ms)
[ RUN      ] Compressor/CompressorTest.decompress_4096/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.decompress_4096/2 (28 ms)
[ RUN      ] Compressor/CompressorTest.decompress_4096/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.decompress_4096/3 (577 ms)
[ RUN      ] Compressor/CompressorTest.decompress_8192/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.decompress_8192/0 (384 ms)
[ RUN      ] Compressor/CompressorTest.decompress_8192/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.decompress_8192/1 (266 ms)
[ RUN      ] Compressor/CompressorTest.decompress_8192/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.decompress_8192/2 (59 ms)
[ RUN      ] Compressor/CompressorTest.decompress_8192/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.decompress_8192/3 (1024 ms)
[ RUN      ] Compressor/CompressorTest.decompress_16384/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.decompress_16384/0 (552 ms)
[ RUN      ] Compressor/CompressorTest.decompress_16384/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.decompress_16384/1 (396 ms)
[ RUN      ] Compressor/CompressorTest.decompress_16384/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.decompress_16384/2 (132 ms)
[ RUN      ] Compressor/CompressorTest.decompress_16384/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.decompress_16384/3 (1449 ms)
[----------] 64 tests from Compressor/CompressorTest (29128 ms total)


Greets,
Stefan

Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for 
>>>>> bluestore in particular.  The problem was that for each compression run 
>>>>> there is a relatively high start-up cost initializing the zstd 
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>>>> execution time... primarily because bluestore is generally compressing 
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
> 
> make unittest_compression && bin/unittest_compression
> 
> should do it!
> 
> sage
> 

Re: ceph zstd not for bluestor due to performance reasons

[Sage Weil]

On Wed, 25 Oct 2017, Sage Weil wrote:
> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
> > Hello,
> > 
> > in the lumious release notes is stated that zstd is not supported by
> > bluestor due to performance reason. I'm wondering why btrfs instead
> > states that zstd is as fast as lz4 but compresses as good as zlib.
> > 
> > Why is zlib than supported by bluestor? And why does btrfs / facebook
> > behave different?
> > 
> > "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
> > also supports zstd for RGW compression but zstd is not recommended for
> > BlueStore for performance reasons.)"
> 
> zstd will work but in our testing the performance wasn't great for 
> bluestore in particular.  The problem was that for each compression run 
> there is a relatively high start-up cost initializing the zstd 
> context/state (IIRC a memset of a huge memory buffer) that dominated the 
> execution time... primarily because bluestore is generally compressing 
> pretty small chunks of data at a time, not big buffers or streams.
> 
> Take a look at unittest_compression timings on compressing 16KB buffers 
> (smaller than bluestore needs usually, but illustrated of the problem):
> 
> [ RUN      ] Compressor/CompressorTest.compress_16384/0
> [plugin zlib (zlib/isal)]
> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/1
> [plugin zlib (zlib/noisal)]
> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/2
> [plugin snappy (snappy)]
> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/3
> [plugin zstd (zstd)]
> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
> 
> It's an order of magnitude slower than zlib or snappy, which probably 
> isn't acceptable--even if it is a bit smaller.

Update!  Zstd developer Yann Collet debugged this and it turns out it was 
a build issue, fixed by https://github.com/ceph/ceph/pull/18879/files 
(missing quotes!  yeesh).  The results now look quite good!

[ RUN      ] Compressor/CompressorTest.compress_16384/0
[plugin zlib (zlib/isal)]
[       OK ] Compressor/CompressorTest.compress_16384/0 (370 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/1
[plugin zlib (zlib/noisal)]
[       OK ] Compressor/CompressorTest.compress_16384/1 (1926 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/2
[plugin snappy (snappy)]
[       OK ] Compressor/CompressorTest.compress_16384/2 (163 ms)
[ RUN      ] Compressor/CompressorTest.compress_16384/3
[plugin zstd (zstd)]
[       OK ] Compressor/CompressorTest.compress_16384/3 (723 ms)

Not as fast as snappy, but somewhere between intel-accellerated zlib and 
non-accellerated zlib, with better compression ratios.

Also, the zstd compression level is currently hard-coded to level 5.  
That should be fixed at some point.

We can backport this to luminous so it's available in 12.2.3.

sage

Re: ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG]

Am 12.11.2017 um 17:55 schrieb Sage Weil:
> On Wed, 25 Oct 2017, Sage Weil wrote:
>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>> Hello,
>>>
>>> in the lumious release notes is stated that zstd is not supported by
>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>
>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>> behave different?
>>>
>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>> also supports zstd for RGW compression but zstd is not recommended for
>>> BlueStore for performance reasons.)"
>>
>> zstd will work but in our testing the performance wasn't great for 
>> bluestore in particular.  The problem was that for each compression run 
>> there is a relatively high start-up cost initializing the zstd 
>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>> execution time... primarily because bluestore is generally compressing 
>> pretty small chunks of data at a time, not big buffers or streams.
>>
>> Take a look at unittest_compression timings on compressing 16KB buffers 
>> (smaller than bluestore needs usually, but illustrated of the problem):
>>
>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>> [plugin zlib (zlib/isal)]
>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>> [plugin zlib (zlib/noisal)]
>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>> [plugin snappy (snappy)]
>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>> [plugin zstd (zstd)]
>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>
>> It's an order of magnitude slower than zlib or snappy, which probably 
>> isn't acceptable--even if it is a bit smaller.
> 
> Update!  Zstd developer Yann Collet debugged this and it turns out it was 
> a build issue, fixed by https://github.com/ceph/ceph/pull/18879/files 
> (missing quotes!  yeesh).  The results now look quite good!
> 
> [ RUN      ] Compressor/CompressorTest.compress_16384/0
> [plugin zlib (zlib/isal)]
> [       OK ] Compressor/CompressorTest.compress_16384/0 (370 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/1
> [plugin zlib (zlib/noisal)]
> [       OK ] Compressor/CompressorTest.compress_16384/1 (1926 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/2
> [plugin snappy (snappy)]
> [       OK ] Compressor/CompressorTest.compress_16384/2 (163 ms)
> [ RUN      ] Compressor/CompressorTest.compress_16384/3
> [plugin zstd (zstd)]
> [       OK ] Compressor/CompressorTest.compress_16384/3 (723 ms)
> 
> Not as fast as snappy, but somewhere between intel-accellerated zlib and 
> non-accellerated zlib, with better compression ratios.
> 
> Also, the zstd compression level is currently hard-coded to level 5.  
> That should be fixed at some point.
> 
> We can backport this to luminous so it's available in 12.2.3.

thanks a lot - i already ported your improvements and the fix to my
local branch but will also change the compression level to 3 or may be 2.

5 is still far too slow and also higher than most apps using zstd.

I'm happy that my post to the zstd github project had so much success.

Greets,
Stefan