* Re: [ceph-users] ceph zstd not for bluestor due to performance reasons
2017-11-04 20:10 ` [ceph-users] " Sage Weil
@ 2017-11-04 20:23 ` Stefan Priebe - Profihost AG
2017-11-04 21:21 ` Stefan Priebe - Profihost AG
[not found] ` <alpine.DEB.2.11.1711042010330.23234-qHenpvqtifaMSRpgCs4c+g@public.gmane.org>
` (2 subsequent siblings)
3 siblings, 1 reply; 15+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-11-04 20:23 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-users, ceph-devel
Thanks - not a C++ guy what's wrong with it?
/build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
'virtual int ZstdCompressor::compress(const bufferlist&,
ceph::bufferlist&)':
/build/ceph/src/compressor/zstd/ZstdCompressor.h:48:28: error:
'ZSTD_resetCStream' was not declared in this scope
ZSTD_resetCStream(cs, 0);
^
/build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
'virtual int ZstdCompressor::decompress(ceph::buffer::list::iterator&,
size_t, ceph::bufferlist&)':
/build/ceph/src/compressor/zstd/ZstdCompressor.h:89:28: error:
'ZSTD_resetDStream' was not declared in this scope
ZSTD_resetDStream(ds, 0);
zstd.h which is included has the function:
src/zstd/lib/zstd.h:ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream*
zcs, unsigned long long pledgedSrcSize); /**< re-use compression
parameters from previous init; skip dictionary loading stage; zcs must
be init at least once before */
Stefan
Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for
>>>>> bluestore in particular. The problem was that for each compression run
>>>>> there is a relatively high start-up cost initializing the zstd
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the
>>>>> execution time... primarily because bluestore is generally compressing
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
>
> make unittest_compression && bin/unittest_compression
>
> should do it!
>
> sage
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ceph-users] ceph zstd not for bluestor due to performance reasons
2017-11-04 20:23 ` Stefan Priebe - Profihost AG
@ 2017-11-04 21:21 ` Stefan Priebe - Profihost AG
0 siblings, 0 replies; 15+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-11-04 21:21 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-users, ceph-devel
OK zstd.h needs:
#define ZSTD_STATIC_LINKING_ONLY
to export that one.
Stefan
Am 04.11.2017 um 21:23 schrieb Stefan Priebe - Profihost AG:
> Thanks - not a C++ guy what's wrong with it?
>
> /build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
> 'virtual int ZstdCompressor::compress(const bufferlist&,
> ceph::bufferlist&)':
> /build/ceph/src/compressor/zstd/ZstdCompressor.h:48:28: error:
> 'ZSTD_resetCStream' was not declared in this scope
> ZSTD_resetCStream(cs, 0);
> ^
> /build/ceph/src/compressor/zstd/ZstdCompressor.h: In member function
> 'virtual int ZstdCompressor::decompress(ceph::buffer::list::iterator&,
> size_t, ceph::bufferlist&)':
> /build/ceph/src/compressor/zstd/ZstdCompressor.h:89:28: error:
> 'ZSTD_resetDStream' was not declared in this scope
> ZSTD_resetDStream(ds, 0);
>
>
> zstd.h which is included has the function:
>
> src/zstd/lib/zstd.h:ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream*
> zcs, unsigned long long pledgedSrcSize); /**< re-use compression
> parameters from previous init; skip dictionary loading stage; zcs must
> be init at least once before */
>
> Stefan
>
> Am 04.11.2017 um 21:10 schrieb Sage Weil:
>> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>>> Hi Sage,
>>>
>>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>> Hi Sage,
>>>>>
>>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>>
>>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>>> behave different?
>>>>>>>
>>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>>> BlueStore for performance reasons.)"
>>>>>>
>>>>>> zstd will work but in our testing the performance wasn't great for
>>>>>> bluestore in particular. The problem was that for each compression run
>>>>>> there is a relatively high start-up cost initializing the zstd
>>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the
>>>>>> execution time... primarily because bluestore is generally compressing
>>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>>
>>>>>> Take a look at unittest_compression timings on compressing 16KB buffers
>>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>>
>>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/0
>>>>>> [plugin zlib (zlib/isal)]
>>>>>> [ OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/1
>>>>>> [plugin zlib (zlib/noisal)]
>>>>>> [ OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/2
>>>>>> [plugin snappy (snappy)]
>>>>>> [ OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/3
>>>>>> [plugin zstd (zstd)]
>>>>>> [ OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>>
>>>>>> It's an order of magnitude slower than zlib or snappy, which probably
>>>>>> isn't acceptable--even if it is a bit smaller.
>>>
>>> i've fixed the zstd compression plugin to use reset stream instead of
>>> initializing new objects.
>>>
>>> What's needed to run only / just the unittest_compression test?
>>
>> make unittest_compression && bin/unittest_compression
>>
>> should do it!
>>
>> sage
>>
^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <alpine.DEB.2.11.1711042010330.23234-qHenpvqtifaMSRpgCs4c+g@public.gmane.org>]
* Re: ceph zstd not for bluestor due to performance reasons
[not found] ` <alpine.DEB.2.11.1711042010330.23234-qHenpvqtifaMSRpgCs4c+g@public.gmane.org>
@ 2017-11-05 6:51 ` Stefan Priebe - Profihost AG
2017-11-05 7:03 ` Stefan Priebe - Profihost AG
1 sibling, 0 replies; 15+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-11-05 6:51 UTC (permalink / raw)
To: Sage Weil
Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, ceph-devel-u79uwXL29TY76Z2rM5mHXA
Hi Sage,
this does not work for me.
>> What's needed to run only / just the unittest_compression test?
>
> make unittest_compression && bin/unittest_compression
>
> should do it!
#:/build/ceph/src# make unittest_compression && bin/unittest_compression
make: *** No rule to make target 'unittest_compression'. Stop.
Stefan
Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for
>>>>> bluestore in particular. The problem was that for each compression run
>>>>> there is a relatively high start-up cost initializing the zstd
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the
>>>>> execution time... primarily because bluestore is generally compressing
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
>
> make unittest_compression && bin/unittest_compression
>
> should do it!
>
> sage
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: ceph zstd not for bluestor due to performance reasons
[not found] ` <alpine.DEB.2.11.1711042010330.23234-qHenpvqtifaMSRpgCs4c+g@public.gmane.org>
2017-11-05 6:51 ` Stefan Priebe - Profihost AG
@ 2017-11-05 7:03 ` Stefan Priebe - Profihost AG
1 sibling, 0 replies; 15+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-11-05 7:03 UTC (permalink / raw)
To: Sage Weil
Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, ceph-devel-u79uwXL29TY76Z2rM5mHXA
Hi,
arg sorry i got it. Building the test right now. Will report results
shortly.
Greets,
Stefan
Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for
>>>>> bluestore in particular. The problem was that for each compression run
>>>>> there is a relatively high start-up cost initializing the zstd
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the
>>>>> execution time... primarily because bluestore is generally compressing
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
>
> make unittest_compression && bin/unittest_compression
>
> should do it!
>
> sage
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ceph-users] ceph zstd not for bluestor due to performance reasons
2017-11-04 20:10 ` [ceph-users] " Sage Weil
2017-11-04 20:23 ` Stefan Priebe - Profihost AG
[not found] ` <alpine.DEB.2.11.1711042010330.23234-qHenpvqtifaMSRpgCs4c+g@public.gmane.org>
@ 2017-11-05 8:07 ` Stefan Priebe - Profihost AG
2017-11-05 8:18 ` Stefan Priebe - Profihost AG
3 siblings, 0 replies; 15+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-11-05 8:07 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-users, ceph-devel
The compression unit test does not work for me:
bin/unittest_compression
2017-11-05 08:05:36.734567 7f7e8322c180 -1 did not load config file,
using default settings.
2017-11-05 08:05:36.735744 7f7e8322c180 -1 Errors while parsing config file!
2017-11-05 08:05:36.735760 7f7e8322c180 -1 parse_file: cannot open
/etc/ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.735761 7f7e8322c180 -1 parse_file: cannot open
~/.ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.735762 7f7e8322c180 -1 parse_file: cannot open
ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737096 7f7e8322c180 -1 Errors while parsing config file!
2017-11-05 08:05:36.737109 7f7e8322c180 -1 parse_file: cannot open
/etc/ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737110 7f7e8322c180 -1 parse_file: cannot open
~/.ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737110 7f7e8322c180 -1 parse_file: cannot open
ceph.conf: (2) No such file or directory
[==========] Running 68 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from ZlibCompressor
[ RUN ] ZlibCompressor.zlib_isal_compatibility
[ OK ] ZlibCompressor.zlib_isal_compatibility (3 ms)
[ RUN ] ZlibCompressor.isal_compress_zlib_decompress_random
[ OK ] ZlibCompressor.isal_compress_zlib_decompress_random (76 ms)
[ RUN ] ZlibCompressor.isal_compress_zlib_decompress_walk
[ OK ] ZlibCompressor.isal_compress_zlib_decompress_walk (65 ms)
[----------] 3 tests from ZlibCompressor (144 ms total)
[----------] 1 test from CompressionPlugin
[ RUN ] CompressionPlugin.all
/build/ceph/src/test/compressor/test_compression.cc:389: Failure
Value of: factory
Actual: false
Expected: true
2017-11-05 08:05:36.884467 7f7e8322c180 -1 load failed dlopen():
"/usr/lib/ceph/compressor/libceph_invalid.so: cannot open shared object
file: No such file or directory" or "/usr/lib/ceph/libceph_invalid.so:
cannot open shared object file: No such file or directory"
2017-11-05 08:05:36.884528 7f7e8322c180 -1 load failed dlopen():
"/usr/lib/ceph/compressor/libceph_example.so: cannot open shared object
file: No such file or directory" or "/usr/lib/ceph/libceph_example.so:
cannot open shared object file: No such file or directory"
*** Caught signal (Segmentation fault) **
i've no idea why it tries to use libceph_invalid and libceph_example
Stefan
Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for
>>>>> bluestore in particular. The problem was that for each compression run
>>>>> there is a relatively high start-up cost initializing the zstd
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the
>>>>> execution time... primarily because bluestore is generally compressing
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
>
> make unittest_compression && bin/unittest_compression
>
> should do it!
>
> sage
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ceph-users] ceph zstd not for bluestor due to performance reasons
2017-11-04 20:10 ` [ceph-users] " Sage Weil
` (2 preceding siblings ...)
2017-11-05 8:07 ` [ceph-users] " Stefan Priebe - Profihost AG
@ 2017-11-05 8:18 ` Stefan Priebe - Profihost AG
3 siblings, 0 replies; 15+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-11-05 8:18 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-users, ceph-devel
Hi,
ok got it running by using:
bin/unittest_compression --compression-dir=lib --plugin-dir=lib
Results:
[----------] 64 tests from Compressor/CompressorTest
[ RUN ] Compressor/CompressorTest.load_plugin/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.load_plugin/0 (3 ms)
[ RUN ] Compressor/CompressorTest.load_plugin/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.load_plugin/1 (2 ms)
[ RUN ] Compressor/CompressorTest.load_plugin/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.load_plugin/2 (2 ms)
[ RUN ] Compressor/CompressorTest.load_plugin/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.load_plugin/3 (3 ms)
[ RUN ] Compressor/CompressorTest.small_round_trip/0
[plugin zlib (zlib/isal)]
orig 77 compressed 168 with zlib/isal
[ OK ] Compressor/CompressorTest.small_round_trip/0 (3 ms)
[ RUN ] Compressor/CompressorTest.small_round_trip/1
[plugin zlib (zlib/noisal)]
orig 77 compressed 66 with zlib/noisal
[ OK ] Compressor/CompressorTest.small_round_trip/1 (2 ms)
[ RUN ] Compressor/CompressorTest.small_round_trip/2
[plugin snappy (snappy)]
orig 77 compressed 74 with snappy
[ OK ] Compressor/CompressorTest.small_round_trip/2 (2 ms)
[ RUN ] Compressor/CompressorTest.small_round_trip/3
[plugin zstd (zstd)]
orig 77 compressed 79 with zstd
[ OK ] Compressor/CompressorTest.small_round_trip/3 (2 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_repeated/0
[plugin zlib (zlib/isal)]
orig 4194344 compressed 54878 with zlib/isal
[ OK ] Compressor/CompressorTest.big_round_trip_repeated/0 (41 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_repeated/1
[plugin zlib (zlib/noisal)]
orig 4194344 compressed 14306 with zlib/noisal
[ OK ] Compressor/CompressorTest.big_round_trip_repeated/1 (64 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_repeated/2
[plugin snappy (snappy)]
orig 4194344 compressed 201194 with snappy
[ OK ] Compressor/CompressorTest.big_round_trip_repeated/2 (22 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_repeated/3
[plugin zstd (zstd)]
orig 4194344 compressed 472 with zstd
[ OK ] Compressor/CompressorTest.big_round_trip_repeated/3 (21 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_randomish/0
[plugin zlib (zlib/isal)]
orig 10485760 compressed 5989211 with zlib/isal
[ OK ] Compressor/CompressorTest.big_round_trip_randomish/0 (415 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_randomish/1
[plugin zlib (zlib/noisal)]
orig 10485760 compressed 4928804 with zlib/noisal
[ OK ] Compressor/CompressorTest.big_round_trip_randomish/1 (1151 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_randomish/2
[plugin snappy (snappy)]
orig 10485760 compressed 8683058 with snappy
[ OK ] Compressor/CompressorTest.big_round_trip_randomish/2 (300 ms)
[ RUN ] Compressor/CompressorTest.big_round_trip_randomish/3
[plugin zstd (zstd)]
orig 10485760 compressed 4829428 with zstd
[ OK ] Compressor/CompressorTest.big_round_trip_randomish/3 (1444 ms)
[ RUN ] Compressor/CompressorTest.compress_decompress/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.compress_decompress/0 (3 ms)
[ RUN ] Compressor/CompressorTest.compress_decompress/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.compress_decompress/1 (2 ms)
[ RUN ] Compressor/CompressorTest.compress_decompress/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.compress_decompress/2 (1 ms)
[ RUN ] Compressor/CompressorTest.compress_decompress/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.compress_decompress/3 (1 ms)
[ RUN ] Compressor/CompressorTest.sharded_input_decompress/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.sharded_input_decompress/0 (3 ms)
[ RUN ] Compressor/CompressorTest.sharded_input_decompress/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.sharded_input_decompress/1 (3 ms)
[ RUN ] Compressor/CompressorTest.sharded_input_decompress/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.sharded_input_decompress/2 (2 ms)
[ RUN ] Compressor/CompressorTest.sharded_input_decompress/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.sharded_input_decompress/3 (1 ms)
[ RUN ] Compressor/CompressorTest.compress_1024/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.compress_1024/0 (73 ms)
[ RUN ] Compressor/CompressorTest.compress_1024/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.compress_1024/1 (614 ms)
[ RUN ] Compressor/CompressorTest.compress_1024/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.compress_1024/2 (15 ms)
[ RUN ] Compressor/CompressorTest.compress_1024/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.compress_1024/3 (566 ms)
[ RUN ] Compressor/CompressorTest.compress_2048/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.compress_2048/0 (102 ms)
[ RUN ] Compressor/CompressorTest.compress_2048/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.compress_2048/1 (819 ms)
[ RUN ] Compressor/CompressorTest.compress_2048/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.compress_2048/2 (18 ms)
[ RUN ] Compressor/CompressorTest.compress_2048/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.compress_2048/3 (1721 ms)
[ RUN ] Compressor/CompressorTest.compress_4096/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.compress_4096/0 (150 ms)
[ RUN ] Compressor/CompressorTest.compress_4096/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.compress_4096/1 (1103 ms)
[ RUN ] Compressor/CompressorTest.compress_4096/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.compress_4096/2 (61 ms)
[ RUN ] Compressor/CompressorTest.compress_4096/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.compress_4096/3 (2206 ms)
[ RUN ] Compressor/CompressorTest.compress_8192/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.compress_8192/0 (193 ms)
[ RUN ] Compressor/CompressorTest.compress_8192/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.compress_8192/1 (1457 ms)
[ RUN ] Compressor/CompressorTest.compress_8192/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.compress_8192/2 (114 ms)
[ RUN ] Compressor/CompressorTest.compress_8192/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.compress_8192/3 (3121 ms)
[ RUN ] Compressor/CompressorTest.compress_16384/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.compress_16384/0 (288 ms)
[ RUN ] Compressor/CompressorTest.compress_16384/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.compress_16384/1 (2090 ms)
[ RUN ] Compressor/CompressorTest.compress_16384/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.compress_16384/2 (202 ms)
[ RUN ] Compressor/CompressorTest.compress_16384/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.compress_16384/3 (4561 ms)
[ RUN ] Compressor/CompressorTest.decompress_1024/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.decompress_1024/0 (176 ms)
[ RUN ] Compressor/CompressorTest.decompress_1024/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.decompress_1024/1 (108 ms)
[ RUN ] Compressor/CompressorTest.decompress_1024/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.decompress_1024/2 (7 ms)
[ RUN ] Compressor/CompressorTest.decompress_1024/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.decompress_1024/3 (12 ms)
[ RUN ] Compressor/CompressorTest.decompress_2048/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.decompress_2048/0 (220 ms)
[ RUN ] Compressor/CompressorTest.decompress_2048/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.decompress_2048/1 (145 ms)
[ RUN ] Compressor/CompressorTest.decompress_2048/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.decompress_2048/2 (9 ms)
[ RUN ] Compressor/CompressorTest.decompress_2048/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.decompress_2048/3 (133 ms)
[ RUN ] Compressor/CompressorTest.decompress_4096/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.decompress_4096/0 (287 ms)
[ RUN ] Compressor/CompressorTest.decompress_4096/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.decompress_4096/1 (195 ms)
[ RUN ] Compressor/CompressorTest.decompress_4096/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.decompress_4096/2 (28 ms)
[ RUN ] Compressor/CompressorTest.decompress_4096/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.decompress_4096/3 (577 ms)
[ RUN ] Compressor/CompressorTest.decompress_8192/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.decompress_8192/0 (384 ms)
[ RUN ] Compressor/CompressorTest.decompress_8192/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.decompress_8192/1 (266 ms)
[ RUN ] Compressor/CompressorTest.decompress_8192/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.decompress_8192/2 (59 ms)
[ RUN ] Compressor/CompressorTest.decompress_8192/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.decompress_8192/3 (1024 ms)
[ RUN ] Compressor/CompressorTest.decompress_16384/0
[plugin zlib (zlib/isal)]
[ OK ] Compressor/CompressorTest.decompress_16384/0 (552 ms)
[ RUN ] Compressor/CompressorTest.decompress_16384/1
[plugin zlib (zlib/noisal)]
[ OK ] Compressor/CompressorTest.decompress_16384/1 (396 ms)
[ RUN ] Compressor/CompressorTest.decompress_16384/2
[plugin snappy (snappy)]
[ OK ] Compressor/CompressorTest.decompress_16384/2 (132 ms)
[ RUN ] Compressor/CompressorTest.decompress_16384/3
[plugin zstd (zstd)]
[ OK ] Compressor/CompressorTest.decompress_16384/3 (1449 ms)
[----------] 64 tests from Compressor/CompressorTest (29128 ms total)
Greets,
Stefan
Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for
>>>>> bluestore in particular. The problem was that for each compression run
>>>>> there is a relatively high start-up cost initializing the zstd
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the
>>>>> execution time... primarily because bluestore is generally compressing
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [ OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
>
> make unittest_compression && bin/unittest_compression
>
> should do it!
>
> sage
>
^ permalink raw reply [flat|nested] 15+ messages in thread