Re: [ceph-users] ceph zstd not for bluestor due to performance reasons

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

The compression unit test does not work for me:
 bin/unittest_compression
2017-11-05 08:05:36.734567 7f7e8322c180 -1 did not load config file,
using default settings.
2017-11-05 08:05:36.735744 7f7e8322c180 -1 Errors while parsing config file!
2017-11-05 08:05:36.735760 7f7e8322c180 -1 parse_file: cannot open
/etc/ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.735761 7f7e8322c180 -1 parse_file: cannot open
~/.ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.735762 7f7e8322c180 -1 parse_file: cannot open
ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737096 7f7e8322c180 -1 Errors while parsing config file!
2017-11-05 08:05:36.737109 7f7e8322c180 -1 parse_file: cannot open
/etc/ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737110 7f7e8322c180 -1 parse_file: cannot open
~/.ceph/ceph.conf: (2) No such file or directory
2017-11-05 08:05:36.737110 7f7e8322c180 -1 parse_file: cannot open
ceph.conf: (2) No such file or directory
[==========] Running 68 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from ZlibCompressor
[ RUN      ] ZlibCompressor.zlib_isal_compatibility
[       OK ] ZlibCompressor.zlib_isal_compatibility (3 ms)
[ RUN      ] ZlibCompressor.isal_compress_zlib_decompress_random
[       OK ] ZlibCompressor.isal_compress_zlib_decompress_random (76 ms)
[ RUN      ] ZlibCompressor.isal_compress_zlib_decompress_walk
[       OK ] ZlibCompressor.isal_compress_zlib_decompress_walk (65 ms)
[----------] 3 tests from ZlibCompressor (144 ms total)

[----------] 1 test from CompressionPlugin
[ RUN      ] CompressionPlugin.all
/build/ceph/src/test/compressor/test_compression.cc:389: Failure
Value of: factory
  Actual: false
Expected: true
2017-11-05 08:05:36.884467 7f7e8322c180 -1 load failed dlopen():
"/usr/lib/ceph/compressor/libceph_invalid.so: cannot open shared object
file: No such file or directory" or "/usr/lib/ceph/libceph_invalid.so:
cannot open shared object file: No such file or directory"
2017-11-05 08:05:36.884528 7f7e8322c180 -1 load failed dlopen():
"/usr/lib/ceph/compressor/libceph_example.so: cannot open shared object
file: No such file or directory" or "/usr/lib/ceph/libceph_example.so:
cannot open shared object file: No such file or directory"
*** Caught signal (Segmentation fault) **

i've no idea why it tries to use libceph_invalid and libceph_example

Stefan

Am 04.11.2017 um 21:10 schrieb Sage Weil:
> On Sat, 4 Nov 2017, Stefan Priebe - Profihost AG wrote:
>> Hi Sage,
>>
>> Am 26.10.2017 um 13:58 schrieb Sage Weil:
>>> On Thu, 26 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hi Sage,
>>>>
>>>> Am 25.10.2017 um 21:54 schrieb Sage Weil:
>>>>> On Wed, 25 Oct 2017, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> in the lumious release notes is stated that zstd is not supported by
>>>>>> bluestor due to performance reason. I'm wondering why btrfs instead
>>>>>> states that zstd is as fast as lz4 but compresses as good as zlib.
>>>>>>
>>>>>> Why is zlib than supported by bluestor? And why does btrfs / facebook
>>>>>> behave different?
>>>>>>
>>>>>> "BlueStore supports inline compression using zlib, snappy, or LZ4. (Ceph
>>>>>> also supports zstd for RGW compression but zstd is not recommended for
>>>>>> BlueStore for performance reasons.)"
>>>>>
>>>>> zstd will work but in our testing the performance wasn't great for 
>>>>> bluestore in particular.  The problem was that for each compression run 
>>>>> there is a relatively high start-up cost initializing the zstd 
>>>>> context/state (IIRC a memset of a huge memory buffer) that dominated the 
>>>>> execution time... primarily because bluestore is generally compressing 
>>>>> pretty small chunks of data at a time, not big buffers or streams.
>>>>>
>>>>> Take a look at unittest_compression timings on compressing 16KB buffers 
>>>>> (smaller than bluestore needs usually, but illustrated of the problem):
>>>>>
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/0
>>>>> [plugin zlib (zlib/isal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/0 (294 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/1
>>>>> [plugin zlib (zlib/noisal)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/1 (1755 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/2
>>>>> [plugin snappy (snappy)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/2 (169 ms)
>>>>> [ RUN      ] Compressor/CompressorTest.compress_16384/3
>>>>> [plugin zstd (zstd)]
>>>>> [       OK ] Compressor/CompressorTest.compress_16384/3 (4528 ms)
>>>>>
>>>>> It's an order of magnitude slower than zlib or snappy, which probably 
>>>>> isn't acceptable--even if it is a bit smaller.
>>
>> i've fixed the zstd compression plugin to use reset stream instead of
>> initializing new objects.
>>
>> What's needed to run only / just the unittest_compression test?
> 
> make unittest_compression && bin/unittest_compression
> 
> should do it!
> 
> sage
>