* [GIT PULL] zstd support (lib, btrfs, squashfs)
@ 2017-09-08 19:33 Chris Mason
2017-09-09 0:43 ` Chris Mason
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Chris Mason @ 2017-09-08 19:33 UTC (permalink / raw)
To: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Herbert Xu, Nick Terrell, Phillip Lougher, Sean Purcell
Cc: David Sterba
Hi Linus,
Nick Terrell's patch series to add zstd support to the kernel has been
floating around for a while. After talking with Dave Sterba, Herbert and
Phillip, we decided to send the whole thing in as one pull request.
I have it in my zstd branch:
git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git zstd
There's a trivial conflict with the main btrfs pull that Dave Sterba just
sent. His pull deletes BTRFS_COMPRESS_LAST in fs/btrfs/compression.h, and
I've put the sample resolution in a branch named zstd-4.14-merge. My
idea was that you'd take our main btrfs pull first and this one second,
but the conflicts are small enough it's not a big deal.
zstd is a big win in speed over zlib and in compression ratio over lzo, and
the compression team here at FB has gotten great results using it in production.
Nick will continue to update the kernel side with new improvements from the
open source zstd userland code.
Nick has a number of benchmarks for the main zstd code in his lib/zstd
commit:
============
I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD. I benchmarked using `silesia.tar` [3], which is
211,988,480 B large. Run the following commands for the benchmark:
sudo modprobe zstd_compress_test
sudo mknod zstd_compress_test c 245 0
sudo cp silesia.tar zstd_compress_test
The time is reported by the time of the userland `cp`.
The MB/s is computed with
1,536,217,008 B / time(buffer size, hash)
which includes the time to copy from userland.
The Adjusted MB/s is computed with
1,536,217,088 B / (time(buffer size, hash) - time(buffer size, none)).
The memory reported is the amount of memory the compressor requests.
| Method | Size (B) | Time (s) | Ratio | MB/s | Adj MB/s | Mem (MB) |
|----------|----------|----------|-------|---------|----------|----------|
| none | 11988480 | 0.100 | 1 | 2119.88 | - | - |
| zstd -1 | 73645762 | 1.044 | 2.878 | 203.05 | 224.56 | 1.23 |
| zstd -3 | 66988878 | 1.761 | 3.165 | 120.38 | 127.63 | 2.47 |
| zstd -5 | 65001259 | 2.563 | 3.261 | 82.71 | 86.07 | 2.86 |
| zstd -10 | 60165346 | 13.242 | 3.523 | 16.01 | 16.13 | 13.22 |
| zstd -15 | 58009756 | 47.601 | 3.654 | 4.45 | 4.46 | 21.61 |
| zstd -19 | 54014593 | 102.835 | 3.925 | 2.06 | 2.06 | 60.15 |
| zlib -1 | 77260026 | 2.895 | 2.744 | 73.23 | 75.85 | 0.27 |
| zlib -3 | 72972206 | 4.116 | 2.905 | 51.50 | 52.79 | 0.27 |
| zlib -6 | 68190360 | 9.633 | 3.109 | 22.01 | 22.24 | 0.27 |
| zlib -9 | 67613382 | 22.554 | 3.135 | 9.40 | 9.44 | 0.27 |
I benchmarked zstd decompression using the same method on the same machine.
The benchmark file is located in the upstream zstd repo under
`contrib/linux-kernel/zstd_decompress_test.c` [4]. The memory reported is
the amount of memory required to decompress data compressed with the given
compression level. If you know the maximum size of your input, you can
reduce the memory usage of decompression irrespective of the compression
level.
| Method | Time (s) | MB/s | Adjusted MB/s | Memory (MB) |
|----------|----------|---------|---------------|-------------|
| none | 0.025 | 8479.54 | - | - |
| zstd -1 | 0.358 | 592.15 | 636.60 | 0.84 |
| zstd -3 | 0.396 | 535.32 | 571.40 | 1.46 |
| zstd -5 | 0.396 | 535.32 | 571.40 | 1.46 |
| zstd -10 | 0.374 | 566.81 | 607.42 | 2.51 |
| zstd -15 | 0.379 | 559.34 | 598.84 | 4.61 |
| zstd -19 | 0.412 | 514.54 | 547.77 | 8.80 |
| zlib -1 | 0.940 | 225.52 | 231.68 | 0.04 |
| zlib -3 | 0.883 | 240.08 | 247.07 | 0.04 |
| zlib -6 | 0.844 | 251.17 | 258.84 | 0.04 |
| zlib -9 | 0.837 | 253.27 | 287.64 | 0.04 |
=======================
I ran a long series of tests and benchmarks on the btrfs side and
the gains are very similar to the core benchmarks Nick ran.
Nick Terrell (4) commits (+14578/-12):
crypto: Add zstd support (+356/-0)
btrfs: Add zstd support (+468/-12)
lib: Add zstd modules (+13014/-0)
lib: Add xxhash module (+740/-0)
Sean Purcell (1) commits (+178/-0):
squashfs: Add zstd support
Total: (5) commits (+14756/-12)
crypto/Kconfig | 9 +
crypto/Makefile | 1 +
crypto/testmgr.c | 10 +
crypto/testmgr.h | 71 +
crypto/zstd.c | 265 ++++
fs/btrfs/Kconfig | 2 +
fs/btrfs/Makefile | 2 +-
fs/btrfs/compression.c | 1 +
fs/btrfs/compression.h | 6 +-
fs/btrfs/ctree.h | 1 +
fs/btrfs/disk-io.c | 2 +
fs/btrfs/ioctl.c | 6 +-
fs/btrfs/props.c | 6 +
fs/btrfs/super.c | 12 +-
fs/btrfs/sysfs.c | 2 +
fs/btrfs/zstd.c | 432 ++++++
fs/squashfs/Kconfig | 14 +
fs/squashfs/Makefile | 1 +
fs/squashfs/decompressor.c | 7 +
fs/squashfs/decompressor.h | 4 +
fs/squashfs/squashfs_fs.h | 1 +
fs/squashfs/zstd_wrapper.c | 151 ++
include/linux/xxhash.h | 236 +++
include/linux/zstd.h | 1157 +++++++++++++++
include/uapi/linux/btrfs.h | 8 +-
lib/Kconfig | 11 +
lib/Makefile | 3 +
lib/xxhash.c | 500 +++++++
lib/zstd/Makefile | 18 +
lib/zstd/bitstream.h | 374 +++++
lib/zstd/compress.c | 3484 ++++++++++++++++++++++++++++++++++++++++++++
lib/zstd/decompress.c | 2528 ++++++++++++++++++++++++++++++++
lib/zstd/entropy_common.c | 243 +++
lib/zstd/error_private.h | 53 +
lib/zstd/fse.h | 575 ++++++++
lib/zstd/fse_compress.c | 795 ++++++++++
lib/zstd/fse_decompress.c | 332 +++++
lib/zstd/huf.h | 212 +++
lib/zstd/huf_compress.c | 770 ++++++++++
lib/zstd/huf_decompress.c | 960 ++++++++++++
lib/zstd/mem.h | 151 ++
lib/zstd/zstd_common.c | 75 +
lib/zstd/zstd_internal.h | 263 ++++
lib/zstd/zstd_opt.h | 1014 +++++++++++++
44 files changed, 14756 insertions(+), 12 deletions(-)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [GIT PULL] zstd support (lib, btrfs, squashfs)
2017-09-08 19:33 [GIT PULL] zstd support (lib, btrfs, squashfs) Chris Mason
@ 2017-09-09 0:43 ` Chris Mason
2017-09-09 1:35 ` Herbert Xu
2017-09-11 19:24 ` [GIT PULL v2] zstd support (lib, btrfs, squashfs, nocrypto) Chris Mason
2 siblings, 0 replies; 9+ messages in thread
From: Chris Mason @ 2017-09-09 0:43 UTC (permalink / raw)
To: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Herbert Xu, Nick Terrell, Phillip Lougher, Sean Purcell
Cc: David Sterba
On 09/08/2017 03:33 PM, Chris Mason wrote:
> Hi Linus,
>
> Nick Terrell's patch series to add zstd support to the kernel has been
> floating around for a while. After talking with Dave Sterba, Herbert and
> Phillip, we decided to send the whole thing in as one pull request.
>
> I have it in my zstd branch:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git zstd
>
> There's a trivial conflict with the main btrfs pull that Dave Sterba just
> sent. His pull deletes BTRFS_COMPRESS_LAST in fs/btrfs/compression.h, and
> I've put the sample resolution in a branch named zstd-4.14-merge. My
> idea was that you'd take our main btrfs pull first and this one second,
> but the conflicts are small enough it's not a big deal.
>
> zstd is a big win in speed over zlib and in compression ratio over lzo, and
> the compression team here at FB has gotten great results using it in production.
> Nick will continue to update the kernel side with new improvements from the
> open source zstd userland code.
Just to clarify, we've been testing the kernel side of this here at FB,
but our zstd use in prod is limited to the application side.
-chris
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [GIT PULL] zstd support (lib, btrfs, squashfs)
2017-09-08 19:33 [GIT PULL] zstd support (lib, btrfs, squashfs) Chris Mason
2017-09-09 0:43 ` Chris Mason
@ 2017-09-09 1:35 ` Herbert Xu
2017-09-09 2:19 ` Chris Mason
2017-09-09 2:48 ` Nick Terrell
2017-09-11 19:24 ` [GIT PULL v2] zstd support (lib, btrfs, squashfs, nocrypto) Chris Mason
2 siblings, 2 replies; 9+ messages in thread
From: Herbert Xu @ 2017-09-09 1:35 UTC (permalink / raw)
To: Chris Mason
Cc: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Nick Terrell, Phillip Lougher, Sean Purcell, David Sterba
On Fri, Sep 08, 2017 at 03:33:05PM -0400, Chris Mason wrote:
>
> crypto/Kconfig | 9 +
> crypto/Makefile | 1 +
> crypto/testmgr.c | 10 +
> crypto/testmgr.h | 71 +
> crypto/zstd.c | 265 ++++
Is there anyone going to use zstd through the crypto API? If not
then I don't see the point in adding it at this point. Especially
as the compression API is still in a state of flux.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [GIT PULL] zstd support (lib, btrfs, squashfs)
2017-09-09 1:35 ` Herbert Xu
@ 2017-09-09 2:19 ` Chris Mason
2017-09-09 2:48 ` Nick Terrell
1 sibling, 0 replies; 9+ messages in thread
From: Chris Mason @ 2017-09-09 2:19 UTC (permalink / raw)
To: Herbert Xu
Cc: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Nick Terrell, Phillip Lougher, Sean Purcell, David Sterba
On Sat, Sep 09, 2017 at 09:35:59AM +0800, Herbert Xu wrote:
>On Fri, Sep 08, 2017 at 03:33:05PM -0400, Chris Mason wrote:
>>
>> crypto/Kconfig | 9 +
>> crypto/Makefile | 1 +
>> crypto/testmgr.c | 10 +
>> crypto/testmgr.h | 71 +
>> crypto/zstd.c | 265 ++++
>
>Is there anyone going to use zstd through the crypto API? If not
>then I don't see the point in adding it at this point. Especially
>as the compression API is still in a state of flux.
That part was requested by intel, but I'm happy to leave it out for
another time. The rest of the patch series doesn't depend on it at all.
-chris
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [GIT PULL] zstd support (lib, btrfs, squashfs)
2017-09-09 1:35 ` Herbert Xu
@ 2017-09-09 2:48 ` Nick Terrell
2017-09-09 2:48 ` Nick Terrell
1 sibling, 0 replies; 9+ messages in thread
From: Nick Terrell @ 2017-09-09 2:48 UTC (permalink / raw)
To: Herbert Xu, Chris Mason
Cc: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Phillip Lougher, Sean Purcell, David Sterba
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 853 bytes --]
On 9/8/17, 6:36 PM, "Herbert Xu" <herbert@gondor.apana.org.au> wrote:
> On Fri, Sep 08, 2017 at 03:33:05PM -0400, Chris Mason wrote:
> >
> > crypto/Kconfig | 9 +
> > crypto/Makefile | 1 +
> > crypto/testmgr.c | 10 +
> > crypto/testmgr.h | 71 +
> > crypto/zstd.c | 265 ++++
>
> Is there anyone going to use zstd through the crypto API? If not
> then I don't see the point in adding it at this point. Especially
> as the compression API is still in a state of flux.
There is a patch [1] floating around adding zstd support to compressed RAM
that uses the crypto API.
[1] https://lkml.kernel.org/r/20170824014936.4738-1-sergey.senozhatsky%40gmail.com
ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨èÚ&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~††Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [GIT PULL] zstd support (lib, btrfs, squashfs)
@ 2017-09-09 2:48 ` Nick Terrell
0 siblings, 0 replies; 9+ messages in thread
From: Nick Terrell @ 2017-09-09 2:48 UTC (permalink / raw)
To: Herbert Xu, Chris Mason
Cc: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Phillip Lougher, Sean Purcell, David Sterba
On 9/8/17, 6:36 PM, "Herbert Xu" <herbert@gondor.apana.org.au> wrote:
> On Fri, Sep 08, 2017 at 03:33:05PM -0400, Chris Mason wrote:
> >
> > crypto/Kconfig | 9 +
> > crypto/Makefile | 1 +
> > crypto/testmgr.c | 10 +
> > crypto/testmgr.h | 71 +
> > crypto/zstd.c | 265 ++++
>
> Is there anyone going to use zstd through the crypto API? If not
> then I don't see the point in adding it at this point. Especially
> as the compression API is still in a state of flux.
There is a patch [1] floating around adding zstd support to compressed RAM
that uses the crypto API.
[1] https://lkml.kernel.org/r/20170824014936.4738-1-sergey.senozhatsky%40gmail.com
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [GIT PULL] zstd support (lib, btrfs, squashfs)
2017-09-09 2:48 ` Nick Terrell
(?)
@ 2017-09-09 4:14 ` Herbert Xu
-1 siblings, 0 replies; 9+ messages in thread
From: Herbert Xu @ 2017-09-09 4:14 UTC (permalink / raw)
To: Nick Terrell
Cc: Chris Mason, Linus Torvalds, Linux Kernel Mailing List,
linux-btrfs, Phillip Lougher, Sean Purcell, David Sterba
On Sat, Sep 09, 2017 at 02:48:55AM +0000, Nick Terrell wrote:
>
> There is a patch [1] floating around adding zstd support to compressed RAM
> that uses the crypto API.
>
> [1] https://lkml.kernel.org/r/20170824014936.4738-1-sergey.senozhatsky%40gmail.com
Yes but I think it would make more sense for the crypto API patches
to be submitted together with its first user. It wouldn't be the
first time for an algorithm to be added with an intended user which
itself never actually gets merged into the kernel.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 9+ messages in thread
* [GIT PULL v2] zstd support (lib, btrfs, squashfs, nocrypto)
2017-09-08 19:33 [GIT PULL] zstd support (lib, btrfs, squashfs) Chris Mason
2017-09-09 0:43 ` Chris Mason
2017-09-09 1:35 ` Herbert Xu
@ 2017-09-11 19:24 ` Chris Mason
2017-09-12 6:30 ` Phillip Lougher
2 siblings, 1 reply; 9+ messages in thread
From: Chris Mason @ 2017-09-11 19:24 UTC (permalink / raw)
To: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Herbert Xu, Nick Terrell, Phillip Lougher, Sean Purcell
Cc: David Sterba
Hi Linus,
Nick Terrell's patch series to add zstd support to the kernel has been
floating around for a while. After talking with Dave Sterba, Herbert
and Phillip, we decided to send the whole thing in as one pull request.
Herbert had asked about the crypto patch when we discussed the pull, but
I didn't realize he really meant not-right-now. I've rebased it out of
this branch, and none of the other patches depended on it.
I have things in my zstd-minimal branch:
git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git zstd-minimal
There's a trivial conflict with the main btrfs pull from last week.
Dave's pull deletes BTRFS_COMPRESS_LAST in fs/btrfs/compression.h, and
I've put the sample resolution in a branch named zstd-4.14-merge.
zstd is a big win in speed over zlib and in compression ratio over lzo,
and the compression team here at FB has gotten great results using it in
production. Nick will continue to update the kernel side with new
improvements from the open source zstd userland code.
Nick has a number of benchmarks for the main zstd code in his lib/zstd
commit:
============
I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD. I benchmarked using `silesia.tar` [3], which is
211,988,480 B large. Run the following commands for the benchmark:
sudo modprobe zstd_compress_test
sudo mknod zstd_compress_test c 245 0
sudo cp silesia.tar zstd_compress_test
The time is reported by the time of the userland `cp`.
The MB/s is computed with
1,536,217,008 B / time(buffer size, hash)
which includes the time to copy from userland.
The Adjusted MB/s is computed with
1,536,217,088 B / (time(buffer size, hash) - time(buffer size, none)).
The memory reported is the amount of memory the compressor requests.
| Method | Size (B) | Time (s) | Ratio | MB/s | Adj MB/s | Mem (MB) |
|----------|----------|----------|-------|---------|----------|----------|
| none | 11988480 | 0.100 | 1 | 2119.88 | - | - |
| zstd -1 | 73645762 | 1.044 | 2.878 | 203.05 | 224.56 | 1.23 |
| zstd -3 | 66988878 | 1.761 | 3.165 | 120.38 | 127.63 | 2.47 |
| zstd -5 | 65001259 | 2.563 | 3.261 | 82.71 | 86.07 | 2.86 |
| zstd -10 | 60165346 | 13.242 | 3.523 | 16.01 | 16.13 | 13.22 |
| zstd -15 | 58009756 | 47.601 | 3.654 | 4.45 | 4.46 | 21.61 |
| zstd -19 | 54014593 | 102.835 | 3.925 | 2.06 | 2.06 | 60.15 |
| zlib -1 | 77260026 | 2.895 | 2.744 | 73.23 | 75.85 | 0.27 |
| zlib -3 | 72972206 | 4.116 | 2.905 | 51.50 | 52.79 | 0.27 |
| zlib -6 | 68190360 | 9.633 | 3.109 | 22.01 | 22.24 | 0.27 |
| zlib -9 | 67613382 | 22.554 | 3.135 | 9.40 | 9.44 | 0.27 |
I benchmarked zstd decompression using the same method on the same machine.
The benchmark file is located in the upstream zstd repo under
`contrib/linux-kernel/zstd_decompress_test.c` [4]. The memory reported is
the amount of memory required to decompress data compressed with the given
compression level. If you know the maximum size of your input, you can
reduce the memory usage of decompression irrespective of the compression
level.
| Method | Time (s) | MB/s | Adjusted MB/s | Memory (MB) |
|----------|----------|---------|---------------|-------------|
| none | 0.025 | 8479.54 | - | - |
| zstd -1 | 0.358 | 592.15 | 636.60 | 0.84 |
| zstd -3 | 0.396 | 535.32 | 571.40 | 1.46 |
| zstd -5 | 0.396 | 535.32 | 571.40 | 1.46 |
| zstd -10 | 0.374 | 566.81 | 607.42 | 2.51 |
| zstd -15 | 0.379 | 559.34 | 598.84 | 4.61 |
| zstd -19 | 0.412 | 514.54 | 547.77 | 8.80 |
| zlib -1 | 0.940 | 225.52 | 231.68 | 0.04 |
| zlib -3 | 0.883 | 240.08 | 247.07 | 0.04 |
| zlib -6 | 0.844 | 251.17 | 258.84 | 0.04 |
| zlib -9 | 0.837 | 253.27 | 287.64 | 0.04 |
=======================
I ran a long series of tests and benchmarks on the btrfs side and
the gains are very similar to the core benchmarks Nick ran.
Nick Terrell (3) commits (+14222/-12):
btrfs: Add zstd support (+468/-12)
lib: Add zstd modules (+13014/-0)
lib: Add xxhash module (+740/-0)
Sean Purcell (1) commits (+178/-0):
squashfs: Add zstd support
Total: (4) commits (+14400/-12)
fs/btrfs/Kconfig | 2 +
fs/btrfs/Makefile | 2 +-
fs/btrfs/compression.c | 1 +
fs/btrfs/compression.h | 6 +-
fs/btrfs/ctree.h | 1 +
fs/btrfs/disk-io.c | 2 +
fs/btrfs/ioctl.c | 6 +-
fs/btrfs/props.c | 6 +
fs/btrfs/super.c | 12 +-
fs/btrfs/sysfs.c | 2 +
fs/btrfs/zstd.c | 432 ++++++
fs/squashfs/Kconfig | 14 +
fs/squashfs/Makefile | 1 +
fs/squashfs/decompressor.c | 7 +
fs/squashfs/decompressor.h | 4 +
fs/squashfs/squashfs_fs.h | 1 +
fs/squashfs/zstd_wrapper.c | 151 ++
include/linux/xxhash.h | 236 +++
include/linux/zstd.h | 1157 +++++++++++++++
include/uapi/linux/btrfs.h | 8 +-
lib/Kconfig | 11 +
lib/Makefile | 3 +
lib/xxhash.c | 500 +++++++
lib/zstd/Makefile | 18 +
lib/zstd/bitstream.h | 374 +++++
lib/zstd/compress.c | 3484 ++++++++++++++++++++++++++++++++++++++++++++
lib/zstd/decompress.c | 2528 ++++++++++++++++++++++++++++++++
lib/zstd/entropy_common.c | 243 +++
lib/zstd/error_private.h | 53 +
lib/zstd/fse.h | 575 ++++++++
lib/zstd/fse_compress.c | 795 ++++++++++
lib/zstd/fse_decompress.c | 332 +++++
lib/zstd/huf.h | 212 +++
lib/zstd/huf_compress.c | 770 ++++++++++
lib/zstd/huf_decompress.c | 960 ++++++++++++
lib/zstd/mem.h | 151 ++
lib/zstd/zstd_common.c | 75 +
lib/zstd/zstd_internal.h | 263 ++++
lib/zstd/zstd_opt.h | 1014 +++++++++++++
39 files changed, 14400 insertions(+), 12 deletions(-)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [GIT PULL v2] zstd support (lib, btrfs, squashfs, nocrypto)
2017-09-11 19:24 ` [GIT PULL v2] zstd support (lib, btrfs, squashfs, nocrypto) Chris Mason
@ 2017-09-12 6:30 ` Phillip Lougher
0 siblings, 0 replies; 9+ messages in thread
From: Phillip Lougher @ 2017-09-12 6:30 UTC (permalink / raw)
To: Chris Mason
Cc: Linus Torvalds, Linux Kernel Mailing List, linux-btrfs,
Herbert Xu, Nick Terrell, Phillip Lougher, Sean Purcell,
David Sterba
On Mon, Sep 11, 2017 at 8:24 PM, Chris Mason <clm@fb.com> wrote:
> Hi Linus,
>
> Nick Terrell's patch series to add zstd support to the kernel has been
> floating around for a while. After talking with Dave Sterba, Herbert
> and Phillip, we decided to send the whole thing in as one pull request.
As the Squashfs maintainer I have ACKed the Squashfs patch, and I
confirm that I've agreed to have it included in this one pull request.
Phillip
>
> Herbert had asked about the crypto patch when we discussed the pull, but
> I didn't realize he really meant not-right-now. I've rebased it out of
> this branch, and none of the other patches depended on it.
>
> I have things in my zstd-minimal branch:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git zstd-minimal
>
> There's a trivial conflict with the main btrfs pull from last week.
> Dave's pull deletes BTRFS_COMPRESS_LAST in fs/btrfs/compression.h, and
> I've put the sample resolution in a branch named zstd-4.14-merge.
>
> zstd is a big win in speed over zlib and in compression ratio over lzo,
> and the compression team here at FB has gotten great results using it in
> production. Nick will continue to update the kernel side with new
> improvements from the open source zstd userland code.
>
> Nick has a number of benchmarks for the main zstd code in his lib/zstd
> commit:
>
> ============
> I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
> The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
> 16 GB of RAM, and a SSD. I benchmarked using `silesia.tar` [3], which is
> 211,988,480 B large. Run the following commands for the benchmark:
>
> sudo modprobe zstd_compress_test
> sudo mknod zstd_compress_test c 245 0
> sudo cp silesia.tar zstd_compress_test
>
> The time is reported by the time of the userland `cp`.
> The MB/s is computed with
>
> 1,536,217,008 B / time(buffer size, hash)
>
> which includes the time to copy from userland.
> The Adjusted MB/s is computed with
>
> 1,536,217,088 B / (time(buffer size, hash) - time(buffer size, none)).
>
> The memory reported is the amount of memory the compressor requests.
>
> | Method | Size (B) | Time (s) | Ratio | MB/s | Adj MB/s | Mem (MB) |
> |----------|----------|----------|-------|---------|----------|----------|
> | none | 11988480 | 0.100 | 1 | 2119.88 | - | - |
> | zstd -1 | 73645762 | 1.044 | 2.878 | 203.05 | 224.56 | 1.23 |
> | zstd -3 | 66988878 | 1.761 | 3.165 | 120.38 | 127.63 | 2.47 |
> | zstd -5 | 65001259 | 2.563 | 3.261 | 82.71 | 86.07 | 2.86 |
> | zstd -10 | 60165346 | 13.242 | 3.523 | 16.01 | 16.13 | 13.22 |
> | zstd -15 | 58009756 | 47.601 | 3.654 | 4.45 | 4.46 | 21.61 |
> | zstd -19 | 54014593 | 102.835 | 3.925 | 2.06 | 2.06 | 60.15 |
> | zlib -1 | 77260026 | 2.895 | 2.744 | 73.23 | 75.85 | 0.27 |
> | zlib -3 | 72972206 | 4.116 | 2.905 | 51.50 | 52.79 | 0.27 |
> | zlib -6 | 68190360 | 9.633 | 3.109 | 22.01 | 22.24 | 0.27 |
> | zlib -9 | 67613382 | 22.554 | 3.135 | 9.40 | 9.44 | 0.27 |
>
> I benchmarked zstd decompression using the same method on the same machine.
> The benchmark file is located in the upstream zstd repo under
> `contrib/linux-kernel/zstd_decompress_test.c` [4]. The memory reported is
> the amount of memory required to decompress data compressed with the given
> compression level. If you know the maximum size of your input, you can
> reduce the memory usage of decompression irrespective of the compression
> level.
>
> | Method | Time (s) | MB/s | Adjusted MB/s | Memory (MB) |
> |----------|----------|---------|---------------|-------------|
> | none | 0.025 | 8479.54 | - | - |
> | zstd -1 | 0.358 | 592.15 | 636.60 | 0.84 |
> | zstd -3 | 0.396 | 535.32 | 571.40 | 1.46 |
> | zstd -5 | 0.396 | 535.32 | 571.40 | 1.46 |
> | zstd -10 | 0.374 | 566.81 | 607.42 | 2.51 |
> | zstd -15 | 0.379 | 559.34 | 598.84 | 4.61 |
> | zstd -19 | 0.412 | 514.54 | 547.77 | 8.80 |
> | zlib -1 | 0.940 | 225.52 | 231.68 | 0.04 |
> | zlib -3 | 0.883 | 240.08 | 247.07 | 0.04 |
> | zlib -6 | 0.844 | 251.17 | 258.84 | 0.04 |
> | zlib -9 | 0.837 | 253.27 | 287.64 | 0.04 |
>
> =======================
>
> I ran a long series of tests and benchmarks on the btrfs side and
> the gains are very similar to the core benchmarks Nick ran.
>
> Nick Terrell (3) commits (+14222/-12):
> btrfs: Add zstd support (+468/-12)
> lib: Add zstd modules (+13014/-0)
> lib: Add xxhash module (+740/-0)
>
> Sean Purcell (1) commits (+178/-0):
> squashfs: Add zstd support
>
> Total: (4) commits (+14400/-12)
>
> fs/btrfs/Kconfig | 2 +
> fs/btrfs/Makefile | 2 +-
> fs/btrfs/compression.c | 1 +
> fs/btrfs/compression.h | 6 +-
> fs/btrfs/ctree.h | 1 +
> fs/btrfs/disk-io.c | 2 +
> fs/btrfs/ioctl.c | 6 +-
> fs/btrfs/props.c | 6 +
> fs/btrfs/super.c | 12 +-
> fs/btrfs/sysfs.c | 2 +
> fs/btrfs/zstd.c | 432 ++++++
> fs/squashfs/Kconfig | 14 +
> fs/squashfs/Makefile | 1 +
> fs/squashfs/decompressor.c | 7 +
> fs/squashfs/decompressor.h | 4 +
> fs/squashfs/squashfs_fs.h | 1 +
> fs/squashfs/zstd_wrapper.c | 151 ++
> include/linux/xxhash.h | 236 +++
> include/linux/zstd.h | 1157 +++++++++++++++
> include/uapi/linux/btrfs.h | 8 +-
> lib/Kconfig | 11 +
> lib/Makefile | 3 +
> lib/xxhash.c | 500 +++++++
> lib/zstd/Makefile | 18 +
> lib/zstd/bitstream.h | 374 +++++
> lib/zstd/compress.c | 3484 ++++++++++++++++++++++++++++++++++++++++++++
> lib/zstd/decompress.c | 2528 ++++++++++++++++++++++++++++++++
> lib/zstd/entropy_common.c | 243 +++
> lib/zstd/error_private.h | 53 +
> lib/zstd/fse.h | 575 ++++++++
> lib/zstd/fse_compress.c | 795 ++++++++++
> lib/zstd/fse_decompress.c | 332 +++++
> lib/zstd/huf.h | 212 +++
> lib/zstd/huf_compress.c | 770 ++++++++++
> lib/zstd/huf_decompress.c | 960 ++++++++++++
> lib/zstd/mem.h | 151 ++
> lib/zstd/zstd_common.c | 75 +
> lib/zstd/zstd_internal.h | 263 ++++
> lib/zstd/zstd_opt.h | 1014 +++++++++++++
> 39 files changed, 14400 insertions(+), 12 deletions(-)
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-09-12 6:30 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-08 19:33 [GIT PULL] zstd support (lib, btrfs, squashfs) Chris Mason
2017-09-09 0:43 ` Chris Mason
2017-09-09 1:35 ` Herbert Xu
2017-09-09 2:19 ` Chris Mason
2017-09-09 2:48 ` Nick Terrell
2017-09-09 2:48 ` Nick Terrell
2017-09-09 4:14 ` Herbert Xu
2017-09-11 19:24 ` [GIT PULL v2] zstd support (lib, btrfs, squashfs, nocrypto) Chris Mason
2017-09-12 6:30 ` Phillip Lougher
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.