From: "HAGIO KAZUHITO(萩尾 一仁)" <k-hagio-ab@nec.com>
To: Tao Liu <ltao@redhat.com>,
"kexec@lists.infradead.org" <kexec@lists.infradead.org>
Subject: RE: [PATCH 00/11] makedumpfile: Add zstd support for makedumpfile
Date: Tue, 14 Sep 2021 07:04:24 +0000 [thread overview]
Message-ID: <TYYPR01MB677786B29FF77911EE5E06F2DDDA9@TYYPR01MB6777.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <20210910103318.292017-1-ltao@redhat.com>
Hi Tao Liu,
Thanks for the patchset!
-----Original Message-----
> This patch set adds ZSTD compression support to makedumpfile. With ZSTD compression
> support, the vmcore dump size and time consumption can have a better balance than
> zlib/lzo/snappy.
>
> How to build:
>
> Build using make:
> $ make USEZSTD=on
>
> Performance Comparison:
>
> How to measure
>
> I took a x86_64 machine which had 4T memory, and the compression level
> range from (-3 to 4) for ZSTD, as well as zlib/lzo/snappy compression.
> All testing was done by makedumpfile single thread mode.
>
> As for compression performance testing, in order to avoid the performance
> bottle neck of disk I/O, I used the following makedumpfile cmd, which took
> lzo compression as an example. "--dry-run" will not write any data to disk,
> "--show-stat" will output the vmcore size after compression, and the time
> consumption can be collected from the output logs.
>
> $ makedumpfile -d 0 -l /proc/kcore vmcore --dry-run --show-stat
>
>
> As for decompression performance testing, I only tested the (-d 31) case,
> because the vmcore size of (-d 0) case is too big to fit in the disk, in
> addtion, to read a oversized file from disk will encounter the disk I/O
> bottle neck.
>
> I triggered a kernel crash and collected a vmcore. Then I converted the
> vmcore into specific compression format using the following makedumpfile
> cmd, which would get a lzo format vmcore as an example:
>
> $ makedumpfile -l vmcore vmcore.lzo
>
> After all vmcores were ready, I used the following cmd to perform the
> decompression, the time consumption can be collected from the logs.
>
> $ makedumpfile -F vmcore.lzo --dry-run --show-stat
>
>
> Result charts
>
> For compression:
>
> makedumpfile -d31 | makedumpfile -d0
> Compression time vmcore size | Compression time vmcore size
> zstd-3 325.516446 5285179595 | 8205.452248 51715430204
> zstd-2 332.069432 5319726604 | 8057.381371 51732062793
> zstd-1 309.942773 5730516274 | 8138.060786 52136191571
> zstd0 439.773076 4673859661 | 8873.059963 50993669657
> zstd1 406.68036 4700959521 | 8259.417132 51036900055
> zstd2 397.195643 4699263608 | 8230.308291 51030410942
> zstd3 436.491632 4673306398 | 8803.970103 51043393637
> zstd4 543.363928 4668419304 | 8991.240244 51058088514
> zlib 561.217381 8514803195 | 14381.755611 78199283893
> lzo 248.175953 16696411879 | 6057.528781 90020895741
> snappy 231.868312 11782236674 | 5290.919894 245661288355
>
> For decompression:
>
> makedumpfile -d31
> decompress time vmcore size
> zstd-3 477.543396 5289373448
> zstd-2 478.034534 5327454123
> zstd-1 459.066807 5748037931
> zstd0 561.687525 4680009013
> zstd1 547.248917 4706358547
> zstd2 544.219758 4704780719
> zstd3 555.726343 4680009013
> zstd4 558.031721 4675545933
> zlib 630.965426 8555376229
> lzo 427.292107 16849457649
> snappy 446.542806 11841407957
>
> Discussion
>
> For zstd range from -3 to 4, compression level 2 (ZSTD_dfast) has
> the best time consumption and vmcore dump size balance.
Do you have a result of -d 1 compression test? I think -d 0 is not
practical, I would like to see a -d 1 result of such a large vmcore.
And just out of curiosity, what version of zstd are you using?
When I tested zstd last time, compression level 1 was faster than 2, iirc.
btw, ZSTD_dfast is an enum of ZSTD_strategy, not for compression level?
(no need to update for now, I will review later)
Thanks,
Kazu
>
> For zstd2/zlib/lzo/snappy, zstd2 has the smallest vmcore size, also
> the best time consumption and vmcore dump size balance.
>
> Tao Liu (11):
> Add dump header for zstd.
> Add command-line processing for zstd
> Add zstd build support
> Notify zstd unsupporting when disabled
> Add single thread zstd compression processing
> Add parallel threads zstd compression processing
> Add single thread zstd uncompression processing
> Add parallel threads zstd uncompression processing
> Add zstd help message
> Add zstd manual description
> Add zstd README description
>
> Makefile | 5 +++
> README | 5 ++-
> diskdump_mod.h | 1 +
> makedumpfile.8 | 7 ++--
> makedumpfile.c | 101 +++++++++++++++++++++++++++++++++++++++++++++----
> makedumpfile.h | 10 +++++
> print_info.c | 30 ++++++++++-----
> 7 files changed, 138 insertions(+), 21 deletions(-)
>
> --
> 2.29.2
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
next prev parent reply other threads:[~2021-09-14 7:04 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-10 10:33 [PATCH 00/11] makedumpfile: Add zstd support for makedumpfile Tao Liu
2021-09-10 10:33 ` [PATCH 01/11] makedumpfile: Add dump header for zstd Tao Liu
2021-09-10 10:33 ` [PATCH 02/11] makedumpfile: Add command-line processing " Tao Liu
2021-09-10 10:33 ` [PATCH 03/11] makedumpfile: Add zstd build support Tao Liu
2021-09-10 10:33 ` [PATCH 04/11] makedumpfile: Notify zstd unsupporting when disabled Tao Liu
2021-09-10 10:33 ` [PATCH 05/11] makedumpfile: Add single thread zstd compression processing Tao Liu
2021-09-10 10:33 ` [PATCH 06/11] makedumpfile: Add parallel threads " Tao Liu
2021-09-10 10:33 ` [PATCH 07/11] makedumpfile: Add single thread zstd uncompression processing Tao Liu
2021-09-10 10:33 ` [PATCH 08/11] makedumpfile: Add parallel threads " Tao Liu
2021-09-10 10:33 ` [PATCH 09/11] makedumpfile: Add zstd help message Tao Liu
2021-09-10 10:33 ` [PATCH 10/11] makedumpfile: Add zstd manual description Tao Liu
2021-09-10 10:33 ` [PATCH 11/11] makedumpfile: Add zstd README description Tao Liu
2021-09-14 7:04 ` HAGIO KAZUHITO(萩尾 一仁) [this message]
2021-09-14 8:33 ` [PATCH 00/11] makedumpfile: Add zstd support for makedumpfile Tao Liu
2021-09-17 1:34 ` HAGIO KAZUHITO(萩尾 一仁)
2021-09-17 2:31 ` HAGIO KAZUHITO(萩尾 一仁)
2021-09-17 7:03 ` HAGIO KAZUHITO(萩尾 一仁)
2021-09-21 9:26 ` Tao Liu
2021-09-22 2:21 ` HAGIO KAZUHITO(萩尾 一仁)
2021-09-22 8:16 ` HAGIO KAZUHITO(萩尾 一仁)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=TYYPR01MB677786B29FF77911EE5E06F2DDDA9@TYYPR01MB6777.jpnprd01.prod.outlook.com \
--to=k-hagio-ab@nec.com \
--cc=kexec@lists.infradead.org \
--cc=ltao@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.