All of lore.kernel.org
 help / color / mirror / Atom feed
From: Nick Terrell <nickrterrell@gmail.com>
To: Herbert Xu <herbert@gondor.apana.org.au>
Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org,
	squashfs-devel@lists.sourceforge.net,
	linux-f2fs-devel@lists.sourceforge.net,
	linux-kernel@vger.kernel.org, Kernel Team <Kernel-team@fb.com>,
	Nick Terrell <nickrterrell@gmail.com>,
	Nick Terrell <terrelln@fb.com>, Chris Mason <clm@fb.com>,
	Petr Malat <oss@malat.biz>, Johannes Weiner <jweiner@fb.com>,
	Niket Agarwal <niketa@fb.com>, Yann Collet <cyan@fb.com>,
	Christoph Hellwig <hch@infradead.org>
Subject: [PATCH v4 8/9] lib: unzstd: Switch to the zstd-1.4.6 API
Date: Tue, 29 Sep 2020 23:53:17 -0700	[thread overview]
Message-ID: <20200930065318.3326526-9-nickrterrell@gmail.com> (raw)
In-Reply-To: <20200930065318.3326526-1-nickrterrell@gmail.com>

From: Nick Terrell <terrelln@fb.com>

Move away from the compatibility wrapper to the zstd-1.4.6 API. This
code is functionally equivalent.

Signed-off-by: Nick Terrell <terrelln@fb.com>
---
 lib/decompress_unzstd.c | 40 ++++++++++++++--------------------------
 1 file changed, 14 insertions(+), 26 deletions(-)

diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index a79f705f236d..d4685df0e120 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -73,7 +73,8 @@
 
 #include <linux/decompress/mm.h>
 #include <linux/kernel.h>
-#include <linux/zstd_compat.h>
+#include <linux/zstd.h>
+#include <linux/zstd_errors.h>
 
 /* 128MB is the maximum window size supported by zstd. */
 #define ZSTD_WINDOWSIZE_MAX	(1 << ZSTD_WINDOWLOG_MAX)
@@ -120,9 +121,9 @@ static int INIT decompress_single(const u8 *in_buf, long in_len, u8 *out_buf,
 				  long out_len, long *in_pos,
 				  void (*error)(char *x))
 {
-	const size_t wksp_size = ZSTD_DCtxWorkspaceBound();
+	const size_t wksp_size = ZSTD_estimateDCtxSize();
 	void *wksp = large_malloc(wksp_size);
-	ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size);
+	ZSTD_DCtx *dctx = ZSTD_initStaticDCtx(wksp, wksp_size);
 	int err;
 	size_t ret;
 
@@ -165,7 +166,6 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
 {
 	ZSTD_inBuffer in;
 	ZSTD_outBuffer out;
-	ZSTD_frameParams params;
 	void *in_allocated = NULL;
 	void *out_allocated = NULL;
 	void *wksp = NULL;
@@ -229,36 +229,24 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
 	out.size = out_len;
 
 	/*
-	 * We need to know the window size to allocate the ZSTD_DStream.
-	 * Since we are streaming, we need to allocate a buffer for the sliding
-	 * window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
-	 * (8 MB), so it is important to use the actual value so as not to
-	 * waste memory when it is smaller.
+	 * Zstd determines the workspace size from the window size written
+	 * into the frame header. This ensures that we use the minimum value
+	 * possible, since the window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
+	 * (1 GB), so it is very important to use the actual value.
 	 */
-	ret = ZSTD_getFrameParams(&params, in.src, in.size);
+	wksp_size = ZSTD_estimateDStreamSize_fromFrame(in.src, in.size);
 	err = handle_zstd_error(ret, error);
 	if (err)
 		goto out;
-	if (ret != 0) {
-		error("ZSTD-compressed data has an incomplete frame header");
-		err = -1;
-		goto out;
-	}
-	if (params.windowSize > ZSTD_WINDOWSIZE_MAX) {
-		error("ZSTD-compressed data has too large a window size");
+	wksp = large_malloc(wksp_size);
+	if (wksp == NULL) {
+		error("Out of memory while allocating ZSTD_DStream");
 		err = -1;
 		goto out;
 	}
-
-	/*
-	 * Allocate the ZSTD_DStream now that we know how much memory is
-	 * required.
-	 */
-	wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize);
-	wksp = large_malloc(wksp_size);
-	dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size);
+	dstream = ZSTD_initStaticDStream(wksp, wksp_size);
 	if (dstream == NULL) {
-		error("Out of memory while allocating ZSTD_DStream");
+		error("ZSTD_initStaticDStream failed");
 		err = -1;
 		goto out;
 	}
-- 
2.28.0


WARNING: multiple messages have this Message-ID (diff)
From: Nick Terrell <nickrterrell@gmail.com>
To: Herbert Xu <herbert@gondor.apana.org.au>
Cc: squashfs-devel@lists.sourceforge.net,
	Johannes Weiner <jweiner@fb.com>,
	Christoph Hellwig <hch@infradead.org>,
	Nick Terrell <nickrterrell@gmail.com>, Yann Collet <cyan@fb.com>,
	linux-kernel@vger.kernel.org,
	linux-f2fs-devel@lists.sourceforge.net,
	Petr Malat <oss@malat.biz>, Chris Mason <clm@fb.com>,
	Nick Terrell <terrelln@fb.com>,
	linux-crypto@vger.kernel.org, Kernel Team <Kernel-team@fb.com>,
	Niket Agarwal <niketa@fb.com>,
	linux-btrfs@vger.kernel.org
Subject: [f2fs-dev] [PATCH v4 8/9] lib: unzstd: Switch to the zstd-1.4.6 API
Date: Tue, 29 Sep 2020 23:53:17 -0700	[thread overview]
Message-ID: <20200930065318.3326526-9-nickrterrell@gmail.com> (raw)
In-Reply-To: <20200930065318.3326526-1-nickrterrell@gmail.com>

From: Nick Terrell <terrelln@fb.com>

Move away from the compatibility wrapper to the zstd-1.4.6 API. This
code is functionally equivalent.

Signed-off-by: Nick Terrell <terrelln@fb.com>
---
 lib/decompress_unzstd.c | 40 ++++++++++++++--------------------------
 1 file changed, 14 insertions(+), 26 deletions(-)

diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index a79f705f236d..d4685df0e120 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -73,7 +73,8 @@
 
 #include <linux/decompress/mm.h>
 #include <linux/kernel.h>
-#include <linux/zstd_compat.h>
+#include <linux/zstd.h>
+#include <linux/zstd_errors.h>
 
 /* 128MB is the maximum window size supported by zstd. */
 #define ZSTD_WINDOWSIZE_MAX	(1 << ZSTD_WINDOWLOG_MAX)
@@ -120,9 +121,9 @@ static int INIT decompress_single(const u8 *in_buf, long in_len, u8 *out_buf,
 				  long out_len, long *in_pos,
 				  void (*error)(char *x))
 {
-	const size_t wksp_size = ZSTD_DCtxWorkspaceBound();
+	const size_t wksp_size = ZSTD_estimateDCtxSize();
 	void *wksp = large_malloc(wksp_size);
-	ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size);
+	ZSTD_DCtx *dctx = ZSTD_initStaticDCtx(wksp, wksp_size);
 	int err;
 	size_t ret;
 
@@ -165,7 +166,6 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
 {
 	ZSTD_inBuffer in;
 	ZSTD_outBuffer out;
-	ZSTD_frameParams params;
 	void *in_allocated = NULL;
 	void *out_allocated = NULL;
 	void *wksp = NULL;
@@ -229,36 +229,24 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
 	out.size = out_len;
 
 	/*
-	 * We need to know the window size to allocate the ZSTD_DStream.
-	 * Since we are streaming, we need to allocate a buffer for the sliding
-	 * window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
-	 * (8 MB), so it is important to use the actual value so as not to
-	 * waste memory when it is smaller.
+	 * Zstd determines the workspace size from the window size written
+	 * into the frame header. This ensures that we use the minimum value
+	 * possible, since the window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
+	 * (1 GB), so it is very important to use the actual value.
 	 */
-	ret = ZSTD_getFrameParams(&params, in.src, in.size);
+	wksp_size = ZSTD_estimateDStreamSize_fromFrame(in.src, in.size);
 	err = handle_zstd_error(ret, error);
 	if (err)
 		goto out;
-	if (ret != 0) {
-		error("ZSTD-compressed data has an incomplete frame header");
-		err = -1;
-		goto out;
-	}
-	if (params.windowSize > ZSTD_WINDOWSIZE_MAX) {
-		error("ZSTD-compressed data has too large a window size");
+	wksp = large_malloc(wksp_size);
+	if (wksp == NULL) {
+		error("Out of memory while allocating ZSTD_DStream");
 		err = -1;
 		goto out;
 	}
-
-	/*
-	 * Allocate the ZSTD_DStream now that we know how much memory is
-	 * required.
-	 */
-	wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize);
-	wksp = large_malloc(wksp_size);
-	dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size);
+	dstream = ZSTD_initStaticDStream(wksp, wksp_size);
 	if (dstream == NULL) {
-		error("Out of memory while allocating ZSTD_DStream");
+		error("ZSTD_initStaticDStream failed");
 		err = -1;
 		goto out;
 	}
-- 
2.28.0



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

  parent reply	other threads:[~2020-09-30  6:50 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-30  6:53 [PATCH v4 0/9] Update to zstd-1.4.6 Nick Terrell
2020-09-30  6:53 ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 1/9] lib: zstd: Add zstd compatibility wrapper Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 2/9] lib: zstd: Add decompress_sources.h for decompress_unzstd Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 3/9] lib: zstd: Upgrade to latest upstream zstd version 1.4.6 Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-10-01 20:50   ` kernel test robot
2020-10-01 20:50     ` kernel test robot
2020-10-01 20:50   ` [PATCH] lib: zstd: fix semicolon.cocci warnings kernel test robot
2020-10-01 20:50     ` kernel test robot
2020-10-01 20:50     ` [f2fs-dev] " kernel test robot
2020-09-30  6:53 ` [PATCH v4 4/9] crypto: zstd: Switch to zstd-1.4.6 API Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 5/9] btrfs: zstd: Switch to the " Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 6/9] f2fs: " Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 7/9] squashfs: " Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` Nick Terrell [this message]
2020-09-30  6:53   ` [f2fs-dev] [PATCH v4 8/9] lib: unzstd: " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 9/9] lib: zstd: Remove zstd compatibility wrapper Nick Terrell
2020-09-30  6:53   ` [f2fs-dev] " Nick Terrell
2020-09-30  6:53 ` [PATCH v4 0/9] Update to zstd-1.4.6 Christoph Hellwig
2020-09-30  6:53   ` [f2fs-dev] " Christoph Hellwig
2020-09-30 20:05   ` Nick Terrell
2020-09-30 20:05     ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-10-02  6:54     ` Christoph Hellwig
2020-10-02  6:54       ` [f2fs-dev] " Christoph Hellwig
2020-10-02 13:42       ` Chris Mason
2020-10-02 13:42         ` [f2fs-dev] " Chris Mason via Linux-f2fs-devel
2020-09-30 20:49 ` [GIT PULL][PATCH " Nick Terrell
2020-09-30 20:49   ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-10-01 10:18   ` David Sterba
2020-10-01 10:18     ` [f2fs-dev] " David Sterba
2020-10-01 18:35     ` Nick Terrell
2020-10-01 18:35       ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-10-02  6:56       ` Christoph Hellwig
2020-10-02  6:56         ` [f2fs-dev] " Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200930065318.3326526-9-nickrterrell@gmail.com \
    --to=nickrterrell@gmail.com \
    --cc=Kernel-team@fb.com \
    --cc=clm@fb.com \
    --cc=cyan@fb.com \
    --cc=hch@infradead.org \
    --cc=herbert@gondor.apana.org.au \
    --cc=jweiner@fb.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=linux-crypto@vger.kernel.org \
    --cc=linux-f2fs-devel@lists.sourceforge.net \
    --cc=linux-kernel@vger.kernel.org \
    --cc=niketa@fb.com \
    --cc=oss@malat.biz \
    --cc=squashfs-devel@lists.sourceforge.net \
    --cc=terrelln@fb.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.