From: Chao Yu <chao@kernel.org> To: Jaegeuk Kim <jaegeuk@kernel.org>, linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, kernel-team@android.com Subject: Re: [f2fs-dev] [PATCH] f2fs: change maximum zstd compression buffer size Date: Tue, 5 May 2020 10:08:37 +0800 [thread overview] Message-ID: <5dadaf04-ef9e-f6a8-c132-8fa072e8e6fe@kernel.org> (raw) In-Reply-To: <7177aab9-630e-e077-7005-0023c93134b3@kernel.org> On 2020-5-5 9:52, Chao Yu wrote: > On 2020-5-4 22:30, Jaegeuk Kim wrote: >> From: Daeho Jeong <daehojeong@google.com> >> >> Current zstd compression buffer size is one page and header size less >> than cluster size. By this, zstd compression always succeeds even if >> the real compression data is failed to fit into the buffer size, and >> eventually reading the cluster returns I/O error with the corrupted >> compression data. > > What's the root cause of this issue? I didn't get it. > >> >> Signed-off-by: Daeho Jeong <daehojeong@google.com> >> --- >> fs/f2fs/compress.c | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c >> index 4c7eaeee52336..a9fa8049b295f 100644 >> --- a/fs/f2fs/compress.c >> +++ b/fs/f2fs/compress.c >> @@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) >> cc->private = workspace; >> cc->private2 = stream; >> >> - cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE; >> + cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size); Max size of intermediate buffer has been set as below? - zstd_init_compress_ctx - params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0) - ZSTD_CStreamWorkspaceBound(params.cParams) - outBuffSize = ZSTD_compressBound(blockSize) + 1; - workspace_size = ... + outBuffSize + ... - workspace = f2fs_kvmalloc(workspace_size) > > In my machine, the value is 66572 which is much larger than size of dst buffer, > so, in where we can tell the real size of dst buffer to zstd compressor? > Otherwise, if compressed data size is larger than dst buffer size, when we flush > compressed data into dst buffer, we may suffer overflow. > >> return 0; >> } >> >> @@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc) >> ZSTD_inBuffer inbuf; >> ZSTD_outBuffer outbuf; >> int src_size = cc->rlen; >> - int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; >> + int dst_size = cc->clen; >> int ret; >> >> inbuf.pos = 0; >> > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
WARNING: multiple messages have this Message-ID (diff)
From: Chao Yu <chao@kernel.org> To: Jaegeuk Kim <jaegeuk@kernel.org>, linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, kernel-team@android.com Subject: Re: [f2fs-dev] [PATCH] f2fs: change maximum zstd compression buffer size Date: Tue, 5 May 2020 10:08:37 +0800 [thread overview] Message-ID: <5dadaf04-ef9e-f6a8-c132-8fa072e8e6fe@kernel.org> (raw) In-Reply-To: <7177aab9-630e-e077-7005-0023c93134b3@kernel.org> On 2020-5-5 9:52, Chao Yu wrote: > On 2020-5-4 22:30, Jaegeuk Kim wrote: >> From: Daeho Jeong <daehojeong@google.com> >> >> Current zstd compression buffer size is one page and header size less >> than cluster size. By this, zstd compression always succeeds even if >> the real compression data is failed to fit into the buffer size, and >> eventually reading the cluster returns I/O error with the corrupted >> compression data. > > What's the root cause of this issue? I didn't get it. > >> >> Signed-off-by: Daeho Jeong <daehojeong@google.com> >> --- >> fs/f2fs/compress.c | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c >> index 4c7eaeee52336..a9fa8049b295f 100644 >> --- a/fs/f2fs/compress.c >> +++ b/fs/f2fs/compress.c >> @@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) >> cc->private = workspace; >> cc->private2 = stream; >> >> - cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE; >> + cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size); Max size of intermediate buffer has been set as below? - zstd_init_compress_ctx - params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0) - ZSTD_CStreamWorkspaceBound(params.cParams) - outBuffSize = ZSTD_compressBound(blockSize) + 1; - workspace_size = ... + outBuffSize + ... - workspace = f2fs_kvmalloc(workspace_size) > > In my machine, the value is 66572 which is much larger than size of dst buffer, > so, in where we can tell the real size of dst buffer to zstd compressor? > Otherwise, if compressed data size is larger than dst buffer size, when we flush > compressed data into dst buffer, we may suffer overflow. > >> return 0; >> } >> >> @@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc) >> ZSTD_inBuffer inbuf; >> ZSTD_outBuffer outbuf; >> int src_size = cc->rlen; >> - int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; >> + int dst_size = cc->clen; >> int ret; >> >> inbuf.pos = 0; >> > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
next prev parent reply other threads:[~2020-05-05 2:08 UTC|newest] Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-05-04 14:30 [PATCH] f2fs: change maximum zstd compression buffer size Jaegeuk Kim 2020-05-04 14:30 ` [f2fs-dev] " Jaegeuk Kim 2020-05-05 1:52 ` Chao Yu 2020-05-05 1:52 ` Chao Yu 2020-05-05 2:08 ` Chao Yu [this message] 2020-05-05 2:08 ` Chao Yu 2020-05-05 23:05 ` Jaegeuk Kim 2020-05-05 23:05 ` Jaegeuk Kim 2020-05-06 7:45 ` Chao Yu 2020-05-06 7:45 ` Chao Yu 2020-05-06 14:56 ` Jaegeuk Kim 2020-05-06 14:56 ` Jaegeuk Kim 2020-05-07 2:48 ` Chao Yu 2020-05-07 2:48 ` Chao Yu 2020-05-07 9:16 ` Chao Yu 2020-05-07 9:16 ` Chao Yu
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=5dadaf04-ef9e-f6a8-c132-8fa072e8e6fe@kernel.org \ --to=chao@kernel.org \ --cc=jaegeuk@kernel.org \ --cc=kernel-team@android.com \ --cc=linux-f2fs-devel@lists.sourceforge.net \ --cc=linux-kernel@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.