* Re: [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression
@ 2024-01-22 20:45 Klara Modin
2024-01-22 20:59 ` David Sterba
0 siblings, 1 reply; 5+ messages in thread
From: Klara Modin @ 2024-01-22 20:45 UTC (permalink / raw)
To: wqu; +Cc: linux-btrfs, terrelln, dsterba, josef, clm
Hi,
With this patch [1], small zstd compressed files on btrfs return
garbage when read on my x86_64 system. Reverting this on top of
next-20240122 resolves the issue for me.
From what I can see so far it only directly affects zstd and not zlib
or lzo (with their respective patches) and only when it fits within
the page size. Larger files don't seem to be affected (and does not
trigger my breakpoint on zstd_decompress).
Note that if the files are defragmented, the garbage data seems to be
used (this triggers the breakpoint on zstd_decompress) which makes it
permanent.
This patch seems to be queued for 6.8-rc2.
Please tell me if there's anything else you need.
Kind regards,
Klara Modin
[1]: https://lore.kernel.org/all/0e4ae269b3fd0caf99964c16c98bdd67dbab7150.1704704328.git.wqu@suse.com/
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression
2024-01-22 20:45 [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression Klara Modin
@ 2024-01-22 20:59 ` David Sterba
2024-01-22 23:54 ` David Sterba
0 siblings, 1 reply; 5+ messages in thread
From: David Sterba @ 2024-01-22 20:59 UTC (permalink / raw)
To: Klara Modin; +Cc: wqu, linux-btrfs, terrelln, dsterba, josef, clm
On Mon, Jan 22, 2024 at 09:45:42PM +0100, Klara Modin wrote:
> Hi,
>
> With this patch [1], small zstd compressed files on btrfs return
> garbage when read on my x86_64 system. Reverting this on top of
> next-20240122 resolves the issue for me.
Thanks for the report, this is serious. The patches have been in testing
for some time but haven't uncovered the problems you mention.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression
2024-01-22 20:59 ` David Sterba
@ 2024-01-22 23:54 ` David Sterba
2024-01-23 10:50 ` Klara Modin
0 siblings, 1 reply; 5+ messages in thread
From: David Sterba @ 2024-01-22 23:54 UTC (permalink / raw)
To: David Sterba; +Cc: Klara Modin, wqu, linux-btrfs, terrelln, dsterba, josef, clm
On Mon, Jan 22, 2024 at 09:59:47PM +0100, David Sterba wrote:
> On Mon, Jan 22, 2024 at 09:45:42PM +0100, Klara Modin wrote:
> > Hi,
> >
> > With this patch [1], small zstd compressed files on btrfs return
> > garbage when read on my x86_64 system. Reverting this on top of
> > next-20240122 resolves the issue for me.
>
> Thanks for the report, this is serious. The patches have been in testing
> for some time but haven't uncovered the problems you mention.
Current status is that the commit is reverted in master branch as Linus
also hit the bug. The cause and a fix are known
https://lore.kernel.org/linux-btrfs/b55a95be-38e8-4db7-9653-f864788b475c@gmx.com/T/#m873962a3b205625045feb9a4f8db70e75f66e418
For the time being the bug described in the changelog will be present
(affecting the subpage case), with the fix for zstd revisited in the
future.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression
2024-01-22 23:54 ` David Sterba
@ 2024-01-23 10:50 ` Klara Modin
0 siblings, 0 replies; 5+ messages in thread
From: Klara Modin @ 2024-01-23 10:50 UTC (permalink / raw)
To: dsterba; +Cc: wqu, linux-btrfs, terrelln, dsterba, josef, clm
Den tis 23 jan. 2024 kl 00:54 skrev David Sterba <dsterba@suse.cz>:
>
> On Mon, Jan 22, 2024 at 09:59:47PM +0100, David Sterba wrote:
> > On Mon, Jan 22, 2024 at 09:45:42PM +0100, Klara Modin wrote:
> > > Hi,
> > >
> > > With this patch [1], small zstd compressed files on btrfs return
> > > garbage when read on my x86_64 system. Reverting this on top of
> > > next-20240122 resolves the issue for me.
> >
> > Thanks for the report, this is serious. The patches have been in testing
> > for some time but haven't uncovered the problems you mention.
>
> Current status is that the commit is reverted in master branch as Linus
> also hit the bug. The cause and a fix are known
>
> https://lore.kernel.org/linux-btrfs/b55a95be-38e8-4db7-9653-f864788b475c@gmx.com/T/#m873962a3b205625045feb9a4f8db70e75f66e418
>
> For the time being the bug described in the changelog will be present
> (affecting the subpage case), with the fix for zstd revisited in the
> future.
Thanks for the update.
I tried the proposed fix,
- memcpy_to_page(dest_page, dest_pgoff + to_copy,
workspace->out_buf.dst, to_copy);
+ memcpy_to_page(dest_page, dest_pgoff, workspace->out_buf.dst, to_copy);
and it resolves the issue for me.
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression
2024-01-08 9:08 [PATCH 0/3] btrfs: fix and simplify the inline extent decompression path for subpage Qu Wenruo
@ 2024-01-08 9:08 ` Qu Wenruo
0 siblings, 0 replies; 5+ messages in thread
From: Qu Wenruo @ 2024-01-08 9:08 UTC (permalink / raw)
To: linux-btrfs
[BUG]
If we have a filesystem with 4k sectorsize, and an inlined compressed
extent created like this:
item 4 key (257 INODE_ITEM 0) itemoff 15863 itemsize 160
generation 8 transid 8 size 4096 nbytes 4096
block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
sequence 1 flags 0x0(none)
item 5 key (257 INODE_REF 256) itemoff 15839 itemsize 24
index 2 namelen 14 name: source_inlined
item 6 key (257 EXTENT_DATA 0) itemoff 15770 itemsize 69
generation 8 type 0 (inline)
inline extent data size 48 ram_bytes 4096 compression 3 (zstd)
Then trying to reflink that extent in an aarch64 system with 64K page
size, the reflink would just fail:
# xfs_io -f -c "reflink $mnt/source_inlined 0 60k 4k" $mnt/dest
XFS_IOC_CLONE_RANGE: Input/output error
[CAUSE]
In zstd_decompress(), we didn't treat @start_byte as just a page offset,
but also use it as an indicator on whether we should error out, without
any proper explanation (this is copied from other decompression code).
In reality, for subpage cases, although @start_byte can be non-zero,
we should never switch input/output buffer nor error out, since the whole
input/output buffer should never exceed one sector, thus we should not
need to do any buffer switch.
Thus the current code using @start_byte as a condition to switch
input/output buffer or finish the decompression is completely incorrect.
[FIX]
The fix involves several modification:
- Rename @start_byte to @dest_pgoff to properly express its meaning
- Use @sectorsize other than PAGE_SIZE to properly initialize the
output buffer size
- Use correct destination offset inside the destination page
- Simplify the main loop
Since the input/output buffer should never switch, we only need one
zstd_decompress_stream() call.
- Consider early end as an error
After the fix, even on 64K page sized aarch64, above reflink now
works as expected:
# xfs_io -f -c "reflink $mnt/source_inlined 0 60k 4k" $mnt/dest
linked 4096/4096 bytes at offset 61440
And results the correct file layout:
item 9 key (258 INODE_ITEM 0) itemoff 15542 itemsize 160
generation 10 transid 10 size 65536 nbytes 4096
block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
sequence 1 flags 0x0(none)
item 10 key (258 INODE_REF 256) itemoff 15528 itemsize 14
index 3 namelen 4 name: dest
item 11 key (258 XATTR_ITEM 3817753667) itemoff 15445 itemsize 83
location key (0 UNKNOWN.0 0) type XATTR
transid 10 data_len 37 name_len 16
name: security.selinux
data unconfined_u:object_r:unlabeled_t:s0
item 12 key (258 EXTENT_DATA 61440) itemoff 15392 itemsize 53
generation 10 type 1 (regular)
extent data disk byte 13631488 nr 4096
extent data offset 0 nr 4096 ram 4096
extent compression 0 (none)
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
fs/btrfs/compression.h | 2 +-
fs/btrfs/zstd.c | 74 +++++++++++++-----------------------------
2 files changed, 23 insertions(+), 53 deletions(-)
diff --git a/fs/btrfs/compression.h b/fs/btrfs/compression.h
index afd7e50d073d..97fe3ebf11a2 100644
--- a/fs/btrfs/compression.h
+++ b/fs/btrfs/compression.h
@@ -169,7 +169,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping,
unsigned long *total_in, unsigned long *total_out);
int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb);
int zstd_decompress(struct list_head *ws, const u8 *data_in,
- struct page *dest_page, unsigned long start_byte, size_t srclen,
+ struct page *dest_page, unsigned long dest_pgoff, size_t srclen,
size_t destlen);
void zstd_init_workspace_manager(void);
void zstd_cleanup_workspace_manager(void);
diff --git a/fs/btrfs/zstd.c b/fs/btrfs/zstd.c
index 0d66db8bc1d4..812e3bd43889 100644
--- a/fs/btrfs/zstd.c
+++ b/fs/btrfs/zstd.c
@@ -20,6 +20,7 @@
#include "misc.h"
#include "compression.h"
#include "ctree.h"
+#include "super.h"
#define ZSTD_BTRFS_MAX_WINDOWLOG 17
#define ZSTD_BTRFS_MAX_INPUT (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
@@ -618,80 +619,49 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
}
int zstd_decompress(struct list_head *ws, const u8 *data_in,
- struct page *dest_page, unsigned long start_byte, size_t srclen,
+ struct page *dest_page, unsigned long dest_pgoff, size_t srclen,
size_t destlen)
{
struct workspace *workspace = list_entry(ws, struct workspace, list);
+ struct btrfs_fs_info *fs_info = btrfs_sb(dest_page->mapping->host->i_sb);
+ const u32 sectorsize = fs_info->sectorsize;
zstd_dstream *stream;
int ret = 0;
- size_t ret2;
- unsigned long total_out = 0;
- unsigned long pg_offset = 0;
+ unsigned long to_copy = 0;
stream = zstd_init_dstream(
ZSTD_BTRFS_MAX_INPUT, workspace->mem, workspace->size);
if (!stream) {
pr_warn("BTRFS: zstd_init_dstream failed\n");
- ret = -EIO;
goto finish;
}
- destlen = min_t(size_t, destlen, PAGE_SIZE);
-
workspace->in_buf.src = data_in;
workspace->in_buf.pos = 0;
workspace->in_buf.size = srclen;
workspace->out_buf.dst = workspace->buf;
workspace->out_buf.pos = 0;
- workspace->out_buf.size = PAGE_SIZE;
+ workspace->out_buf.size = sectorsize;
- ret2 = 1;
- while (pg_offset < destlen
- && workspace->in_buf.pos < workspace->in_buf.size) {
- unsigned long buf_start;
- unsigned long buf_offset;
- unsigned long bytes;
-
- /* Check if the frame is over and we still need more input */
- if (ret2 == 0) {
- pr_debug("BTRFS: zstd_decompress_stream ended early\n");
- ret = -EIO;
- goto finish;
- }
- ret2 = zstd_decompress_stream(stream, &workspace->out_buf,
- &workspace->in_buf);
- if (zstd_is_error(ret2)) {
- pr_debug("BTRFS: zstd_decompress_stream returned %d\n",
- zstd_get_error_code(ret2));
- ret = -EIO;
- goto finish;
- }
-
- buf_start = total_out;
- total_out += workspace->out_buf.pos;
- workspace->out_buf.pos = 0;
-
- if (total_out <= start_byte)
- continue;
-
- if (total_out > start_byte && buf_start < start_byte)
- buf_offset = start_byte - buf_start;
- else
- buf_offset = 0;
-
- bytes = min_t(unsigned long, destlen - pg_offset,
- workspace->out_buf.size - buf_offset);
-
- memcpy_to_page(dest_page, pg_offset,
- workspace->out_buf.dst + buf_offset, bytes);
-
- pg_offset += bytes;
+ /*
+ * Since both input and output buffer should not exceed one sector,
+ * One call should end the decompression.
+ */
+ ret = zstd_decompress_stream(stream, &workspace->out_buf, &workspace->in_buf);
+ if (zstd_is_error(ret)) {
+ pr_warn_ratelimited("BTRFS: zstd_decompress_stream return %d\n",
+ zstd_get_error_code(ret));
+ goto finish;
}
- ret = 0;
+ to_copy = workspace->out_buf.pos;
+ memcpy_to_page(dest_page, dest_pgoff + to_copy, workspace->out_buf.dst,
+ to_copy);
finish:
- if (pg_offset < destlen) {
- memzero_page(dest_page, pg_offset, destlen - pg_offset);
+ /* Error or early end. */
+ if (to_copy < destlen) {
+ ret = -EIO;
+ memzero_page(dest_page, dest_pgoff + to_copy, destlen - to_copy);
}
return ret;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-01-23 10:50 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-01-22 20:45 [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression Klara Modin
2024-01-22 20:59 ` David Sterba
2024-01-22 23:54 ` David Sterba
2024-01-23 10:50 ` Klara Modin
-- strict thread matches above, loose matches on Subject: below --
2024-01-08 9:08 [PATCH 0/3] btrfs: fix and simplify the inline extent decompression path for subpage Qu Wenruo
2024-01-08 9:08 ` [PATCH 3/3] btrfs: zstd: fix and simplify the inline extent decompression Qu Wenruo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).