[f2fs-dev] [Bug 216162] New: F2FS filesystem with compression broken after kernel version 5.18.1

[f2fs-dev] [Bug 216162] New: F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

            Bug ID: 216162
           Summary: F2FS filesystem with compression broken after kernel
                    version 5.18.1
           Product: File System
           Version: 2.5
    Kernel Version: 5.18.3+
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: f2fs
          Assignee: filesystem_f2fs@kernel-bugs.kernel.org
          Reporter: ghtm2@protonmail.com
        Regression: No

I've been unable to boot two of my systems after upgrading past kernel version
5.18.1.
Both of them are using F2FS with compression enabled.

This was the command used to create them back on 5.13.6:
mkfs.f2fs -l root -O extra_attr,inode_checksum,sb_checksum,compression
/dev/sda3

The command used to mount:
mount -o compress_algorithm=lz4,compress_chksum,atgc,gc_merge,noatime /dev/sda3
/mnt

A simple 
ls /mnt/usr/bin will report "Structure needs cleaning" on almost all of the
critical system files on kernels past 5.18.1.
Downgrading to 5.18.1 fixes everything.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #1 from ghtm2@protonmail.com ---
Created attachment 301323
  --> https://bugzilla.kernel.org/attachment.cgi?id=301323&action=edit
broken f2fs image

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

ghtm2@protonmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Regression|No                          |Yes

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #2 from ghtm2@protonmail.com ---
I haven't had the time to bisect, but I've made a small image to demonstrate.
It's mountable on 5.18.1 but not 5.18.3 and after.
Due to limited time I was not able to test 5.18.2.

Maybe someone more knowledgeable than me could take a look?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

Jaegeuk Kim (jaegeuk@kernel.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jaegeuk@kernel.org

--- Comment #3 from Jaegeuk Kim (jaegeuk@kernel.org) ---
I suspect one issue recent reported with respect to compression vs.
inline_data. Could you please run fsck from this?
https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/?h=dev

Especially this patch should fix the corruption.
  "fsck.f2fs: drop compression bit if inline_data is set"

And, in order to avoid the runtime issue, we need this from Linus's tree.
https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git/commit/?h=dev&id=29eeafc661f283a7412ecf8d9899a5ffd0c80f59

I think v5.18 stable branch hasn't merge the fixes yet. By any chance, could
you please apply the below from the Linus tree on top of 5.18?

  "f2fs: attach inline_data after setting compression"

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #4 from ghtm2@protonmail.com ---
I did compile and run f2fs-tools-git just now, it doesn't do anything
unfortunately.

The mentioned patch is in the stable tree as of 5.18.8 and doesn't fix this
problem either.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #5 from ghtm2@protonmail.com ---
According to git bisect that I just ran,
198fd9faa271dd54dca6fc8eb6873f42dfd3b4d8 is the first bad commit.
"f2fs: fix to do sanity check for inline inode"

I reverted that on linux-stable v5.18.9 and the problem is indeed no longer
present.

Wile doing that I've found that my initial description as well es the attached
image were missing a command:
    chattr -R +c /mnt/etc
Without it the compression wouldn't be enabled and the errors wouldn't be
visible.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #6 from ghtm2@protonmail.com ---
Another week, another point release and the issue still persists on 5.18.10.

Can I ask for the offending commit to please be reverted?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

Andrej (surae@yandex.ru) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |surae@yandex.ru

--- Comment #7 from Andrej (surae@yandex.ru) ---
I confirm for 5.18.11 bug persists

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

Chao Yu (chao@kernel.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |chao@kernel.org

--- Comment #8 from Chao Yu (chao@kernel.org) ---
(In reply to ghtm2 from comment #5)
> According to git bisect that I just ran,
> 198fd9faa271dd54dca6fc8eb6873f42dfd3b4d8 is the first bad commit.
> "f2fs: fix to do sanity check for inline inode"
> 
> I reverted that on linux-stable v5.18.9 and the problem is indeed no longer
> present.
> 
> Wile doing that I've found that my initial description as well es the
> attached image were missing a command:
>     chattr -R +c /mnt/etc
> Without it the compression wouldn't be enabled and the errors wouldn't be
> visible.

I can reproduce your issue and repair this w/ last f2fs-tools:

[reproduce]
mount -o loop ./broken.img /mnt/f2fs/
touch /mnt/f2fs/file
lsattr /mnt/f2fs/file 
-----------------N-- /mnt/f2fs/file
chattr +c /mnt/f2fs/file 
lsattr /mnt/f2fs/file 
--------c--------N-- /mnt/f2fs/file
umount /mnt/f2fs 
mount -o loop ./broken.img /mnt/f2fs/
lsattr /mnt/f2fs/file 
lsattr: Structure needs cleaning while trying to stat /mnt/f2fs/file

[repair]
fsck.f2fs ./broken.img

output:

[FIX] (fsck_chk_inode_blk: 728)  --> [0x505] i_flags=0x4 -> 0x0
[ASSERT] (fsck_chk_inode_blk:1182)  --> ino: 0x505 chksum:0xe7b707a8, but
calculated one is: 0x3f6daf58
[FIX] (fsck_chk_inode_blk:1188)  --> ino: 0x505 recover, i_inode_checksum=
0xe7b707a8 -> 0x3f6daf58

[verify]
mount -o loop ./broken.img /mnt/f2fs/
stat /mnt/f2fs/file 
  File: /mnt/f2fs/file
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 70fh/1807d      Inode: 1285        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-07-17 11:51:41.705230802 +0800
Modify: 2022-07-17 11:51:41.705230802 +0800
Change: 2022-07-17 11:51:57.641110482 +0800
 Birth: -
lsattr /mnt/f2fs/file 
-----------------N-- /mnt/f2fs/file         ---- compressed flag was removed by
fsck.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

ghtm2@protonmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #301323|0                           |1
        is obsolete|                            |

--- Comment #9 from ghtm2@protonmail.com ---
Created attachment 301448
  --> https://bugzilla.kernel.org/attachment.cgi?id=301448&action=edit
broken f2fs image

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #10 from ghtm2@protonmail.com ---
Which f2fs-tools are those?
On my end neither f2fs-tools-1.15 nor f2fs-tools-git commit
c1144bf472f69fdfac8f3fd7634aab19b771c32e fix anything.

This is the output:
Info: MKFS version
  "Linux version 5.13.6-arch1-1 (linux@archlinux) (gcc (GCC) 11.1.0, GNU ld
(GNU Binutils) 2.36.1) #1 SMP PREEMPT Thu, 29 Jul 2021 00:21:06 +0000"
Info: FSCK version
  from "Linux version 5.13.6-arch1-1 (linux@archlinux) (gcc (GCC) 11.1.0, GNU
ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Thu, 29 Jul 2021 00:21:06 +0000"
    to "Linux version 5.18.7-arch1-1 (linux@archlinux) (gcc (GCC) 12.1.0, GNU
ld (GNU Binutils) 2.38) #1 SMP PREEMPT_DYNAMIC Sat, 25 Jun 2022 20:22:01 +0000"
Info: superblock features = 2828 :  extra_attr inode_checksum sb_checksum
compression
Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
Info: Segments per section = 1
Info: Sections per zone = 1
Info: total FS sectors = 262144 (128 MB)
Info: CKPT version = 1a027a03
Info: Checked valid nat_bits in checkpoint
Info: checkpoint state = d5 :  nat_bits crc fsck compacted_summary unmount
[FSCK] Check node 1 / 1282 (0.08%)
[FSCK] Check node 129 / 1282 (10.08%)
[FSCK] Check node 257 / 1282 (20.08%)
[FSCK] Check node 385 / 1282 (30.08%)
[FSCK] Check node 513 / 1282 (40.08%)
[FSCK] Check node 641 / 1282 (50.08%)
[FSCK] Check node 769 / 1282 (60.08%)
[FSCK] Check node 897 / 1282 (70.08%)
[FSCK] Check node 1025 / 1282 (80.08%)
[FSCK] Check node 1153 / 1282 (90.08%)
[FSCK] Check node 1281 / 1282 (100.08%)

[FSCK] Max image size: 36 MB, Free space: 52 MB
[FSCK] Unreachable nat entries                        [Ok..] [0x0]
[FSCK] SIT valid block bitmap checking                [Ok..]
[FSCK] Hard link checking for regular file            [Ok..] [0x0]
[FSCK] valid_block_count matching with CP             [Ok..] [0x9be]
[FSCK] valid_node_count matching with CP (de lookup)  [Ok..] [0x502]
[FSCK] valid_node_count matching with CP (nat lookup) [Ok..] [0x502]
[FSCK] valid_inode_count matched with CP              [Ok..] [0x502]
[FSCK] free segment_count matched with CP             [Ok..] [0x2f]
[FSCK] next block offset is free                      [Ok..]
[FSCK] fixing SIT types
[FSCK] other corrupted bugs                           [Ok..]

Done: 0.000000 secs

I've updated the broken image attached to this bug report.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #11 from Chao Yu (chao@kernel.org) ---
Please switch to dev-test/dev branch, due to fixing patch has not been
committed to master branch now...

https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/?h=dev-test

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 216162] F2FS filesystem with compression broken after kernel version 5.18.1

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216162

--- Comment #12 from ghtm2@protonmail.com ---
Ah, I somehow completely missed that the dev branched was linked before in
comment 3, sorry.
That branch does indeed fix the issue, thanks for the hint.

Maybe the fix warrants a push to master and a new release?
Most distros aren't exactly fast in packaging up new releases, the majority are
still on f2fs-tools-1. 14 or older...

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel