From: Jiri Slaby <jirislaby@kernel.org>
To: Linus Torvalds <torvalds@linux-foundation.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Cc: minchan@kernel.org, ngupta@vflare.org,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Jan Kara <jack@suse.com>, Ted Ts'o <tytso@mit.edu>,
Andreas Dilger <adilger.kernel@dilger.ca>,
Ext4 Developers List <linux-ext4@vger.kernel.org>,
avromanov@sberdevices.ru, ddrokosov@sberdevices.ru,
ngupta@vflare.org
Subject: Re: ext2/zram issue [was: Linux 5.19]
Date: Tue, 9 Aug 2022 09:59:01 +0200 [thread overview]
Message-ID: <c7c69f77-4ea2-3375-33f3-322a3d35def5@kernel.org> (raw)
In-Reply-To: <702b3187-14bf-b733-263b-20272f53105d@kernel.org>
On 09. 08. 22, 8:03, Jiri Slaby wrote:
> Hi,
>
> On 31. 07. 22, 23:43, Linus Torvalds wrote:
>> So here we are, one week late, and 5.19 is tagged and pushed out.
>>
>> The full shortlog (just from rc8, obviously not all of 5.19) is below,
>> but I can happily report that there is nothing really interesting in
>> there. A lot of random small stuff.
>
> Note: I originally reported this downstream for tracking at:
> https://bugzilla.suse.com/show_bug.cgi?id=1202203
>
> 5.19 behaves pretty weird in openSUSE's openQA (opposing to 5.18, or
> 5.18.15). It's all qemu-kvm "HW"¹⁾:
> https://openqa.opensuse.org/tests/2502148
> loop2: detected capacity change from 0 to 72264
> EXT4-fs warning (device zram0): ext4_end_bio:343: I/O error 10 writing
> to inode 57375 starting block 137216)
> Buffer I/O error on device zram0, logical block 137216
> Buffer I/O error on device zram0, logical block 137217
> ...
> SQUASHFS error: xz decompression failed, data probably corrupt
> SQUASHFS error: Failed to read block 0x2e41680: -5
> SQUASHFS error: xz decompression failed, data probably corrupt
> SQUASHFS error: Failed to read block 0x2e41680: -5
> Bus error
>
>
>
> https://openqa.opensuse.org/tests/2502145
> FS-Cache: Loaded
> begin 644 ldconfig.core.pid_2094.sig_7.time_1659859442
>
>
>
> https://openqa.opensuse.org/tests/2502146
> FS-Cache: Loaded
> begin 644 Xorg.bin.core.pid_3733.sig_6.time_1659858784
>
>
>
> https://openqa.opensuse.org/tests/2502148
> EXT4-fs warning (device zram0): ext4_end_bio:343: I/O error 10 writing
> to inode 57375 starting block 137216)
> Buffer I/O error on device zram0, logical block 137216
> Buffer I/O error on device zram0, logical block 137217
>
>
>
> https://openqa.opensuse.org/tests/2502154
> [ 13.158090][ T634] FS-Cache: Loaded
> ...
> [ 525.627024][ C0] sysrq: Show State
>
>
>
> Those are various failures -- crashes of ldconfig, Xorg; I/O failures on
> zram; the last one is a lockup likely, something invoked sysrq after
> 500s stall.
>
> Interestingly, I've also hit this twice locally:
> > init[1]: segfault at 18 ip 00007fb6154b4c81 sp 00007ffc243ed600 error
> 6 in libc.so.6[7fb61543f000+185000]
> > Code: 41 5f c3 66 0f 1f 44 00 00 42 f6 44 10 08 01 0f 84 04 01 00 00
> 48 83 e1 fe 48 89 48 08 49 8b 47 70 49 89 5f 70 66 48 0f 6e c0 <48> 89
> 58 18 0f 16 44 24 08 48 81 fd ff 03 00 00 76 08 66 0f ef c9
> > *** signal 11 ***
> > malloc(): unsorted double linked list corrupted
> > traps: init[1] general protection fault ip:7fb61543f8b9
> sp:7ffc243ebf40 error:0 in libc.so.6[7fb61543f000+185000]
> > Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> > CPU: 0 PID: 1 Comm: init Not tainted 5.19.0-1-default #1 openSUSE
> Tumbleweed e1df13166a33f423514290c702e43cfbb2b5b575
>
> KASAN is not helpful either, so it's unlikely a memory corruption
> (unless it is "HW" related; should I try to turn on IOMMU in qemu?):
>> kasan: KernelAddressSanitizer initialized
>> ...
>> zram: module verification failed: signature and/or required key
>> missing - tainting kernel
>> zram: Added device: zram0
>> zram0: detected capacity change from 0 to 2097152
>> EXT4-fs (zram0): mounting ext2 file system using the ext4 subsystem
>> EXT4-fs (zram0): mounted filesystem without journal. Quota mode: none.
>> EXT4-fs warning (device zram0): ext4_end_bio:343: I/O error 10 writing
>> to inode 16386 starting block 159744)
>> Buffer I/O error on device zram0, logical block 159744
>> Buffer I/O error on device zram0, logical block 159745
>
>
>
> They all occur to me like a zram failure. The installer apparently
> creates an ext2 FS and after it mounts it using ext4 module, the issue
> starts occurring.
>
> Any tests I/you could run on 5.19 to exercise zram and ext2? Otherwise I
> am unable to reproduce easily, except using the openSUSE installer :/.
Ah, now I can. It's easy when one lowers memory available to qemu. -m
800M in this case:
echo $((1000*1024*1024)) > /sys/block/zram0/disksize
mkfs.ext2 /dev/zram0
mount /dev/zram0 /mnt/a/
dd if=/dev/urandom of=/mnt/a/stuff
[ 200.334277][ T8] EXT4-fs warning (device zram0): ext4_end_bio:343:
I/O error 10 writing to inode 12 starting block 8192)
[ 200.340198][ T8] Buffer I/O error on device zram0, logical block 8192
So currently, I blame:
commit e7be8d1dd983156bbdd22c0319b71119a8fbb697
Author: Alexey Romanov <avromanov@sberdevices.ru>
Date: Thu May 12 20:23:07 2022 -0700
zram: remove double compression logic
/me needs to confirm.
> Any other ideas? Or is this known already?
>
> ¹⁾ main are uefi boot and virtio-blk (it likely happens with virtio-scsi
> too). The cmdline _I_ use: qemu-kvm -device intel-hda -device hda-duplex
> -drive file=/tmp/pokus.qcow2,if=none,id=hd -device
> virtio-blk-pci,drive=hd -drive
> if=pflash,format=raw,unit=0,readonly=on,file=/usr/share/qemu/ovmf-x86_64-opensuse-code.bin -drive if=pflash,format=raw,unit=1,file=/tmp/vars.bin -cdrom /tmp/cd1.iso -m 1G -smp 1 -net user -net nic,model=virtio -serial pty -device virtio-rng-pci -device qemu-xhci,p2=4,p3=4 -usbdevice tablet
>
>
> thanks,
--
js
suse labs
next prev parent reply other threads:[~2022-08-09 7:59 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-07-31 21:43 Linux 5.19 Linus Torvalds
2022-08-01 12:47 ` Build regressions/improvements in v5.19 Geert Uytterhoeven
2022-08-02 9:14 ` Geert Uytterhoeven
2022-08-01 16:52 ` Linux 5.19 Tony Luck
2022-08-01 16:59 ` Linus Torvalds
2022-08-05 17:00 ` Please consider Linux 5.20 because it means "I love Linux" in Chinese (Re: Linux 5.19) Zhang Boyang
2022-08-07 17:21 ` David Laight
2022-08-09 6:03 ` ext2/zram issue [was: Linux 5.19] Jiri Slaby
2022-08-09 7:59 ` Jiri Slaby [this message]
2022-08-09 8:12 ` Jiri Slaby
2022-08-09 8:43 ` Sergey Senozhatsky
2022-08-09 9:11 ` Sergey Senozhatsky
2022-08-09 9:20 ` Sergey Senozhatsky
2022-08-09 10:20 ` Dmitry Rokosov
2022-08-09 11:53 ` Sergey Senozhatsky
2022-08-09 13:15 ` Aleksey Romanov
2022-08-09 13:29 ` Sergey Senozhatsky
2022-08-10 7:06 ` [PATCH] Revert "zram: remove double compression logic" Jiri Slaby
2022-08-10 7:14 ` Sergey Senozhatsky
2022-08-09 12:35 ` ext2/zram issue [was: Linux 5.19] Jiri Slaby
2022-08-09 12:45 ` Jiri Slaby
2022-08-09 12:57 ` Sergey Senozhatsky
2022-08-09 13:07 ` Sergey Senozhatsky
2022-08-09 9:12 ` Lukas Czerner
2022-08-09 9:15 ` Sergey Senozhatsky
2022-08-09 9:53 ` Lukas Czerner
[not found] ` <20220830214626.26544-1-charlie39@cock.li>
2022-08-31 7:55 ` Jiri Slaby
2022-08-11 14:02 ` [RESEND] Please consider name next Linux release "I love Linux" (Re: Linux 5.19) Zhang Boyang
2022-08-12 2:39 ` Baoquan He
2022-08-12 3:28 ` Huacai Chen
2022-08-12 6:31 ` Gao Xiang
2022-08-12 8:22 ` YanTeng Si
2022-08-13 17:17 ` Zhang Boyang
2022-08-14 15:22 ` WANG Xuerui
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c7c69f77-4ea2-3375-33f3-322a3d35def5@kernel.org \
--to=jirislaby@kernel.org \
--cc=adilger.kernel@dilger.ca \
--cc=avromanov@sberdevices.ru \
--cc=ddrokosov@sberdevices.ru \
--cc=jack@suse.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=minchan@kernel.org \
--cc=ngupta@vflare.org \
--cc=senozhatsky@chromium.org \
--cc=torvalds@linux-foundation.org \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).