From: "Alex Xu (Hello71)" <alex_y_xu@yahoo.ca>
To: linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org,
dm-crypt@saout.de, linux-nvme@lists.infradead.org,
linux-block@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
Changheun Lee <nanich.lee@samsung.com>,
bvanassche@acm.org, yi.zhang@redhat.com, ming.lei@redhat.com,
bgoncalv@redhat.com, hch@lst.de, jaegeuk@kernel.org
Subject: Re: regression: data corruption with ext4 on LUKS on nvme with torvalds master
Date: Sat, 08 May 2021 22:29:57 -0400 [thread overview]
Message-ID: <1620526887.tg1zx7w5np.none@localhost> (raw)
In-Reply-To: <1620493841.bxdq8r5haw.none@localhost>
Excerpts from Alex Xu (Hello71)'s message of May 8, 2021 1:54 pm:
> Hi all,
>
> Using torvalds master, I recently encountered data corruption on my ext4
> volume on LUKS on NVMe. Specifically, during heavy writes, the system
> partially hangs; SysRq-W shows that processes are blocked in the kernel
> on I/O. After forcibly rebooting, chunks of files are replaced with
> other, unrelated data. I'm not sure exactly what the data is; some of it
> is unknown binary data, but in at least one case, a list of file paths
> was inserted into a file, indicating that the data is misdirected after
> encryption.
>
> This issue appears to affect files receiving writes in the temporal
> vicinity of the hang, but affects both new and old data: for example, my
> shell history file was corrupted up to many months before.
>
> The drive reports no SMART issues.
>
> I believe this is a regression in the kernel related to something merged
> in the last few days, as it consistently occurs with my most recent
> kernel versions, but disappears when reverting to an older kernel.
>
> I haven't investigated further, such as by bisecting. I hope this is
> sufficient information to give someone a lead on the issue, and if it is
> a bug, nail it down before anybody else loses data.
>
> Regards,
> Alex.
>
I found the following test to reproduce a hang, which I guess may be the
cause:
host$ cd /tmp
host$ truncate -s 10G drive
host$ qemu-system-x86_64 -drive format=raw,file=drive,if=none,id=drive -device nvme,drive=drive,serial=1 [... more VM setup options]
guest$ cryptsetup luksFormat /dev/nvme0n1
[accept warning, use any password]
guest$ cryptsetup open /dev/nvme0n1
[enter password]
guest$ mkfs.ext4 /dev/mapper/test
[normal output...]
Creating journal (16384 blocks): [hangs forever]
I bisected this issue to:
cd2c7545ae1beac3b6aae033c7f31193b3255946 is the first bad commit
commit cd2c7545ae1beac3b6aae033c7f31193b3255946
Author: Changheun Lee <nanich.lee@samsung.com>
Date: Mon May 3 18:52:03 2021 +0900
bio: limit bio max size
I didn't try reverting this commit or further reducing the test case.
Let me know if you need my kernel config or other information.
Regards,
Alex.
next prev parent reply other threads:[~2021-05-09 2:30 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1620493841.bxdq8r5haw.none.ref@localhost>
2021-05-08 17:54 ` regression: data corruption with ext4 on LUKS on nvme with torvalds master Alex Xu (Hello71)
2021-05-09 2:29 ` Alex Xu (Hello71) [this message]
2021-05-09 3:51 ` Jens Axboe
2021-05-09 14:47 ` Alex Xu (Hello71)
[not found] <CGME20210513100034epcas1p4b23892cd77bde73c777eea6dc51c16a4@epcas1p4.samsung.com>
2021-05-13 9:42 ` Changheun Lee
2021-05-13 14:15 ` Theodore Ts'o
2021-05-13 15:59 ` Bart Van Assche
[not found] ` <0e7b0b6e-e78c-f22d-af8d-d7bdcb597bea@gmail.com>
2021-05-13 19:22 ` Mikulas Patocka
2021-05-13 21:18 ` Bart Van Assche
2021-05-14 9:43 ` Mikulas Patocka
2021-05-14 9:50 ` Mikulas Patocka
[not found] ` <CGME20210514104426epcas1p3ee2f22f8e18c961118795c356e6a14ae@epcas1p3.samsung.com>
2021-05-14 10:26 ` Changheun Lee
2021-07-09 20:45 ` Samuel Mendoza-Jonas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1620526887.tg1zx7w5np.none@localhost \
--to=alex_y_xu@yahoo.ca \
--cc=axboe@kernel.dk \
--cc=bgoncalv@redhat.com \
--cc=bvanassche@acm.org \
--cc=dm-crypt@saout.de \
--cc=hch@lst.de \
--cc=jaegeuk@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=ming.lei@redhat.com \
--cc=nanich.lee@samsung.com \
--cc=yi.zhang@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).