archive mirror
 help / color / mirror / Atom feed
From: "Alex Xu (Hello71)" <>
To: Jens Axboe <>,,,,,,,,,,,
	Changheun Lee <>,
Subject: Re: regression: data corruption with ext4 on LUKS on nvme with torvalds master
Date: Sun, 09 May 2021 10:47:26 -0400	[thread overview]
Message-ID: <1620571445.2k94orj8ee.none@localhost> (raw)
In-Reply-To: <>

Excerpts from Jens Axboe's message of May 8, 2021 11:51 pm:
> On 5/8/21 8:29 PM, Alex Xu (Hello71) wrote:
>> Excerpts from Alex Xu (Hello71)'s message of May 8, 2021 1:54 pm:
>>> Hi all,
>>> Using torvalds master, I recently encountered data corruption on my ext4 
>>> volume on LUKS on NVMe. Specifically, during heavy writes, the system 
>>> partially hangs; SysRq-W shows that processes are blocked in the kernel 
>>> on I/O. After forcibly rebooting, chunks of files are replaced with 
>>> other, unrelated data. I'm not sure exactly what the data is; some of it 
>>> is unknown binary data, but in at least one case, a list of file paths 
>>> was inserted into a file, indicating that the data is misdirected after 
>>> encryption.
>>> This issue appears to affect files receiving writes in the temporal 
>>> vicinity of the hang, but affects both new and old data: for example, my 
>>> shell history file was corrupted up to many months before.
>>> The drive reports no SMART issues.
>>> I believe this is a regression in the kernel related to something merged 
>>> in the last few days, as it consistently occurs with my most recent 
>>> kernel versions, but disappears when reverting to an older kernel.
>>> I haven't investigated further, such as by bisecting. I hope this is 
>>> sufficient information to give someone a lead on the issue, and if it is 
>>> a bug, nail it down before anybody else loses data.
>>> Regards,
>>> Alex.
>> I found the following test to reproduce a hang, which I guess may be the 
>> cause:
>> host$ cd /tmp
>> host$ truncate -s 10G drive
>> host$ qemu-system-x86_64 -drive format=raw,file=drive,if=none,id=drive -device nvme,drive=drive,serial=1 [... more VM setup options]
>> guest$ cryptsetup luksFormat /dev/nvme0n1
>> [accept warning, use any password]
>> guest$ cryptsetup open /dev/nvme0n1
>> [enter password]
>> guest$ mkfs.ext4 /dev/mapper/test
>> [normal output...]
>> Creating journal (16384 blocks): [hangs forever]
>> I bisected this issue to:
>> cd2c7545ae1beac3b6aae033c7f31193b3255946 is the first bad commit
>> commit cd2c7545ae1beac3b6aae033c7f31193b3255946
>> Author: Changheun Lee <>
>> Date:   Mon May 3 18:52:03 2021 +0900
>>     bio: limit bio max size
>> I didn't try reverting this commit or further reducing the test case. 
>> Let me know if you need my kernel config or other information.
> If you have time, please do test with that reverted. I'd be anxious to
> get this revert queued up for 5.13-rc1.
> -- 
> Jens Axboe

I tested reverting it on top of b741596468b010af2846b75f5e75a842ce344a6e 
("Merge tag 'riscv-for-linus-5.13-mw1' of 
git://"), causing it 
to no longer hang. I didn't check if this fixes the data corruption, but 
I assume so.

I also tested a 1 GB image (works either way), and a virtio-blk 
interface (works either way)

The Show Blocked State from the VM (without revert):

sysrq: Show Blocked State
task:kworker/u2:0    state:D stack:    0 pid:    7 ppid:     2 flags:0x00004000
Workqueue: kcryptd/252:0 kcryptd_crypt
Call Trace:
 ? lock_timer_base+0x80/0x80
 ? __wake_up_common_lock+0x90/0x90
 ? process_one_work+0x300/0x300
 ? __kthread_create_worker+0x100/0x100
task:mkfs.ext4       state:D stack:    0 pid:  979 ppid:   964 flags:0x00004000
Call Trace:
 ? __schedule+0x1aa/0x4f0
 ? do_syscall_64+0x6e/0x80


  reply	other threads:[~2021-05-09 14:47 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <1620493841.bxdq8r5haw.none.ref@localhost>
2021-05-08 17:54 ` regression: data corruption with ext4 on LUKS on nvme with torvalds master Alex Xu (Hello71)
2021-05-09  2:29   ` Alex Xu (Hello71)
2021-05-09  3:51     ` Jens Axboe
2021-05-09 14:47       ` Alex Xu (Hello71) [this message]
     [not found] <>
2021-05-13  9:42 ` Changheun Lee
2021-05-13 14:15   ` Theodore Ts'o
2021-05-13 15:59     ` Bart Van Assche
     [not found]       ` <>
2021-05-13 19:22         ` Mikulas Patocka
2021-05-13 21:18           ` Bart Van Assche
2021-05-14  9:43             ` Mikulas Patocka
2021-05-14  9:50           ` Mikulas Patocka
     [not found]             ` <>
2021-05-14 10:26               ` Changheun Lee
2021-07-09 20:45                 ` Samuel Mendoza-Jonas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1620571445.2k94orj8ee.none@localhost \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).