From: Ignat Korchagin <ignat@cloudflare.com>
To: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "Maciej S. Szmigiero" <mail@maciej.szmigiero.name>,
Mike Snitzer <snitzer@redhat.com>,
kernel-team <kernel-team@cloudflare.com>,
dm-crypt@saout.de, Damien Le Moal <Damien.LeMoal@wdc.com>,
linux-kernel <linux-kernel@vger.kernel.org>,
Nobuto Murata <nobuto.murata@canonical.com>,
Eric Biggers <ebiggers@kernel.org>, Chris Mason <clm@fb.com>,
device-mapper development <dm-devel@redhat.com>,
Mikulas Patocka <mpatocka@redhat.com>,
linux-btrfs@vger.kernel.org, David Sterba <dsterba@suse.com>,
Josef Bacik <josef@toxicpanda.com>,
Alasdair G Kergon <agk@redhat.com>,
linux-crypto <linux-crypto@vger.kernel.org>
Subject: Re: [dm-crypt] dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG()
Date: Thu, 24 Dec 2020 18:46:18 +0000 [thread overview]
Message-ID: <CALrw=nFRLxpG+Qzy=wki1m6HnQUqPK9CQFGEEnB1tjSF0ex4UQ@mail.gmail.com> (raw)
In-Reply-To: <20201223205642.GA19817@gondor.apana.org.au>
On Wed, Dec 23, 2020 at 8:57 PM Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Wed, Dec 23, 2020 at 04:37:34PM +0100, Maciej S. Szmigiero wrote:
> >
> > It looks like to me that the skcipher API might not be safe to
> > call from a softirq context, after all.
>
> skcipher is safe to use in a softirq. The problem is only in
> dm-crypt where it tries to allocate memory with GFP_NOIO.
Hm.. After eliminating the GFP_NOIO (as well as some other sleeping
paths) from dm-crypt softirq code I still hit an occasional crash in
my extreme setup (QEMU with 1 CPU and cryptd_max_cpu_qlen set to 1)
(decoded with stacktrace_decode.sh):
[ 89.324723] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 89.325713] #PF: supervisor write access in kernel mode
[ 89.326460] #PF: error_code(0x0002) - not-present page
[ 89.327211] PGD 0 P4D 0
[ 89.327589] Oops: 0002 [#1] PREEMPT SMP PTI
[ 89.328200] CPU: 0 PID: 21 Comm: kworker/0:1 Not tainted 5.10.0+ #79
[ 89.329109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 0.0.0 02/06/2015
[ 89.330284] Workqueue: cryptd cryptd_queue_worker
[ 89.330999] RIP: 0010:crypto_dequeue_request
(/cfsetup_build/./include/linux/list.h:112
/cfsetup_build/./include/linux/list.h:135
/cfsetup_build/./include/linux/list.h:146
/cfsetup_build/crypto/algapi.c:957)
[ 89.331757] Code: e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 e8 c2 88 fe ff 4c
8b 23 48 c7 c6 e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 49 8b 14 24 49 8b 44
24 08 <48> 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
48
All code
========
0: e9 c9 d0 a8 48 jmpq 0x48a8d0ce
5: c7 c7 f9 c9 d0 a8 mov $0xa8d0c9f9,%edi
b: e8 c2 88 fe ff callq 0xfffffffffffe88d2
10: 4c 8b 23 mov (%rbx),%r12
13: 48 c7 c6 e9 c9 d0 a8 mov $0xffffffffa8d0c9e9,%rsi
1a: 48 c7 c7 f9 c9 d0 a8 mov $0xffffffffa8d0c9f9,%rdi
21: 49 8b 14 24 mov (%r12),%rdx
25: 49 8b 44 24 08 mov 0x8(%r12),%rax
2a:* 48 89 42 08 mov %rax,0x8(%rdx) <-- trapping instruction
2e: 48 89 10 mov %rdx,(%rax)
31: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
38: 00 ad de
3b: 49 89 04 24 mov %rax,(%r12)
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: 48 89 42 08 mov %rax,0x8(%rdx)
4: 48 89 10 mov %rdx,(%rax)
7: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
e: 00 ad de
11: 49 89 04 24 mov %rax,(%r12)
15: 48 rex.W
[ 89.334414] RSP: 0018:ffffba64c00bbe68 EFLAGS: 00010246
[ 89.335165] RAX: 0000000000000000 RBX: ffff9b9d6fc28d88 RCX: 0000000000000000
[ 89.336182] RDX: 0000000000000000 RSI: ffffffffa8d0c9e9 RDI: ffffffffa8d0c9f9
[ 89.337204] RBP: 0000000000000000 R08: ffffffffa906e708 R09: 0000000000000058
[ 89.338208] R10: ffffffffa9068720 R11: 00000000fffffc00 R12: ffff9b9a43797478
[ 89.339216] R13: 0000000000000020 R14: ffff9b9d6fc28e00 R15: 0000000000000000
[ 89.340231] FS: 0000000000000000(0000) GS:ffff9b9d6fc00000(0000)
knlGS:0000000000000000
[ 89.341376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 89.342207] CR2: 0000000000000008 CR3: 000000014cd76002 CR4: 0000000000170ef0
[ 89.343238] Call Trace:
[ 89.343609] cryptd_queue_worker (/cfsetup_build/crypto/cryptd.c:172)
[ 89.344218] process_one_work
(/cfsetup_build/./arch/x86/include/asm/preempt.h:26
/cfsetup_build/kernel/workqueue.c:2284)
[ 89.344821] ? rescuer_thread (/cfsetup_build/kernel/workqueue.c:2364)
[ 89.345399] worker_thread
(/cfsetup_build/./include/linux/list.h:282
/cfsetup_build/kernel/workqueue.c:2422)
[ 89.345923] ? rescuer_thread (/cfsetup_build/kernel/workqueue.c:2364)
[ 89.346504] kthread (/cfsetup_build/kernel/kthread.c:292)
[ 89.346986] ? kthread_create_worker_on_cpu
(/cfsetup_build/kernel/kthread.c:245)
[ 89.347713] ret_from_fork (/cfsetup_build/arch/x86/entry/entry_64.S:302)
[ 89.348255] Modules linked in:
[ 89.348708] CR2: 0000000000000008
[ 89.349197] ---[ end trace b7e9618b4122ed3b ]---
[ 89.349863] RIP: 0010:crypto_dequeue_request
(/cfsetup_build/./include/linux/list.h:112
/cfsetup_build/./include/linux/list.h:135
/cfsetup_build/./include/linux/list.h:146
/cfsetup_build/crypto/algapi.c:957)
[ 89.350606] Code: e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 e8 c2 88 fe ff 4c
8b 23 48 c7 c6 e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 49 8b 14 24 49 8b 44
24 08 <48> 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
48
All code
========
0: e9 c9 d0 a8 48 jmpq 0x48a8d0ce
5: c7 c7 f9 c9 d0 a8 mov $0xa8d0c9f9,%edi
b: e8 c2 88 fe ff callq 0xfffffffffffe88d2
10: 4c 8b 23 mov (%rbx),%r12
13: 48 c7 c6 e9 c9 d0 a8 mov $0xffffffffa8d0c9e9,%rsi
1a: 48 c7 c7 f9 c9 d0 a8 mov $0xffffffffa8d0c9f9,%rdi
21: 49 8b 14 24 mov (%r12),%rdx
25: 49 8b 44 24 08 mov 0x8(%r12),%rax
2a:* 48 89 42 08 mov %rax,0x8(%rdx) <-- trapping instruction
2e: 48 89 10 mov %rdx,(%rax)
31: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
38: 00 ad de
3b: 49 89 04 24 mov %rax,(%r12)
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: 48 89 42 08 mov %rax,0x8(%rdx)
4: 48 89 10 mov %rdx,(%rax)
7: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
e: 00 ad de
11: 49 89 04 24 mov %rax,(%r12)
15: 48 rex.W
[ 89.353266] RSP: 0018:ffffba64c00bbe68 EFLAGS: 00010246
[ 89.354003] RAX: 0000000000000000 RBX: ffff9b9d6fc28d88 RCX: 0000000000000000
[ 89.355048] RDX: 0000000000000000 RSI: ffffffffa8d0c9e9 RDI: ffffffffa8d0c9f9
[ 89.356063] RBP: 0000000000000000 R08: ffffffffa906e708 R09: 0000000000000058
[ 89.357082] R10: ffffffffa9068720 R11: 00000000fffffc00 R12: ffff9b9a43797478
[ 89.358088] R13: 0000000000000020 R14: ffff9b9d6fc28e00 R15: 0000000000000000
[ 89.359127] FS: 0000000000000000(0000) GS:ffff9b9d6fc00000(0000)
knlGS:0000000000000000
[ 89.360296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 89.361129] CR2: 0000000000000008 CR3: 000000014cd76002 CR4: 0000000000170ef0
[ 89.362160] Kernel panic - not syncing: Fatal exception in interrupt
[ 89.363145] Kernel Offset: 0x26000000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 89.364730] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
This happens when running dm-crypt with no_read_workqueues on top of
an emulated NVME in QEMU (NVME driver "completes" IO in IRQ context).
Somehow sending decryption requests to cryptd in some fashion in
softirq context corrupts the crypto queue it seems.
Regards,
Ignat
> Cheers,
> --
> Email: Herbert Xu <herbert@gondor.apana.org.au>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt
next prev parent reply other threads:[~2020-12-25 21:57 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-14 18:11 [dm-crypt] dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG() Maciej S. Szmigiero
2020-12-14 18:11 ` Maciej S. Szmigiero
2020-12-23 15:37 ` Maciej S. Szmigiero
2020-12-23 20:56 ` Herbert Xu
2020-12-24 18:46 ` Ignat Korchagin [this message]
2020-12-24 18:52 ` Maciej S. Szmigiero
2020-12-23 21:09 ` Ignat Korchagin
2020-12-23 21:19 ` Maciej S. Szmigiero
2020-12-23 21:31 ` Ignat Korchagin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALrw=nFRLxpG+Qzy=wki1m6HnQUqPK9CQFGEEnB1tjSF0ex4UQ@mail.gmail.com' \
--to=ignat@cloudflare.com \
--cc=Damien.LeMoal@wdc.com \
--cc=agk@redhat.com \
--cc=clm@fb.com \
--cc=dm-crypt@saout.de \
--cc=dm-devel@redhat.com \
--cc=dsterba@suse.com \
--cc=ebiggers@kernel.org \
--cc=herbert@gondor.apana.org.au \
--cc=josef@toxicpanda.com \
--cc=kernel-team@cloudflare.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-crypto@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mail@maciej.szmigiero.name \
--cc=mpatocka@redhat.com \
--cc=nobuto.murata@canonical.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).