All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ignat Korchagin <ignat@cloudflare.com>
To: "Maciej S. Szmigiero" <mail@maciej.szmigiero.name>
Cc: Alasdair G Kergon <agk@redhat.com>,
	Mike Snitzer <snitzer@redhat.com>,
	device-mapper development <dm-devel@redhat.com>,
	dm-crypt@saout.de, linux-kernel <linux-kernel@vger.kernel.org>,
	Eric Biggers <ebiggers@kernel.org>,
	Damien Le Moal <Damien.LeMoal@wdc.com>,
	Mikulas Patocka <mpatocka@redhat.com>,
	Herbert Xu <herbert@gondor.apana.org.au>,
	kernel-team <kernel-team@cloudflare.com>,
	Nobuto Murata <nobuto.murata@canonical.com>,
	Chris Mason <clm@fb.com>, Josef Bacik <josef@toxicpanda.com>,
	David Sterba <dsterba@suse.com>,
	linux-btrfs@vger.kernel.org,
	linux-crypto <linux-crypto@vger.kernel.org>
Subject: Re: dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG()
Date: Wed, 23 Dec 2020 21:31:29 +0000	[thread overview]
Message-ID: <CALrw=nEuMx4YDCFEa89kxn-pw+65h61g1grSsvY8sd=JeGAHjA@mail.gmail.com> (raw)
In-Reply-To: <fc27dc51-65a7-f2fa-6b29-01a1d5eaec6c@maciej.szmigiero.name>

On Wed, Dec 23, 2020 at 9:20 PM Maciej S. Szmigiero
<mail@maciej.szmigiero.name> wrote:
>
> On 23.12.2020 22:09, Ignat Korchagin wrote:
> (..)
> > I've been looking into this for the last couple of days because of
> > other reports [1].
> > Just finished testing a possible solution. Will submit soon.
>
> Thanks for looking into it.
>
> By the way, on a bare metal I am actually hitting a different problem
> (scheduling while atomic) when scrubbing a btrfs filesystem, just as one
> of your users from that GitHub report had [1].

That is because dm-crypt calls "wait_for_completion" in rare cases
when Crypto API (cryptd) backlogs the decryption request. I've
reproduced that one as well (even with no FS).
We never hit these problems in the original testing probably due to
the fact we mostly used xtsproxy custom crypto module, which is
totally synchronous.
I did test it later with standard crypto, but did not encounter these
problems as well most likely because it is also depends which storage
driver underneath we are using: most of them do not submit read
requests to dm-crypt in irq/softirq context in the first place

> I've pasted that backtrace in my original Dec 14 message.
>
> Thanks,
> Maciej
>
> [1]: https://github.com/cloudflare/linux/issues/1#issuecomment-736734243

Regards,
Ignat

WARNING: multiple messages have this Message-ID (diff)
From: Ignat Korchagin <ignat@cloudflare.com>
To: "Maciej S. Szmigiero" <mail@maciej.szmigiero.name>
Cc: Damien Le Moal <Damien.LeMoal@wdc.com>,
	Herbert Xu <herbert@gondor.apana.org.au>,
	Mike Snitzer <snitzer@redhat.com>,
	kernel-team <kernel-team@cloudflare.com>,
	dm-crypt@saout.de, linux-kernel <linux-kernel@vger.kernel.org>,
	Nobuto Murata <nobuto.murata@canonical.com>,
	Eric Biggers <ebiggers@kernel.org>, Chris Mason <clm@fb.com>,
	device-mapper development <dm-devel@redhat.com>,
	Mikulas Patocka <mpatocka@redhat.com>,
	linux-btrfs@vger.kernel.org, David Sterba <dsterba@suse.com>,
	Josef Bacik <josef@toxicpanda.com>,
	Alasdair G Kergon <agk@redhat.com>,
	linux-crypto <linux-crypto@vger.kernel.org>
Subject: Re: [dm-crypt] dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG()
Date: Wed, 23 Dec 2020 21:31:29 +0000	[thread overview]
Message-ID: <CALrw=nEuMx4YDCFEa89kxn-pw+65h61g1grSsvY8sd=JeGAHjA@mail.gmail.com> (raw)
In-Reply-To: <fc27dc51-65a7-f2fa-6b29-01a1d5eaec6c@maciej.szmigiero.name>

On Wed, Dec 23, 2020 at 9:20 PM Maciej S. Szmigiero
<mail@maciej.szmigiero.name> wrote:
>
> On 23.12.2020 22:09, Ignat Korchagin wrote:
> (..)
> > I've been looking into this for the last couple of days because of
> > other reports [1].
> > Just finished testing a possible solution. Will submit soon.
>
> Thanks for looking into it.
>
> By the way, on a bare metal I am actually hitting a different problem
> (scheduling while atomic) when scrubbing a btrfs filesystem, just as one
> of your users from that GitHub report had [1].

That is because dm-crypt calls "wait_for_completion" in rare cases
when Crypto API (cryptd) backlogs the decryption request. I've
reproduced that one as well (even with no FS).
We never hit these problems in the original testing probably due to
the fact we mostly used xtsproxy custom crypto module, which is
totally synchronous.
I did test it later with standard crypto, but did not encounter these
problems as well most likely because it is also depends which storage
driver underneath we are using: most of them do not submit read
requests to dm-crypt in irq/softirq context in the first place

> I've pasted that backtrace in my original Dec 14 message.
>
> Thanks,
> Maciej
>
> [1]: https://github.com/cloudflare/linux/issues/1#issuecomment-736734243

Regards,
Ignat
_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt

WARNING: multiple messages have this Message-ID (diff)
From: Ignat Korchagin <ignat@cloudflare.com>
To: "Maciej S. Szmigiero" <mail@maciej.szmigiero.name>
Cc: Damien Le Moal <Damien.LeMoal@wdc.com>,
	Herbert Xu <herbert@gondor.apana.org.au>,
	Mike Snitzer <snitzer@redhat.com>,
	kernel-team <kernel-team@cloudflare.com>,
	dm-crypt@saout.de, linux-kernel <linux-kernel@vger.kernel.org>,
	Nobuto Murata <nobuto.murata@canonical.com>,
	Eric Biggers <ebiggers@kernel.org>, Chris Mason <clm@fb.com>,
	device-mapper development <dm-devel@redhat.com>,
	Mikulas Patocka <mpatocka@redhat.com>,
	linux-btrfs@vger.kernel.org, David Sterba <dsterba@suse.com>,
	Josef Bacik <josef@toxicpanda.com>,
	Alasdair G Kergon <agk@redhat.com>,
	linux-crypto <linux-crypto@vger.kernel.org>
Subject: Re: [dm-devel] dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG()
Date: Wed, 23 Dec 2020 21:31:29 +0000	[thread overview]
Message-ID: <CALrw=nEuMx4YDCFEa89kxn-pw+65h61g1grSsvY8sd=JeGAHjA@mail.gmail.com> (raw)
In-Reply-To: <fc27dc51-65a7-f2fa-6b29-01a1d5eaec6c@maciej.szmigiero.name>

On Wed, Dec 23, 2020 at 9:20 PM Maciej S. Szmigiero
<mail@maciej.szmigiero.name> wrote:
>
> On 23.12.2020 22:09, Ignat Korchagin wrote:
> (..)
> > I've been looking into this for the last couple of days because of
> > other reports [1].
> > Just finished testing a possible solution. Will submit soon.
>
> Thanks for looking into it.
>
> By the way, on a bare metal I am actually hitting a different problem
> (scheduling while atomic) when scrubbing a btrfs filesystem, just as one
> of your users from that GitHub report had [1].

That is because dm-crypt calls "wait_for_completion" in rare cases
when Crypto API (cryptd) backlogs the decryption request. I've
reproduced that one as well (even with no FS).
We never hit these problems in the original testing probably due to
the fact we mostly used xtsproxy custom crypto module, which is
totally synchronous.
I did test it later with standard crypto, but did not encounter these
problems as well most likely because it is also depends which storage
driver underneath we are using: most of them do not submit read
requests to dm-crypt in irq/softirq context in the first place

> I've pasted that backtrace in my original Dec 14 message.
>
> Thanks,
> Maciej
>
> [1]: https://github.com/cloudflare/linux/issues/1#issuecomment-736734243

Regards,
Ignat

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel


  reply	other threads:[~2020-12-23 21:32 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-14 18:11 dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG() Maciej S. Szmigiero
2020-12-14 18:11 ` [dm-devel] " Maciej S. Szmigiero
2020-12-14 18:11 ` [dm-crypt] " Maciej S. Szmigiero
2020-12-14 18:11 ` Maciej S. Szmigiero
2020-12-23 15:37 ` Maciej S. Szmigiero
2020-12-23 15:37   ` [dm-devel] " Maciej S. Szmigiero
2020-12-23 15:37   ` [dm-crypt] " Maciej S. Szmigiero
2020-12-23 20:56   ` Herbert Xu
2020-12-23 20:56     ` [dm-devel] " Herbert Xu
2020-12-23 20:56     ` [dm-crypt] " Herbert Xu
2020-12-24 18:46     ` Ignat Korchagin
2020-12-24 18:46       ` [dm-devel] " Ignat Korchagin
2020-12-24 18:46       ` [dm-crypt] " Ignat Korchagin
2020-12-24 18:52       ` Maciej S. Szmigiero
2020-12-24 18:52         ` [dm-devel] " Maciej S. Szmigiero
2020-12-24 18:52         ` [dm-crypt] " Maciej S. Szmigiero
2020-12-23 21:09   ` Ignat Korchagin
2020-12-23 21:09     ` [dm-devel] " Ignat Korchagin
2020-12-23 21:09     ` [dm-crypt] " Ignat Korchagin
2020-12-23 21:19     ` Maciej S. Szmigiero
2020-12-23 21:19       ` [dm-devel] " Maciej S. Szmigiero
2020-12-23 21:19       ` [dm-crypt] " Maciej S. Szmigiero
2020-12-23 21:31       ` Ignat Korchagin [this message]
2020-12-23 21:31         ` [dm-devel] " Ignat Korchagin
2020-12-23 21:31         ` [dm-crypt] " Ignat Korchagin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CALrw=nEuMx4YDCFEa89kxn-pw+65h61g1grSsvY8sd=JeGAHjA@mail.gmail.com' \
    --to=ignat@cloudflare.com \
    --cc=Damien.LeMoal@wdc.com \
    --cc=agk@redhat.com \
    --cc=clm@fb.com \
    --cc=dm-crypt@saout.de \
    --cc=dm-devel@redhat.com \
    --cc=dsterba@suse.com \
    --cc=ebiggers@kernel.org \
    --cc=herbert@gondor.apana.org.au \
    --cc=josef@toxicpanda.com \
    --cc=kernel-team@cloudflare.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=linux-crypto@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mail@maciej.szmigiero.name \
    --cc=mpatocka@redhat.com \
    --cc=nobuto.murata@canonical.com \
    --cc=snitzer@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.