From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84BE9C433E0 for ; Wed, 23 Dec 2020 21:32:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 565E922273 for ; Wed, 23 Dec 2020 21:32:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726583AbgLWVcb (ORCPT ); Wed, 23 Dec 2020 16:32:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726037AbgLWVcb (ORCPT ); Wed, 23 Dec 2020 16:32:31 -0500 Received: from mail-il1-x135.google.com (mail-il1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D0F9C0617A6 for ; Wed, 23 Dec 2020 13:31:41 -0800 (PST) Received: by mail-il1-x135.google.com with SMTP id 75so346294ilv.13 for ; Wed, 23 Dec 2020 13:31:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QbxsWfgRlzCJgOBxqB1SwxZEGeePTfdrMkj+YGC6JAo=; b=Z9K6Fn4hT/nSk9i8CErJ6bmQCsMMOcNp7Bt7j+01+MHkB6QJ/NqED08RvBk4ZkwDs4 ep+q+MMmoC3gCyMErKrgsjZHvjFx7zubx/VOHVC5THt50fvDcCaq04C137s/L91FSa6l XfRTwdgR+Q1Kz7Yulcnw4FYxkwjUFPJX9v73k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QbxsWfgRlzCJgOBxqB1SwxZEGeePTfdrMkj+YGC6JAo=; b=TStz7Z7fEAoEYFyQbUkUjrL3Euv/AI0mTuw9P5X1WeuNEitXP+Q885XQNTSc8bYtDc a3cBNHbOXUe/B40tjAq6TnZsHAQxxgEyxFEAYLq1oRTYLJukZpEO7MD6w8qQ4biW3Fhb RCSgS37zOvKiRLs9nqYrpC39lU7QH2tfNdtR6e4FtI+Gu/NMYnfem4WLbKxlwNzVTGiQ U47+6OeEigOcRSt1cPR2ik39Ornws2OLpO9/xLzDOon/8gH2Pcc20gHqR6khImPA8YpR C8BY+p7SuuqJqQozg3kpA9YZkJfkZGWtB2Ucuaz5oSUU0JPJkUO4YoMfYM8YZD18njLp DAJw== X-Gm-Message-State: AOAM532V6SWU5q8UIhnEilSoRBOYIiz29APNquFGG6232FBTFIb8dzPN /ChYkDiyG2OKDKjy8gZWW6jPi7svvpHK/zxtk3R7pA== X-Google-Smtp-Source: ABdhPJw7V/eEElU2vOuKPcbbwjTNnQSkE2GjRl2RJ/IHsDvus/FqzOOyUqBW6cMF5+UWgLfMLa4buvJS20HzeW05lDI= X-Received: by 2002:a05:6e02:10c3:: with SMTP id s3mr27679093ilj.269.1608759100500; Wed, 23 Dec 2020 13:31:40 -0800 (PST) MIME-Version: 1.0 References: <16ffadab-42ba-f9c7-8203-87fda3dc9b44@maciej.szmigiero.name> <74c7129b-a437-ebc4-1466-7fb9f034e006@maciej.szmigiero.name> In-Reply-To: From: Ignat Korchagin Date: Wed, 23 Dec 2020 21:31:29 +0000 Message-ID: Subject: Re: dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG() To: "Maciej S. Szmigiero" Cc: Alasdair G Kergon , Mike Snitzer , device-mapper development , dm-crypt@saout.de, linux-kernel , Eric Biggers , Damien Le Moal , Mikulas Patocka , Herbert Xu , kernel-team , Nobuto Murata , Chris Mason , Josef Bacik , David Sterba , linux-btrfs@vger.kernel.org, linux-crypto Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org On Wed, Dec 23, 2020 at 9:20 PM Maciej S. Szmigiero wrote: > > On 23.12.2020 22:09, Ignat Korchagin wrote: > (..) > > I've been looking into this for the last couple of days because of > > other reports [1]. > > Just finished testing a possible solution. Will submit soon. > > Thanks for looking into it. > > By the way, on a bare metal I am actually hitting a different problem > (scheduling while atomic) when scrubbing a btrfs filesystem, just as one > of your users from that GitHub report had [1]. That is because dm-crypt calls "wait_for_completion" in rare cases when Crypto API (cryptd) backlogs the decryption request. I've reproduced that one as well (even with no FS). We never hit these problems in the original testing probably due to the fact we mostly used xtsproxy custom crypto module, which is totally synchronous. I did test it later with standard crypto, but did not encounter these problems as well most likely because it is also depends which storage driver underneath we are using: most of them do not submit read requests to dm-crypt in irq/softirq context in the first place > I've pasted that backtrace in my original Dec 14 message. > > Thanks, > Maciej > > [1]: https://github.com/cloudflare/linux/issues/1#issuecomment-736734243 Regards, Ignat From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51BCAC433DB for ; Wed, 23 Dec 2020 23:23:19 +0000 (UTC) Received: from mail.server123.net (mail.server123.net [78.46.64.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BAD252225E for ; Wed, 23 Dec 2020 23:23:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BAD252225E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=cloudflare.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dm-crypt-bounces@saout.de X-Virus-Scanned: amavisd-new at saout.de Authentication-Results: mail.server123.net (amavisd-new); dkim=pass (1024-bit key) header.d=cloudflare.com Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::132; helo=mail-il1-x132.google.com; envelope-from=ignat@cloudflare.com; receiver= Received: from mail-il1-x132.google.com (mail-il1-x132.google.com [IPv6:2607:f8b0:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.server123.net (Postfix) with ESMTPS for ; Wed, 23 Dec 2020 22:31:42 +0100 (CET) Received: by mail-il1-x132.google.com with SMTP id g1so373743ilk.7 for ; Wed, 23 Dec 2020 13:31:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QbxsWfgRlzCJgOBxqB1SwxZEGeePTfdrMkj+YGC6JAo=; b=Z9K6Fn4hT/nSk9i8CErJ6bmQCsMMOcNp7Bt7j+01+MHkB6QJ/NqED08RvBk4ZkwDs4 ep+q+MMmoC3gCyMErKrgsjZHvjFx7zubx/VOHVC5THt50fvDcCaq04C137s/L91FSa6l XfRTwdgR+Q1Kz7Yulcnw4FYxkwjUFPJX9v73k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QbxsWfgRlzCJgOBxqB1SwxZEGeePTfdrMkj+YGC6JAo=; b=O/nLG5ZgxitnRUVj0fdFW7bEAkUCu77Aj94hKQOeWxtFQ45WF9FXEtB+fAxCox31FH OcgKhDLOdTfKak0Zyq5h9yNN8NxZlSeufngjgtCUfcNmuNl/x/qaYNPIMURJyEy0C5nK 4mJGycHXSPz6HMplEfvfqL1pnpFAiIrK9/FxVN/+nDSrbZlaSc/U4zVQwjZQOnStf1l3 +dG2VAGpNHXI18+NHaCflGPlQpH1KbSpI7lMVODKoZFMY18aEb1q3eud+1PfYAZ/9Cqq ThrIyUOGIjSYtoJ7ya8dzInhL8YIFAuv3rha/UccDASI9l1yZZUeq4hfpSUypmqlHVEn Rruw== X-Gm-Message-State: AOAM533OzXQjPc2nAvJaT3Ifx+4mZWN5f3IyI2RYYUUt9N8R/HS5Rq3T DZLKw6sxN9YY+p26CbOGt1k1IKLZ1JmrbisDNvSkXg== X-Google-Smtp-Source: ABdhPJw7V/eEElU2vOuKPcbbwjTNnQSkE2GjRl2RJ/IHsDvus/FqzOOyUqBW6cMF5+UWgLfMLa4buvJS20HzeW05lDI= X-Received: by 2002:a05:6e02:10c3:: with SMTP id s3mr27679093ilj.269.1608759100500; Wed, 23 Dec 2020 13:31:40 -0800 (PST) MIME-Version: 1.0 References: <16ffadab-42ba-f9c7-8203-87fda3dc9b44@maciej.szmigiero.name> <74c7129b-a437-ebc4-1466-7fb9f034e006@maciej.szmigiero.name> In-Reply-To: From: Ignat Korchagin Date: Wed, 23 Dec 2020 21:31:29 +0000 Message-ID: To: "Maciej S. Szmigiero" X-Mailman-Approved-At: Thu, 24 Dec 2020 00:15:13 +0100 Subject: Re: [dm-crypt] dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG() X-BeenThere: dm-crypt@saout.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Damien Le Moal , Herbert Xu , Mike Snitzer , kernel-team , dm-crypt@saout.de, linux-kernel , Nobuto Murata , Eric Biggers , Chris Mason , device-mapper development , Mikulas Patocka , linux-btrfs@vger.kernel.org, David Sterba , Josef Bacik , Alasdair G Kergon , linux-crypto Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: dm-crypt-bounces@saout.de Sender: "dm-crypt" On Wed, Dec 23, 2020 at 9:20 PM Maciej S. Szmigiero wrote: > > On 23.12.2020 22:09, Ignat Korchagin wrote: > (..) > > I've been looking into this for the last couple of days because of > > other reports [1]. > > Just finished testing a possible solution. Will submit soon. > > Thanks for looking into it. > > By the way, on a bare metal I am actually hitting a different problem > (scheduling while atomic) when scrubbing a btrfs filesystem, just as one > of your users from that GitHub report had [1]. That is because dm-crypt calls "wait_for_completion" in rare cases when Crypto API (cryptd) backlogs the decryption request. I've reproduced that one as well (even with no FS). We never hit these problems in the original testing probably due to the fact we mostly used xtsproxy custom crypto module, which is totally synchronous. I did test it later with standard crypto, but did not encounter these problems as well most likely because it is also depends which storage driver underneath we are using: most of them do not submit read requests to dm-crypt in irq/softirq context in the first place > I've pasted that backtrace in my original Dec 14 message. > > Thanks, > Maciej > > [1]: https://github.com/cloudflare/linux/issues/1#issuecomment-736734243 Regards, Ignat _______________________________________________ dm-crypt mailing list dm-crypt@saout.de https://www.saout.de/mailman/listinfo/dm-crypt From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 125A3C433E0 for ; Wed, 23 Dec 2020 21:32:00 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 771C822273 for ; Wed, 23 Dec 2020 21:31:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 771C822273 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=cloudflare.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-377-an85rj3rP32_C5FRFbIpMQ-1; Wed, 23 Dec 2020 16:31:55 -0500 X-MC-Unique: an85rj3rP32_C5FRFbIpMQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D568710054FF; Wed, 23 Dec 2020 21:31:50 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 842472BFE3; Wed, 23 Dec 2020 21:31:50 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id C5A7B180954D; Wed, 23 Dec 2020 21:31:49 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 0BNLVlGo006217 for ; Wed, 23 Dec 2020 16:31:47 -0500 Received: by smtp.corp.redhat.com (Postfix) id 4CFC31102E05; Wed, 23 Dec 2020 21:31:47 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast02.extmail.prod.ext.rdu2.redhat.com [10.11.55.18]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 48AB31102E03 for ; Wed, 23 Dec 2020 21:31:45 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0FFC28007D9 for ; Wed, 23 Dec 2020 21:31:45 +0000 (UTC) Received: from mail-il1-f178.google.com (mail-il1-f178.google.com [209.85.166.178]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-449-t2c1PYohNQ6qQAHJxyy0YQ-1; Wed, 23 Dec 2020 16:31:41 -0500 X-MC-Unique: t2c1PYohNQ6qQAHJxyy0YQ-1 Received: by mail-il1-f178.google.com with SMTP id v3so387784ilo.5 for ; Wed, 23 Dec 2020 13:31:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QbxsWfgRlzCJgOBxqB1SwxZEGeePTfdrMkj+YGC6JAo=; b=asHoYByvazcqN2qg47PeCON9KZvIrD4EVwZz8TJqOFDp3OsvqNv3L3xixSpJwBj2LV KF9FldF/b6R2RwfKSrASwlz/8S2N2grunbBhFkYk8Awuo3k1EmHkWcK+kFzFy8Mca0D2 3ilItrVs9X75zE+Vv83OolCZvkuzWTr5YtLc+mxbgu78uNdvrlwUJrSblz3sEpnhH8Op KsmI80bN9QTSOZHip/WWBydunDLdiDpseogYzlNXSxfukRZTeMUQjZx3sKZ1zHURg2BI GRuKJQtsBIgD3qteUqtmU30Kt7SHMZJMy/YrOc9+2n7iCOxtl+G0NoibN3b8g7eADhYL 9NiQ== X-Gm-Message-State: AOAM532UzeKxo2GRj5E5lrjJFeA+up53Nby9lQoDtiluBW5tS6JT41be L6IsugFT+ciqMUbTs5UG4pRKQA2zwrBQWi6SXuq6bw== X-Google-Smtp-Source: ABdhPJw7V/eEElU2vOuKPcbbwjTNnQSkE2GjRl2RJ/IHsDvus/FqzOOyUqBW6cMF5+UWgLfMLa4buvJS20HzeW05lDI= X-Received: by 2002:a05:6e02:10c3:: with SMTP id s3mr27679093ilj.269.1608759100500; Wed, 23 Dec 2020 13:31:40 -0800 (PST) MIME-Version: 1.0 References: <16ffadab-42ba-f9c7-8203-87fda3dc9b44@maciej.szmigiero.name> <74c7129b-a437-ebc4-1466-7fb9f034e006@maciej.szmigiero.name> In-Reply-To: From: Ignat Korchagin Date: Wed, 23 Dec 2020 21:31:29 +0000 Message-ID: To: "Maciej S. Szmigiero" X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-loop: dm-devel@redhat.com Cc: Damien Le Moal , Herbert Xu , Mike Snitzer , kernel-team , dm-crypt@saout.de, linux-kernel , Nobuto Murata , Eric Biggers , Chris Mason , device-mapper development , Mikulas Patocka , linux-btrfs@vger.kernel.org, David Sterba , Josef Bacik , Alasdair G Kergon , linux-crypto Subject: Re: [dm-devel] dm-crypt with no_read_workqueue and no_write_workqueue + btrfs scrub = BUG() X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Wed, Dec 23, 2020 at 9:20 PM Maciej S. Szmigiero wrote: > > On 23.12.2020 22:09, Ignat Korchagin wrote: > (..) > > I've been looking into this for the last couple of days because of > > other reports [1]. > > Just finished testing a possible solution. Will submit soon. > > Thanks for looking into it. > > By the way, on a bare metal I am actually hitting a different problem > (scheduling while atomic) when scrubbing a btrfs filesystem, just as one > of your users from that GitHub report had [1]. That is because dm-crypt calls "wait_for_completion" in rare cases when Crypto API (cryptd) backlogs the decryption request. I've reproduced that one as well (even with no FS). We never hit these problems in the original testing probably due to the fact we mostly used xtsproxy custom crypto module, which is totally synchronous. I did test it later with standard crypto, but did not encounter these problems as well most likely because it is also depends which storage driver underneath we are using: most of them do not submit read requests to dm-crypt in irq/softirq context in the first place > I've pasted that backtrace in my original Dec 14 message. > > Thanks, > Maciej > > [1]: https://github.com/cloudflare/linux/issues/1#issuecomment-736734243 Regards, Ignat -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel