From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4C37C4363C for ; Wed, 7 Oct 2020 19:41:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5002C20872 for ; Wed, 7 Oct 2020 19:41:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kcN3zsM/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728414AbgJGTlX (ORCPT ); Wed, 7 Oct 2020 15:41:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728412AbgJGTlX (ORCPT ); Wed, 7 Oct 2020 15:41:23 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FF35C061755; Wed, 7 Oct 2020 12:41:22 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id n6so3390721wrm.13; Wed, 07 Oct 2020 12:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BlXTl6Hst6j2ZWlhRy8AiZeKdtWqE6ffJKL4bR3bTTQ=; b=kcN3zsM/ZtIXY6Z3zNNaFiikNc/iRaBZnxWvNYxoXEUtzNiHbwXZE4CR024aHVJEt3 dXZoZeL6K3E3uWH8Xp9tFUjWpzrtRdRM9v1kwZdo7Ed4w0h6bFcL0v1ixtz9pjUhzzkE TU/3b5FJtRRkNVZVxbNiInzha9rZEHRsxWTUTlUN2rhDPp6S+3m1pN6xficRXceeERJ7 QCkvpeB5aJ4MuBZnnXgrkAep6Rt5RGtqj1tYUiMsArrJSipocmOvPJ0Z6gelvX/+YHaq bBPMlYl0XA9kURPuUbQuH42W/milxSPmX4C/iLklEzeXB9/C12CppD3jJbhAvWIhw09i a+Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BlXTl6Hst6j2ZWlhRy8AiZeKdtWqE6ffJKL4bR3bTTQ=; b=JmmGgUb5jJhH8KXazu1Z10EHToVqUIu7kLExO/L0e7PlepPq/kIhjqVeg/AFgiqyYO heZzZqp78dB92yadWJ+pDGpUa/q7T4LVkNovO2j+y4LI05jMfRkf+gLU/tMs7gMy6V8U /4dab/j99xxfYGBdX1H2Wgh0x97///Bw7F1NWHgs1PR2DmJFmryROstF/sRrErg5Pjpp OODRGgUOuRco9LlMb9k2Y/+VZpqpCK/8nCkiidLoQMxmgacexYdbi2/v86NJQ/jc/Q/S uI4qPcLKTsLWHJT9qQwTtaIKC55290co8rKTHD0R/tb7QtQ0Ktg8aNbp2Pq3t/JXhQUK Y9ng== X-Gm-Message-State: AOAM530qtLiA+K/nQViK/wKe4z/M8GVfclsTSkT3eP8fIrJWY6WQxwlQ wss44cqzmgPd7v91qdvyMkQ= X-Google-Smtp-Source: ABdhPJy/7viZdAja+X9JBIUZBanX5mxDMsnzvuFIPvUeDKF3xHLhGKTLJJ0EwcgvuZBD0cR6EcQ74A== X-Received: by 2002:adf:ce01:: with SMTP id p1mr4928562wrn.33.1602099680845; Wed, 07 Oct 2020 12:41:20 -0700 (PDT) Received: from Red ([2a01:cb1d:3d5:a100:264b:feff:fe03:2806]) by smtp.googlemail.com with ESMTPSA id c1sm4156555wru.49.2020.10.07.12.41.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Oct 2020 12:41:19 -0700 (PDT) Date: Wed, 7 Oct 2020 21:41:17 +0200 From: Corentin Labbe To: Daniel Jordan Cc: Robin Murphy , mark.rutland@arm.com, jiangshanlai@gmail.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, tj@kernel.org, Will Deacon , linux-arm-kernel@lists.infradead.org Subject: Re: WARNING: at kernel/workqueue.c:1473 __queue_work+0x3b8/0x3d0 Message-ID: <20201007194117.GA4859@Red> References: <20200220090350.GA19858@Red> <20200221174223.r3y6tugavp3k5jdl@ca-dmjordan1.us.oracle.com> <20200228123311.GE3275@willie-the-truck> <20200228153331.uimy62rat2tdxxod@ca-dmjordan1.us.oracle.com> <20200301175351.GA11684@Red> <20200302172510.fspofleipqjcdxak@ca-dmjordan1.us.oracle.com> <20200303213017.tanczhqd3nhpeeak@ca-dmjordan1.us.oracle.com> <20201001175022.GA22444@Red> <20201005170910.vxwrdwnzlw3ahkb4@ca-dmjordan1.us.oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201005170910.vxwrdwnzlw3ahkb4@ca-dmjordan1.us.oracle.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org On Mon, Oct 05, 2020 at 01:09:10PM -0400, Daniel Jordan wrote: > On Thu, Oct 01, 2020 at 07:50:22PM +0200, Corentin Labbe wrote: > > On Tue, Mar 03, 2020 at 04:30:17PM -0500, Daniel Jordan wrote: > > > Barring other ideas, Corentin, would you be willing to boot with > > > > > > trace_event=initcall:*,module:* trace_options=stacktrace > > > > > > and > > > > > > diff --git a/kernel/module.c b/kernel/module.c > > > index 33569a01d6e1..393be6979a27 100644 > > > --- a/kernel/module.c > > > +++ b/kernel/module.c > > > @@ -3604,8 +3604,11 @@ static noinline int do_init_module(struct module *mod) > > > * be cleaned up needs to sync with the queued work - ie > > > * rcu_barrier() > > > */ > > > - if (llist_add(&freeinit->node, &init_free_list)) > > > + if (llist_add(&freeinit->node, &init_free_list)) { > > > + pr_warn("%s: schedule_work for mod=%s\n", __func__, mod->name); > > > + dump_stack(); > > > schedule_work(&init_free_wq); > > > + } > > > > > > mutex_unlock(&module_mutex); > > > wake_up_all(&module_wq); > > > > > > but not my earlier fix and share the dmesg and ftrace output to see if the > > > theory holds? > > > > > > Also, could you attach your config? Curious now what your crypto options look > > > like after fiddling with some of them today while trying and failing to see > > > this on x86. > > > > > > thanks, > > > Daniel > > > > Hello > > > > Sorry for the very delayed answer. > > > > I fail to reproduce it on x86 (qemu and real hw) and arm. > > It seems to only happen on arm64. > > Thanks for the config and dmesg, but there's no ftrace. I see it's not > configured in your kernel, so could you boot with my earlier debug patch plus > this one and the kernel argument initcall_debug instead? > > I'm trying to see whether it really is a request module call from the crypto > tests that's triggering this warning. Preeetty likely that's what's happening, > but want to be sure since I can't reproduce this. Then I can post the fix. > I have added CONFIG_FTRACE=y and your second patch. The boot log can be seen at http://kernel.montjoie.ovh/108789.log But it seems the latest dump_stack addition flood a bit. I have started to read ftrace documentation, but if you have a quick what to do in /sys/kernel/debug/tracing, it will be helpfull. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB97EC4363C for ; Wed, 7 Oct 2020 19:42:54 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3FD09206F0 for ; Wed, 7 Oct 2020 19:42:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="TELSaFHC"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kcN3zsM/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3FD09206F0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FVrnUSQayftdRtAVBw7zlyDhhJnF9JwXMmS4OR6aXgI=; b=TELSaFHCTwTypbEfXaGBoeO3o VYFNvM4b69R+03Bf7ZQ/xIwwny2PGy1VUlZ2SRTNeSnRlfY6MzVY6XY+YC/QyXvRkYI9U1SQnh31Z sYY2QltX9zOSYmX13GoC3ozOljrJDXioUDpKy//R2s9nW+RVthgoucrawB6dsv2XNU8ibf1/CQP/F BP1MpJyiehxvrubPeYALjhAG8QH1Q2hVXQiADYjM3PpUNdiIschhxptKEqGdOztmGAS9ejl93zoEH qiL4qxQUzjam+mhW4xTcgPaFoBjLNeW1mr/PpqJXb3adXaHPoZm5ZFqCtRZH9ryAFNUrHC01+C6bm 6TDmDKvYQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kQFJW-0005rj-H7; Wed, 07 Oct 2020 19:41:26 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kQFJT-0005qf-ER for linux-arm-kernel@lists.infradead.org; Wed, 07 Oct 2020 19:41:24 +0000 Received: by mail-wr1-x443.google.com with SMTP id k10so3688802wru.6 for ; Wed, 07 Oct 2020 12:41:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BlXTl6Hst6j2ZWlhRy8AiZeKdtWqE6ffJKL4bR3bTTQ=; b=kcN3zsM/ZtIXY6Z3zNNaFiikNc/iRaBZnxWvNYxoXEUtzNiHbwXZE4CR024aHVJEt3 dXZoZeL6K3E3uWH8Xp9tFUjWpzrtRdRM9v1kwZdo7Ed4w0h6bFcL0v1ixtz9pjUhzzkE TU/3b5FJtRRkNVZVxbNiInzha9rZEHRsxWTUTlUN2rhDPp6S+3m1pN6xficRXceeERJ7 QCkvpeB5aJ4MuBZnnXgrkAep6Rt5RGtqj1tYUiMsArrJSipocmOvPJ0Z6gelvX/+YHaq bBPMlYl0XA9kURPuUbQuH42W/milxSPmX4C/iLklEzeXB9/C12CppD3jJbhAvWIhw09i a+Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BlXTl6Hst6j2ZWlhRy8AiZeKdtWqE6ffJKL4bR3bTTQ=; b=iDfX0Py2kGM3O5vYc/1jL5D1eJ2fIgJHHTqfjUSKz6WLm7/z4uxP6bl6aVmNicR2pB sJY0o341XlDysxpDlxzL4/tSJ1vmtQCwjr5DO257b5zqwkjKPs6BcWbY0ufi/nB6Rc4k afrx4DaAIudkGGbXos1HtVqrynsAu/qxTMtrA+x0yZBYsYlBftWAnEB+rMUkjEnR6xLK dJQ5ZSn66EKJhTNQFeqqfU1ViuKuGkKVn3nZir67vPiutPT5TvUUjRayFer8NkZbzCpG 72uoIetONV+HqIh2w0iR9Q8rcxIHNJH54VNMrpq+tWGU5GwqJwDBQrZzN7PZP3tEZvHZ r7ZA== X-Gm-Message-State: AOAM5325jtYkWc9p81FJP+DjRbtdZ2Urtpow+9t8RhlfWaj14B+WwL4X QS1Gb9IftDfK9ndcH/lwgXE= X-Google-Smtp-Source: ABdhPJy/7viZdAja+X9JBIUZBanX5mxDMsnzvuFIPvUeDKF3xHLhGKTLJJ0EwcgvuZBD0cR6EcQ74A== X-Received: by 2002:adf:ce01:: with SMTP id p1mr4928562wrn.33.1602099680845; Wed, 07 Oct 2020 12:41:20 -0700 (PDT) Received: from Red ([2a01:cb1d:3d5:a100:264b:feff:fe03:2806]) by smtp.googlemail.com with ESMTPSA id c1sm4156555wru.49.2020.10.07.12.41.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Oct 2020 12:41:19 -0700 (PDT) Date: Wed, 7 Oct 2020 21:41:17 +0200 From: Corentin Labbe To: Daniel Jordan Subject: Re: WARNING: at kernel/workqueue.c:1473 __queue_work+0x3b8/0x3d0 Message-ID: <20201007194117.GA4859@Red> References: <20200220090350.GA19858@Red> <20200221174223.r3y6tugavp3k5jdl@ca-dmjordan1.us.oracle.com> <20200228123311.GE3275@willie-the-truck> <20200228153331.uimy62rat2tdxxod@ca-dmjordan1.us.oracle.com> <20200301175351.GA11684@Red> <20200302172510.fspofleipqjcdxak@ca-dmjordan1.us.oracle.com> <20200303213017.tanczhqd3nhpeeak@ca-dmjordan1.us.oracle.com> <20201001175022.GA22444@Red> <20201005170910.vxwrdwnzlw3ahkb4@ca-dmjordan1.us.oracle.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20201005170910.vxwrdwnzlw3ahkb4@ca-dmjordan1.us.oracle.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201007_154123_514928_A8E7DB78 X-CRM114-Status: GOOD ( 28.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, Will Deacon , jiangshanlai@gmail.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, tj@kernel.org, Robin Murphy , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Oct 05, 2020 at 01:09:10PM -0400, Daniel Jordan wrote: > On Thu, Oct 01, 2020 at 07:50:22PM +0200, Corentin Labbe wrote: > > On Tue, Mar 03, 2020 at 04:30:17PM -0500, Daniel Jordan wrote: > > > Barring other ideas, Corentin, would you be willing to boot with > > > > > > trace_event=initcall:*,module:* trace_options=stacktrace > > > > > > and > > > > > > diff --git a/kernel/module.c b/kernel/module.c > > > index 33569a01d6e1..393be6979a27 100644 > > > --- a/kernel/module.c > > > +++ b/kernel/module.c > > > @@ -3604,8 +3604,11 @@ static noinline int do_init_module(struct module *mod) > > > * be cleaned up needs to sync with the queued work - ie > > > * rcu_barrier() > > > */ > > > - if (llist_add(&freeinit->node, &init_free_list)) > > > + if (llist_add(&freeinit->node, &init_free_list)) { > > > + pr_warn("%s: schedule_work for mod=%s\n", __func__, mod->name); > > > + dump_stack(); > > > schedule_work(&init_free_wq); > > > + } > > > > > > mutex_unlock(&module_mutex); > > > wake_up_all(&module_wq); > > > > > > but not my earlier fix and share the dmesg and ftrace output to see if the > > > theory holds? > > > > > > Also, could you attach your config? Curious now what your crypto options look > > > like after fiddling with some of them today while trying and failing to see > > > this on x86. > > > > > > thanks, > > > Daniel > > > > Hello > > > > Sorry for the very delayed answer. > > > > I fail to reproduce it on x86 (qemu and real hw) and arm. > > It seems to only happen on arm64. > > Thanks for the config and dmesg, but there's no ftrace. I see it's not > configured in your kernel, so could you boot with my earlier debug patch plus > this one and the kernel argument initcall_debug instead? > > I'm trying to see whether it really is a request module call from the crypto > tests that's triggering this warning. Preeetty likely that's what's happening, > but want to be sure since I can't reproduce this. Then I can post the fix. > I have added CONFIG_FTRACE=y and your second patch. The boot log can be seen at http://kernel.montjoie.ovh/108789.log But it seems the latest dump_stack addition flood a bit. I have started to read ftrace documentation, but if you have a quick what to do in /sys/kernel/debug/tracing, it will be helpfull. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel