From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25F25C6379F for ; Fri, 20 Nov 2020 18:02:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA0612245A for ; Fri, 20 Nov 2020 18:02:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729274AbgKTSCO (ORCPT ); Fri, 20 Nov 2020 13:02:14 -0500 Received: from foss.arm.com ([217.140.110.172]:53070 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728438AbgKTSCN (ORCPT ); Fri, 20 Nov 2020 13:02:13 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 483481042; Fri, 20 Nov 2020 10:02:12 -0800 (PST) Received: from C02TD0UTHF1T.local (unknown [10.57.27.176]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DBCEC3F719; Fri, 20 Nov 2020 10:02:08 -0800 (PST) Date: Fri, 20 Nov 2020 18:02:06 +0000 From: Mark Rutland To: "Paul E. McKenney" Cc: Marco Elver , Steven Rostedt , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, Peter Zijlstra , Tejun Heo , Lai Jiangshan , linux-arm-kernel@lists.infradead.org Subject: Re: linux-next: stall warnings and deadlock on Arm64 (was: [PATCH] kfence: Avoid stalling...) Message-ID: <20201120180206.GF2328@C02TD0UTHF1T.local> References: <20201119125357.GA2084963@elver.google.com> <20201119151409.GU1437@paulmck-ThinkPad-P72> <20201119170259.GA2134472@elver.google.com> <20201119184854.GY1437@paulmck-ThinkPad-P72> <20201119193819.GA2601289@elver.google.com> <20201119213512.GB1437@paulmck-ThinkPad-P72> <20201120141928.GB3120165@elver.google.com> <20201120143928.GH1437@paulmck-ThinkPad-P72> <20201120152200.GD2328@C02TD0UTHF1T.local> <20201120173824.GJ1437@paulmck-ThinkPad-P72> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201120173824.GJ1437@paulmck-ThinkPad-P72> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Nov 20, 2020 at 09:38:24AM -0800, Paul E. McKenney wrote: > On Fri, Nov 20, 2020 at 03:22:00PM +0000, Mark Rutland wrote: > > On Fri, Nov 20, 2020 at 06:39:28AM -0800, Paul E. McKenney wrote: > > > On Fri, Nov 20, 2020 at 03:19:28PM +0100, Marco Elver wrote: > > > > I found that disabling ftrace for some of kernel/rcu (see below) solved > > > > the stalls (and any mention of deadlocks as a side-effect I assume), > > > > resulting in successful boot. > > > > > > > > Does that provide any additional clues? I tried to narrow it down to 1-2 > > > > files, but that doesn't seem to work. > > > > > > There were similar issues during the x86/entry work. Are the ARM guys > > > doing arm64/entry work now? > > > > I'm currently looking at it. I had been trying to shift things to C for > > a while, and right now I'm trying to fix the lockdep state tracking, > > which is requiring untangling lockdep/rcu/tracing. > > > > The main issue I see remaining atm is that we don't save/restore the > > lockdep state over exceptions taken from kernel to kernel. That could > > result in lockdep thinking IRQs are disabled when they're actually > > enabled (because code in the nested context might do a save/restore > > while IRQs are disabled, then return to a context where IRQs are > > enabled), but AFAICT shouldn't result in the inverse in most cases since > > the non-NMI handlers all call lockdep_hardirqs_disabled(). > > > > I'm at a loss to explaim the rcu vs ftrace bits, so if you have any > > pointers to the issuies ween with the x86 rework that'd be quite handy. > > There were several over a number of months. I especially recall issues > with the direct-from-idle execution of smp_call_function*() handlers, > and also with some of the special cases in the entry code, for example, > reentering the kernel from the kernel. This latter could cause RCU to > not be watching when it should have been or vice versa. Ah; those are precisely the cases I'm currently fixing, so if we're lucky this is an indirect result of one of those rather than a novel source of pain... > I would of course be most aware of the issues that impinged on RCU > and that were located by rcutorture. This is actually not hard to run, > especially if the ARM bits in the scripting have managed to avoid bitrot. > The "modprobe rcutorture" approach has fewer dependencies. Either way: > https://paulmck.livejournal.com/57769.html and later posts. That is a very good idea. I'd been relying on Syzkaller to tickle the issue, but the torture infrastructure is a much better fit for this problem. I hadn't realise how comprehensive the scripting was, thanks for this! I'll see about giving that a go once I have the irq-from-idle cases sorted, as those are very obviously broken if you hack trace_hardirqs_{on,off}() to check that RCU is watching. Thanks, Mark. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53DC6C2D0E4 for ; Fri, 20 Nov 2020 18:03:25 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CB172206B7 for ; Fri, 20 Nov 2020 18:03:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="BvR5C+gj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CB172206B7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ojW0vNxp7raCB27xP1aEqJ/a5vVocdzdNXei6eIE1EE=; b=BvR5C+gjnCoRF1nMJIynubOEh NDoX9oJCTndmT/XJc7zqT/AbKEG1IWvshdQJhKKhsAIZ9ez9Kj5BJ5uE4s2550ll9l9xW06JG++oy 1NHgKpYA8Cu3yOsyumvgbCAXxklawkvXnQ12CX7VtRAhv7DXsnq5diI2Cb8cbXxu1l0k66dgMswS4 7Wwr7Nih9Npl1w/4CDFztbC+wki4YRbY/nbgI+PxygIGsMn0geK1Rsg6rN0ftDSijdwpYF3tUy+Ud +rt8YvhP9Fq/jHITEenfJ+627L5Gd6L8AJdCJmmM/bTHUWII9MmAxs3L7ZFRSSNAPJo6xyCRztEVl hYMagMxzQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kgAjj-000683-Kc; Fri, 20 Nov 2020 18:02:19 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kgAjg-00067D-Ep for linux-arm-kernel@lists.infradead.org; Fri, 20 Nov 2020 18:02:18 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 483481042; Fri, 20 Nov 2020 10:02:12 -0800 (PST) Received: from C02TD0UTHF1T.local (unknown [10.57.27.176]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DBCEC3F719; Fri, 20 Nov 2020 10:02:08 -0800 (PST) Date: Fri, 20 Nov 2020 18:02:06 +0000 From: Mark Rutland To: "Paul E. McKenney" Subject: Re: linux-next: stall warnings and deadlock on Arm64 (was: [PATCH] kfence: Avoid stalling...) Message-ID: <20201120180206.GF2328@C02TD0UTHF1T.local> References: <20201119125357.GA2084963@elver.google.com> <20201119151409.GU1437@paulmck-ThinkPad-P72> <20201119170259.GA2134472@elver.google.com> <20201119184854.GY1437@paulmck-ThinkPad-P72> <20201119193819.GA2601289@elver.google.com> <20201119213512.GB1437@paulmck-ThinkPad-P72> <20201120141928.GB3120165@elver.google.com> <20201120143928.GH1437@paulmck-ThinkPad-P72> <20201120152200.GD2328@C02TD0UTHF1T.local> <20201120173824.GJ1437@paulmck-ThinkPad-P72> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20201120173824.GJ1437@paulmck-ThinkPad-P72> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201120_130216_920490_E0D5F02B X-CRM114-Status: GOOD ( 32.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arm-kernel@lists.infradead.org, Marco Elver , Anders Roxell , Jann Horn , Peter Zijlstra , Lai Jiangshan , Linux Kernel Mailing List , Steven Rostedt , rcu@vger.kernel.org, Linux-MM , Alexander Potapenko , kasan-dev , Tejun Heo , Andrew Morton , Dmitry Vyukov Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Nov 20, 2020 at 09:38:24AM -0800, Paul E. McKenney wrote: > On Fri, Nov 20, 2020 at 03:22:00PM +0000, Mark Rutland wrote: > > On Fri, Nov 20, 2020 at 06:39:28AM -0800, Paul E. McKenney wrote: > > > On Fri, Nov 20, 2020 at 03:19:28PM +0100, Marco Elver wrote: > > > > I found that disabling ftrace for some of kernel/rcu (see below) solved > > > > the stalls (and any mention of deadlocks as a side-effect I assume), > > > > resulting in successful boot. > > > > > > > > Does that provide any additional clues? I tried to narrow it down to 1-2 > > > > files, but that doesn't seem to work. > > > > > > There were similar issues during the x86/entry work. Are the ARM guys > > > doing arm64/entry work now? > > > > I'm currently looking at it. I had been trying to shift things to C for > > a while, and right now I'm trying to fix the lockdep state tracking, > > which is requiring untangling lockdep/rcu/tracing. > > > > The main issue I see remaining atm is that we don't save/restore the > > lockdep state over exceptions taken from kernel to kernel. That could > > result in lockdep thinking IRQs are disabled when they're actually > > enabled (because code in the nested context might do a save/restore > > while IRQs are disabled, then return to a context where IRQs are > > enabled), but AFAICT shouldn't result in the inverse in most cases since > > the non-NMI handlers all call lockdep_hardirqs_disabled(). > > > > I'm at a loss to explaim the rcu vs ftrace bits, so if you have any > > pointers to the issuies ween with the x86 rework that'd be quite handy. > > There were several over a number of months. I especially recall issues > with the direct-from-idle execution of smp_call_function*() handlers, > and also with some of the special cases in the entry code, for example, > reentering the kernel from the kernel. This latter could cause RCU to > not be watching when it should have been or vice versa. Ah; those are precisely the cases I'm currently fixing, so if we're lucky this is an indirect result of one of those rather than a novel source of pain... > I would of course be most aware of the issues that impinged on RCU > and that were located by rcutorture. This is actually not hard to run, > especially if the ARM bits in the scripting have managed to avoid bitrot. > The "modprobe rcutorture" approach has fewer dependencies. Either way: > https://paulmck.livejournal.com/57769.html and later posts. That is a very good idea. I'd been relying on Syzkaller to tickle the issue, but the torture infrastructure is a much better fit for this problem. I hadn't realise how comprehensive the scripting was, thanks for this! I'll see about giving that a go once I have the irq-from-idle cases sorted, as those are very obviously broken if you hack trace_hardirqs_{on,off}() to check that RCU is watching. Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel