From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30650C4332F for ; Thu, 30 Sep 2021 09:00:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1BF8861881 for ; Thu, 30 Sep 2021 09:00:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349203AbhI3JCe (ORCPT ); Thu, 30 Sep 2021 05:02:34 -0400 Received: from foss.arm.com ([217.140.110.172]:50766 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348335AbhI3JCd (ORCPT ); Thu, 30 Sep 2021 05:02:33 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0AF56D6E; Thu, 30 Sep 2021 02:00:51 -0700 (PDT) Received: from e113632-lin (e113632-lin.cambridge.arm.com [10.1.196.57]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 084F13F793; Thu, 30 Sep 2021 02:00:47 -0700 (PDT) From: Valentin Schneider To: Thomas Gleixner , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, rcu@vger.kernel.org, linux-rt-users@vger.kernel.org Cc: Catalin Marinas , Will Deacon , Ingo Molnar , Peter Zijlstra , Steven Rostedt , Daniel Bristot de Oliveira , Sebastian Andrzej Siewior , "Paul E. McKenney" , Frederic Weisbecker , Josh Triplett , Mathieu Desnoyers , Davidlohr Bueso , Lai Jiangshan , Joel Fernandes , Anshuman Khandual , Vincenzo Frascino , Steven Price , Ard Biesheuvel , Boqun Feng , Mike Galbraith Subject: Re: rcu/tree: Protect rcu_rdp_is_offloaded() invocations on RT In-Reply-To: <87pmt163al.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain References: <20210811201354.1976839-1-valentin.schneider@arm.com> <20210811201354.1976839-4-valentin.schneider@arm.com> <874kae6n3g.ffs@tglx> <87pmt163al.ffs@tglx> Date: Thu, 30 Sep 2021 10:00:39 +0100 Message-ID: <87h7e21lqg.mognet@arm.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On 21/09/21 23:12, Thomas Gleixner wrote: > Valentin reported warnings about suspicious RCU usage on RT kernels. Those > happen when offloading of RCU callbacks is enabled: > > WARNING: suspicious RCU usage > 5.13.0-rt1 #20 Not tainted > ----------------------------- > kernel/rcu/tree_plugin.h:69 Unsafe read of RCU_NOCB offloaded state! > > rcu_rdp_is_offloaded (kernel/rcu/tree_plugin.h:69 kernel/rcu/tree_plugin.h:58) > rcu_core (kernel/rcu/tree.c:2332 kernel/rcu/tree.c:2398 kernel/rcu/tree.c:2777) > rcu_cpu_kthread (./include/linux/bottom_half.h:32 kernel/rcu/tree.c:2876) > > The reason is that rcu_rdp_is_offloaded() is invoked without one of the > required protections on RT enabled kernels because local_bh_disable() does > not disable preemption on RT. > > Valentin proposed to add a local lock to the code in question, but that's > suboptimal in several aspects: > > 1) local locks add extra code to !RT kernels for no value. > > 2) All possible callsites have to audited and amended when affected > possible at an outer function level due to lock nesting issues. > > 3) As the local lock has to be taken at the outer functions it's required > to release and reacquire them in the inner code sections which might > voluntary schedule, e.g. rcu_do_batch(). > > Both callsites of rcu_rdp_is_offloaded() which trigger this check invoke > rcu_rdp_is_offloaded() in the variable declaration section right at the top > of the functions. But the actual usage of the result is either within a > section which provides the required protections or after such a section. > > So the obvious solution is to move the invocation into the code sections > which provide the proper protections, which solves the problem for RT and > does not have any impact on !RT kernels. > Thanks for taking a look at this! My reasoning for adding protection in the outer functions was to prevent impaired unlocks of rcu_nocb_{un}lock_irqsave(), as that too depends on the offload state. Cf. Frederic's writeup: http://lore.kernel.org/r/20210727230814.GC283787@lothringen Anywho, I see Frederic has sent a fancy new series, let me go stare at it. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F00C8C433EF for ; Thu, 30 Sep 2021 09:03:35 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B1EB66044F for ; Thu, 30 Sep 2021 09:03:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B1EB66044F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Message-ID:Date:References:MIME-Version :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WoVvpC7fY5d78tXdnOXn6tG9T055UdG6Xyr3MhS6g4Q=; b=0WKoprJnKWM/6u PF83LTqRkrX3I6JyBHGZ3w7DmExI2Y6LJs09ivv+wHlPLltvUfDlZajphsjfXbTvUtV17JuH3IHvw Zy7qz2Luf3wm8nlMeQ5fZuhRzaZAr2t+ne5wsjN55fRqRNuYLuNb6Ad4I4yy2uPuC3/Dx4zPIupu0 jEMEwm4v25/BH6CZafKI6lLVm4Impxyx7FAhPwIN3wZKFtAPC83tT73bkci754gCPT0TwId/T2ukU w/xRIBpggoV8wlrSqV68IsEONkoRxAFZIeZX9nxPOVn3BOCmNuRpy9NwJmXGWwUw++ZtiC+MLUGOU UlXLOlrZ3Bq+CIKVab7A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mVrw2-00DVaF-91; Thu, 30 Sep 2021 09:00:58 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mVrvy-00DVYy-K9 for linux-arm-kernel@lists.infradead.org; Thu, 30 Sep 2021 09:00:56 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0AF56D6E; Thu, 30 Sep 2021 02:00:51 -0700 (PDT) Received: from e113632-lin (e113632-lin.cambridge.arm.com [10.1.196.57]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 084F13F793; Thu, 30 Sep 2021 02:00:47 -0700 (PDT) From: Valentin Schneider To: Thomas Gleixner , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, rcu@vger.kernel.org, linux-rt-users@vger.kernel.org Cc: Catalin Marinas , Will Deacon , Ingo Molnar , Peter Zijlstra , Steven Rostedt , Daniel Bristot de Oliveira , Sebastian Andrzej Siewior , "Paul E. McKenney" , Frederic Weisbecker , Josh Triplett , Mathieu Desnoyers , Davidlohr Bueso , Lai Jiangshan , Joel Fernandes , Anshuman Khandual , Vincenzo Frascino , Steven Price , Ard Biesheuvel , Boqun Feng , Mike Galbraith Subject: Re: rcu/tree: Protect rcu_rdp_is_offloaded() invocations on RT In-Reply-To: <87pmt163al.ffs@tglx> MIME-Version: 1.0 References: <20210811201354.1976839-1-valentin.schneider@arm.com> <20210811201354.1976839-4-valentin.schneider@arm.com> <874kae6n3g.ffs@tglx> <87pmt163al.ffs@tglx> Date: Thu, 30 Sep 2021 10:00:39 +0100 Message-ID: <87h7e21lqg.mognet@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210930_020054_750843_3BDF4E9A X-CRM114-Status: GOOD ( 16.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, On 21/09/21 23:12, Thomas Gleixner wrote: > Valentin reported warnings about suspicious RCU usage on RT kernels. Those > happen when offloading of RCU callbacks is enabled: > > WARNING: suspicious RCU usage > 5.13.0-rt1 #20 Not tainted > ----------------------------- > kernel/rcu/tree_plugin.h:69 Unsafe read of RCU_NOCB offloaded state! > > rcu_rdp_is_offloaded (kernel/rcu/tree_plugin.h:69 kernel/rcu/tree_plugin.h:58) > rcu_core (kernel/rcu/tree.c:2332 kernel/rcu/tree.c:2398 kernel/rcu/tree.c:2777) > rcu_cpu_kthread (./include/linux/bottom_half.h:32 kernel/rcu/tree.c:2876) > > The reason is that rcu_rdp_is_offloaded() is invoked without one of the > required protections on RT enabled kernels because local_bh_disable() does > not disable preemption on RT. > > Valentin proposed to add a local lock to the code in question, but that's > suboptimal in several aspects: > > 1) local locks add extra code to !RT kernels for no value. > > 2) All possible callsites have to audited and amended when affected > possible at an outer function level due to lock nesting issues. > > 3) As the local lock has to be taken at the outer functions it's required > to release and reacquire them in the inner code sections which might > voluntary schedule, e.g. rcu_do_batch(). > > Both callsites of rcu_rdp_is_offloaded() which trigger this check invoke > rcu_rdp_is_offloaded() in the variable declaration section right at the top > of the functions. But the actual usage of the result is either within a > section which provides the required protections or after such a section. > > So the obvious solution is to move the invocation into the code sections > which provide the proper protections, which solves the problem for RT and > does not have any impact on !RT kernels. > Thanks for taking a look at this! My reasoning for adding protection in the outer functions was to prevent impaired unlocks of rcu_nocb_{un}lock_irqsave(), as that too depends on the offload state. Cf. Frederic's writeup: http://lore.kernel.org/r/20210727230814.GC283787@lothringen Anywho, I see Frederic has sent a fancy new series, let me go stare at it. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel