From: Frederic Weisbecker <frederic@kernel.org> To: Valentin Schneider <valentin.schneider@arm.com> Cc: Thomas Gleixner <tglx@linutronix.de>, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, rcu@vger.kernel.org, linux-rt-users@vger.kernel.org, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Ingo Molnar <mingo@kernel.org>, Peter Zijlstra <peterz@infradead.org>, Steven Rostedt <rostedt@goodmis.org>, Daniel Bristot de Oliveira <bristot@redhat.com>, Sebastian Andrzej Siewior <bigeasy@linutronix.de>, "Paul E. McKenney" <paulmck@kernel.org>, Josh Triplett <josh@joshtriplett.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>, Davidlohr Bueso <dave@stgolabs.net>, Lai Jiangshan <jiangshanlai@gmail.com>, Joel Fernandes <joel@joelfernandes.org>, Anshuman Khandual <anshuman.khandual@arm.com>, Vincenzo Frascino <vincenzo.frascino@arm.com>, Steven Price <steven.price@arm.com>, Ard Biesheuvel <ardb@kernel.org>, Boqun Feng <boqun.feng@gmail.com>, Mike Galbraith <efault@gmx.de> Subject: Re: rcu/tree: Protect rcu_rdp_is_offloaded() invocations on RT Date: Thu, 30 Sep 2021 12:53:40 +0200 [thread overview] Message-ID: <20210930105340.GA232241@lothringen> (raw) In-Reply-To: <87h7e21lqg.mognet@arm.com> On Thu, Sep 30, 2021 at 10:00:39AM +0100, Valentin Schneider wrote: > Hi, > > On 21/09/21 23:12, Thomas Gleixner wrote: > > Valentin reported warnings about suspicious RCU usage on RT kernels. Those > > happen when offloading of RCU callbacks is enabled: > > > > WARNING: suspicious RCU usage > > 5.13.0-rt1 #20 Not tainted > > ----------------------------- > > kernel/rcu/tree_plugin.h:69 Unsafe read of RCU_NOCB offloaded state! > > > > rcu_rdp_is_offloaded (kernel/rcu/tree_plugin.h:69 kernel/rcu/tree_plugin.h:58) > > rcu_core (kernel/rcu/tree.c:2332 kernel/rcu/tree.c:2398 kernel/rcu/tree.c:2777) > > rcu_cpu_kthread (./include/linux/bottom_half.h:32 kernel/rcu/tree.c:2876) > > > > The reason is that rcu_rdp_is_offloaded() is invoked without one of the > > required protections on RT enabled kernels because local_bh_disable() does > > not disable preemption on RT. > > > > Valentin proposed to add a local lock to the code in question, but that's > > suboptimal in several aspects: > > > > 1) local locks add extra code to !RT kernels for no value. > > > > 2) All possible callsites have to audited and amended when affected > > possible at an outer function level due to lock nesting issues. > > > > 3) As the local lock has to be taken at the outer functions it's required > > to release and reacquire them in the inner code sections which might > > voluntary schedule, e.g. rcu_do_batch(). > > > > Both callsites of rcu_rdp_is_offloaded() which trigger this check invoke > > rcu_rdp_is_offloaded() in the variable declaration section right at the top > > of the functions. But the actual usage of the result is either within a > > section which provides the required protections or after such a section. > > > > So the obvious solution is to move the invocation into the code sections > > which provide the proper protections, which solves the problem for RT and > > does not have any impact on !RT kernels. > > > > Thanks for taking a look at this! > > My reasoning for adding protection in the outer functions was to prevent > impaired unlocks of rcu_nocb_{un}lock_irqsave(), as that too depends on the > offload state. Cf. Frederic's writeup: > > http://lore.kernel.org/r/20210727230814.GC283787@lothringen I was wrong about that BTW! Because rcu_nocb_lock() always require IRQs to be disabled, which of course disables preemption, so the offloaded state can't change between rcu_nocb_lock[_irqsave]() and rcu_nocb_unlock[_irqrestore]() but anyway there were many other issues to fix :-) > > Anywho, I see Frederic has sent a fancy new series, let me go stare at it.
WARNING: multiple messages have this Message-ID (diff)
From: Frederic Weisbecker <frederic@kernel.org> To: Valentin Schneider <valentin.schneider@arm.com> Cc: Thomas Gleixner <tglx@linutronix.de>, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, rcu@vger.kernel.org, linux-rt-users@vger.kernel.org, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Ingo Molnar <mingo@kernel.org>, Peter Zijlstra <peterz@infradead.org>, Steven Rostedt <rostedt@goodmis.org>, Daniel Bristot de Oliveira <bristot@redhat.com>, Sebastian Andrzej Siewior <bigeasy@linutronix.de>, "Paul E. McKenney" <paulmck@kernel.org>, Josh Triplett <josh@joshtriplett.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>, Davidlohr Bueso <dave@stgolabs.net>, Lai Jiangshan <jiangshanlai@gmail.com>, Joel Fernandes <joel@joelfernandes.org>, Anshuman Khandual <anshuman.khandual@arm.com>, Vincenzo Frascino <vincenzo.frascino@arm.com>, Steven Price <steven.price@arm.com>, Ard Biesheuvel <ardb@kernel.org>, Boqun Feng <boqun.feng@gmail.com>, Mike Galbraith <efault@gmx.de> Subject: Re: rcu/tree: Protect rcu_rdp_is_offloaded() invocations on RT Date: Thu, 30 Sep 2021 12:53:40 +0200 [thread overview] Message-ID: <20210930105340.GA232241@lothringen> (raw) In-Reply-To: <87h7e21lqg.mognet@arm.com> On Thu, Sep 30, 2021 at 10:00:39AM +0100, Valentin Schneider wrote: > Hi, > > On 21/09/21 23:12, Thomas Gleixner wrote: > > Valentin reported warnings about suspicious RCU usage on RT kernels. Those > > happen when offloading of RCU callbacks is enabled: > > > > WARNING: suspicious RCU usage > > 5.13.0-rt1 #20 Not tainted > > ----------------------------- > > kernel/rcu/tree_plugin.h:69 Unsafe read of RCU_NOCB offloaded state! > > > > rcu_rdp_is_offloaded (kernel/rcu/tree_plugin.h:69 kernel/rcu/tree_plugin.h:58) > > rcu_core (kernel/rcu/tree.c:2332 kernel/rcu/tree.c:2398 kernel/rcu/tree.c:2777) > > rcu_cpu_kthread (./include/linux/bottom_half.h:32 kernel/rcu/tree.c:2876) > > > > The reason is that rcu_rdp_is_offloaded() is invoked without one of the > > required protections on RT enabled kernels because local_bh_disable() does > > not disable preemption on RT. > > > > Valentin proposed to add a local lock to the code in question, but that's > > suboptimal in several aspects: > > > > 1) local locks add extra code to !RT kernels for no value. > > > > 2) All possible callsites have to audited and amended when affected > > possible at an outer function level due to lock nesting issues. > > > > 3) As the local lock has to be taken at the outer functions it's required > > to release and reacquire them in the inner code sections which might > > voluntary schedule, e.g. rcu_do_batch(). > > > > Both callsites of rcu_rdp_is_offloaded() which trigger this check invoke > > rcu_rdp_is_offloaded() in the variable declaration section right at the top > > of the functions. But the actual usage of the result is either within a > > section which provides the required protections or after such a section. > > > > So the obvious solution is to move the invocation into the code sections > > which provide the proper protections, which solves the problem for RT and > > does not have any impact on !RT kernels. > > > > Thanks for taking a look at this! > > My reasoning for adding protection in the outer functions was to prevent > impaired unlocks of rcu_nocb_{un}lock_irqsave(), as that too depends on the > offload state. Cf. Frederic's writeup: > > http://lore.kernel.org/r/20210727230814.GC283787@lothringen I was wrong about that BTW! Because rcu_nocb_lock() always require IRQs to be disabled, which of course disables preemption, so the offloaded state can't change between rcu_nocb_lock[_irqsave]() and rcu_nocb_unlock[_irqrestore]() but anyway there were many other issues to fix :-) > > Anywho, I see Frederic has sent a fancy new series, let me go stare at it. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-09-30 10:53 UTC|newest] Thread overview: 102+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-08-11 20:13 [PATCH v3 0/4] rcu, arm64: PREEMPT_RT fixlets Valentin Schneider 2021-08-11 20:13 ` Valentin Schneider 2021-08-11 20:13 ` [PATCH v3 1/4] rcutorture: Don't disable softirqs with preemption disabled when PREEMPT_RT Valentin Schneider 2021-08-11 20:13 ` Valentin Schneider 2021-08-12 16:47 ` Paul E. McKenney 2021-08-12 16:47 ` Paul E. McKenney 2021-08-17 12:13 ` Sebastian Andrzej Siewior 2021-08-17 12:13 ` Sebastian Andrzej Siewior 2021-08-17 13:17 ` Sebastian Andrzej Siewior 2021-08-17 13:17 ` Sebastian Andrzej Siewior 2021-08-17 14:40 ` [PATCH] rcutorture: Avoid problematic critical section nesting on RT Sebastian Andrzej Siewior 2021-08-17 14:40 ` Sebastian Andrzej Siewior 2021-08-18 22:46 ` Paul E. McKenney 2021-08-18 22:46 ` Paul E. McKenney 2021-08-19 15:35 ` Sebastian Andrzej Siewior 2021-08-19 15:35 ` Sebastian Andrzej Siewior 2021-08-19 15:39 ` Sebastian Andrzej Siewior 2021-08-19 15:39 ` Sebastian Andrzej Siewior 2021-08-19 15:47 ` Sebastian Andrzej Siewior 2021-08-19 15:47 ` Sebastian Andrzej Siewior 2021-08-19 18:20 ` Paul E. McKenney 2021-08-19 18:20 ` Paul E. McKenney 2021-08-19 18:45 ` Sebastian Andrzej Siewior 2021-08-19 18:45 ` Sebastian Andrzej Siewior 2021-08-20 4:11 ` Scott Wood 2021-08-20 4:11 ` Scott Wood 2021-08-20 7:11 ` Sebastian Andrzej Siewior 2021-08-20 7:11 ` Sebastian Andrzej Siewior 2021-08-20 7:42 ` [PATCH v2] rcutorture: Avoid problematic critical section nesting on PREEMPT_RT Sebastian Andrzej Siewior 2021-08-20 7:42 ` Sebastian Andrzej Siewior 2021-08-20 22:10 ` Paul E. McKenney 2021-08-20 22:10 ` Paul E. McKenney 2021-08-20 3:23 ` [PATCH] rcutorture: Avoid problematic critical section nesting on RT Scott Wood 2021-08-20 3:23 ` Scott Wood 2021-08-20 6:54 ` Sebastian Andrzej Siewior 2021-08-20 6:54 ` Sebastian Andrzej Siewior 2021-08-11 20:13 ` [PATCH v3 2/4] sched: Introduce migratable() Valentin Schneider 2021-08-11 20:13 ` Valentin Schneider 2021-08-17 14:43 ` Sebastian Andrzej Siewior 2021-08-17 14:43 ` Sebastian Andrzej Siewior 2021-08-22 17:31 ` Valentin Schneider 2021-08-22 17:31 ` Valentin Schneider 2021-08-17 17:09 ` Sebastian Andrzej Siewior 2021-08-17 17:09 ` Sebastian Andrzej Siewior 2021-08-17 19:30 ` Phil Auld 2021-08-17 19:30 ` Phil Auld 2021-08-22 18:14 ` Valentin Schneider 2021-08-22 18:14 ` Valentin Schneider 2022-01-26 16:56 ` Sebastian Andrzej Siewior 2022-01-26 16:56 ` Sebastian Andrzej Siewior 2022-01-26 18:10 ` Valentin Schneider 2022-01-26 18:10 ` Valentin Schneider 2022-01-27 10:07 ` Sebastian Andrzej Siewior 2022-01-27 10:07 ` Sebastian Andrzej Siewior 2022-01-27 18:23 ` Valentin Schneider 2022-01-27 18:23 ` Valentin Schneider 2022-01-27 19:27 ` Valentin Schneider 2022-01-27 19:27 ` Valentin Schneider 2022-02-04 9:24 ` Sebastian Andrzej Siewior 2022-02-04 9:24 ` Sebastian Andrzej Siewior 2021-08-11 20:13 ` [PATCH v3 3/4] rcu/nocb: Protect NOCB state via local_lock() under PREEMPT_RT Valentin Schneider 2021-08-11 20:13 ` Valentin Schneider 2021-08-13 0:20 ` Paul E. McKenney 2021-08-13 0:20 ` Paul E. McKenney 2021-08-13 18:48 ` Valentin Schneider 2021-08-13 18:48 ` Valentin Schneider 2021-08-24 13:00 ` Frederic Weisbecker 2021-08-17 15:36 ` Sebastian Andrzej Siewior 2021-08-17 15:36 ` Sebastian Andrzej Siewior 2021-08-22 18:15 ` Valentin Schneider 2021-08-22 18:15 ` Valentin Schneider 2021-09-21 14:05 ` Thomas Gleixner 2021-09-21 14:05 ` Thomas Gleixner 2021-09-21 21:12 ` rcu/tree: Protect rcu_rdp_is_offloaded() invocations on RT Thomas Gleixner 2021-09-21 23:36 ` Frederic Weisbecker 2021-09-21 23:36 ` Frederic Weisbecker 2021-09-22 2:18 ` Paul E. McKenney 2021-09-22 2:18 ` Paul E. McKenney 2021-09-22 11:31 ` Frederic Weisbecker 2021-09-22 11:31 ` Frederic Weisbecker 2021-09-21 23:45 ` Frederic Weisbecker 2021-09-21 23:45 ` Frederic Weisbecker 2021-09-22 6:32 ` Sebastian Andrzej Siewior 2021-09-22 6:32 ` Sebastian Andrzej Siewior 2021-09-22 11:10 ` Frederic Weisbecker 2021-09-22 11:10 ` Frederic Weisbecker 2021-09-22 11:27 ` Sebastian Andrzej Siewior 2021-09-22 11:27 ` Sebastian Andrzej Siewior 2021-09-22 11:38 ` Frederic Weisbecker 2021-09-22 11:38 ` Frederic Weisbecker 2021-09-22 13:02 ` Sebastian Andrzej Siewior 2021-09-22 13:02 ` Sebastian Andrzej Siewior 2021-09-23 10:02 ` Frederic Weisbecker 2021-09-23 10:02 ` Frederic Weisbecker 2021-09-30 9:00 ` Valentin Schneider 2021-09-30 9:00 ` Valentin Schneider 2021-09-30 10:53 ` Frederic Weisbecker [this message] 2021-09-30 10:53 ` Frederic Weisbecker 2021-09-30 13:22 ` Valentin Schneider 2021-09-30 13:22 ` Valentin Schneider 2021-08-11 20:13 ` [PATCH v3 4/4] arm64: mm: Make arch_faults_on_old_pte() check for migratability Valentin Schneider 2021-08-11 20:13 ` Valentin Schneider
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210930105340.GA232241@lothringen \ --to=frederic@kernel.org \ --cc=anshuman.khandual@arm.com \ --cc=ardb@kernel.org \ --cc=bigeasy@linutronix.de \ --cc=boqun.feng@gmail.com \ --cc=bristot@redhat.com \ --cc=catalin.marinas@arm.com \ --cc=dave@stgolabs.net \ --cc=efault@gmx.de \ --cc=jiangshanlai@gmail.com \ --cc=joel@joelfernandes.org \ --cc=josh@joshtriplett.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-rt-users@vger.kernel.org \ --cc=mathieu.desnoyers@efficios.com \ --cc=mingo@kernel.org \ --cc=paulmck@kernel.org \ --cc=peterz@infradead.org \ --cc=rcu@vger.kernel.org \ --cc=rostedt@goodmis.org \ --cc=steven.price@arm.com \ --cc=tglx@linutronix.de \ --cc=valentin.schneider@arm.com \ --cc=vincenzo.frascino@arm.com \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.