From: "Paul E. McKenney" <paulmck@kernel.org> To: Thomas Gleixner <tglx@linutronix.de> Cc: LKML <linux-kernel@vger.kernel.org>, Peter Zijlstra <peterz@infradead.org>, Linus Torvalds <torvalds@linux-foundation.org>, Ingo Molnar <mingo@kernel.org>, Will Deacon <will@kernel.org>, Joel Fernandes <joel@joelfernandes.org>, Steven Rostedt <rostedt@goodmis.org>, Randy Dunlap <rdunlap@infradead.org>, Sebastian Andrzej Siewior <bigeasy@linutronix.de>, Logan Gunthorpe <logang@deltatee.com>, Kurt Schwemmer <kurt.schwemmer@microsemi.com>, Bjorn Helgaas <bhelgaas@google.com>, linux-pci@vger.kernel.org, Felipe Balbi <balbi@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, linux-usb@vger.kernel.org, Kalle Valo <kvalo@codeaurora.org>, "David S. Miller" <davem@davemloft.net>, linux-wireless@vger.kernel.org, netdev@vger.kernel.org, Oleg Nesterov <oleg@redhat.com>, Davidlohr Bueso <dave@stgolabs.net>, Michael Ellerman <mpe@ellerman.id.au>, Arnd Bergmann <arnd@arndb.de>, linuxppc-dev@lists.ozlabs.org Subject: Re: [patch V2 08/15] Documentation: Add lock ordering and nesting documentation Date: Fri, 20 Mar 2020 14:02:43 -0700 [thread overview] Message-ID: <20200320210243.GT3199@paulmck-ThinkPad-P72> (raw) In-Reply-To: <87mu8apzxr.fsf@nanos.tec.linutronix.de> On Fri, Mar 20, 2020 at 08:51:44PM +0100, Thomas Gleixner wrote: > "Paul E. McKenney" <paulmck@kernel.org> writes: > > > > - The soft interrupt related suffix (_bh()) still disables softirq > > handlers. However, unlike non-PREEMPT_RT kernels (which disable > > preemption to get this effect), PREEMPT_RT kernels use a per-CPU > > lock to exclude softirq handlers. > > I've made that: > > - The soft interrupt related suffix (_bh()) still disables softirq > handlers. > > Non-PREEMPT_RT kernels disable preemption to get this effect. > > PREEMPT_RT kernels use a per-CPU lock for serialization. The lock > disables softirq handlers and prevents reentrancy by a preempting > task. That works! At the end, I would instead say "prevents reentrancy due to task preemption", but what you have works. > On non-RT this is implicit through preemption disable, but it's non > obvious for RT as preemption stays enabled. > > > PREEMPT_RT kernels preserve all other spinlock_t semantics: > > > > - Tasks holding a spinlock_t do not migrate. Non-PREEMPT_RT kernels > > avoid migration by disabling preemption. PREEMPT_RT kernels instead > > disable migration, which ensures that pointers to per-CPU variables > > remain valid even if the task is preempted. > > > > - Task state is preserved across spinlock acquisition, ensuring that the > > task-state rules apply to all kernel configurations. In non-PREEMPT_RT > > kernels leave task state untouched. However, PREEMPT_RT must change > > task state if the task blocks during acquisition. Therefore, the > > corresponding lock wakeup restores the task state. Note that regular > > (not lock related) wakeups do not restore task state. > > - Task state is preserved across spinlock acquisition, ensuring that the > task-state rules apply to all kernel configurations. Non-PREEMPT_RT > kernels leave task state untouched. However, PREEMPT_RT must change > task state if the task blocks during acquisition. Therefore, it > saves the current task state before blocking and the corresponding > lock wakeup restores it. A regular not lock related wakeup sets the > task state to RUNNING. If this happens while the task is blocked on > a spinlock then the saved task state is changed so that correct > state is restored on lock wakeup. > > Hmm? I of course cannot resist editing the last two sentences: ... Other types of wakeups unconditionally set task state to RUNNING. If this happens while a task is blocked while acquiring a spinlock, then the task state is restored to its pre-acquisition value at lock-wakeup time. > > But this code failes on PREEMPT_RT kernels because the memory allocator > > is fully preemptible and therefore cannot be invoked from truly atomic > > contexts. However, it is perfectly fine to invoke the memory allocator > > while holding a normal non-raw spinlocks because they do not disable > > preemption:: > > > >> + spin_lock(&lock); > >> + p = kmalloc(sizeof(*p), GFP_ATOMIC); > >> + > >> +Most places which use GFP_ATOMIC allocations are safe on PREEMPT_RT as the > >> +execution is forced into thread context and the lock substitution is > >> +ensuring preemptibility. > > > > Interestingly enough, most uses of GFP_ATOMIC allocations are > > actually safe on PREEMPT_RT because the the lock substitution ensures > > preemptibility. Only those GFP_ATOMIC allocations that are invoke > > while holding a raw spinlock or with preemption otherwise disabled need > > adjustment to work correctly on PREEMPT_RT. > > > > [ I am not as confident of the above as I would like to be... ] > > I'd leave that whole paragraph out. This documents the rules and from > the above code examples it's pretty clear what works and what not :) Works for me! ;-) > > And meeting time, will continue later! > > Enjoy! Not bad, actually, as meetings go. Thanx, Paul
WARNING: multiple messages have this Message-ID (diff)
From: "Paul E. McKenney" <paulmck@kernel.org> To: Thomas Gleixner <tglx@linutronix.de> Cc: Randy Dunlap <rdunlap@infradead.org>, Peter Zijlstra <peterz@infradead.org>, linux-pci@vger.kernel.org, Sebastian Andrzej Siewior <bigeasy@linutronix.de>, Oleg Nesterov <oleg@redhat.com>, Joel Fernandes <joel@joelfernandes.org>, Will Deacon <will@kernel.org>, Ingo Molnar <mingo@kernel.org>, Davidlohr Bueso <dave@stgolabs.net>, Logan Gunthorpe <logang@deltatee.com>, Arnd Bergmann <arnd@arndb.de>, linuxppc-dev@lists.ozlabs.org, Steven Rostedt <rostedt@goodmis.org>, Bjorn Helgaas <bhelgaas@google.com>, Kurt Schwemmer <kurt.schwemmer@microsemi.com>, Kalle Valo <kvalo@codeaurora.org>, Felipe Balbi <balbi@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, linux-usb@vger.kernel.org, linux-wireless@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>, netdev@vger.kernel.org, Linus Torvalds <torvalds@linux-foundation.org>, "David S. Miller" <davem@davemloft.net> Subject: Re: [patch V2 08/15] Documentation: Add lock ordering and nesting documentation Date: Fri, 20 Mar 2020 14:02:43 -0700 [thread overview] Message-ID: <20200320210243.GT3199@paulmck-ThinkPad-P72> (raw) In-Reply-To: <87mu8apzxr.fsf@nanos.tec.linutronix.de> On Fri, Mar 20, 2020 at 08:51:44PM +0100, Thomas Gleixner wrote: > "Paul E. McKenney" <paulmck@kernel.org> writes: > > > > - The soft interrupt related suffix (_bh()) still disables softirq > > handlers. However, unlike non-PREEMPT_RT kernels (which disable > > preemption to get this effect), PREEMPT_RT kernels use a per-CPU > > lock to exclude softirq handlers. > > I've made that: > > - The soft interrupt related suffix (_bh()) still disables softirq > handlers. > > Non-PREEMPT_RT kernels disable preemption to get this effect. > > PREEMPT_RT kernels use a per-CPU lock for serialization. The lock > disables softirq handlers and prevents reentrancy by a preempting > task. That works! At the end, I would instead say "prevents reentrancy due to task preemption", but what you have works. > On non-RT this is implicit through preemption disable, but it's non > obvious for RT as preemption stays enabled. > > > PREEMPT_RT kernels preserve all other spinlock_t semantics: > > > > - Tasks holding a spinlock_t do not migrate. Non-PREEMPT_RT kernels > > avoid migration by disabling preemption. PREEMPT_RT kernels instead > > disable migration, which ensures that pointers to per-CPU variables > > remain valid even if the task is preempted. > > > > - Task state is preserved across spinlock acquisition, ensuring that the > > task-state rules apply to all kernel configurations. In non-PREEMPT_RT > > kernels leave task state untouched. However, PREEMPT_RT must change > > task state if the task blocks during acquisition. Therefore, the > > corresponding lock wakeup restores the task state. Note that regular > > (not lock related) wakeups do not restore task state. > > - Task state is preserved across spinlock acquisition, ensuring that the > task-state rules apply to all kernel configurations. Non-PREEMPT_RT > kernels leave task state untouched. However, PREEMPT_RT must change > task state if the task blocks during acquisition. Therefore, it > saves the current task state before blocking and the corresponding > lock wakeup restores it. A regular not lock related wakeup sets the > task state to RUNNING. If this happens while the task is blocked on > a spinlock then the saved task state is changed so that correct > state is restored on lock wakeup. > > Hmm? I of course cannot resist editing the last two sentences: ... Other types of wakeups unconditionally set task state to RUNNING. If this happens while a task is blocked while acquiring a spinlock, then the task state is restored to its pre-acquisition value at lock-wakeup time. > > But this code failes on PREEMPT_RT kernels because the memory allocator > > is fully preemptible and therefore cannot be invoked from truly atomic > > contexts. However, it is perfectly fine to invoke the memory allocator > > while holding a normal non-raw spinlocks because they do not disable > > preemption:: > > > >> + spin_lock(&lock); > >> + p = kmalloc(sizeof(*p), GFP_ATOMIC); > >> + > >> +Most places which use GFP_ATOMIC allocations are safe on PREEMPT_RT as the > >> +execution is forced into thread context and the lock substitution is > >> +ensuring preemptibility. > > > > Interestingly enough, most uses of GFP_ATOMIC allocations are > > actually safe on PREEMPT_RT because the the lock substitution ensures > > preemptibility. Only those GFP_ATOMIC allocations that are invoke > > while holding a raw spinlock or with preemption otherwise disabled need > > adjustment to work correctly on PREEMPT_RT. > > > > [ I am not as confident of the above as I would like to be... ] > > I'd leave that whole paragraph out. This documents the rules and from > the above code examples it's pretty clear what works and what not :) Works for me! ;-) > > And meeting time, will continue later! > > Enjoy! Not bad, actually, as meetings go. Thanx, Paul
next prev parent reply other threads:[~2020-03-20 21:02 UTC|newest] Thread overview: 148+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-03-18 20:43 [patch V2 00/15] Lock ordering documentation and annotation for lockdep Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 20:43 ` [patch V2 01/15] PCI/switchtec: Fix init_completion race condition with poll_wait() Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 21:25 ` Bjorn Helgaas 2020-03-18 21:25 ` Bjorn Helgaas 2020-03-18 20:43 ` [patch V2 02/15] pci/switchtec: Replace completion wait queue usage for poll Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 21:26 ` Bjorn Helgaas 2020-03-18 21:26 ` Bjorn Helgaas 2020-03-18 22:11 ` Logan Gunthorpe 2020-03-18 22:11 ` Logan Gunthorpe 2020-03-18 20:43 ` [patch V2 03/15] usb: gadget: Use completion interface instead of open coding it Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-19 8:41 ` Greg Kroah-Hartman 2020-03-19 8:41 ` Greg Kroah-Hartman 2020-03-18 20:43 ` [patch V2 04/15] orinoco_usb: Use the regular completion interfaces Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-19 8:40 ` Greg Kroah-Hartman 2020-03-19 8:40 ` Greg Kroah-Hartman 2020-03-18 20:43 ` [patch V2 05/15] acpi: Remove header dependency Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 20:43 ` [patch V2 06/15] rcuwait: Add @state argument to rcuwait_wait_event() Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-20 5:36 ` Davidlohr Bueso 2020-03-20 5:36 ` Davidlohr Bueso 2020-03-20 8:45 ` Sebastian Andrzej Siewior 2020-03-20 8:45 ` Sebastian Andrzej Siewior 2020-03-20 8:58 ` Davidlohr Bueso 2020-03-20 8:58 ` Davidlohr Bueso 2020-03-20 9:48 ` [PATCH 0/5] Remove mm.h from arch/*/include/asm/uaccess.h Sebastian Andrzej Siewior 2020-03-20 9:48 ` Sebastian Andrzej Siewior 2020-03-20 9:48 ` [PATCH 1/5] nds32: Remove mm.h from asm/uaccess.h Sebastian Andrzej Siewior 2020-03-20 9:48 ` Sebastian Andrzej Siewior 2020-03-20 9:48 ` [PATCH 2/5] csky: " Sebastian Andrzej Siewior 2020-03-20 9:48 ` Sebastian Andrzej Siewior 2020-03-21 11:24 ` Guo Ren 2020-03-21 11:24 ` Guo Ren 2020-03-21 12:08 ` Thomas Gleixner 2020-03-21 12:08 ` Thomas Gleixner 2020-03-21 14:11 ` Guo Ren 2020-03-21 14:11 ` Guo Ren 2020-03-20 9:48 ` [PATCH 3/5] hexagon: " Sebastian Andrzej Siewior 2020-03-20 9:48 ` Sebastian Andrzej Siewior 2020-03-20 9:48 ` [PATCH 4/5] ia64: " Sebastian Andrzej Siewior 2020-03-20 9:48 ` Sebastian Andrzej Siewior 2020-03-20 9:48 ` Sebastian Andrzej Siewior 2020-03-20 9:48 ` [PATCH 5/5] microblaze: " Sebastian Andrzej Siewior 2020-03-20 9:48 ` Sebastian Andrzej Siewior 2020-03-18 20:43 ` [patch V2 07/15] powerpc/ps3: Convert half completion to rcuwait Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-19 9:00 ` Sebastian Andrzej Siewior 2020-03-19 9:00 ` Sebastian Andrzej Siewior 2020-03-19 9:18 ` Peter Zijlstra 2020-03-19 9:18 ` Peter Zijlstra 2020-03-19 9:21 ` Davidlohr Bueso 2020-03-19 9:21 ` Davidlohr Bueso 2020-03-19 10:04 ` Christoph Hellwig 2020-03-19 10:04 ` Christoph Hellwig 2020-03-19 10:26 ` Sebastian Andrzej Siewior 2020-03-19 10:26 ` Sebastian Andrzej Siewior 2020-03-20 0:01 ` Geoff Levand 2020-03-20 0:01 ` Geoff Levand 2020-03-20 0:45 ` Michael Ellerman 2020-03-20 0:45 ` Michael Ellerman 2020-03-21 10:41 ` Thomas Gleixner 2020-03-21 10:41 ` Thomas Gleixner 2020-03-18 20:43 ` [patch V2 08/15] Documentation: Add lock ordering and nesting documentation Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 22:31 ` Paul E. McKenney 2020-03-18 22:31 ` Paul E. McKenney 2020-03-19 18:02 ` Thomas Gleixner 2020-03-19 18:02 ` Thomas Gleixner 2020-03-20 16:01 ` Paul E. McKenney 2020-03-20 16:01 ` Paul E. McKenney 2020-03-20 19:51 ` Thomas Gleixner 2020-03-20 19:51 ` Thomas Gleixner 2020-03-20 21:02 ` Paul E. McKenney [this message] 2020-03-20 21:02 ` Paul E. McKenney 2020-03-20 22:36 ` Thomas Gleixner 2020-03-20 22:36 ` Thomas Gleixner 2020-03-21 2:29 ` Paul E. McKenney 2020-03-21 2:29 ` Paul E. McKenney 2020-03-21 10:26 ` Thomas Gleixner 2020-03-21 10:26 ` Thomas Gleixner 2020-03-21 17:23 ` Paul E. McKenney 2020-03-21 17:23 ` Paul E. McKenney 2020-03-19 8:51 ` Davidlohr Bueso 2020-03-19 8:51 ` Davidlohr Bueso 2020-03-19 15:04 ` Jonathan Corbet 2020-03-19 15:04 ` Jonathan Corbet 2020-03-19 18:04 ` Thomas Gleixner 2020-03-19 18:04 ` Thomas Gleixner 2020-03-21 21:21 ` Joel Fernandes 2020-03-21 21:21 ` Joel Fernandes 2020-03-21 21:49 ` Thomas Gleixner 2020-03-21 21:49 ` Thomas Gleixner 2020-03-22 1:36 ` Joel Fernandes 2020-03-22 1:36 ` Joel Fernandes 2020-03-18 20:43 ` [patch V2 09/15] timekeeping: Split jiffies seqlock Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 20:43 ` [patch V2 10/15] sched/swait: Prepare usage in completions Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 20:43 ` [patch V2 11/15] completion: Use simple wait queues Thomas Gleixner 2020-03-18 20:43 ` Thomas Gleixner 2020-03-18 22:28 ` Logan Gunthorpe 2020-03-18 22:28 ` Logan Gunthorpe 2020-03-19 0:33 ` Joel Fernandes 2020-03-19 0:33 ` Joel Fernandes 2020-03-19 0:44 ` Thomas Gleixner 2020-03-19 0:44 ` Thomas Gleixner 2020-03-19 8:42 ` Greg Kroah-Hartman 2020-03-19 8:42 ` Greg Kroah-Hartman 2020-03-19 17:12 ` Linus Torvalds 2020-03-19 17:12 ` Linus Torvalds 2020-03-19 23:25 ` Julian Calaby 2020-03-19 23:25 ` Julian Calaby 2020-03-20 6:59 ` Christoph Hellwig 2020-03-20 6:59 ` Christoph Hellwig 2020-03-20 9:01 ` Davidlohr Bueso 2020-03-20 9:01 ` Davidlohr Bueso 2020-03-18 20:43 ` [patch V2 13/15] lockdep: Add hrtimer context tracing bits Thomas Gleixner 2020-03-18 20:43 ` [patch V2 14/15] lockdep: Annotate irq_work Thomas Gleixner 2020-03-18 20:43 ` [patch V2 15/15] lockdep: Add posixtimer context tracing bits Thomas Gleixner 2020-03-20 8:50 ` [patch V2 00/15] Lock ordering documentation and annotation for lockdep Davidlohr Bueso 2020-03-20 8:50 ` Davidlohr Bueso 2020-03-20 8:55 ` [PATCH 16/15] rcuwait: Get rid of stale name comment Davidlohr Bueso 2020-03-20 8:55 ` Davidlohr Bueso 2020-03-20 8:55 ` [PATCH 17/15] rcuwait: Inform rcuwait_wake_up() users if a wakeup was attempted Davidlohr Bueso 2020-03-20 8:55 ` Davidlohr Bueso 2020-03-20 9:13 ` Sebastian Andrzej Siewior 2020-03-20 9:13 ` Sebastian Andrzej Siewior 2020-03-20 10:44 ` Peter Zijlstra 2020-03-20 10:44 ` Peter Zijlstra 2020-03-20 8:55 ` [PATCH 18/15] kvm: Replace vcpu->swait with rcuwait Davidlohr Bueso 2020-03-20 8:55 ` Davidlohr Bueso 2020-03-20 11:20 ` Paolo Bonzini 2020-03-20 11:20 ` Paolo Bonzini 2020-03-20 12:54 ` Peter Zijlstra 2020-03-20 12:54 ` Peter Zijlstra 2020-03-22 16:33 ` Davidlohr Bueso 2020-03-22 16:33 ` Davidlohr Bueso 2020-03-22 22:32 ` Peter Zijlstra 2020-03-22 22:32 ` Peter Zijlstra 2020-03-20 8:55 ` [PATCH 19/15] sched/swait: Reword some of the main description Davidlohr Bueso 2020-03-20 8:55 ` Davidlohr Bueso 2020-03-20 9:19 ` Sebastian Andrzej Siewior 2020-03-20 9:19 ` Sebastian Andrzej Siewior
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200320210243.GT3199@paulmck-ThinkPad-P72 \ --to=paulmck@kernel.org \ --cc=arnd@arndb.de \ --cc=balbi@kernel.org \ --cc=bhelgaas@google.com \ --cc=bigeasy@linutronix.de \ --cc=dave@stgolabs.net \ --cc=davem@davemloft.net \ --cc=gregkh@linuxfoundation.org \ --cc=joel@joelfernandes.org \ --cc=kurt.schwemmer@microsemi.com \ --cc=kvalo@codeaurora.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-pci@vger.kernel.org \ --cc=linux-usb@vger.kernel.org \ --cc=linux-wireless@vger.kernel.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=logang@deltatee.com \ --cc=mingo@kernel.org \ --cc=mpe@ellerman.id.au \ --cc=netdev@vger.kernel.org \ --cc=oleg@redhat.com \ --cc=peterz@infradead.org \ --cc=rdunlap@infradead.org \ --cc=rostedt@goodmis.org \ --cc=tglx@linutronix.de \ --cc=torvalds@linux-foundation.org \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.