From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> To: Ingo Molnar <mingo@kernel.org> Cc: Manfred Spraul <manfred@colorfullife.com>, Peter Zijlstra <peterz@infradead.org>, David Laight <David.Laight@ACULAB.COM>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "netfilter-devel@vger.kernel.org" <netfilter-devel@vger.kernel.org>, "netdev@vger.kernel.org" <netdev@vger.kernel.org>, "oleg@redhat.com" <oleg@redhat.com>, "akpm@linux-foundation.org" <akpm@linux-foundation.org>, "mingo@redhat.com" <mingo@redhat.com>, "dave@stgolabs.net" <dave@stgolabs.net>, "tj@kernel.org" <tj@kernel.org>, "arnd@arndb.de" <arnd@arndb.de>, "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>, "will.deacon@arm.com" <will.deacon@arm.com>, "stern@rowland.harvard.edu" <stern@rowland.harvard.edu>, "parri.andrea@gmail.com" <parri.andrea@gmail.com>, "torvalds@linux-foundation.org" <torvalds@linux-foundation.org> Subject: Re: [PATCH v2 0/9] Remove spin_unlock_wait() Date: Sat, 8 Jul 2017 07:45:50 -0700 [thread overview] Message-ID: <20170708144550.GQ2393@linux.vnet.ibm.com> (raw) In-Reply-To: <20170708123018.qiwmyf3x43i4pgsg@gmail.com> On Sat, Jul 08, 2017 at 02:30:19PM +0200, Ingo Molnar wrote: > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote: > > > On Sat, Jul 08, 2017 at 10:35:43AM +0200, Ingo Molnar wrote: > > > > > > * Manfred Spraul <manfred@colorfullife.com> wrote: > > > > > > > Hi Ingo, > > > > > > > > On 07/07/2017 10:31 AM, Ingo Molnar wrote: > > > > > > > > > > There's another, probably just as significant advantage: queued_spin_unlock_wait() > > > > > is 'read-only', while spin_lock()+spin_unlock() dirties the lock cache line. On > > > > > any bigger system this should make a very measurable difference - if > > > > > spin_unlock_wait() is ever used in a performance critical code path. > > > > At least for ipc/sem: > > > > Dirtying the cacheline (in the slow path) allows to remove a smp_mb() in the > > > > hot path. > > > > So for sem_lock(), I either need a primitive that dirties the cacheline or > > > > sem_lock() must continue to use spin_lock()/spin_unlock(). > > > > > > Technically you could use spin_trylock()+spin_unlock() and avoid the lock acquire > > > spinning on spin_unlock() and get very close to the slow path performance of a > > > pure cacheline-dirtying behavior. > > > > > > But adding something like spin_barrier(), which purely dirties the lock cacheline, > > > would be even faster, right? > > > > Interestingly enough, the arm64 and powerpc implementations of > > spin_unlock_wait() were very close to what it sounds like you are > > describing. > > So could we perhaps solve all our problems by defining the generic version thusly: > > void spin_unlock_wait(spinlock_t *lock) > { > if (spin_trylock(lock)) > spin_unlock(lock); > } > > ... and perhaps rename it to spin_barrier() [or whatever proper name there would > be]? As lockdep, 0day Test Robot, Linus Torvalds, and several others let me know in response to my original (thankfully RFC!) patch series, this needs to disable irqs to work in the general case. For example, if the lock in question is an irq-disabling lock, you take an interrupt just after a successful spin_trylock(), and that interrupt acquires the same lock, the actuarial statistics of your kernel degrade sharply and suddenly. What I get for sending out untested patches! :-/ > Architectures can still optimize it, to remove the small window where the lock is > held locally - as long as the ordering is at least as strong as the generic > version. > > This would have various advantages: > > - semantics are well-defined > > - the generic implementation is already pretty well optimized (no spinning) > > - it would make it usable for the IPC performance optimization > > - architectures could still optimize it to eliminate the window where the lock is > held locally - if there's such instructions available. > > Was this proposed before, or am I missing something? It was sort of proposed... https://marc.info/?l=linux-arch&m=149912878628355&w=2 But do we have a situation where normal usage of spin_lock() and spin_unlock() is causing performance or scalability trouble? (We do have at least one situation in fnic that appears to be buggy use of spin_is_locked(), and proposing a patch for that case in on my todo list.) Thanx, Paul
WARNING: multiple messages have this Message-ID (diff)
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> To: Ingo Molnar <mingo@kernel.org> Cc: Manfred Spraul <manfred@colorfullife.com>, Peter Zijlstra <peterz@infradead.org>, David Laight <David.Laight@ACULAB.COM>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "netfilter-devel@vger.kernel.org" <netfilter-devel@vger.kernel.org>, "netdev@vger.kernel.org" <netdev@vger.kernel.org>, "oleg@redhat.com" <oleg@redhat.com>, "akpm@linux-foundation.org" <akpm@linux-foundation.org>, "mingo@redhat.com" <mingo@redhat.com>, "dave@stgolabs.net" <dave@stgolabs.net>, "tj@kernel.org" <tj@kernel.org>, "arnd@arndb.de" <arnd@arndb.de>, "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>, "will.deacon@arm.com" <will.deacon@arm.com>, "stern@rowland.harvard.edu" <stern@rowland.harvard.edu>, "parri.andrea@gmail.com" <parri.andrea@gmail.com>, "torvalds@linux-fou Subject: Re: [PATCH v2 0/9] Remove spin_unlock_wait() Date: Sat, 8 Jul 2017 07:45:50 -0700 [thread overview] Message-ID: <20170708144550.GQ2393@linux.vnet.ibm.com> (raw) In-Reply-To: <20170708123018.qiwmyf3x43i4pgsg@gmail.com> On Sat, Jul 08, 2017 at 02:30:19PM +0200, Ingo Molnar wrote: > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote: > > > On Sat, Jul 08, 2017 at 10:35:43AM +0200, Ingo Molnar wrote: > > > > > > * Manfred Spraul <manfred@colorfullife.com> wrote: > > > > > > > Hi Ingo, > > > > > > > > On 07/07/2017 10:31 AM, Ingo Molnar wrote: > > > > > > > > > > There's another, probably just as significant advantage: queued_spin_unlock_wait() > > > > > is 'read-only', while spin_lock()+spin_unlock() dirties the lock cache line. On > > > > > any bigger system this should make a very measurable difference - if > > > > > spin_unlock_wait() is ever used in a performance critical code path. > > > > At least for ipc/sem: > > > > Dirtying the cacheline (in the slow path) allows to remove a smp_mb() in the > > > > hot path. > > > > So for sem_lock(), I either need a primitive that dirties the cacheline or > > > > sem_lock() must continue to use spin_lock()/spin_unlock(). > > > > > > Technically you could use spin_trylock()+spin_unlock() and avoid the lock acquire > > > spinning on spin_unlock() and get very close to the slow path performance of a > > > pure cacheline-dirtying behavior. > > > > > > But adding something like spin_barrier(), which purely dirties the lock cacheline, > > > would be even faster, right? > > > > Interestingly enough, the arm64 and powerpc implementations of > > spin_unlock_wait() were very close to what it sounds like you are > > describing. > > So could we perhaps solve all our problems by defining the generic version thusly: > > void spin_unlock_wait(spinlock_t *lock) > { > if (spin_trylock(lock)) > spin_unlock(lock); > } > > ... and perhaps rename it to spin_barrier() [or whatever proper name there would > be]? As lockdep, 0day Test Robot, Linus Torvalds, and several others let me know in response to my original (thankfully RFC!) patch series, this needs to disable irqs to work in the general case. For example, if the lock in question is an irq-disabling lock, you take an interrupt just after a successful spin_trylock(), and that interrupt acquires the same lock, the actuarial statistics of your kernel degrade sharply and suddenly. What I get for sending out untested patches! :-/ > Architectures can still optimize it, to remove the small window where the lock is > held locally - as long as the ordering is at least as strong as the generic > version. > > This would have various advantages: > > - semantics are well-defined > > - the generic implementation is already pretty well optimized (no spinning) > > - it would make it usable for the IPC performance optimization > > - architectures could still optimize it to eliminate the window where the lock is > held locally - if there's such instructions available. > > Was this proposed before, or am I missing something? It was sort of proposed... https://marc.info/?l=linux-arch&m=149912878628355&w=2 But do we have a situation where normal usage of spin_lock() and spin_unlock() is causing performance or scalability trouble? (We do have at least one situation in fnic that appears to be buggy use of spin_is_locked(), and proposing a patch for that case in on my todo list.) Thanx, Paul
next prev parent reply other threads:[~2017-07-08 14:46 UTC|newest] Thread overview: 211+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-06-29 23:59 [PATCH RFC 0/26] Remove spin_unlock_wait() Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 01/26] netfilter: Replace spin_unlock_wait() with lock/unlock pair Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney [not found] ` <a6642feb-2f3a-980f-5ed6-2deb79563e6b@colorfullife.com> 2017-07-02 2:00 ` Paul E. McKenney 2017-07-03 14:39 ` Alan Stern 2017-07-03 14:39 ` Alan Stern 2017-07-03 17:14 ` Paul E. McKenney 2017-07-03 19:01 ` Manfred Spraul 2017-07-03 19:57 ` Alan Stern 2017-07-03 19:57 ` Alan Stern 2017-07-06 18:43 ` Manfred Spraul 2017-07-06 18:43 ` Manfred Spraul 2017-07-06 18:43 ` Manfred Spraul 2017-07-03 20:04 ` Alan Stern 2017-07-03 20:04 ` Alan Stern 2017-07-03 20:53 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 02/26] task_work: " Paul E. McKenney 2017-06-30 11:04 ` Oleg Nesterov 2017-06-30 12:50 ` Paul E. McKenney 2017-06-30 15:20 ` Oleg Nesterov 2017-06-30 16:16 ` Paul E. McKenney 2017-06-30 17:21 ` Paul E. McKenney 2017-06-30 19:21 ` Oleg Nesterov 2017-06-30 19:50 ` Alan Stern 2017-06-30 19:50 ` Alan Stern 2017-06-30 20:04 ` Paul E. McKenney 2017-06-30 20:02 ` Paul E. McKenney 2017-06-30 20:19 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 03/26] sched: " Paul E. McKenney 2017-06-30 10:31 ` Arnd Bergmann 2017-06-30 12:35 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 04/26] completion: " Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 05/26] exit: " Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 06/26] ipc: " Paul E. McKenney 2017-07-01 19:23 ` Manfred Spraul 2017-07-02 3:16 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 07/26] drivers/ata: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 08/26] locking: Remove spin_unlock_wait() generic definitions Paul E. McKenney 2017-06-30 9:19 ` Will Deacon 2017-06-30 12:38 ` Paul E. McKenney 2017-06-30 13:13 ` Will Deacon 2017-06-30 22:18 ` Paul E. McKenney 2017-07-03 13:15 ` Will Deacon 2017-07-03 16:18 ` Paul E. McKenney 2017-07-03 16:40 ` Linus Torvalds 2017-07-03 17:13 ` Will Deacon 2017-07-03 22:30 ` Paul E. McKenney 2017-07-03 22:49 ` Linus Torvalds 2017-07-04 0:39 ` Paul E. McKenney 2017-07-04 0:54 ` Paul E. McKenney 2017-07-03 21:10 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 09/26] alpha: Remove spin_unlock_wait() arch-specific definitions Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 10/26] arc: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 11/26] arm: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 12/26] arm64: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 9:20 ` Will Deacon 2017-06-30 9:20 ` Will Deacon 2017-06-30 17:29 ` Paul E. McKenney 2017-06-30 17:29 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 13/26] blackfin: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 14/26] hexagon: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 15/26] ia64: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 16/26] m32r: " Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 17/26] metag: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 18/26] mips: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 19/26] mn10300: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 20/26] parisc: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 21/26] powerpc: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-07-02 3:58 ` Boqun Feng 2017-07-05 23:57 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 22/26] s390: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 23/26] sh: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 24/26] sparc: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 25/26] tile: " Paul E. McKenney 2017-06-30 0:06 ` Linus Torvalds 2017-06-30 0:09 ` Paul E. McKenney 2017-06-30 0:14 ` Paul E. McKenney 2017-06-30 0:10 ` Linus Torvalds 2017-06-30 0:24 ` Paul E. McKenney 2017-06-30 0:01 ` [PATCH RFC 26/26] xtensa: " Paul E. McKenney 2017-06-30 0:01 ` Paul E. McKenney 2017-07-05 23:29 ` [PATCH v2 0/9] Remove spin_unlock_wait() Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 1/9] net/netfilter/nf_conntrack_core: Fix net_conntrack_lock() Paul E. McKenney 2017-07-05 23:31 ` Paul E. McKenney 2017-07-06 18:45 ` Manfred Spraul 2017-07-06 18:45 ` Manfred Spraul 2017-07-06 18:45 ` Manfred Spraul 2017-07-06 20:26 ` Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 2/9] task_work: Replace spin_unlock_wait() with lock/unlock pair Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 3/9] sched: " Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 4/9] completion: " Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 5/9] exit: " Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 6/9] ipc: " Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 7/9] drivers/ata: " Paul E. McKenney 2017-07-05 23:31 ` Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 8/9] locking: Remove spin_unlock_wait() generic definitions Paul E. McKenney 2017-07-05 23:31 ` [PATCH v2 9/9] arch: Remove spin_unlock_wait() arch-specific definitions Paul E. McKenney 2017-07-06 14:12 ` [PATCH v2 0/9] Remove spin_unlock_wait() David Laight 2017-07-06 15:21 ` Paul E. McKenney 2017-07-06 15:21 ` Paul E. McKenney 2017-07-06 15:21 ` Paul E. McKenney 2017-07-06 16:10 ` Peter Zijlstra 2017-07-06 16:10 ` Peter Zijlstra 2017-07-06 16:10 ` Peter Zijlstra 2017-07-06 16:24 ` Paul E. McKenney 2017-07-06 16:24 ` Paul E. McKenney 2017-07-06 16:24 ` Paul E. McKenney 2017-07-06 16:41 ` Peter Zijlstra 2017-07-06 16:41 ` Peter Zijlstra 2017-07-06 16:41 ` Peter Zijlstra 2017-07-06 17:03 ` Paul E. McKenney 2017-07-06 17:03 ` Paul E. McKenney 2017-07-06 17:03 ` Paul E. McKenney 2017-07-06 16:49 ` Alan Stern 2017-07-06 16:54 ` Peter Zijlstra 2017-07-06 16:54 ` Peter Zijlstra 2017-07-06 19:37 ` Alan Stern 2017-07-06 19:37 ` Alan Stern 2017-07-06 16:05 ` Peter Zijlstra 2017-07-06 16:05 ` Peter Zijlstra 2017-07-06 16:05 ` Peter Zijlstra 2017-07-06 16:20 ` Paul E. McKenney 2017-07-06 16:20 ` Paul E. McKenney 2017-07-06 16:20 ` Paul E. McKenney 2017-07-06 16:50 ` Peter Zijlstra 2017-07-06 16:50 ` Peter Zijlstra 2017-07-06 16:50 ` Peter Zijlstra 2017-07-06 17:08 ` Will Deacon 2017-07-06 17:08 ` Will Deacon 2017-07-06 17:08 ` Will Deacon 2017-07-06 17:29 ` Paul E. McKenney 2017-07-06 17:29 ` Paul E. McKenney 2017-07-06 17:29 ` Paul E. McKenney 2017-07-06 17:18 ` Paul E. McKenney 2017-07-06 17:18 ` Paul E. McKenney 2017-07-06 17:18 ` Paul E. McKenney 2017-07-07 8:31 ` Ingo Molnar 2017-07-07 8:31 ` Ingo Molnar 2017-07-07 8:31 ` Ingo Molnar 2017-07-07 8:44 ` Peter Zijlstra 2017-07-07 8:44 ` Peter Zijlstra 2017-07-07 8:44 ` Peter Zijlstra 2017-07-07 10:33 ` Ingo Molnar 2017-07-07 10:33 ` Ingo Molnar 2017-07-07 10:33 ` Ingo Molnar 2017-07-07 11:23 ` Peter Zijlstra 2017-07-07 11:23 ` Peter Zijlstra 2017-07-07 11:23 ` Peter Zijlstra 2017-07-07 14:41 ` Paul E. McKenney 2017-07-07 14:41 ` Paul E. McKenney 2017-07-08 8:43 ` Ingo Molnar 2017-07-08 8:43 ` Ingo Molnar 2017-07-08 11:41 ` Paul E. McKenney 2017-07-08 11:41 ` Paul E. McKenney 2017-07-07 17:47 ` Manfred Spraul 2017-07-07 17:47 ` Manfred Spraul 2017-07-07 17:47 ` Manfred Spraul 2017-07-08 8:35 ` Ingo Molnar 2017-07-08 8:35 ` Ingo Molnar 2017-07-08 11:39 ` Paul E. McKenney 2017-07-08 11:39 ` Paul E. McKenney 2017-07-08 12:30 ` Ingo Molnar 2017-07-08 12:30 ` Ingo Molnar 2017-07-08 14:45 ` Paul E. McKenney [this message] 2017-07-08 14:45 ` Paul E. McKenney 2017-07-08 16:21 ` Alan Stern 2017-07-08 16:21 ` Alan Stern 2017-07-10 17:22 ` Manfred Spraul 2017-07-07 8:06 ` Ingo Molnar 2017-07-07 8:06 ` Ingo Molnar 2017-07-07 8:06 ` Ingo Molnar 2017-07-07 9:32 ` Ingo Molnar 2017-07-07 9:32 ` Ingo Molnar 2017-07-07 9:32 ` Ingo Molnar 2017-07-07 19:27 ` [PATCH v3 " Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 1/9] net/netfilter/nf_conntrack_core: Fix net_conntrack_lock() Paul E. McKenney 2017-07-07 19:28 ` Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 2/9] task_work: Replace spin_unlock_wait() with lock/unlock pair Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 3/9] sched: " Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 4/9] completion: " Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 5/9] exit: " Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 6/9] ipc: " Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 7/9] drivers/ata: " Paul E. McKenney 2017-07-07 19:28 ` Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 8/9] locking: Remove spin_unlock_wait() generic definitions Paul E. McKenney 2017-07-07 19:28 ` [PATCH v3 9/9] arch: Remove spin_unlock_wait() arch-specific definitions Paul E. McKenney
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20170708144550.GQ2393@linux.vnet.ibm.com \ --to=paulmck@linux.vnet.ibm.com \ --cc=David.Laight@ACULAB.COM \ --cc=akpm@linux-foundation.org \ --cc=arnd@arndb.de \ --cc=dave@stgolabs.net \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=manfred@colorfullife.com \ --cc=mingo@kernel.org \ --cc=mingo@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=netfilter-devel@vger.kernel.org \ --cc=oleg@redhat.com \ --cc=parri.andrea@gmail.com \ --cc=peterz@infradead.org \ --cc=stern@rowland.harvard.edu \ --cc=tj@kernel.org \ --cc=torvalds@linux-foundation.org \ --cc=will.deacon@arm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.