From: Laurent Dufour <ldufour@linux.ibm.com> To: Peter Zijlstra <peterz@infradead.org> Cc: akpm@linux-foundation.org, mhocko@kernel.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox <willy@infradead.org>, aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, hpa@zytor.com, Will Deacon <will.deacon@arm.com>, Sergey Senozhatsky <sergey.senozhatsky@gmail.com>, sergey.senozhatsky.work@gmail.com, Andrea Arcangeli <aarcange@redhat.com>, Alexei Starovoitov <alexei.starovoitov@gmail.com>, kemi.wang@intel.com, Daniel Jordan <daniel.m.jordan@oracle.com>, David Rientjes <rientjes@google.com>, Jerome Glisse <jglisse@redhat.com>, Ganesh Mahendran <opensource.ganesh@gmail.com>, Minchan Kim <minchan@kernel.org>, Punit Agrawal <punitagrawal@gmail.com>, vinayak menon <vinayakm.list@gmail.com>, Yang Shi <yang.shi@linux.alibaba.com>, zhong jiang <zhongjiang@huawei.com>, Haiyan Song <haiyanx.song@intel.com>, Balbir Singh <bsingharora@gmail.com>, sj38.park@gmail.com, Michel Lespinasse <walken@google.com>, Mike Rapoport <rppt@linux.ibm.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen <tim.c.chen@linux.intel.com>, linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: Re: [PATCH v12 21/31] mm: Introduce find_vma_rcu() Date: Wed, 24 Apr 2019 09:57:05 +0200 [thread overview] Message-ID: <ea141f88-a0d3-df97-5141-706fa18b7ad3@linux.ibm.com> (raw) In-Reply-To: <20190423092710.GI11158@hirez.programming.kicks-ass.net> Le 23/04/2019 à 11:27, Peter Zijlstra a écrit : > On Tue, Apr 16, 2019 at 03:45:12PM +0200, Laurent Dufour wrote: >> This allows to search for a VMA structure without holding the mmap_sem. >> >> The search is repeated while the mm seqlock is changing and until we found >> a valid VMA. >> >> While under the RCU protection, a reference is taken on the VMA, so the >> caller must call put_vma() once it not more need the VMA structure. >> >> At the time a VMA is inserted in the MM RB tree, in vma_rb_insert(), a >> reference is taken to the VMA by calling get_vma(). >> >> When removing a VMA from the MM RB tree, the VMA is not release immediately >> but at the end of the RCU grace period through vm_rcu_put(). This ensures >> that the VMA remains allocated until the end the RCU grace period. >> >> Since the vm_file pointer, if valid, is released in put_vma(), there is no >> guarantee that the file pointer will be valid on the returned VMA. > > What I'm missing here, and in the previous patch introducing the > refcount (also see refcount_t), is _why_ we need the refcount thing at > all. The need for the VMA's refcount is to ensure that the VMA will remain until the end of the SPF handler. This is a consequence of the use of RCU instead of SRCU to protect the RB tree. I was not aware of the refcount_t type, it would be better here to avoid wrapping. > My original plan was to use SRCU, which at the time was not complete > enough so I abused/hacked preemptible RCU, but that is no longer the > case, SRCU has all the required bits and pieces. When I did test using SRCU it was involving a lot a scheduling to run the SRCU callback mechanism. In some workload the impact on the perfomance was significant [1]. I can't see this overhead using RCU. > > Also; the initial motivation was prefaulting large VMAs and the > contention on mmap was killing things; but similarly, the contention on > the refcount (I did try that) killed things just the same. Doing prefaulting should be doable, I'll try to think further about that. Regarding the refcount, I should I missed something, this is an atomic counter, so there should not be contention on it but cache exclusivity, not ideal I agree but I can't see what else to use here. > So I'm really sad to see the refcount return; and without any apparent > justification. I'm not opposed to use another mechanism here, but SRCU didn't show good performance with some workload, and I can't see how to use RCU without a reference counter here. So please, advise. Thanks, Laurent. [1] https://lore.kernel.org/linux-mm/7ca80231-fe02-a3a7-84bc-ce81690ea051@intel.com/
WARNING: multiple messages have this Message-ID (diff)
From: Laurent Dufour <ldufour@linux.ibm.com> To: Peter Zijlstra <peterz@infradead.org> Cc: jack@suse.cz, sergey.senozhatsky.work@gmail.com, Will Deacon <will.deacon@arm.com>, mhocko@kernel.org, linux-mm@kvack.org, paulus@samba.org, Punit Agrawal <punitagrawal@gmail.com>, hpa@zytor.com, Michel Lespinasse <walken@google.com>, Alexei Starovoitov <alexei.starovoitov@gmail.com>, Andrea Arcangeli <aarcange@redhat.com>, ak@linux.intel.com, Minchan Kim <minchan@kernel.org>, aneesh.kumar@linux.ibm.com, x86@kernel.org, Matthew Wilcox <willy@infradead.org>, Daniel Jordan <daniel.m.jordan@oracle.com>, Ingo Molnar <mingo@redhat.com>, David Rientjes <rientjes@google.com>, paulmck@linux.vnet.ibm.com, Haiyan Song <haiyanx.song@intel.com>, npiggin@gmail.com, sj38.park@gmail.com, Jerome Glisse <jglisse@redhat.com>, dave@stgolabs.net, kemi.wang@intel.com, kirill@shutemov.name, Thomas Gleixner <tglx@linutronix.de>, zhong jiang <zhongjiang@huawei.com>, Ganesh Mahendran <opensource.ganesh@gmail.com>, Yang Shi <yang.shi@linux.alibaba.com>, Mike Rapoport <rppt@linux.ibm.com>, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky <sergey.senozhatsky@gmail.com>, vinayak menon <vinayakm.list@gmail.com>, akpm@linux-foundation.org, Tim Chen <tim.c.chen@linux.intel.com>, haren@linux.vnet.ibm.com Subject: Re: [PATCH v12 21/31] mm: Introduce find_vma_rcu() Date: Wed, 24 Apr 2019 09:57:05 +0200 [thread overview] Message-ID: <ea141f88-a0d3-df97-5141-706fa18b7ad3@linux.ibm.com> (raw) In-Reply-To: <20190423092710.GI11158@hirez.programming.kicks-ass.net> Le 23/04/2019 à 11:27, Peter Zijlstra a écrit : > On Tue, Apr 16, 2019 at 03:45:12PM +0200, Laurent Dufour wrote: >> This allows to search for a VMA structure without holding the mmap_sem. >> >> The search is repeated while the mm seqlock is changing and until we found >> a valid VMA. >> >> While under the RCU protection, a reference is taken on the VMA, so the >> caller must call put_vma() once it not more need the VMA structure. >> >> At the time a VMA is inserted in the MM RB tree, in vma_rb_insert(), a >> reference is taken to the VMA by calling get_vma(). >> >> When removing a VMA from the MM RB tree, the VMA is not release immediately >> but at the end of the RCU grace period through vm_rcu_put(). This ensures >> that the VMA remains allocated until the end the RCU grace period. >> >> Since the vm_file pointer, if valid, is released in put_vma(), there is no >> guarantee that the file pointer will be valid on the returned VMA. > > What I'm missing here, and in the previous patch introducing the > refcount (also see refcount_t), is _why_ we need the refcount thing at > all. The need for the VMA's refcount is to ensure that the VMA will remain until the end of the SPF handler. This is a consequence of the use of RCU instead of SRCU to protect the RB tree. I was not aware of the refcount_t type, it would be better here to avoid wrapping. > My original plan was to use SRCU, which at the time was not complete > enough so I abused/hacked preemptible RCU, but that is no longer the > case, SRCU has all the required bits and pieces. When I did test using SRCU it was involving a lot a scheduling to run the SRCU callback mechanism. In some workload the impact on the perfomance was significant [1]. I can't see this overhead using RCU. > > Also; the initial motivation was prefaulting large VMAs and the > contention on mmap was killing things; but similarly, the contention on > the refcount (I did try that) killed things just the same. Doing prefaulting should be doable, I'll try to think further about that. Regarding the refcount, I should I missed something, this is an atomic counter, so there should not be contention on it but cache exclusivity, not ideal I agree but I can't see what else to use here. > So I'm really sad to see the refcount return; and without any apparent > justification. I'm not opposed to use another mechanism here, but SRCU didn't show good performance with some workload, and I can't see how to use RCU without a reference counter here. So please, advise. Thanks, Laurent. [1] https://lore.kernel.org/linux-mm/7ca80231-fe02-a3a7-84bc-ce81690ea051@intel.com/
next prev parent reply other threads:[~2019-04-24 7:57 UTC|newest] Thread overview: 197+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-04-16 13:44 [PATCH v12 00/31] Speculative page faults Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-16 13:44 ` [PATCH v12 01/31] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-18 21:47 ` Jerome Glisse 2019-04-18 21:47 ` Jerome Glisse 2019-04-23 15:21 ` Laurent Dufour 2019-04-23 15:21 ` Laurent Dufour 2019-04-16 13:44 ` [PATCH v12 02/31] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-18 21:48 ` Jerome Glisse 2019-04-18 21:48 ` Jerome Glisse 2019-04-16 13:44 ` [PATCH v12 03/31] powerpc/mm: set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-18 21:49 ` Jerome Glisse 2019-04-18 21:49 ` Jerome Glisse 2019-04-16 13:44 ` [PATCH v12 04/31] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-16 14:27 ` Mark Rutland 2019-04-16 14:27 ` Mark Rutland 2019-04-16 14:31 ` Laurent Dufour 2019-04-16 14:31 ` Laurent Dufour 2019-04-16 14:41 ` Mark Rutland 2019-04-16 14:41 ` Mark Rutland 2019-04-18 21:51 ` Jerome Glisse 2019-04-18 21:51 ` Jerome Glisse 2019-04-23 15:36 ` Laurent Dufour 2019-04-23 15:36 ` Laurent Dufour 2019-04-23 16:19 ` Mark Rutland 2019-04-23 16:19 ` Mark Rutland 2019-04-24 10:34 ` Laurent Dufour 2019-04-24 10:34 ` Laurent Dufour 2019-04-16 13:44 ` [PATCH v12 05/31] mm: prepare for FAULT_FLAG_SPECULATIVE Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-18 22:04 ` Jerome Glisse 2019-04-18 22:04 ` Jerome Glisse 2019-04-23 15:45 ` Laurent Dufour 2019-04-23 15:45 ` Laurent Dufour 2019-04-16 13:44 ` [PATCH v12 06/31] mm: introduce pte_spinlock " Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-18 22:05 ` Jerome Glisse 2019-04-18 22:05 ` Jerome Glisse 2019-04-16 13:44 ` [PATCH v12 07/31] mm: make pte_unmap_same compatible with SPF Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-18 22:10 ` Jerome Glisse 2019-04-18 22:10 ` Jerome Glisse 2019-04-23 15:43 ` Matthew Wilcox 2019-04-23 15:43 ` Matthew Wilcox 2019-04-23 15:47 ` Laurent Dufour 2019-04-23 15:47 ` Laurent Dufour 2019-04-16 13:44 ` [PATCH v12 08/31] mm: introduce INIT_VMA() Laurent Dufour 2019-04-16 13:44 ` Laurent Dufour 2019-04-18 22:22 ` Jerome Glisse 2019-04-18 22:22 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 09/31] mm: VMA sequence count Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-18 22:48 ` Jerome Glisse 2019-04-18 22:48 ` Jerome Glisse 2019-04-19 15:45 ` Laurent Dufour 2019-04-19 15:45 ` Laurent Dufour 2019-04-22 15:51 ` Jerome Glisse 2019-04-22 15:51 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 10/31] mm: protect VMA modifications using " Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 19:43 ` Jerome Glisse 2019-04-22 19:43 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 11/31] mm: protect mremap() against SPF hanlder Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 19:51 ` Jerome Glisse 2019-04-22 19:51 ` Jerome Glisse 2019-04-23 15:51 ` Laurent Dufour 2019-04-23 15:51 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 12/31] mm: protect SPF handler against anon_vma changes Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 19:53 ` Jerome Glisse 2019-04-22 19:53 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 13/31] mm: cache some VMA fields in the vm_fault structure Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:06 ` Jerome Glisse 2019-04-22 20:06 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 14/31] mm/migrate: Pass vm_fault pointer to migrate_misplaced_page() Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:09 ` Jerome Glisse 2019-04-22 20:09 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 15/31] mm: introduce __lru_cache_add_active_or_unevictable Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:11 ` Jerome Glisse 2019-04-22 20:11 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 16/31] mm: introduce __vm_normal_page() Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:15 ` Jerome Glisse 2019-04-22 20:15 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 17/31] mm: introduce __page_add_new_anon_rmap() Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:18 ` Jerome Glisse 2019-04-22 20:18 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 18/31] mm: protect against PTE changes done by dup_mmap() Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:32 ` Jerome Glisse 2019-04-22 20:32 ` Jerome Glisse 2019-04-24 10:33 ` Laurent Dufour 2019-04-24 10:33 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 19/31] mm: protect the RB tree with a sequence lock Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:33 ` Jerome Glisse 2019-04-22 20:33 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 20/31] mm: introduce vma reference counter Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:36 ` Jerome Glisse 2019-04-22 20:36 ` Jerome Glisse 2019-04-24 14:26 ` Laurent Dufour 2019-04-24 14:26 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 21/31] mm: Introduce find_vma_rcu() Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 20:57 ` Jerome Glisse 2019-04-22 20:57 ` Jerome Glisse 2019-04-24 14:39 ` Laurent Dufour 2019-04-24 14:39 ` Laurent Dufour 2019-04-23 9:27 ` Peter Zijlstra 2019-04-23 9:27 ` Peter Zijlstra 2019-04-23 18:13 ` Davidlohr Bueso 2019-04-23 18:13 ` Davidlohr Bueso 2019-04-24 7:57 ` Laurent Dufour [this message] 2019-04-24 7:57 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 22/31] mm: provide speculative fault infrastructure Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 21:26 ` Jerome Glisse 2019-04-22 21:26 ` Jerome Glisse 2019-04-24 14:56 ` Laurent Dufour 2019-04-24 14:56 ` Laurent Dufour 2019-04-24 15:13 ` Jerome Glisse 2019-04-24 15:13 ` Jerome Glisse 2019-04-16 13:45 ` [PATCH v12 23/31] mm: don't do swap readahead during speculative page fault Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 21:36 ` Jerome Glisse 2019-04-22 21:36 ` Jerome Glisse 2019-04-24 14:57 ` Laurent Dufour 2019-04-24 14:57 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 24/31] mm: adding speculative page fault failure trace events Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 25/31] perf: add a speculative page fault sw event Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 26/31] perf tools: add support for the SPF perf event Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 27/31] mm: add speculative page fault vmstats Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 28/31] x86/mm: add speculative pagefault handling Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 29/31] powerpc/mm: add speculative page fault Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 30/31] arm64/mm: " Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-16 13:45 ` [PATCH v12 31/31] mm: Add a speculative page fault switch in sysctl Laurent Dufour 2019-04-16 13:45 ` Laurent Dufour 2019-04-22 21:29 ` [PATCH v12 00/31] Speculative page faults Michel Lespinasse 2019-04-22 21:29 ` Michel Lespinasse 2019-04-22 21:29 ` Michel Lespinasse 2019-04-23 9:38 ` Peter Zijlstra 2019-04-23 9:38 ` Peter Zijlstra 2019-04-24 7:33 ` Laurent Dufour 2019-04-24 7:33 ` Laurent Dufour 2019-04-27 1:53 ` Michel Lespinasse 2019-04-27 1:53 ` Michel Lespinasse 2019-04-23 10:47 ` Michal Hocko 2019-04-23 10:47 ` Michal Hocko 2019-04-23 12:41 ` Matthew Wilcox 2019-04-23 12:41 ` Matthew Wilcox 2019-04-23 12:48 ` Peter Zijlstra 2019-04-23 12:48 ` Peter Zijlstra 2019-04-23 13:42 ` Michal Hocko 2019-04-23 13:42 ` Michal Hocko 2019-04-24 18:01 ` Laurent Dufour 2019-04-24 18:01 ` Laurent Dufour 2019-04-27 6:00 ` Michel Lespinasse 2019-04-27 6:00 ` Michel Lespinasse 2019-04-23 11:35 ` Anshuman Khandual 2019-04-23 11:35 ` Anshuman Khandual 2019-06-06 6:51 ` Haiyan Song 2019-06-06 6:51 ` Haiyan Song 2019-06-14 8:37 ` Laurent Dufour 2019-06-14 8:37 ` Laurent Dufour 2019-06-14 8:44 ` Laurent Dufour 2019-06-14 8:44 ` Laurent Dufour 2019-06-20 8:19 ` Haiyan Song 2019-06-20 8:19 ` Haiyan Song 2020-07-06 9:25 ` Chinwen Chang 2020-07-06 9:25 ` Chinwen Chang 2020-07-06 12:27 ` Laurent Dufour 2020-07-06 12:27 ` Laurent Dufour 2020-07-07 5:31 ` Chinwen Chang 2020-07-07 5:31 ` Chinwen Chang 2020-12-14 2:03 ` Joel Fernandes 2020-12-14 2:03 ` Joel Fernandes 2020-12-14 9:36 ` Laurent Dufour 2020-12-14 9:36 ` Laurent Dufour 2020-12-14 18:10 ` Joel Fernandes 2020-12-14 18:10 ` Joel Fernandes
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=ea141f88-a0d3-df97-5141-706fa18b7ad3@linux.ibm.com \ --to=ldufour@linux.ibm.com \ --cc=aarcange@redhat.com \ --cc=ak@linux.intel.com \ --cc=akpm@linux-foundation.org \ --cc=alexei.starovoitov@gmail.com \ --cc=aneesh.kumar@linux.ibm.com \ --cc=benh@kernel.crashing.org \ --cc=bsingharora@gmail.com \ --cc=daniel.m.jordan@oracle.com \ --cc=dave@stgolabs.net \ --cc=haiyanx.song@intel.com \ --cc=haren@linux.vnet.ibm.com \ --cc=hpa@zytor.com \ --cc=jack@suse.cz \ --cc=jglisse@redhat.com \ --cc=kemi.wang@intel.com \ --cc=kirill@shutemov.name \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=mhocko@kernel.org \ --cc=minchan@kernel.org \ --cc=mingo@redhat.com \ --cc=mpe@ellerman.id.au \ --cc=npiggin@gmail.com \ --cc=opensource.ganesh@gmail.com \ --cc=paulmck@linux.vnet.ibm.com \ --cc=paulus@samba.org \ --cc=peterz@infradead.org \ --cc=punitagrawal@gmail.com \ --cc=rientjes@google.com \ --cc=rppt@linux.ibm.com \ --cc=sergey.senozhatsky.work@gmail.com \ --cc=sergey.senozhatsky@gmail.com \ --cc=sj38.park@gmail.com \ --cc=tglx@linutronix.de \ --cc=tim.c.chen@linux.intel.com \ --cc=vinayakm.list@gmail.com \ --cc=walken@google.com \ --cc=will.deacon@arm.com \ --cc=willy@infradead.org \ --cc=x86@kernel.org \ --cc=yang.shi@linux.alibaba.com \ --cc=zhongjiang@huawei.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.