All of lore.kernel.org
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: Tejun Heo <tj@kernel.org>
Cc: Petr Mladek <pmladek@suse.com>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	akpm@linux-foundation.org, linux-mm@kvack.org,
	Cong Wang <xiyou.wangcong@gmail.com>,
	Dave Hansen <dave.hansen@intel.com>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Mel Gorman <mgorman@suse.de>, Michal Hocko <mhocko@kernel.org>,
	Vlastimil Babka <vbabka@suse.cz>,
	Peter Zijlstra <peterz@infradead.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Jan Kara <jack@suse.cz>,
	Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
	rostedt@home.goodmis.org, Byungchul Park <byungchul.park@lge.com>,
	Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	Pavel Machek <pavel@ucw.cz>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v5 0/2] printk: Console owner and waiter logic cleanup
Date: Wed, 10 Jan 2018 13:41:57 -0500	[thread overview]
Message-ID: <20180110134157.1c3ce4b9@vmware.local.home> (raw)
In-Reply-To: <20180110181252.GK3668920@devbig577.frc2.facebook.com>

On Wed, 10 Jan 2018 10:12:52 -0800
Tejun Heo <tj@kernel.org> wrote:

> Hello, Steven.
> 
> So, everything else on your message, sure.  You do what you have to
> do, but I really don't understand the following part, and this has
> been the main source of frustration in the whole discussion.
> 
> On Wed, Jan 10, 2018 at 01:05:17PM -0500, Steven Rostedt wrote:
> > You on the other hand are showing unrealistic scenarios, and crying
> > that it's what you see in production, with no proof of it.  
> 
> I've explained the same scenario multiple times.  Unless you're
> assuming that I'm lying, it should be amply clear that the scenario is
> unrealistic - we've been seeing them taking place repeatedly for quite
> a while.

The one scenario you did show was the recursive OOM messages, and as
Peter Zijlstra pointed out that's more of a bug in the net console than
a printk bug.

> 
> What I don't understand is why we can't address this seemingly obvious
> problem.  If there are technical reasons and the consensus is to not
> solve this within flushing logic, sure, we can deal with it otherwise,
> but we at least have to be able to agree that there are actual issues
> here, no?

The issue with the solution you want to do with printk is that it can
break existing printk usages. As Petr said, people want printk to do two
things. 1 - print out data ASAP, 2 - not lock up the system. The two
are fighting each other. You care more about 2 where I (and others,
like Peter Zijlstra and Linus) care more about 1.

My solution can help with 2 without doing anything to hurt 1.

You are NACKing my solution because it doesn't solve this bug with net
console. I believe net console should be fixed. You believe that printk
should have a work around to not let net console type bugs occur. Which
to me is papering over the real bugs.

-- Steve

WARNING: multiple messages have this Message-ID (diff)
From: Steven Rostedt <rostedt@goodmis.org>
To: Tejun Heo <tj@kernel.org>
Cc: Petr Mladek <pmladek@suse.com>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	akpm@linux-foundation.org, linux-mm@kvack.org,
	Cong Wang <xiyou.wangcong@gmail.com>,
	Dave Hansen <dave.hansen@intel.com>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Mel Gorman <mgorman@suse.de>, Michal Hocko <mhocko@kernel.org>,
	Vlastimil Babka <vbabka@suse.cz>,
	Peter Zijlstra <peterz@infradead.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Jan Kara <jack@suse.cz>,
	Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
	rostedt@home.goodmis.org, Byungchul Park <byungchul.park@lge.com>,
	Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	Pavel Machek <pavel@ucw.cz>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v5 0/2] printk: Console owner and waiter logic cleanup
Date: Wed, 10 Jan 2018 13:41:57 -0500	[thread overview]
Message-ID: <20180110134157.1c3ce4b9@vmware.local.home> (raw)
In-Reply-To: <20180110181252.GK3668920@devbig577.frc2.facebook.com>

On Wed, 10 Jan 2018 10:12:52 -0800
Tejun Heo <tj@kernel.org> wrote:

> Hello, Steven.
> 
> So, everything else on your message, sure.  You do what you have to
> do, but I really don't understand the following part, and this has
> been the main source of frustration in the whole discussion.
> 
> On Wed, Jan 10, 2018 at 01:05:17PM -0500, Steven Rostedt wrote:
> > You on the other hand are showing unrealistic scenarios, and crying
> > that it's what you see in production, with no proof of it.  
> 
> I've explained the same scenario multiple times.  Unless you're
> assuming that I'm lying, it should be amply clear that the scenario is
> unrealistic - we've been seeing them taking place repeatedly for quite
> a while.

The one scenario you did show was the recursive OOM messages, and as
Peter Zijlstra pointed out that's more of a bug in the net console than
a printk bug.

> 
> What I don't understand is why we can't address this seemingly obvious
> problem.  If there are technical reasons and the consensus is to not
> solve this within flushing logic, sure, we can deal with it otherwise,
> but we at least have to be able to agree that there are actual issues
> here, no?

The issue with the solution you want to do with printk is that it can
break existing printk usages. As Petr said, people want printk to do two
things. 1 - print out data ASAP, 2 - not lock up the system. The two
are fighting each other. You care more about 2 where I (and others,
like Peter Zijlstra and Linus) care more about 1.

My solution can help with 2 without doing anything to hurt 1.

You are NACKing my solution because it doesn't solve this bug with net
console. I believe net console should be fixed. You believe that printk
should have a work around to not let net console type bugs occur. Which
to me is papering over the real bugs.

-- Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

  parent reply	other threads:[~2018-01-10 18:42 UTC|newest]

Thread overview: 280+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-01-10 13:24 [PATCH v5 0/2] printk: Console owner and waiter logic cleanup Petr Mladek
2018-01-10 13:24 ` Petr Mladek
2018-01-10 13:24 ` [PATCH v5 1/2] printk: Add console owner and waiter logic to load balance console writes Petr Mladek
2018-01-10 13:24   ` Petr Mladek
2018-01-10 16:50   ` Steven Rostedt
2018-01-10 16:50     ` Steven Rostedt
2018-01-12 16:54   ` Steven Rostedt
2018-01-12 16:54     ` Steven Rostedt
2018-01-12 17:11     ` Steven Rostedt
2018-01-12 17:11       ` Steven Rostedt
2018-01-17 19:13       ` Rasmus Villemoes
2018-01-17 19:13         ` Rasmus Villemoes
2018-01-17 19:33         ` Steven Rostedt
2018-01-17 19:33           ` Steven Rostedt
2018-01-19  9:51         ` Sergey Senozhatsky
2018-01-19  9:51           ` Sergey Senozhatsky
2018-01-18 22:03     ` Pavel Machek
2018-01-18 22:03       ` Pavel Machek
2018-01-19  0:20       ` Steven Rostedt
2018-01-19  0:20         ` Steven Rostedt
2018-01-17  2:19   ` Byungchul Park
2018-01-17  2:19     ` Byungchul Park
2018-01-17  4:54     ` Byungchul Park
2018-01-17  4:54       ` Byungchul Park
2018-01-17  7:34     ` Byungchul Park
2018-01-17  7:34       ` Byungchul Park
2018-01-17 12:04     ` Petr Mladek
2018-01-17 12:04       ` Petr Mladek
2018-01-18  1:53       ` Byungchul Park
2018-01-18  1:53         ` Byungchul Park
2018-01-18  1:57         ` Byungchul Park
2018-01-18  1:57           ` Byungchul Park
2018-01-18  2:19         ` Steven Rostedt
2018-01-18  2:19           ` Steven Rostedt
2018-01-18  4:01           ` Byungchul Park
2018-01-18  4:01             ` Byungchul Park
2018-01-18 15:21             ` Steven Rostedt
2018-01-18 15:21               ` Steven Rostedt
2018-01-19  2:37               ` Byungchul Park
2018-01-19  2:37                 ` Byungchul Park
2018-01-19  3:27                 ` Steven Rostedt
2018-01-19  3:27                   ` Steven Rostedt
2018-01-22  2:31                   ` Byungchul Park
2018-01-22  2:31                     ` Byungchul Park
2018-01-10 13:24 ` [PATCH v5 2/2] printk: Hide console waiter logic into helpers Petr Mladek
2018-01-10 13:24   ` Petr Mladek
2018-01-10 17:52   ` Steven Rostedt
2018-01-10 17:52     ` Steven Rostedt
2018-01-11 12:03     ` Petr Mladek
2018-01-11 12:03       ` Petr Mladek
2018-01-12 15:37       ` Steven Rostedt
2018-01-12 15:37         ` Steven Rostedt
2018-01-12 16:08         ` Petr Mladek
2018-01-12 16:08           ` Petr Mladek
2018-01-12 16:36           ` Steven Rostedt
2018-01-12 16:36             ` Steven Rostedt
2018-01-15 16:08             ` Petr Mladek
2018-01-15 16:08               ` Petr Mladek
2018-01-16  5:05               ` Sergey Senozhatsky
2018-01-16  5:05                 ` Sergey Senozhatsky
2018-01-10 14:05 ` [PATCH v5 0/2] printk: Console owner and waiter logic cleanup Tejun Heo
2018-01-10 14:05   ` Tejun Heo
2018-01-10 16:29   ` Petr Mladek
2018-01-10 16:29     ` Petr Mladek
2018-01-10 17:02     ` Tejun Heo
2018-01-10 17:02       ` Tejun Heo
2018-01-10 18:21       ` Peter Zijlstra
2018-01-10 18:21         ` Peter Zijlstra
2018-01-10 18:30         ` Tejun Heo
2018-01-10 18:30           ` Tejun Heo
2018-01-10 18:41           ` Peter Zijlstra
2018-01-10 18:41             ` Peter Zijlstra
2018-01-10 19:05             ` Tejun Heo
2018-01-10 19:05               ` Tejun Heo
2018-01-11  5:15         ` Sergey Senozhatsky
2018-01-11  5:15           ` Sergey Senozhatsky
2018-01-10 18:22       ` Steven Rostedt
2018-01-10 18:22         ` Steven Rostedt
2018-01-10 18:36         ` Tejun Heo
2018-01-10 18:36           ` Tejun Heo
2018-01-10 18:40       ` Mathieu Desnoyers
2018-01-10 18:40         ` Mathieu Desnoyers
2018-01-11  7:36         ` Sergey Senozhatsky
2018-01-11  7:36           ` Sergey Senozhatsky
2018-01-11 11:24           ` Petr Mladek
2018-01-11 11:24             ` Petr Mladek
2018-01-11 13:19             ` Sergey Senozhatsky
2018-01-11 13:19               ` Sergey Senozhatsky
2018-01-24  9:36       ` Peter Zijlstra
2018-01-24  9:36         ` Peter Zijlstra
2018-01-24 18:46         ` Tejun Heo
2018-01-24 18:46           ` Tejun Heo
2018-05-09  8:58       ` Sergey Senozhatsky
2018-05-09  8:58         ` Sergey Senozhatsky
2018-01-10 18:54     ` Steven Rostedt
2018-01-10 18:54       ` Steven Rostedt
2018-01-11  5:10     ` Sergey Senozhatsky
2018-01-11  5:10       ` Sergey Senozhatsky
2018-01-10 18:05   ` Steven Rostedt
2018-01-10 18:05     ` Steven Rostedt
2018-01-10 18:12     ` Tejun Heo
2018-01-10 18:12       ` Tejun Heo
2018-01-10 18:14       ` Tejun Heo
2018-01-10 18:14         ` Tejun Heo
2018-01-10 18:45         ` Steven Rostedt
2018-01-10 18:45           ` Steven Rostedt
2018-01-10 18:41       ` Steven Rostedt [this message]
2018-01-10 18:41         ` Steven Rostedt
2018-01-10 18:57         ` Tejun Heo
2018-01-10 18:57           ` Tejun Heo
2018-01-10 19:17           ` Steven Rostedt
2018-01-10 19:17             ` Steven Rostedt
2018-01-10 19:34             ` Tejun Heo
2018-01-10 19:34               ` Tejun Heo
2018-01-10 19:44               ` Steven Rostedt
2018-01-10 19:44                 ` Steven Rostedt
2018-01-10 22:44                 ` Tejun Heo
2018-01-10 22:44                   ` Tejun Heo
2018-01-11  5:35             ` Sergey Senozhatsky
2018-01-11  5:35               ` Sergey Senozhatsky
2018-01-11  4:58     ` Sergey Senozhatsky
2018-01-11  4:58       ` Sergey Senozhatsky
2018-01-11  9:34       ` Petr Mladek
2018-01-11  9:34         ` Petr Mladek
2018-01-11 10:38         ` Sergey Senozhatsky
2018-01-11 10:38           ` Sergey Senozhatsky
2018-01-11 11:50           ` Petr Mladek
2018-01-11 11:50             ` Petr Mladek
2018-01-11 16:29           ` Steven Rostedt
2018-01-11 16:29             ` Steven Rostedt
2018-01-12  1:30             ` Steven Rostedt
2018-01-12  1:30               ` Steven Rostedt
2018-01-12  2:55               ` Steven Rostedt
2018-01-12  2:55                 ` Steven Rostedt
2018-01-12  4:20                 ` Steven Rostedt
2018-01-12  4:20                   ` Steven Rostedt
2018-01-16 19:44                 ` Tejun Heo
2018-01-16 19:44                   ` Tejun Heo
2018-01-17  9:12                   ` Petr Mladek
2018-01-17  9:12                     ` Petr Mladek
2018-01-17 15:15                     ` Tejun Heo
2018-01-17 15:15                       ` Tejun Heo
2018-01-17 17:12                       ` Steven Rostedt
2018-01-17 17:12                         ` Steven Rostedt
2018-01-17 18:42                         ` Steven Rostedt
2018-01-17 18:42                           ` Steven Rostedt
2018-01-19 18:20                           ` Steven Rostedt
2018-01-19 18:20                             ` Steven Rostedt
2018-01-20  7:14                             ` Sergey Senozhatsky
2018-01-20  7:14                               ` Sergey Senozhatsky
2018-01-20 15:49                               ` Steven Rostedt
2018-01-20 15:49                                 ` Steven Rostedt
2018-01-21 14:15                                 ` Sergey Senozhatsky
2018-01-21 14:15                                   ` Sergey Senozhatsky
2018-01-21 21:04                                   ` Steven Rostedt
2018-01-21 21:04                                     ` Steven Rostedt
2018-01-22  8:56                                     ` Sergey Senozhatsky
2018-01-22  8:56                                       ` Sergey Senozhatsky
2018-01-22 10:28                                       ` Sergey Senozhatsky
2018-01-22 10:28                                         ` Sergey Senozhatsky
2018-01-22 10:36                                         ` Sergey Senozhatsky
2018-01-22 10:36                                           ` Sergey Senozhatsky
2018-01-23  6:40                                   ` Sergey Senozhatsky
2018-01-23  6:40                                     ` Sergey Senozhatsky
2018-01-23  7:05                                     ` Sergey Senozhatsky
2018-01-23  7:05                                       ` Sergey Senozhatsky
2018-01-23  7:31                                     ` Sergey Senozhatsky
2018-01-23  7:31                                       ` Sergey Senozhatsky
2018-01-23 14:56                                     ` Steven Rostedt
2018-01-23 14:56                                       ` Steven Rostedt
2018-01-23 15:21                                       ` Sergey Senozhatsky
2018-01-23 15:21                                         ` Sergey Senozhatsky
2018-01-23 15:41                                         ` Steven Rostedt
2018-01-23 15:41                                           ` Steven Rostedt
2018-01-23 15:43                                           ` Tejun Heo
2018-01-23 15:43                                             ` Tejun Heo
2018-01-23 16:12                                             ` Sergey Senozhatsky
2018-01-23 16:12                                               ` Sergey Senozhatsky
2018-01-23 16:13                                             ` Steven Rostedt
2018-01-23 16:13                                               ` Steven Rostedt
2018-01-23 17:21                                               ` Tejun Heo
2018-01-23 17:21                                                 ` Tejun Heo
2018-04-23  5:35                                             ` Sergey Senozhatsky
2018-04-23  5:35                                               ` Sergey Senozhatsky
2018-01-23 16:01                                           ` Sergey Senozhatsky
2018-01-23 16:01                                             ` Sergey Senozhatsky
2018-01-23 16:24                                             ` Steven Rostedt
2018-01-23 16:24                                               ` Steven Rostedt
2018-01-24  2:11                                               ` Sergey Senozhatsky
2018-01-24  2:11                                                 ` Sergey Senozhatsky
2018-01-24  2:52                                                 ` Steven Rostedt
2018-01-24  2:52                                                   ` Steven Rostedt
2018-01-24  4:44                                                   ` Sergey Senozhatsky
2018-01-24  4:44                                                     ` Sergey Senozhatsky
2018-01-23 17:22                                             ` Tejun Heo
2018-01-23 17:22                                               ` Tejun Heo
2018-01-20 12:19                             ` Tejun Heo
2018-01-20 12:19                               ` Tejun Heo
2018-01-20 14:51                               ` Steven Rostedt
2018-01-20 14:51                                 ` Steven Rostedt
2018-01-17 20:05                         ` Tejun Heo
2018-01-17 20:05                           ` Tejun Heo
2018-01-18  5:43                           ` Sergey Senozhatsky
2018-01-18  5:43                             ` Sergey Senozhatsky
2018-01-18 11:51                           ` Petr Mladek
2018-01-18 11:51                             ` Petr Mladek
2018-01-18  5:42                         ` Sergey Senozhatsky
2018-01-18  5:42                           ` Sergey Senozhatsky
2018-01-12  3:12               ` Sergey Senozhatsky
2018-01-12  3:12                 ` Sergey Senozhatsky
2018-01-12  2:56             ` Sergey Senozhatsky
2018-01-12  2:56               ` Sergey Senozhatsky
2018-01-12  3:21               ` Steven Rostedt
2018-01-12  3:21                 ` Steven Rostedt
2018-01-12 10:05                 ` Sergey Senozhatsky
2018-01-12 10:05                   ` Sergey Senozhatsky
2018-01-12 12:21                   ` Steven Rostedt
2018-01-12 12:21                     ` Steven Rostedt
2018-01-12 12:55                     ` Petr Mladek
2018-01-12 12:55                       ` Petr Mladek
2018-01-13  7:31                       ` Sergey Senozhatsky
2018-01-13  7:31                         ` Sergey Senozhatsky
2018-01-15  8:51                         ` Petr Mladek
2018-01-15  8:51                           ` Petr Mladek
2018-01-15  9:48                           ` Sergey Senozhatsky
2018-01-15  9:48                             ` Sergey Senozhatsky
2018-01-16  5:16                           ` Sergey Senozhatsky
2018-01-16  5:16                             ` Sergey Senozhatsky
2018-01-16  9:08                             ` Petr Mladek
2018-01-16  9:08                               ` Petr Mladek
2018-01-15 12:08                       ` Steven Rostedt
2018-01-15 12:08                         ` Steven Rostedt
2018-01-16  4:51                         ` Sergey Senozhatsky
2018-01-16  4:51                           ` Sergey Senozhatsky
2018-01-13  7:28                     ` Sergey Senozhatsky
2018-01-13  7:28                       ` Sergey Senozhatsky
2018-01-15 10:17                       ` Petr Mladek
2018-01-15 10:17                         ` Petr Mladek
2018-01-15 11:50                         ` Petr Mladek
2018-01-15 11:50                           ` Petr Mladek
2018-01-16  6:10                           ` Sergey Senozhatsky
2018-01-16  6:10                             ` Sergey Senozhatsky
2018-01-16  9:36                             ` Petr Mladek
2018-01-16  9:36                               ` Petr Mladek
2018-01-16 10:10                               ` Sergey Senozhatsky
2018-01-16 10:10                                 ` Sergey Senozhatsky
2018-01-16 16:06                             ` Steven Rostedt
2018-01-16 16:06                               ` Steven Rostedt
2018-01-16  5:23                         ` Sergey Senozhatsky
2018-01-16  5:23                           ` Sergey Senozhatsky
2018-01-15 12:06                       ` Steven Rostedt
2018-01-15 12:06                         ` Steven Rostedt
2018-01-15 14:45                         ` Petr Mladek
2018-01-15 14:45                           ` Petr Mladek
2018-01-16  2:23                           ` Sergey Senozhatsky
2018-01-16  2:23                             ` Sergey Senozhatsky
2018-01-16  4:47                             ` Sergey Senozhatsky
2018-01-16  4:47                               ` Sergey Senozhatsky
2018-01-16 10:19                               ` Petr Mladek
2018-01-16 10:19                                 ` Petr Mladek
2018-01-17  2:24                                 ` Sergey Senozhatsky
2018-01-17  2:24                                   ` Sergey Senozhatsky
2018-01-16 15:45                               ` Steven Rostedt
2018-01-16 15:45                                 ` Steven Rostedt
2018-01-17  2:18                                 ` Sergey Senozhatsky
2018-01-17  2:18                                   ` Sergey Senozhatsky
2018-01-17 13:04                                   ` Petr Mladek
2018-01-17 13:04                                     ` Petr Mladek
2018-01-17 15:24                                     ` Steven Rostedt
2018-01-17 15:24                                       ` Steven Rostedt
2018-01-18  4:31                                     ` Sergey Senozhatsky
2018-01-18  4:31                                       ` Sergey Senozhatsky
2018-01-18 15:22                                       ` Steven Rostedt
2018-01-18 15:22                                         ` Steven Rostedt
2018-01-16 10:13                             ` Petr Mladek
2018-01-16 10:13                               ` Petr Mladek
2018-01-17  6:29                               ` Sergey Senozhatsky
2018-01-17  6:29                                 ` Sergey Senozhatsky
2018-01-16  1:46                         ` Sergey Senozhatsky
2018-01-16  1:46                           ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180110134157.1c3ce4b9@vmware.local.home \
    --to=rostedt@goodmis.org \
    --cc=akpm@linux-foundation.org \
    --cc=byungchul.park@lge.com \
    --cc=dave.hansen@intel.com \
    --cc=hannes@cmpxchg.org \
    --cc=jack@suse.cz \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mgorman@suse.de \
    --cc=mhocko@kernel.org \
    --cc=pavel@ucw.cz \
    --cc=penguin-kernel@I-love.SAKURA.ne.jp \
    --cc=peterz@infradead.org \
    --cc=pmladek@suse.com \
    --cc=rostedt@home.goodmis.org \
    --cc=sergey.senozhatsky.work@gmail.com \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=tj@kernel.org \
    --cc=torvalds@linux-foundation.org \
    --cc=vbabka@suse.cz \
    --cc=xiyou.wangcong@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.