From: David Howells <dhowells@redhat.com>
To: Trond Myklebust <trondmy@hammerspace.com>,
Anna Schumaker <anna.schumaker@netapp.com>,
Steve French <sfrench@samba.org>,
Dominique Martinet <asmadeus@codewreck.org>
Cc: dhowells@redhat.com, Jeff Layton <jlayton@redhat.com>,
Matthew Wilcox <willy@infradead.org>,
Alexander Viro <viro@zeniv.linux.org.uk>,
linux-cachefs@redhat.com, linux-afs@lists.infradead.org,
linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org,
ceph-devel@vger.kernel.org, v9fs-developer@lists.sourceforge.net,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [RFC PATCH 68/76] netfs: Add write_begin helper
Date: Fri, 20 Nov 2020 15:17:19 +0000 [thread overview]
Message-ID: <160588543960.3465195.2792938973035886168.stgit@warthog.procyon.org.uk> (raw)
In-Reply-To: <160588455242.3465195.3214733858273019178.stgit@warthog.procyon.org.uk>
Add a helper to do the pre-reading work for the netfs write_begin address
space op.
Signed-off-by: David Howells <dhowells@redhat.com>
---
fs/afs/file.c | 8 ++
fs/netfs/internal.h | 2 +
fs/netfs/read_helper.c | 168 ++++++++++++++++++++++++++++++++++++++++++
fs/netfs/stats.c | 10 ++-
include/linux/netfs.h | 8 ++
include/trace/events/netfs.h | 4 +
6 files changed, 196 insertions(+), 4 deletions(-)
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 8772f5e1cbd3..84a222587ca3 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -353,6 +353,13 @@ static void afs_init_rreq(struct netfs_read_request *rreq, struct file *file)
rreq->cookie_debug_id = afs_vnode_cache(vnode)->debug_id;
}
+static bool afs_is_cache_enabled(struct inode *inode)
+{
+ struct fscache_cookie *cookie = afs_vnode_cache(AFS_FS_I(inode));
+
+ return fscache_cookie_enabled(cookie) && !hlist_empty(&cookie->backing_objects);
+}
+
static int afs_begin_cache_operation(struct netfs_read_request *rreq)
{
struct afs_vnode *vnode = AFS_FS_I(rreq->inode);
@@ -368,6 +375,7 @@ static void afs_priv_cleanup(struct address_space *mapping, void *netfs_priv)
static const struct netfs_read_request_ops afs_req_ops = {
.init_rreq = afs_init_rreq,
+ .is_cache_enabled = afs_is_cache_enabled,
.begin_cache_operation = afs_begin_cache_operation,
.issue_op = afs_req_issue_op,
.cleanup = afs_priv_cleanup,
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index d83317b1eb9d..3d7ab3ab5743 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -34,8 +34,10 @@ extern atomic_t netfs_n_rh_read_failed;
extern atomic_t netfs_n_rh_zero;
extern atomic_t netfs_n_rh_short_read;
extern atomic_t netfs_n_rh_write;
+extern atomic_t netfs_n_rh_write_begin;
extern atomic_t netfs_n_rh_write_done;
extern atomic_t netfs_n_rh_write_failed;
+extern atomic_t netfs_n_rh_write_zskip;
static inline void netfs_stat(atomic_t *stat)
diff --git a/fs/netfs/read_helper.c b/fs/netfs/read_helper.c
index e1ce197d8d18..3e59d7d2f77e 100644
--- a/fs/netfs/read_helper.c
+++ b/fs/netfs/read_helper.c
@@ -955,3 +955,171 @@ int netfs_readpage(struct file *file,
return ret;
}
EXPORT_SYMBOL(netfs_readpage);
+
+static void netfs_clear_thp(struct page *page)
+{
+ unsigned int i;
+
+ for (i = 0; i < thp_nr_pages(page); i++)
+ clear_highpage(page + i);
+}
+
+/**
+ * netfs_write_begin - Helper to prepare for writing
+ * @file: The file to read from
+ * @mapping: The mapping to read from
+ * @pos: File position at which the write will begin
+ * @len: The length of the write in this page
+ * @flags: AOP_* flags
+ * @_page: Where to put the resultant page
+ * @_fsdata: Place for the netfs to store a cookie
+ * @ops: The network filesystem's operations for the helper to use
+ * @netfs_priv: Private netfs data to be retained in the request
+ *
+ * Pre-read data for a write-begin request by drawing data from the cache if
+ * possible, or the netfs if not. Space beyond the EOF is zero-filled.
+ * Multiple I/O requests from different sources will get munged together. If
+ * necessary, the readahead window can be expanded in either direction to a
+ * more convenient alighment for RPC efficiency or to make storage in the cache
+ * feasible.
+ *
+ * The calling netfs must provide a table of operations, only one of which,
+ * issue_op, is mandatory.
+ *
+ * The check_write_begin() operation can be provided to check for and flush
+ * conflicting writes once the page is grabbed and locked. It is passed a
+ * pointer to the fsdata cookie that gets returned to the VM to be passed to
+ * write_end. It is permitted to sleep. It should return 0 if the request
+ * should go ahead; unlock the page and return -EAGAIN to cause the page to be
+ * regot; or return an error.
+ *
+ * This is usable whether or not caching is enabled.
+ */
+int netfs_write_begin(struct file *file, struct address_space *mapping,
+ loff_t pos, unsigned int len, unsigned int flags,
+ struct page **_page, void **_fsdata,
+ const struct netfs_read_request_ops *ops,
+ void *netfs_priv)
+{
+ struct netfs_read_request *rreq;
+ struct page *page, *xpage;
+ struct inode *inode = file_inode(file);
+ unsigned int debug_index = 0;
+ pgoff_t index = pos >> PAGE_SHIFT;
+ int pos_in_page = pos & ~PAGE_MASK;
+ loff_t size;
+ int ret;
+
+ struct readahead_control ractl = {
+ .file = file,
+ .mapping = mapping,
+ ._index = index,
+ ._nr_pages = 0,
+ };
+
+retry:
+ page = grab_cache_page_write_begin(mapping, index, 0);
+ if (!page)
+ return -ENOMEM;
+
+ if (ops->check_write_begin) {
+ /* Allow the netfs (eg. ceph) to flush conflicts. */
+ ret = ops->check_write_begin(file, pos, len, page, _fsdata);
+ if (ret < 0) {
+ if (ret == -EAGAIN)
+ goto retry;
+ goto error;
+ }
+ }
+
+ if (PageUptodate(page))
+ goto have_page;
+
+ /* If the page is beyond the EOF, we want to clear it - unless it's
+ * within the cache granule containing the EOF, in which case we need
+ * to preload the granule.
+ */
+ size = i_size_read(inode);
+ if (!ops->is_cache_enabled(inode) &&
+ ((pos_in_page == 0 && len == thp_size(page)) ||
+ (pos >= size) ||
+ (pos_in_page == 0 && (pos + len) >= size))) {
+ netfs_clear_thp(page);
+ SetPageUptodate(page);
+ netfs_stat(&netfs_n_rh_write_zskip);
+ goto have_page_no_wait;
+ }
+
+ ret = -ENOMEM;
+ rreq = netfs_alloc_read_request(ops, netfs_priv, file);
+ if (!rreq)
+ goto error;
+ rreq->mapping = page->mapping;
+ rreq->start = page->index * PAGE_SIZE;
+ rreq->len = thp_size(page);
+ rreq->no_unlock_page = page->index;
+ __set_bit(NETFS_RREQ_NO_UNLOCK_PAGE, &rreq->flags);
+ netfs_priv = NULL;
+
+ netfs_stat(&netfs_n_rh_write_begin);
+ trace_netfs_read(rreq, pos, len, netfs_read_trace_write_begin);
+
+ if (ops->begin_cache_operation)
+ ops->begin_cache_operation(rreq);
+
+ /* Expand the request to meet caching requirements and download
+ * preferences.
+ */
+ ractl._nr_pages = thp_nr_pages(page);
+ netfs_rreq_expand(rreq, &ractl);
+ netfs_get_read_request(rreq);
+
+ /* We hold the page locks, so we can drop the references */
+ while ((xpage = readahead_page(&ractl)))
+ if (xpage != page)
+ put_page(xpage);
+
+ atomic_set(&rreq->nr_rd_ops, 1);
+ do {
+ if (!netfs_rreq_submit_slice(rreq, &debug_index))
+ break;
+
+ } while (rreq->submitted < rreq->len);
+
+ // TODO: If we didn't submit enough readage, we need to clean up
+
+ /* Keep nr_rd_ops incremented so that the ref always belongs to us, and
+ * the service code isn't punted off to a random thread pool to
+ * process.
+ */
+ for (;;) {
+ wait_var_event(&rreq->nr_rd_ops, atomic_read(&rreq->nr_rd_ops) == 1);
+ netfs_rreq_assess(rreq);
+ if (!test_bit(NETFS_RREQ_IN_PROGRESS, &rreq->flags))
+ break;
+ cond_resched();
+ }
+
+ ret = rreq->error;
+ netfs_put_read_request(rreq);
+ if (ret < 0)
+ goto error;
+
+have_page:
+ wait_on_page_fscache(page);
+have_page_no_wait:
+ if (netfs_priv)
+ ops->cleanup(netfs_priv, mapping);
+ *_page = page;
+ _leave(" = 0");
+ return 0;
+
+error:
+ unlock_page(page);
+ put_page(page);
+ if (netfs_priv)
+ ops->cleanup(netfs_priv, mapping);
+ _leave(" = %d", ret);
+ return ret;
+}
+EXPORT_SYMBOL(netfs_write_begin);
diff --git a/fs/netfs/stats.c b/fs/netfs/stats.c
index 3a7a3c10e1cd..cdd09e30ce75 100644
--- a/fs/netfs/stats.c
+++ b/fs/netfs/stats.c
@@ -23,19 +23,23 @@ atomic_t netfs_n_rh_read_failed;
atomic_t netfs_n_rh_zero;
atomic_t netfs_n_rh_short_read;
atomic_t netfs_n_rh_write;
+atomic_t netfs_n_rh_write_begin;
atomic_t netfs_n_rh_write_done;
atomic_t netfs_n_rh_write_failed;
+atomic_t netfs_n_rh_write_zskip;
void netfs_stats_show(struct seq_file *m)
{
- seq_printf(m, "RdHelp : RA=%u RP=%u rr=%u sr=%u\n",
+ seq_printf(m, "RdHelp : RA=%u RP=%u WB=%u rr=%u sr=%u\n",
atomic_read(&netfs_n_rh_readahead),
atomic_read(&netfs_n_rh_readpage),
+ atomic_read(&netfs_n_rh_write_begin),
atomic_read(&netfs_n_rh_rreq),
atomic_read(&netfs_n_rh_sreq));
- seq_printf(m, "RdHelp : ZR=%u sh=%u\n",
+ seq_printf(m, "RdHelp : ZR=%u sh=%u sk=%u\n",
atomic_read(&netfs_n_rh_zero),
- atomic_read(&netfs_n_rh_short_read));
+ atomic_read(&netfs_n_rh_short_read),
+ atomic_read(&netfs_n_rh_write_zskip));
seq_printf(m, "RdHelp : DL=%u ds=%u df=%u di=%u\n",
atomic_read(&netfs_n_rh_download),
atomic_read(&netfs_n_rh_download_done),
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index d9cd9a1d1f6d..aced292b7e6d 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -78,12 +78,15 @@ struct netfs_read_request {
* Operations the network filesystem can/must provide to the helpers.
*/
struct netfs_read_request_ops {
+ bool (*is_cache_enabled)(struct inode *inode);
void (*init_rreq)(struct netfs_read_request *rreq, struct file *file);
int (*begin_cache_operation)(struct netfs_read_request *rreq);
void (*expand_readahead)(struct netfs_read_request *rreq);
bool (*clamp_length)(struct netfs_read_subrequest *subreq);
void (*issue_op)(struct netfs_read_subrequest *subreq);
bool (*is_still_valid)(struct netfs_read_request *rreq);
+ int (*check_write_begin)(struct file *file, loff_t pos, unsigned len,
+ struct page *page, void **_fsdata);
void (*done)(struct netfs_read_request *rreq);
void (*cleanup)(struct address_space *mapping, void *netfs_priv);
};
@@ -96,6 +99,11 @@ extern int netfs_readpage(struct file *,
struct page *,
const struct netfs_read_request_ops *,
void *);
+extern int netfs_write_begin(struct file *, struct address_space *,
+ loff_t, unsigned int, unsigned int, struct page **,
+ void **,
+ const struct netfs_read_request_ops *,
+ void *);
extern void netfs_subreq_terminated(struct netfs_read_subrequest *, ssize_t);
extern void netfs_stats_show(struct seq_file *);
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 56a734a3fb0d..b199dbb59256 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -23,6 +23,7 @@ enum netfs_read_trace {
netfs_read_trace_expanded,
netfs_read_trace_readahead,
netfs_read_trace_readpage,
+ netfs_read_trace_write_begin,
};
enum netfs_rreq_trace {
@@ -51,7 +52,8 @@ enum netfs_sreq_trace {
#define netfs_read_traces \
EM(netfs_read_trace_expanded, "EXPANDED ") \
EM(netfs_read_trace_readahead, "READAHEAD") \
- E_(netfs_read_trace_readpage, "READPAGE ")
+ EM(netfs_read_trace_readpage, "READPAGE ") \
+ E_(netfs_read_trace_write_begin, "WRITEBEGN")
#define netfs_rreq_traces \
EM(netfs_rreq_trace_assess, "ASSESS") \
next prev parent reply other threads:[~2020-11-20 15:18 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-20 15:02 [RFC PATCH 00/76] fscache: Modernisation David Howells
2020-11-20 15:02 ` [RFC PATCH 01/76] nfs, cifs, ceph, 9p: Disable use of fscache prior to its rewrite David Howells
2020-11-20 15:02 ` [RFC PATCH 02/76] afs: Disable use of the fscache I/O routines David Howells
2020-11-20 15:03 ` [RFC PATCH 03/76] fscache: Add a cookie debug ID and use that in traces David Howells
2020-11-20 15:03 ` [RFC PATCH 04/76] fscache: Procfile to display cookies David Howells
2020-11-20 15:03 ` [RFC PATCH 05/76] fscache: Remove the old I/O API David Howells
2020-11-20 15:03 ` [RFC PATCH 06/76] fscache: Remove the netfs data from the cookie David Howells
2020-11-20 15:04 ` [RFC PATCH 07/76] fscache: Remove struct fscache_cookie_def David Howells
2020-11-20 15:04 ` [RFC PATCH 08/76] fscache: Remove store_limit* from struct fscache_object David Howells
2020-11-20 15:04 ` [RFC PATCH 09/76] fscache: Remove fscache_check_consistency() David Howells
2020-11-20 15:04 ` [RFC PATCH 10/76] fscache: Remove fscache_attr_changed() David Howells
2020-11-20 15:04 ` [RFC PATCH 11/76] fscache: Remove obsolete stats David Howells
2020-11-20 15:05 ` [RFC PATCH 12/76] fscache: Remove old I/O tracepoints David Howells
2020-11-20 15:05 ` [RFC PATCH 13/76] fscache: Temporarily disable fscache_invalidate() David Howells
2020-11-20 15:05 ` [RFC PATCH 14/76] fscache: Remove the I/O operation manager David Howells
2020-11-20 15:05 ` [RFC PATCH 15/76] fscache: Change %p in format strings to something else David Howells
2020-11-20 15:06 ` [RFC PATCH 16/76] cachefiles: " David Howells
2020-11-20 15:06 ` [RFC PATCH 17/76] iov_iter: Add ITER_XARRAY David Howells
2020-11-20 15:06 ` [RFC PATCH 18/76] vm: Add wait/unlock functions for PG_fscache David Howells
2020-11-20 15:06 ` [RFC PATCH 19/76] mm: Implement readahead_control pageset expansion David Howells
2020-11-20 15:06 ` [RFC PATCH 20/76] mm: Stop generic_file_buffered_read() from grabbing a superfluous page David Howells
2020-11-20 15:07 ` [RFC PATCH 21/76] vfs: Export rw_verify_area() for use by cachefiles David Howells
2020-11-20 15:07 ` [RFC PATCH 22/76] vfs: Provide S_CACHE_FILE inode flag David Howells
2020-11-20 15:07 ` [RFC PATCH 23/76] cachefiles: Remove tree of active files and use " David Howells
2020-11-20 15:07 ` [RFC PATCH 24/76] fscache: Provide a simple thread pool for running ops asynchronously David Howells
2020-11-20 15:08 ` [RFC PATCH 26/76] fscache: Rewrite the I/O API based on iov_iter David Howells
2020-11-20 15:08 ` [RFC PATCH 27/76] fscache: Keep track of size of a file last set independently on the server David Howells
2020-11-20 15:08 ` [RFC PATCH 28/76] fscache, cachefiles: Fix disabled histogram warnings David Howells
2020-11-20 15:08 ` [RFC PATCH 29/76] fscache: Recast assertion in terms of cookie not being an index David Howells
2020-11-20 15:08 ` [RFC PATCH 30/76] vfs, fscache: Force ->write_inode() to occur if cookie pinned for writeback David Howells
2020-11-20 15:09 ` [RFC PATCH 31/76] fscache: Allow ->put_super() to be used to wait for cache operations David Howells
2020-11-20 15:09 ` [RFC PATCH 32/76] netfs: Make a netfs helper module David Howells
2020-11-20 15:09 ` [RFC PATCH 33/76] netfs: Provide readahead and readpage netfs helpers David Howells
2020-11-20 15:09 ` [RFC PATCH 34/76] netfs: Use the cache David Howells
2020-11-20 15:10 ` [RFC PATCH 35/76] fscache: read-helper: Add tracepoints David Howells
2020-11-20 15:10 ` [RFC PATCH 36/76] cachefiles: Remove some redundant checks on unsigned values David Howells
2020-11-20 15:10 ` [RFC PATCH 37/76] cachefiles: trace: Log coherency checks David Howells
2020-11-20 15:10 ` [RFC PATCH 38/76] cachefiles: Split cachefiles_drop_object() up a bit David Howells
2020-11-20 15:11 ` [RFC PATCH 39/76] cachefiles: Implement new fscache I/O backend API David Howells
2020-11-20 15:11 ` [RFC PATCH 40/76] cachefiles: Merge object->backer into object->dentry David Howells
2020-11-20 15:11 ` [RFC PATCH 41/76] cachefiles: Implement a content-present indicator and bitmap David Howells
2020-11-20 15:11 ` [RFC PATCH 42/76] cachefiles: Shape requests from the fscache read helper David Howells
2020-11-20 15:11 ` [RFC PATCH 43/76] cachefiles: Round the cachefile size up to DIO block size David Howells
2020-11-20 15:12 ` [RFC PATCH 44/76] cachefiles: Implement read and write parts of new I/O API David Howells
2020-11-20 15:12 ` [RFC PATCH 45/76] cachefiles: Add I/O tracepoints David Howells
2020-11-20 15:12 ` [RFC PATCH 46/76] fscache: Display cache-specific data in /proc/fs/fscache/objects David Howells
2020-11-20 15:12 ` [RFC PATCH 47/76] fscache: Remove more obsolete stats David Howells
2020-11-20 15:12 ` [RFC PATCH 48/76] fscache: Always create /proc/fs/fscache/stats if configured David Howells
2020-11-20 15:13 ` [RFC PATCH 49/76] netfs: Stats David Howells
2020-11-20 15:13 ` [RFC PATCH 50/76] fscache: New stats David Howells
2020-11-20 15:13 ` [RFC PATCH 51/76] fscache, cachefiles: Rewrite invalidation David Howells
2020-11-20 15:13 ` [RFC PATCH 52/76] fscache: disable cookie when doing an invalidation for DIO write David Howells
2020-11-20 15:13 ` [RFC PATCH 53/76] fscache: Implement "will_modify" parameter on fscache_use_cookie() David Howells
2020-11-20 15:14 ` [RFC PATCH 54/76] fscache: Provide resize operation David Howells
2020-11-20 15:14 ` [RFC PATCH 55/76] fscache: Remove the update operation David Howells
2020-11-20 15:14 ` [RFC PATCH 56/76] afs: Pass page into dirty region helpers to provide THP size David Howells
2020-11-20 15:14 ` [RFC PATCH 57/76] afs: Print the operation debug_id when logging an unexpected data version David Howells
2020-11-20 15:14 ` [RFC PATCH 58/76] afs: Move key to afs_read struct David Howells
2020-11-20 15:15 ` [RFC PATCH 59/76] afs: Don't truncate iter during data fetch David Howells
2020-11-20 15:15 ` [RFC PATCH 60/76] afs: Log remote unmarshalling errors David Howells
2020-11-20 15:15 ` [RFC PATCH 61/76] afs: Set up the iov_iter before calling afs_extract_data() David Howells
2020-11-20 15:15 ` [RFC PATCH 62/76] afs: Use ITER_XARRAY for writing David Howells
2020-11-20 15:16 ` [RFC PATCH 63/76] afs: Wait on PG_fscache before modifying/releasing a page David Howells
2020-11-20 15:16 ` [RFC PATCH 64/76] afs: Extract writeback extension into its own function David Howells
2020-11-20 15:16 ` [RFC PATCH 65/76] afs: Prepare for use of THPs David Howells
2020-11-20 15:16 ` [RFC PATCH 66/76] afs: Use the fs operation ops to handle FetchData completion David Howells
2020-11-20 15:17 ` [RFC PATCH 67/76] afs: Use new fscache read helper API David Howells
2020-11-20 15:17 ` David Howells [this message]
2020-11-20 15:17 ` [RFC PATCH 69/76] fscache: Add support for writing to the cache David Howells
2020-11-20 15:17 ` [RFC PATCH 70/76] afs: Use the fscache_write_begin() helper David Howells
2020-11-20 15:18 ` [RFC PATCH 71/76] afs: Copy local writes to the cache when writing to the server David Howells
2020-11-20 15:18 ` [RFC PATCH 72/76] afs: Invoke fscache_resize_cookie() when handling ATTR_SIZE for setattr David Howells
2020-11-20 15:18 ` [RFC PATCH 73/76] afs: Add O_DIRECT read support David Howells
2020-11-20 15:18 ` [RFC PATCH 74/76] afs: Skip truncation on the server of data we haven't written yet David Howells
2020-11-20 15:19 ` [RFC PATCH 75/76] afs: Make afs_write_begin() return the THP subpage David Howells
2020-11-20 15:19 ` [RFC PATCH 76/76] afs: Fix speculative status fetch going out of order wrt to modifications David Howells
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=160588543960.3465195.2792938973035886168.stgit@warthog.procyon.org.uk \
--to=dhowells@redhat.com \
--cc=anna.schumaker@netapp.com \
--cc=asmadeus@codewreck.org \
--cc=ceph-devel@vger.kernel.org \
--cc=jlayton@redhat.com \
--cc=linux-afs@lists.infradead.org \
--cc=linux-cachefs@redhat.com \
--cc=linux-cifs@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=sfrench@samba.org \
--cc=trondmy@hammerspace.com \
--cc=v9fs-developer@lists.sourceforge.net \
--cc=viro@zeniv.linux.org.uk \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).