* [PATCH] block, writeback: wait for writeback to finish before detaching wb
@ 2017-03-09 2:19 Tahsin Erdogan
2017-03-09 18:26 ` Tejun Heo
0 siblings, 1 reply; 5+ messages in thread
From: Tahsin Erdogan @ 2017-03-09 2:19 UTC (permalink / raw)
To: Alexander Viro, Ilya Dryomov, Jens Axboe, Raghavendra K T, Tejun Heo
Cc: linux-fsdevel, linux-kernel, stable, Tahsin Erdogan
__blkdev_put() could surprise writeback thread by detaching the
wb object from an inode that hasn't cleared the I_SYNC flag yet.
This causes a NULL pointer dereference as seen below:
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: locked_inode_to_wb_and_lock_list+0x38/0x440
PGD 0
Oops: 0000 [#1] SMP
CPU: 0 PID: 34 Comm: kworker/u8:1 Not tainted 4.11.0-rc1+ #202
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Workqueue: writeback wb_workfn (flush-8:16)
task: ffff88013aa780c0 task.stack: ffffc9000012c000
RIP: 0010:locked_inode_to_wb_and_lock_list+0x38/0x440
RSP: 0018:ffffc9000012fb70 EFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000018
RDX: ffff88013aa780c0 RSI: ffff880139a478f8 RDI: ffff88013aa788b8
RBP: ffffc9000012fba0 R08: 0000000000000001 R09: 0000000000000000
R10: 00000000969da8e2 R11: 0000000000000000 R12: ffff880139a47858
R13: ffff880139a478e0 R14: ffff880139a478f8 R15: ffff8801371f4058
FS: 0000000000000000(0000) GS:ffff88013ae00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000001012000 CR4: 00000000000006f0
Call Trace:
writeback_sb_inodes+0x3e1/0x7a0
__writeback_inodes_wb+0x87/0xc0
wb_writeback+0x2e7/0x5c0
wb_workfn+0x2d1/0x9c0
process_one_work+0x1d3/0x620
worker_thread+0x126/0x4a0
kthread+0x10a/0x140
ret_from_fork+0x2e/0x40
RIP: locked_inode_to_wb_and_lock_list+0x38/0x440 RSP: ffffc9000012fb70
CR2: 0000000000000000
---[ end trace e0ea8a2695f4c86c ]---
Make __blkdev_put() wait for the I_SYNC flag to clear before detaching
wb.
Fixes: 43d1c0eb7e11 ("block: detach bdev inode from its wb in __blkdev_put()")
Signed-off-by: Tahsin Erdogan <tahsin@google.com>
---
fs/block_dev.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 2eca00ec4370..70fb82fcedd0 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -95,7 +95,7 @@ void kill_bdev(struct block_device *bdev)
invalidate_bh_lrus();
truncate_inode_pages(mapping, 0);
-}
+}
EXPORT_SYMBOL(kill_bdev);
/* Invalidate clean unused buffers and pagecache. */
@@ -617,13 +617,13 @@ static loff_t block_llseek(struct file *file, loff_t offset, int whence)
inode_unlock(bd_inode);
return retval;
}
-
+
int blkdev_fsync(struct file *filp, loff_t start, loff_t end, int datasync)
{
struct inode *bd_inode = bdev_file_inode(filp);
struct block_device *bdev = I_BDEV(bd_inode);
int error;
-
+
error = filemap_write_and_wait_range(filp->f_mapping, start, end);
if (error)
return error;
@@ -1038,7 +1038,7 @@ void bdput(struct block_device *bdev)
}
EXPORT_SYMBOL(bdput);
-
+
static struct block_device *bd_acquire(struct inode *inode)
{
struct block_device *bdev;
@@ -1880,7 +1880,10 @@ static void __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)
* Detaching bdev inode from its wb in __destroy_inode()
* is too late: the queue which embeds its bdi (along with
* root wb) can be gone as soon as we put_disk() below.
+ * Before detaching wb, wait for any writeback activity for
+ * inode to settle.
*/
+ inode_wait_for_writeback(bdev->bd_inode);
inode_detach_wb(bdev->bd_inode);
}
if (bdev->bd_contains == bdev) {
--
2.12.0.246.ga2ecc84866-goog
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] block, writeback: wait for writeback to finish before detaching wb
2017-03-09 2:19 [PATCH] block, writeback: wait for writeback to finish before detaching wb Tahsin Erdogan
@ 2017-03-09 18:26 ` Tejun Heo
2017-03-09 23:46 ` [PATCH v2] " Tahsin Erdogan
2017-03-09 23:53 ` [PATCH] " Jan Kara
0 siblings, 2 replies; 5+ messages in thread
From: Tejun Heo @ 2017-03-09 18:26 UTC (permalink / raw)
To: Tahsin Erdogan, Jan Kara
Cc: Alexander Viro, Ilya Dryomov, Jens Axboe, Raghavendra K T,
linux-fsdevel, linux-kernel, stable
(cc'ing Jan and quoting the whole message)
On Wed, Mar 08, 2017 at 06:19:37PM -0800, Tahsin Erdogan wrote:
> __blkdev_put() could surprise writeback thread by detaching the
> wb object from an inode that hasn't cleared the I_SYNC flag yet.
> This causes a NULL pointer dereference as seen below:
>
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: locked_inode_to_wb_and_lock_list+0x38/0x440
> PGD 0
> Oops: 0000 [#1] SMP
> CPU: 0 PID: 34 Comm: kworker/u8:1 Not tainted 4.11.0-rc1+ #202
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Workqueue: writeback wb_workfn (flush-8:16)
> task: ffff88013aa780c0 task.stack: ffffc9000012c000
> RIP: 0010:locked_inode_to_wb_and_lock_list+0x38/0x440
> RSP: 0018:ffffc9000012fb70 EFLAGS: 00010202
> RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000018
> RDX: ffff88013aa780c0 RSI: ffff880139a478f8 RDI: ffff88013aa788b8
> RBP: ffffc9000012fba0 R08: 0000000000000001 R09: 0000000000000000
> R10: 00000000969da8e2 R11: 0000000000000000 R12: ffff880139a47858
> R13: ffff880139a478e0 R14: ffff880139a478f8 R15: ffff8801371f4058
> FS: 0000000000000000(0000) GS:ffff88013ae00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000000 CR3: 0000000001012000 CR4: 00000000000006f0
> Call Trace:
> writeback_sb_inodes+0x3e1/0x7a0
> __writeback_inodes_wb+0x87/0xc0
> wb_writeback+0x2e7/0x5c0
> wb_workfn+0x2d1/0x9c0
> process_one_work+0x1d3/0x620
> worker_thread+0x126/0x4a0
> kthread+0x10a/0x140
> ret_from_fork+0x2e/0x40
> RIP: locked_inode_to_wb_and_lock_list+0x38/0x440 RSP: ffffc9000012fb70
> CR2: 0000000000000000
> ---[ end trace e0ea8a2695f4c86c ]---
>
> Make __blkdev_put() wait for the I_SYNC flag to clear before detaching
> wb.
>
> Fixes: 43d1c0eb7e11 ("block: detach bdev inode from its wb in __blkdev_put()")
> Signed-off-by: Tahsin Erdogan <tahsin@google.com>
> ---
> fs/block_dev.c | 11 +++++++----
> 1 file changed, 7 insertions(+), 4 deletions(-)
>
> diff --git a/fs/block_dev.c b/fs/block_dev.c
> index 2eca00ec4370..70fb82fcedd0 100644
> --- a/fs/block_dev.c
> +++ b/fs/block_dev.c
> @@ -95,7 +95,7 @@ void kill_bdev(struct block_device *bdev)
>
> invalidate_bh_lrus();
> truncate_inode_pages(mapping, 0);
> -}
> +}
> EXPORT_SYMBOL(kill_bdev);
>
> /* Invalidate clean unused buffers and pagecache. */
> @@ -617,13 +617,13 @@ static loff_t block_llseek(struct file *file, loff_t offset, int whence)
> inode_unlock(bd_inode);
> return retval;
> }
> -
> +
> int blkdev_fsync(struct file *filp, loff_t start, loff_t end, int datasync)
> {
> struct inode *bd_inode = bdev_file_inode(filp);
> struct block_device *bdev = I_BDEV(bd_inode);
> int error;
> -
> +
> error = filemap_write_and_wait_range(filp->f_mapping, start, end);
> if (error)
> return error;
> @@ -1038,7 +1038,7 @@ void bdput(struct block_device *bdev)
> }
>
> EXPORT_SYMBOL(bdput);
> -
> +
White line contaminations.
> static struct block_device *bd_acquire(struct inode *inode)
> {
> struct block_device *bdev;
> @@ -1880,7 +1880,10 @@ static void __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)
> * Detaching bdev inode from its wb in __destroy_inode()
> * is too late: the queue which embeds its bdi (along with
> * root wb) can be gone as soon as we put_disk() below.
> + * Before detaching wb, wait for any writeback activity for
> + * inode to settle.
> */
> + inode_wait_for_writeback(bdev->bd_inode);
> inode_detach_wb(bdev->bd_inode);
> }
> if (bdev->bd_contains == bdev) {
Given that there's write_inode_now(@sync == true) call right above,
I'm not sure how waiting for I_SYNC once more time would make a
difference. Can you please explain how to trigger the issue?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2] block, writeback: wait for writeback to finish before detaching wb
2017-03-09 18:26 ` Tejun Heo
@ 2017-03-09 23:46 ` Tahsin Erdogan
2017-03-09 23:53 ` Tahsin Erdogan
2017-03-09 23:53 ` [PATCH] " Jan Kara
1 sibling, 1 reply; 5+ messages in thread
From: Tahsin Erdogan @ 2017-03-09 23:46 UTC (permalink / raw)
To: Alexander Viro, Ilya Dryomov, Jens Axboe, Raghavendra K T,
Tejun Heo, Jan Kara
Cc: linux-fsdevel, linux-kernel, stable, Tahsin Erdogan
__blkdev_put() could surprise writeback thread by detaching the
wb object from an inode that hasn't cleared the I_SYNC flag yet.
This causes a NULL pointer dereference as seen below:
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: locked_inode_to_wb_and_lock_list+0x38/0x440
PGD 0
Oops: 0000 [#1] SMP
CPU: 0 PID: 34 Comm: kworker/u8:1 Not tainted 4.11.0-rc1+ #202
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Workqueue: writeback wb_workfn (flush-8:16)
task: ffff88013aa780c0 task.stack: ffffc9000012c000
RIP: 0010:locked_inode_to_wb_and_lock_list+0x38/0x440
RSP: 0018:ffffc9000012fb70 EFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000018
RDX: ffff88013aa780c0 RSI: ffff880139a478f8 RDI: ffff88013aa788b8
RBP: ffffc9000012fba0 R08: 0000000000000001 R09: 0000000000000000
R10: 00000000969da8e2 R11: 0000000000000000 R12: ffff880139a47858
R13: ffff880139a478e0 R14: ffff880139a478f8 R15: ffff8801371f4058
FS: 0000000000000000(0000) GS:ffff88013ae00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000001012000 CR4: 00000000000006f0
Call Trace:
writeback_sb_inodes+0x3e1/0x7a0
__writeback_inodes_wb+0x87/0xc0
wb_writeback+0x2e7/0x5c0
wb_workfn+0x2d1/0x9c0
process_one_work+0x1d3/0x620
worker_thread+0x126/0x4a0
kthread+0x10a/0x140
ret_from_fork+0x2e/0x40
RIP: locked_inode_to_wb_and_lock_list+0x38/0x440 RSP: ffffc9000012fb70
CR2: 0000000000000000
---[ end trace e0ea8a2695f4c86c ]---
Make __blkdev_put() wait for the I_SYNC flag to clear before detaching
wb.
Fixes: 43d1c0eb7e11 ("block: detach bdev inode from its wb in __blkdev_put()")
Signed-off-by: Tahsin Erdogan <tahsin@google.com>
---
v2:
Removed white space clean up changes
fs/block_dev.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 2eca00ec4370..fdc71f9f8003 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -1880,7 +1880,10 @@ static void __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)
* Detaching bdev inode from its wb in __destroy_inode()
* is too late: the queue which embeds its bdi (along with
* root wb) can be gone as soon as we put_disk() below.
+ * Before detaching wb, wait for any writeback activity for
+ * inode to settle.
*/
+ inode_wait_for_writeback(bdev->bd_inode);
inode_detach_wb(bdev->bd_inode);
}
if (bdev->bd_contains == bdev) {
--
2.12.0.246.ga2ecc84866-goog
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v2] block, writeback: wait for writeback to finish before detaching wb
2017-03-09 23:46 ` [PATCH v2] " Tahsin Erdogan
@ 2017-03-09 23:53 ` Tahsin Erdogan
0 siblings, 0 replies; 5+ messages in thread
From: Tahsin Erdogan @ 2017-03-09 23:53 UTC (permalink / raw)
To: Alexander Viro, Ilya Dryomov, Jens Axboe, Raghavendra K T,
Tejun Heo, Jan Kara
Cc: linux-fsdevel, linux-kernel, stable, Tahsin Erdogan
Hi Tejun,
>
> White line contaminations.
Fixed in v2.
> Given that there's write_inode_now(@sync == true) call right above,
> I'm not sure how waiting for I_SYNC once more time would make a
> difference. Can you please explain how to trigger the issue?
bdev_write_inode() calls write_inode_now() only if I_DIRTY is set for
the inode. It is possible that I_DIRTY flags have been cleared by the
writeback thread, but it hasn't run node_sync_complete() yet.
I am able to reproduce the problem by adding articial delays in
__blkdev_put() and writeback_sb_inodes(). Please see the repro patch
below:
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 2eca00ec4370..e3cfbba19a6c 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -33,6 +33,7 @@
#include <linux/task_io_accounting_ops.h>
#include <linux/falloc.h>
#include <linux/uaccess.h>
+#include <linux/delay.h>
#include "internal.h"
struct bdev_inode {
@@ -1875,6 +1876,8 @@ static void __blkdev_put(struct block_device
*bdev, fmode_t mode, int for_part)
sync_blockdev(bdev);
kill_bdev(bdev);
+ /* Allow time for writeback thread to clear I_DIRTY flags */
+ mdelay(200);
bdev_write_inode(bdev);
/*
* Detaching bdev inode from its wb in __destroy_inode()
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index ef600591d96f..55073cbc1823 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -28,6 +28,7 @@
#include <linux/tracepoint.h>
#include <linux/device.h>
#include <linux/memcontrol.h>
+#include <linux/delay.h>
#include "internal.h"
/*
@@ -1585,6 +1586,10 @@ static long writeback_sb_inodes(struct super_block *sb,
cond_resched();
}
+ /* Allow __blkdev_put() to run inode_detach_wb() */
+ if (inode->i_ino == 0)
+ mdelay(200);
+
/*
* Requeue @inode if still dirty. Be careful as @inode may
* have been switched to another wb in the meantime.
diff --git a/wb-detach-repro.sh b/wb-detach-repro.sh
new file mode 100755
index 000000000000..66892d71cd20
--- /dev/null
+++ b/wb-detach-repro.sh
@@ -0,0 +1,10 @@
+#!/bin/bash
+set -ex
+
+echo 100 > /proc/sys/vm/dirty_expire_centisecs
+echo 100 > /proc/sys/vm/dirty_writeback_centisecs
+
+mount /dev/sdb /mnt/sdb
+sleep 0.9
+echo 10 > /proc/sys/vm/dirty_writeback_centisecs
+umount /dev/sdb
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] block, writeback: wait for writeback to finish before detaching wb
2017-03-09 18:26 ` Tejun Heo
2017-03-09 23:46 ` [PATCH v2] " Tahsin Erdogan
@ 2017-03-09 23:53 ` Jan Kara
1 sibling, 0 replies; 5+ messages in thread
From: Jan Kara @ 2017-03-09 23:53 UTC (permalink / raw)
To: Tejun Heo
Cc: Tahsin Erdogan, Jan Kara, Alexander Viro, Ilya Dryomov,
Jens Axboe, Raghavendra K T, linux-fsdevel, linux-kernel, stable
On Thu 09-03-17 13:26:45, Tejun Heo wrote:
> (cc'ing Jan and quoting the whole message)
>
> On Wed, Mar 08, 2017 at 06:19:37PM -0800, Tahsin Erdogan wrote:
> > __blkdev_put() could surprise writeback thread by detaching the
> > wb object from an inode that hasn't cleared the I_SYNC flag yet.
> > This causes a NULL pointer dereference as seen below:
> >
> > BUG: unable to handle kernel NULL pointer dereference at (null)
> > IP: locked_inode_to_wb_and_lock_list+0x38/0x440
> > PGD 0
> > Oops: 0000 [#1] SMP
> > CPU: 0 PID: 34 Comm: kworker/u8:1 Not tainted 4.11.0-rc1+ #202
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> > Workqueue: writeback wb_workfn (flush-8:16)
> > task: ffff88013aa780c0 task.stack: ffffc9000012c000
> > RIP: 0010:locked_inode_to_wb_and_lock_list+0x38/0x440
> > RSP: 0018:ffffc9000012fb70 EFLAGS: 00010202
> > RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000018
> > RDX: ffff88013aa780c0 RSI: ffff880139a478f8 RDI: ffff88013aa788b8
> > RBP: ffffc9000012fba0 R08: 0000000000000001 R09: 0000000000000000
> > R10: 00000000969da8e2 R11: 0000000000000000 R12: ffff880139a47858
> > R13: ffff880139a478e0 R14: ffff880139a478f8 R15: ffff8801371f4058
> > FS: 0000000000000000(0000) GS:ffff88013ae00000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000000000 CR3: 0000000001012000 CR4: 00000000000006f0
> > Call Trace:
> > writeback_sb_inodes+0x3e1/0x7a0
> > __writeback_inodes_wb+0x87/0xc0
> > wb_writeback+0x2e7/0x5c0
> > wb_workfn+0x2d1/0x9c0
> > process_one_work+0x1d3/0x620
> > worker_thread+0x126/0x4a0
> > kthread+0x10a/0x140
> > ret_from_fork+0x2e/0x40
> > RIP: locked_inode_to_wb_and_lock_list+0x38/0x440 RSP: ffffc9000012fb70
> > CR2: 0000000000000000
> > ---[ end trace e0ea8a2695f4c86c ]---
> >
> > Make __blkdev_put() wait for the I_SYNC flag to clear before detaching
> > wb.
> >
> > Fixes: 43d1c0eb7e11 ("block: detach bdev inode from its wb in __blkdev_put()")
> > Signed-off-by: Tahsin Erdogan <tahsin@google.com>
> > ---
> > fs/block_dev.c | 11 +++++++----
> > 1 file changed, 7 insertions(+), 4 deletions(-)
> >
> > diff --git a/fs/block_dev.c b/fs/block_dev.c
> > index 2eca00ec4370..70fb82fcedd0 100644
> > --- a/fs/block_dev.c
> > +++ b/fs/block_dev.c
> > @@ -95,7 +95,7 @@ void kill_bdev(struct block_device *bdev)
> >
> > invalidate_bh_lrus();
> > truncate_inode_pages(mapping, 0);
> > -}
> > +}
> > EXPORT_SYMBOL(kill_bdev);
> >
> > /* Invalidate clean unused buffers and pagecache. */
> > @@ -617,13 +617,13 @@ static loff_t block_llseek(struct file *file, loff_t offset, int whence)
> > inode_unlock(bd_inode);
> > return retval;
> > }
> > -
> > +
> > int blkdev_fsync(struct file *filp, loff_t start, loff_t end, int datasync)
> > {
> > struct inode *bd_inode = bdev_file_inode(filp);
> > struct block_device *bdev = I_BDEV(bd_inode);
> > int error;
> > -
> > +
> > error = filemap_write_and_wait_range(filp->f_mapping, start, end);
> > if (error)
> > return error;
> > @@ -1038,7 +1038,7 @@ void bdput(struct block_device *bdev)
> > }
> >
> > EXPORT_SYMBOL(bdput);
> > -
> > +
>
> White line contaminations.
>
> > static struct block_device *bd_acquire(struct inode *inode)
> > {
> > struct block_device *bdev;
> > @@ -1880,7 +1880,10 @@ static void __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)
> > * Detaching bdev inode from its wb in __destroy_inode()
> > * is too late: the queue which embeds its bdi (along with
> > * root wb) can be gone as soon as we put_disk() below.
> > + * Before detaching wb, wait for any writeback activity for
> > + * inode to settle.
> > */
> > + inode_wait_for_writeback(bdev->bd_inode);
> > inode_detach_wb(bdev->bd_inode);
> > }
> > if (bdev->bd_contains == bdev) {
>
> Given that there's write_inode_now(@sync == true) call right above,
> I'm not sure how waiting for I_SYNC once more time would make a
> difference. Can you please explain how to trigger the issue?
Agreed that waiting for I_SYNC isn't going to cut it. Flusher thread (or
possibly some other process) can be running inode_to_wb() without I_SYNC
set. I have patches that fix this (remove inode_detach_wb() from
__blkdev_put()) - the ones you were reviewing lately. I'll post v4 next
week so that we can get those merged.
But if Tahsin has a reproducer, it would be nice. I was not able to hit the
race in my testing.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-03-09 23:53 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-09 2:19 [PATCH] block, writeback: wait for writeback to finish before detaching wb Tahsin Erdogan
2017-03-09 18:26 ` Tejun Heo
2017-03-09 23:46 ` [PATCH v2] " Tahsin Erdogan
2017-03-09 23:53 ` Tahsin Erdogan
2017-03-09 23:53 ` [PATCH] " Jan Kara
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).