* ORC unwinder bad backtrace
@ 2018-04-18 13:54 Matthew Wilcox
2018-04-18 15:45 ` Josh Poimboeuf
0 siblings, 1 reply; 3+ messages in thread
From: Matthew Wilcox @ 2018-04-18 13:54 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: x86, linux-kernel
[there's no entry in MAINTAINERS for the ORC unwinder. I'm guessing
at the mailing lists that might be interested in this]
I've done something stupid and fallen off the end of the stack. ORC
appears not to handle this very well (see backtrace at the bottom).
Look at this loop:
[ 60.750032] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.750750] ? do_page_fault+0x85/0xf0
[ 60.751327] ? async_page_fault+0x1e/0x30
[ 60.751970] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.752607] ? perf_trace_x86_exceptions+0x44/0x100
The instruction at pte_clear.constprop.18+0x2e/0x2e is a nop (as you might
expect for the last byte of a function), so that can't be right:
ffffffff81061192 <pte_clear.constprop.18>:
...
ffffffff810611bf: 90 nop
ffffffff810611c0 <perf_trace_x86_exceptions>:
I suspect an off-by-one error; you don't really mean to point to the
byte before perf_trace_x86_exception, you mean to point to byte 0 of
perf_trace_x86_exception.
I'm going to archive up this compilation in case there's anything useful
I can extract for you from it later.
[ 60.734865] BUG: stack guard page was hit at 0000000020ffe8f7 (stack is 00000000d090cc16..00000000cd7d843a)
[ 60.736511] kernel stack overflow (double-fault): 0000 [#1] SMP PTI
[ 60.737468] Modules linked in:
[ 60.737979] CPU: 0 PID: 1 Comm: init Not tainted 4.17.0-rc1+ #30
[ 60.738884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.1-1 04/01/2014
[ 60.740143] RIP: 0010:error_entry+0x12/0x100
[ 60.740882] RSP: 0018:ffffb050000d0000 EFLAGS: 00010046
[ 60.741701] RAX: 0000000000000000 RBX: ffffd04fffc042f8 RCX: 0000000000000000
[ 60.742707] RDX: 0000000000000000 RSI: ffffffffb5c010a8 RDI: ffffffffb6425d20
[ 60.743694] RBP: ffffb050000d0090 R08: ffffffffb54611c0 R09: 0000000000000000
[ 60.744687] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffb6425d20
[ 60.745673] R13: ffffd04fffc042f8 R14: 0000000000000000 R15: ffffb050000d00c8
[ 60.746640] FS: 00007fec2d341700(0000) GS:ffff9d42df800000(0000) knlGS:0000000000000000
[ 60.747804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 60.748620] CR2: ffffb050000cfff8 CR3: 000000001c974003 CR4: 00000000003606f0
[ 60.749607] Call Trace:
[ 60.750032] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.750750] ? do_page_fault+0x85/0xf0
[ 60.751327] ? async_page_fault+0x1e/0x30
[ 60.751970] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.752607] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.753250] ? do_page_fault+0x85/0xf0
[ 60.753771] ? async_page_fault+0x1e/0x30
[ 60.754317] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.754920] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.755562] ? do_page_fault+0x85/0xf0
[ 60.756083] ? async_page_fault+0x1e/0x30
[ 60.756688] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.757622] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.758515] ? do_page_fault+0x85/0xf0
[ 60.759174] ? async_page_fault+0x1e/0x30
[ 60.759862] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.760633] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.761450] ? do_page_fault+0x85/0xf0
[ 60.762105] ? async_page_fault+0x1e/0x30
[ 60.762797] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.763562] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.764383] ? do_page_fault+0x85/0xf0
[ 60.765040] ? async_page_fault+0x1e/0x30
[ 60.765730] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.766495] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.767311] ? do_page_fault+0x85/0xf0
[ 60.767967] ? async_page_fault+0x1e/0x30
[ 60.768671] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.769449] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.770271] ? do_page_fault+0x85/0xf0
[ 60.770936] ? async_page_fault+0x1e/0x30
[ 60.771627] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.772398] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.773254] ? do_page_fault+0x85/0xf0
[ 60.773922] ? async_page_fault+0x1e/0x30
[ 60.774711] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.775537] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.776451] ? do_page_fault+0x85/0xf0
[ 60.777140] ? async_page_fault+0x1e/0x30
[ 60.777860] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.778628] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.779444] ? do_page_fault+0x85/0xf0
[ 60.780097] ? async_page_fault+0x1e/0x30
[ 60.780792] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.781571] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.782524] ? do_page_fault+0x85/0xf0
[ 60.783201] ? async_page_fault+0x1e/0x30
[ 60.783919] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.784698] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.785520] ? do_page_fault+0x85/0xf0
[ 60.786176] ? async_page_fault+0x1e/0x30
[ 60.786870] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.787641] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.788462] ? do_page_fault+0x85/0xf0
[ 60.789116] ? async_page_fault+0x1e/0x30
[ 60.789829] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.790652] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.791495] ? do_page_fault+0x85/0xf0
[ 60.792155] ? async_page_fault+0x1e/0x30
[ 60.792849] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.793630] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.794445] ? do_page_fault+0x85/0xf0
[ 60.795101] ? async_page_fault+0x1e/0x30
[ 60.795806] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.796576] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.797413] ? do_page_fault+0x85/0xf0
[ 60.798078] ? async_page_fault+0x1e/0x30
[ 60.798768] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.799550] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.800370] ? do_page_fault+0x85/0xf0
[ 60.801023] ? async_page_fault+0x1e/0x30
[ 60.801745] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.802529] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.803349] ? do_page_fault+0x85/0xf0
[ 60.804015] ? async_page_fault+0x1e/0x30
[ 60.804724] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.805495] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.806313] ? do_page_fault+0x85/0xf0
[ 60.806981] ? async_page_fault+0x1e/0x30
[ 60.807673] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.816987] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.817816] ? do_page_fault+0x85/0xf0
[ 60.818476] ? async_page_fault+0x1e/0x30
[ 60.819173] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.819950] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.820782] ? do_page_fault+0x85/0xf0
[ 60.821442] ? async_page_fault+0x1e/0x30
[ 60.822137] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.822905] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.823721] ? do_page_fault+0x85/0xf0
[ 60.824383] ? async_page_fault+0x1e/0x30
[ 60.825072] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.825832] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.826688] ? do_page_fault+0x85/0xf0
[ 60.827342] ? async_page_fault+0x1e/0x30
[ 60.828033] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.828805] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.829619] ? do_page_fault+0x85/0xf0
[ 60.830286] ? async_page_fault+0x1e/0x30
[ 60.830985] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.831762] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.832582] ? do_page_fault+0x85/0xf0
[ 60.833249] ? async_page_fault+0x1e/0x30
[ 60.833945] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.834723] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.835538] ? do_page_fault+0x85/0xf0
[ 60.836199] ? async_page_fault+0x1e/0x30
[ 60.836895] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.837663] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.838482] ? do_page_fault+0x85/0xf0
[ 60.839126] ? async_page_fault+0x1e/0x30
[ 60.839819] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.840591] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.841408] ? do_page_fault+0x85/0xf0
[ 60.842074] ? async_page_fault+0x1e/0x30
[ 60.842763] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.843550] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.844547] ? do_page_fault+0x85/0xf0
[ 60.845251] ? async_page_fault+0x1e/0x30
[ 60.845959] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.846793] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.847625] ? do_page_fault+0x85/0xf0
[ 60.848291] ? async_page_fault+0x1e/0x30
[ 60.848993] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.849759] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.850576] ? do_page_fault+0x85/0xf0
[ 60.851231] ? async_page_fault+0x1e/0x30
[ 60.851918] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.852698] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.853526] ? do_page_fault+0x85/0xf0
[ 60.854184] ? async_page_fault+0x1e/0x30
[ 60.854886] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.855655] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.856483] ? do_page_fault+0x85/0xf0
[ 60.857069] ? async_page_fault+0x1e/0x30
[ 60.857789] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.858554] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.859376] ? do_page_fault+0x85/0xf0
[ 60.860031] ? async_page_fault+0x1e/0x30
[ 60.860734] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.861512] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.862335] ? do_page_fault+0x85/0xf0
[ 60.863006] ? async_page_fault+0x1e/0x30
[ 60.863698] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.864483] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.865319] ? do_page_fault+0x85/0xf0
[ 60.865980] ? async_page_fault+0x1e/0x30
[ 60.866684] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.867453] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.868277] ? do_page_fault+0x85/0xf0
[ 60.868936] ? async_page_fault+0x1e/0x30
[ 60.869637] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.870404] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.871237] ? do_page_fault+0x85/0xf0
[ 60.871894] ? async_page_fault+0x1e/0x30
[ 60.872588] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.873368] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.874240] ? do_page_fault+0x85/0xf0
[ 60.875060] ? async_page_fault+0x1e/0x30
[ 60.875767] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.876571] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.877295] ? do_page_fault+0x85/0xf0
[ 60.877888] ? async_page_fault+0x1e/0x30
[ 60.878501] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.879177] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.879894] ? do_page_fault+0x85/0xf0
[ 60.880481] ? async_page_fault+0x1e/0x30
[ 60.881090] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.881774] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.882495] ? do_page_fault+0x85/0xf0
[ 60.883074] ? async_page_fault+0x1e/0x30
[ 60.883684] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.884376] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.885091] ? do_page_fault+0x85/0xf0
[ 60.885665] ? async_page_fault+0x1e/0x30
[ 60.886271] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.886944] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.887694] ? do_page_fault+0x85/0xf0
[ 60.888275] ? async_page_fault+0x1e/0x30
[ 60.888883] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.889557] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.890274] ? do_page_fault+0x85/0xf0
[ 60.890849] ? async_page_fault+0x1e/0x30
[ 60.891456] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.892144] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.892890] ? do_page_fault+0x85/0xf0
[ 60.893466] ? async_page_fault+0x1e/0x30
[ 60.894040] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.894646] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.895288] ? acpi_ns_lookup+0x294/0x3b2
[ 60.895834] ? do_page_fault+0x85/0xf0
[ 60.896355] ? async_page_fault+0x1e/0x30
[ 60.896940] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.897615] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.898332] ? __update_load_avg_se.isra.39+0x1cf/0x1e0
[ 60.899088] ? do_page_fault+0x85/0xf0
[ 60.899668] ? async_page_fault+0x1e/0x30
[ 60.900282] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.900958] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.901673] ? do_page_fault+0x85/0xf0
[ 60.902255] ? async_page_fault+0x1e/0x30
[ 60.902867] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.903539] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.904255] ? do_page_fault+0x85/0xf0
[ 60.904839] ? async_page_fault+0x1e/0x30
[ 60.905466] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.906212] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.906932] ? do_select+0x6a6/0x7b0
[ 60.907492] ? do_page_fault+0x85/0xf0
[ 60.908076] ? async_page_fault+0x1e/0x30
[ 60.908706] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.909452] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.910213] ? do_page_fault+0x85/0xf0
[ 60.910794] ? async_page_fault+0x1e/0x30
[ 60.911404] ? pte_clear.constprop.18+0x2e/0x2e
[ 60.912081] ? perf_trace_x86_exceptions+0x44/0x100
[ 60.912797] ? do_page_fault+0x85/0xf0
[ 60.913371] ? async_page_fault+0x1e/0x30
[ 60.913970] ? copy_user_enhanced_fast_string+0xe/0x20
[ 60.914709] ? find_get_entry+0x19/0xc0
[ 60.915291] ? copyout+0x22/0x30
[ 60.915794] ? copy_page_to_iter+0x10c/0x2f0
[ 60.916432] ? generic_file_read_iter+0x44a/0x9c0
[ 60.917132] ? do_mmap+0x3c9/0x5e0
[ 60.917670] ? __vfs_read+0x100/0x170
[ 60.918234] ? vfs_read+0x89/0x130
[ 60.918770] ? ksys_read+0x52/0xc0
[ 60.919308] ? do_syscall_64+0x55/0xf0
[ 60.919885] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 60.920673] Code: 48 0f b3 1c 25 da 10 02 00 eb 05 49 0f ba ee 3f 41 0f 22 de e9 eb f7 ff ff fc 56 48 8b 74 24 08 48 89 7c 24 08 52 31 d2 51 31 c9 <50> 41 50 45 31 c0 41 51 45 31 c9 41 52 45 31 d2 41 53 45 31 db
[ 60.923194] RIP: error_entry+0x12/0x100 RSP: ffffb050000d0000
[ 60.924008] ---[ end trace 8a1ab620615a934d ]---
[ 60.924693] Kernel panic - not syncing: Fatal exception
[ 60.925556] Kernel Offset: 0x34400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: ORC unwinder bad backtrace
2018-04-18 13:54 ORC unwinder bad backtrace Matthew Wilcox
@ 2018-04-18 15:45 ` Josh Poimboeuf
2018-05-01 14:28 ` Josh Poimboeuf
0 siblings, 1 reply; 3+ messages in thread
From: Josh Poimboeuf @ 2018-04-18 15:45 UTC (permalink / raw)
To: Matthew Wilcox; +Cc: x86, linux-kernel
On Wed, Apr 18, 2018 at 06:54:38AM -0700, Matthew Wilcox wrote:
> f81061192 <pte_clear.constprop.18>:
> ...
> ffffffff810611bf: 90 nop
> ffffffff810611c0 <perf_trace_x86_exceptions>:
>
> I suspect an off-by-one error; you don't really mean to point to the
> byte before perf_trace_x86_exception, you mean to point to byte 0 of
> perf_trace_x86_exception.
>
> I'm going to archive up this compilation in case there's anything useful
> I can extract for you from it later.
Thanks for reporting this. So there are really two issues:
1) The question marks mean the ORC unwinder got confused (and had to
fall back to the crude "just print all text addresses on the stack").
This is the real issue.
2) As you found, what should be "perf_trace_x86_exceptions+0x0" is
actually printed as "pte_clear.constprop.18+0x2e". I don't think
this is fixable, because this is printed by the oops fallback code
which just blindly prints out all the text addresses it finds on the
stack when the unwinder fails. It can't know whether the address was
a call return address (the usual case) or was something else (in this
case I suspect it's just a function pointer which just happens to be
on the stack), so it assumes the former, and prints it accordingly.
This isn't fixable per se -- but it will be "fixed" when we fix #1,
which will give a deterministic stack trace instead of using the dumb
fallback code.
Is it possible for you to copy the vmlinux somewhere? That would be the
easiest option for debugging.
Otherwise I may ask for some specifics for you to gather from it.
Is it recreatable? Once I come up with a fix, it would be helpful to
test with the same scenario.
Also has the root cause of the stack recursion been found? It looks
like the perf_trace_x86_exceptions() tracepoint code is doing something
bad.
--
Josh
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: ORC unwinder bad backtrace
2018-04-18 15:45 ` Josh Poimboeuf
@ 2018-05-01 14:28 ` Josh Poimboeuf
0 siblings, 0 replies; 3+ messages in thread
From: Josh Poimboeuf @ 2018-05-01 14:28 UTC (permalink / raw)
To: Matthew Wilcox; +Cc: x86, linux-kernel
On Wed, Apr 18, 2018 at 10:45:48AM -0500, Josh Poimboeuf wrote:
> On Wed, Apr 18, 2018 at 06:54:38AM -0700, Matthew Wilcox wrote:
> > f81061192 <pte_clear.constprop.18>:
> > ...
> > ffffffff810611bf: 90 nop
> > ffffffff810611c0 <perf_trace_x86_exceptions>:
> >
> > I suspect an off-by-one error; you don't really mean to point to the
> > byte before perf_trace_x86_exception, you mean to point to byte 0 of
> > perf_trace_x86_exception.
> >
> > I'm going to archive up this compilation in case there's anything useful
> > I can extract for you from it later.
>
> Thanks for reporting this. So there are really two issues:
>
> 1) The question marks mean the ORC unwinder got confused (and had to
> fall back to the crude "just print all text addresses on the stack").
> This is the real issue.
>
> 2) As you found, what should be "perf_trace_x86_exceptions+0x0" is
> actually printed as "pte_clear.constprop.18+0x2e". I don't think
> this is fixable, because this is printed by the oops fallback code
> which just blindly prints out all the text addresses it finds on the
> stack when the unwinder fails. It can't know whether the address was
> a call return address (the usual case) or was something else (in this
> case I suspect it's just a function pointer which just happens to be
> on the stack), so it assumes the former, and prints it accordingly.
> This isn't fixable per se -- but it will be "fixed" when we fix #1,
> which will give a deterministic stack trace instead of using the dumb
> fallback code.
>
> Is it possible for you to copy the vmlinux somewhere? That would be the
> easiest option for debugging.
>
> Otherwise I may ask for some specifics for you to gather from it.
>
> Is it recreatable? Once I come up with a fix, it would be helpful to
> test with the same scenario.
>
> Also has the root cause of the stack recursion been found? It looks
> like the perf_trace_x86_exceptions() tracepoint code is doing something
> bad.
Matthew, ping?
--
Josh
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2018-05-01 14:28 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-18 13:54 ORC unwinder bad backtrace Matthew Wilcox
2018-04-18 15:45 ` Josh Poimboeuf
2018-05-01 14:28 ` Josh Poimboeuf
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).