* [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space @ 2017-01-23 9:18 Marcin Nowakowski 2017-01-23 9:18 ` Marcin Nowakowski ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: Marcin Nowakowski @ 2017-01-23 9:18 UTC (permalink / raw) To: Ralf Baechle; +Cc: James Hogan, linux-mips With certain EVA configurations it is possible for the kernel address space to overlap user address space, which allows the user to set watchpoints on kernel addresses via ptrace. If a watchpoint is set in the watch exception handling code (after exception level has been cleared) then the system will hang in an infinite loop when hitting a watchpoint while trying to process it. To prevent that simply disallow placing any watchpoints at addresses above start of kernel that overlap userspace. Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com> --- This supersedes "protect watchpoint handling code from setting watchpoints" which originally would only protect part of the kernel code most vulnerable. However, that change was incomplete and it is really difficult to ensure all required sections are properly guarded. Having selective guards on parts of the kernel address space could also be used as a method to circumvent KASLR. --- arch/mips/kernel/ptrace.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/mips/kernel/ptrace.c b/arch/mips/kernel/ptrace.c index c8ba260..7b87493 100644 --- a/arch/mips/kernel/ptrace.c +++ b/arch/mips/kernel/ptrace.c @@ -253,6 +253,11 @@ int ptrace_set_watch_regs(struct task_struct *child, #ifdef CONFIG_32BIT if (lt[i] & __UA_LIMIT) return -EINVAL; + +#ifdef CONFIG_EVA + if (lt[i] >= PAGE_OFFSET) + return -EINVAL; +#endif /* CONFIG_EVA */ #else if (test_tsk_thread_flag(child, TIF_32BIT_ADDR)) { if (lt[i] & 0xffffffff80000000UL) -- 2.7.4 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-23 9:18 [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Marcin Nowakowski @ 2017-01-23 9:18 ` Marcin Nowakowski 2017-01-23 9:18 ` [PATCH v2 2/2] MIPS: ptrace: disable watchpoints if hit in kernel mode Marcin Nowakowski 2017-01-24 17:09 ` [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Maciej W. Rozycki 2 siblings, 0 replies; 16+ messages in thread From: Marcin Nowakowski @ 2017-01-23 9:18 UTC (permalink / raw) To: Ralf Baechle; +Cc: James Hogan, linux-mips With certain EVA configurations it is possible for the kernel address space to overlap user address space, which allows the user to set watchpoints on kernel addresses via ptrace. If a watchpoint is set in the watch exception handling code (after exception level has been cleared) then the system will hang in an infinite loop when hitting a watchpoint while trying to process it. To prevent that simply disallow placing any watchpoints at addresses above start of kernel that overlap userspace. Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com> --- This supersedes "protect watchpoint handling code from setting watchpoints" which originally would only protect part of the kernel code most vulnerable. However, that change was incomplete and it is really difficult to ensure all required sections are properly guarded. Having selective guards on parts of the kernel address space could also be used as a method to circumvent KASLR. --- arch/mips/kernel/ptrace.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/mips/kernel/ptrace.c b/arch/mips/kernel/ptrace.c index c8ba260..7b87493 100644 --- a/arch/mips/kernel/ptrace.c +++ b/arch/mips/kernel/ptrace.c @@ -253,6 +253,11 @@ int ptrace_set_watch_regs(struct task_struct *child, #ifdef CONFIG_32BIT if (lt[i] & __UA_LIMIT) return -EINVAL; + +#ifdef CONFIG_EVA + if (lt[i] >= PAGE_OFFSET) + return -EINVAL; +#endif /* CONFIG_EVA */ #else if (test_tsk_thread_flag(child, TIF_32BIT_ADDR)) { if (lt[i] & 0xffffffff80000000UL) -- 2.7.4 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 2/2] MIPS: ptrace: disable watchpoints if hit in kernel mode 2017-01-23 9:18 [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Marcin Nowakowski 2017-01-23 9:18 ` Marcin Nowakowski @ 2017-01-23 9:18 ` Marcin Nowakowski 2017-01-23 9:18 ` Marcin Nowakowski 2017-01-24 17:09 ` [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Maciej W. Rozycki 2 siblings, 1 reply; 16+ messages in thread From: Marcin Nowakowski @ 2017-01-23 9:18 UTC (permalink / raw) To: Ralf Baechle; +Cc: James Hogan, linux-mips If a watchpoint is hit when in kernel mode it is possible for the system to end up in an infinite loop processing the watchpoint. This can happen if a user sets a watchpoint in the kernel addess space (which is possible in certain EVA configurations) or if a user sets a watchpoint in a user area accessed directly by the kernel (eg. a user buffer accessed via a syscall). To prevent the infinite loop ensure that the watchpoint was hit in userspace, and clear the watchpoint registers otherwise. As this change could mean that a watchpoint is not hit when it should be (when returning to the interrupted traced task on exception exit), the resume_userspace path needs to be extended to conditionally restore the watchpoint configuration. If a task switch occurs when returning to userspace, the watchpoints will be restored in a typical way in the switch_to() handler. Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com> --- arch/mips/kernel/entry.S | 9 ++++++++- arch/mips/kernel/traps.c | 2 +- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S index 7791840..1982fc6 100644 --- a/arch/mips/kernel/entry.S +++ b/arch/mips/kernel/entry.S @@ -55,7 +55,14 @@ resume_userspace: LONG_L a2, TI_FLAGS($28) # current->work andi t0, a2, _TIF_WORK_MASK # (ignoring syscall_trace) bnez t0, work_pending - j restore_all +#ifdef CONFIG_HARDWARE_WATCHPOINTS + li t0, _TIF_LOAD_WATCH + and t0, a2, t0 + beqz t0, 1f + PTR_L a0, TI_TASK($28) + jal mips_install_watch_registers +#endif +1: j restore_all #ifdef CONFIG_PREEMPT resume_kernel: diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c index 6c7f9d7..5371cb0 100644 --- a/arch/mips/kernel/traps.c +++ b/arch/mips/kernel/traps.c @@ -1525,7 +1525,7 @@ asmlinkage void do_watch(struct pt_regs *regs) * their values and send SIGTRAP. Otherwise another thread * left the registers set, clear them and continue. */ - if (test_tsk_thread_flag(current, TIF_LOAD_WATCH)) { + if (user_mode(regs) && test_tsk_thread_flag(current, TIF_LOAD_WATCH)) { mips_read_watch_registers(); local_irq_enable(); force_sig_info(SIGTRAP, &info, current); -- 2.7.4 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 2/2] MIPS: ptrace: disable watchpoints if hit in kernel mode 2017-01-23 9:18 ` [PATCH v2 2/2] MIPS: ptrace: disable watchpoints if hit in kernel mode Marcin Nowakowski @ 2017-01-23 9:18 ` Marcin Nowakowski 0 siblings, 0 replies; 16+ messages in thread From: Marcin Nowakowski @ 2017-01-23 9:18 UTC (permalink / raw) To: Ralf Baechle; +Cc: James Hogan, linux-mips If a watchpoint is hit when in kernel mode it is possible for the system to end up in an infinite loop processing the watchpoint. This can happen if a user sets a watchpoint in the kernel addess space (which is possible in certain EVA configurations) or if a user sets a watchpoint in a user area accessed directly by the kernel (eg. a user buffer accessed via a syscall). To prevent the infinite loop ensure that the watchpoint was hit in userspace, and clear the watchpoint registers otherwise. As this change could mean that a watchpoint is not hit when it should be (when returning to the interrupted traced task on exception exit), the resume_userspace path needs to be extended to conditionally restore the watchpoint configuration. If a task switch occurs when returning to userspace, the watchpoints will be restored in a typical way in the switch_to() handler. Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com> --- arch/mips/kernel/entry.S | 9 ++++++++- arch/mips/kernel/traps.c | 2 +- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S index 7791840..1982fc6 100644 --- a/arch/mips/kernel/entry.S +++ b/arch/mips/kernel/entry.S @@ -55,7 +55,14 @@ resume_userspace: LONG_L a2, TI_FLAGS($28) # current->work andi t0, a2, _TIF_WORK_MASK # (ignoring syscall_trace) bnez t0, work_pending - j restore_all +#ifdef CONFIG_HARDWARE_WATCHPOINTS + li t0, _TIF_LOAD_WATCH + and t0, a2, t0 + beqz t0, 1f + PTR_L a0, TI_TASK($28) + jal mips_install_watch_registers +#endif +1: j restore_all #ifdef CONFIG_PREEMPT resume_kernel: diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c index 6c7f9d7..5371cb0 100644 --- a/arch/mips/kernel/traps.c +++ b/arch/mips/kernel/traps.c @@ -1525,7 +1525,7 @@ asmlinkage void do_watch(struct pt_regs *regs) * their values and send SIGTRAP. Otherwise another thread * left the registers set, clear them and continue. */ - if (test_tsk_thread_flag(current, TIF_LOAD_WATCH)) { + if (user_mode(regs) && test_tsk_thread_flag(current, TIF_LOAD_WATCH)) { mips_read_watch_registers(); local_irq_enable(); force_sig_info(SIGTRAP, &info, current); -- 2.7.4 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-23 9:18 [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Marcin Nowakowski 2017-01-23 9:18 ` Marcin Nowakowski 2017-01-23 9:18 ` [PATCH v2 2/2] MIPS: ptrace: disable watchpoints if hit in kernel mode Marcin Nowakowski @ 2017-01-24 17:09 ` Maciej W. Rozycki 2017-01-24 17:09 ` Maciej W. Rozycki 2017-01-24 18:54 ` James Hogan 2 siblings, 2 replies; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-24 17:09 UTC (permalink / raw) To: Marcin Nowakowski; +Cc: Ralf Baechle, James Hogan, linux-mips On Mon, 23 Jan 2017, Marcin Nowakowski wrote: > With certain EVA configurations it is possible for the kernel address > space to overlap user address space, which allows the user to set > watchpoints on kernel addresses via ptrace. > > If a watchpoint is set in the watch exception handling code (after > exception level has been cleared) then the system will hang in an > infinite loop when hitting a watchpoint while trying to process it. > > To prevent that simply disallow placing any watchpoints at addresses > above start of kernel that overlap userspace. This can be severely crippling for user debugging. Is there no better way? Can't for example the low-level exception handling entry/exit code be moved out of the way of the EVA overlap range and then all watchpoints masked for the duration of kernel mode execution? This would be quite expensive, however it could only be executed if a task flag indicates watchpoints are being used. Alternatively perhaps we could clobber CP0.EntryHi.ASID, at least temporarily; that would be cheaper. Overall I think this situation is asking for a watchpoint flag to be added to inhibit hits in the kernel mode in hardware; for completeness this probably actually ought to be a field to cover the kernel, supervisor and user modes separately -- either a plain bitmask for arbitrary control or an encoded value similar to CP0.Status.KSU which would indicate the most privileged mode to accept a watchpoint in. I had a recollection of such a facility already being available for JTAG debugging, but I can't track it down in the specification, so perhaps it was for another architecture and it would be completely new for ours. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 17:09 ` [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Maciej W. Rozycki @ 2017-01-24 17:09 ` Maciej W. Rozycki 2017-01-24 18:54 ` James Hogan 1 sibling, 0 replies; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-24 17:09 UTC (permalink / raw) To: Marcin Nowakowski; +Cc: Ralf Baechle, James Hogan, linux-mips On Mon, 23 Jan 2017, Marcin Nowakowski wrote: > With certain EVA configurations it is possible for the kernel address > space to overlap user address space, which allows the user to set > watchpoints on kernel addresses via ptrace. > > If a watchpoint is set in the watch exception handling code (after > exception level has been cleared) then the system will hang in an > infinite loop when hitting a watchpoint while trying to process it. > > To prevent that simply disallow placing any watchpoints at addresses > above start of kernel that overlap userspace. This can be severely crippling for user debugging. Is there no better way? Can't for example the low-level exception handling entry/exit code be moved out of the way of the EVA overlap range and then all watchpoints masked for the duration of kernel mode execution? This would be quite expensive, however it could only be executed if a task flag indicates watchpoints are being used. Alternatively perhaps we could clobber CP0.EntryHi.ASID, at least temporarily; that would be cheaper. Overall I think this situation is asking for a watchpoint flag to be added to inhibit hits in the kernel mode in hardware; for completeness this probably actually ought to be a field to cover the kernel, supervisor and user modes separately -- either a plain bitmask for arbitrary control or an encoded value similar to CP0.Status.KSU which would indicate the most privileged mode to accept a watchpoint in. I had a recollection of such a facility already being available for JTAG debugging, but I can't track it down in the specification, so perhaps it was for another architecture and it would be completely new for ours. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 17:09 ` [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Maciej W. Rozycki 2017-01-24 17:09 ` Maciej W. Rozycki @ 2017-01-24 18:54 ` James Hogan 2017-01-24 18:54 ` James Hogan 2017-01-24 20:52 ` Maciej W. Rozycki 1 sibling, 2 replies; 16+ messages in thread From: James Hogan @ 2017-01-24 18:54 UTC (permalink / raw) To: Maciej W. Rozycki; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips [-- Attachment #1: Type: text/plain, Size: 3083 bytes --] Hi Maciej, On Tue, Jan 24, 2017 at 05:09:32PM +0000, Maciej W. Rozycki wrote: > On Mon, 23 Jan 2017, Marcin Nowakowski wrote: > > > With certain EVA configurations it is possible for the kernel address > > space to overlap user address space, which allows the user to set > > watchpoints on kernel addresses via ptrace. > > > > If a watchpoint is set in the watch exception handling code (after > > exception level has been cleared) then the system will hang in an > > infinite loop when hitting a watchpoint while trying to process it. > > > > To prevent that simply disallow placing any watchpoints at addresses > > above start of kernel that overlap userspace. > > This can be severely crippling for user debugging. Is there no better > way? See the v1 patches, but as you say below, an architectural fix is vastly preferable. Other proposed solutions have so far added complexity and fragility, while struggling to completely plug the original problem. > > Can't for example the low-level exception handling entry/exit code be > moved out of the way of the EVA overlap range and then all watchpoints > masked for the duration of kernel mode execution? This would be quite > expensive, however it could only be executed if a task flag indicates > watchpoints are being used. That doesn't cover data watches. RAM would still need accessing, e.g. to save/restore the watch state from the thread context, or even to read the task flag, and stack accesses in C code. The only safe way for it to work would be to somehow disable or inhibit watchpoints before clearing EXL, and re-enable them after setting EXL, though you'd still get a loop of deferred watchpoints if it hit on the way out to user mode unless cleared at the last moment before ERET. > Alternatively perhaps we could clobber > CP0.EntryHi.ASID, at least temporarily; that would be cheaper. Kernel mode still needs to access the user address space. Alternatively we could set WatchHi.ASID to a reserved one, and only clear/set the WatchHi.G bit (to bypass the ASID match) at the first/last moment while EXL=1. It still wouldn't protect against code watches around there exposing the kernel address of that code by the resulting hang though, so would need to move the ebase out of the overlap range too (which would have to be platform specific). Cheers James > > Overall I think this situation is asking for a watchpoint flag to be > added to inhibit hits in the kernel mode in hardware; for completeness > this probably actually ought to be a field to cover the kernel, supervisor > and user modes separately -- either a plain bitmask for arbitrary control > or an encoded value similar to CP0.Status.KSU which would indicate the > most privileged mode to accept a watchpoint in. > > I had a recollection of such a facility already being available for JTAG > debugging, but I can't track it down in the specification, so perhaps it > was for another architecture and it would be completely new for ours. > > Maciej [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 801 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 18:54 ` James Hogan @ 2017-01-24 18:54 ` James Hogan 2017-01-24 20:52 ` Maciej W. Rozycki 1 sibling, 0 replies; 16+ messages in thread From: James Hogan @ 2017-01-24 18:54 UTC (permalink / raw) To: Maciej W. Rozycki; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips [-- Attachment #1: Type: text/plain, Size: 3083 bytes --] Hi Maciej, On Tue, Jan 24, 2017 at 05:09:32PM +0000, Maciej W. Rozycki wrote: > On Mon, 23 Jan 2017, Marcin Nowakowski wrote: > > > With certain EVA configurations it is possible for the kernel address > > space to overlap user address space, which allows the user to set > > watchpoints on kernel addresses via ptrace. > > > > If a watchpoint is set in the watch exception handling code (after > > exception level has been cleared) then the system will hang in an > > infinite loop when hitting a watchpoint while trying to process it. > > > > To prevent that simply disallow placing any watchpoints at addresses > > above start of kernel that overlap userspace. > > This can be severely crippling for user debugging. Is there no better > way? See the v1 patches, but as you say below, an architectural fix is vastly preferable. Other proposed solutions have so far added complexity and fragility, while struggling to completely plug the original problem. > > Can't for example the low-level exception handling entry/exit code be > moved out of the way of the EVA overlap range and then all watchpoints > masked for the duration of kernel mode execution? This would be quite > expensive, however it could only be executed if a task flag indicates > watchpoints are being used. That doesn't cover data watches. RAM would still need accessing, e.g. to save/restore the watch state from the thread context, or even to read the task flag, and stack accesses in C code. The only safe way for it to work would be to somehow disable or inhibit watchpoints before clearing EXL, and re-enable them after setting EXL, though you'd still get a loop of deferred watchpoints if it hit on the way out to user mode unless cleared at the last moment before ERET. > Alternatively perhaps we could clobber > CP0.EntryHi.ASID, at least temporarily; that would be cheaper. Kernel mode still needs to access the user address space. Alternatively we could set WatchHi.ASID to a reserved one, and only clear/set the WatchHi.G bit (to bypass the ASID match) at the first/last moment while EXL=1. It still wouldn't protect against code watches around there exposing the kernel address of that code by the resulting hang though, so would need to move the ebase out of the overlap range too (which would have to be platform specific). Cheers James > > Overall I think this situation is asking for a watchpoint flag to be > added to inhibit hits in the kernel mode in hardware; for completeness > this probably actually ought to be a field to cover the kernel, supervisor > and user modes separately -- either a plain bitmask for arbitrary control > or an encoded value similar to CP0.Status.KSU which would indicate the > most privileged mode to accept a watchpoint in. > > I had a recollection of such a facility already being available for JTAG > debugging, but I can't track it down in the specification, so perhaps it > was for another architecture and it would be completely new for ours. > > Maciej [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 801 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 18:54 ` James Hogan 2017-01-24 18:54 ` James Hogan @ 2017-01-24 20:52 ` Maciej W. Rozycki 2017-01-24 20:52 ` Maciej W. Rozycki 2017-01-24 22:05 ` James Hogan 1 sibling, 2 replies; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-24 20:52 UTC (permalink / raw) To: James Hogan; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips On Tue, 24 Jan 2017, James Hogan wrote: > > Can't for example the low-level exception handling entry/exit code be > > moved out of the way of the EVA overlap range and then all watchpoints > > masked for the duration of kernel mode execution? This would be quite > > expensive, however it could only be executed if a task flag indicates > > watchpoints are being used. > > That doesn't cover data watches. RAM would still need accessing, e.g. to > save/restore the watch state from the thread context, or even to read > the task flag, and stack accesses in C code. All the critical data structures would have to be outside the EVA overlap. > The only safe way for it to work would be to somehow disable or inhibit > watchpoints before clearing EXL, and re-enable them after setting EXL, > though you'd still get a loop of deferred watchpoints if it hit on the > way out to user mode unless cleared at the last moment before ERET. Ah, I forgot about CP0.Cause.WP -- is it not enough to clear the bit to have any pending exception cancelled? If so (and the architecture manual is actually clear that it is) then it looks like we have a solution and we don't have to place any code or data specially, although it'll have to be carefully coded. > > Alternatively perhaps we could clobber > > CP0.EntryHi.ASID, at least temporarily; that would be cheaper. > > Kernel mode still needs to access the user address space. Sure, that's why it would have to be temporary. Low-level exception entry/exit code is not supposed to have a need to access user memory. So we can put aside a certain ASID value, say 0 (for easy pasting with INS from $0), and never use it for a real context. Then it can be cleared right away at the general exception entry point if EVA is in use, say: <d>mfc0 $k0, $10 <d>ins $k0, $zero, 0, 10 <d>mtc0 $k0, $10 (there'll be a hazard here, but we can clear it later on if needed). There is no neeed to save the old ASID as we can retrieve the original from our data structures. Then we can proceed with the usual switch to the kernel mode, switching stacks, saving registers, etc. We can then check CP0.Cause.WP and stash it away for further processing if needed (though discarding it would I think be the usual if not only choice) and clear, with a hazard barrier, right before CP0.Status.EXL is cleared. Now that we're in the regular kernel mode, with ASID still set to 0, we can check if process tracing has been enabled and if so, then iterate over the watchpoints registers masking them all. At this point we can restore the correct ASID in CP0.EntryHi and proceed with the exception handler. And then we'd do the reverse in the exception epilogue, only restoring the ASID as the last instruction before final ERET. > Alternatively we could set WatchHi.ASID to a reserved one, and only > clear/set the WatchHi.G bit (to bypass the ASID match) at the first/last > moment while EXL=1. It still wouldn't protect against code watches > around there exposing the kernel address of that code by the resulting > hang though, so would need to move the ebase out of the overlap range > too (which would have to be platform specific). You'd still have to iterate over all WatchHi registers, a variable number up to 8 architecturally, which I think would be too expensive for the common exception path. Poking at ASID as I described above is just a couple of instructions at entry and exit, and the rest would only be done if tracing is active. Plus you don't actually have to move anything away, except from the final ERET, though likely not even that, owing to the delayed nature of an ASID update. So can you find a flaw in my proposal so far? We'll have to think about the TLB refill handler yet though. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 20:52 ` Maciej W. Rozycki @ 2017-01-24 20:52 ` Maciej W. Rozycki 2017-01-24 22:05 ` James Hogan 1 sibling, 0 replies; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-24 20:52 UTC (permalink / raw) To: James Hogan; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips On Tue, 24 Jan 2017, James Hogan wrote: > > Can't for example the low-level exception handling entry/exit code be > > moved out of the way of the EVA overlap range and then all watchpoints > > masked for the duration of kernel mode execution? This would be quite > > expensive, however it could only be executed if a task flag indicates > > watchpoints are being used. > > That doesn't cover data watches. RAM would still need accessing, e.g. to > save/restore the watch state from the thread context, or even to read > the task flag, and stack accesses in C code. All the critical data structures would have to be outside the EVA overlap. > The only safe way for it to work would be to somehow disable or inhibit > watchpoints before clearing EXL, and re-enable them after setting EXL, > though you'd still get a loop of deferred watchpoints if it hit on the > way out to user mode unless cleared at the last moment before ERET. Ah, I forgot about CP0.Cause.WP -- is it not enough to clear the bit to have any pending exception cancelled? If so (and the architecture manual is actually clear that it is) then it looks like we have a solution and we don't have to place any code or data specially, although it'll have to be carefully coded. > > Alternatively perhaps we could clobber > > CP0.EntryHi.ASID, at least temporarily; that would be cheaper. > > Kernel mode still needs to access the user address space. Sure, that's why it would have to be temporary. Low-level exception entry/exit code is not supposed to have a need to access user memory. So we can put aside a certain ASID value, say 0 (for easy pasting with INS from $0), and never use it for a real context. Then it can be cleared right away at the general exception entry point if EVA is in use, say: <d>mfc0 $k0, $10 <d>ins $k0, $zero, 0, 10 <d>mtc0 $k0, $10 (there'll be a hazard here, but we can clear it later on if needed). There is no neeed to save the old ASID as we can retrieve the original from our data structures. Then we can proceed with the usual switch to the kernel mode, switching stacks, saving registers, etc. We can then check CP0.Cause.WP and stash it away for further processing if needed (though discarding it would I think be the usual if not only choice) and clear, with a hazard barrier, right before CP0.Status.EXL is cleared. Now that we're in the regular kernel mode, with ASID still set to 0, we can check if process tracing has been enabled and if so, then iterate over the watchpoints registers masking them all. At this point we can restore the correct ASID in CP0.EntryHi and proceed with the exception handler. And then we'd do the reverse in the exception epilogue, only restoring the ASID as the last instruction before final ERET. > Alternatively we could set WatchHi.ASID to a reserved one, and only > clear/set the WatchHi.G bit (to bypass the ASID match) at the first/last > moment while EXL=1. It still wouldn't protect against code watches > around there exposing the kernel address of that code by the resulting > hang though, so would need to move the ebase out of the overlap range > too (which would have to be platform specific). You'd still have to iterate over all WatchHi registers, a variable number up to 8 architecturally, which I think would be too expensive for the common exception path. Poking at ASID as I described above is just a couple of instructions at entry and exit, and the rest would only be done if tracing is active. Plus you don't actually have to move anything away, except from the final ERET, though likely not even that, owing to the delayed nature of an ASID update. So can you find a flaw in my proposal so far? We'll have to think about the TLB refill handler yet though. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 20:52 ` Maciej W. Rozycki 2017-01-24 20:52 ` Maciej W. Rozycki @ 2017-01-24 22:05 ` James Hogan 2017-01-24 22:05 ` James Hogan 2017-01-24 23:07 ` Maciej W. Rozycki 1 sibling, 2 replies; 16+ messages in thread From: James Hogan @ 2017-01-24 22:05 UTC (permalink / raw) To: Maciej W. Rozycki; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips [-- Attachment #1: Type: text/plain, Size: 5026 bytes --] On Tue, Jan 24, 2017 at 08:52:22PM +0000, Maciej W. Rozycki wrote: > On Tue, 24 Jan 2017, James Hogan wrote: > > > > Can't for example the low-level exception handling entry/exit code be > > > moved out of the way of the EVA overlap range and then all watchpoints > > > masked for the duration of kernel mode execution? This would be quite > > > expensive, however it could only be executed if a task flag indicates > > > watchpoints are being used. > > > > That doesn't cover data watches. RAM would still need accessing, e.g. to > > save/restore the watch state from the thread context, or even to read > > the task flag, and stack accesses in C code. > > All the critical data structures would have to be outside the EVA > overlap. This in itself is awkward. If a SoC supports multiple RAM sizes, e.g. up to 2GB, you might want a single EVA kernel that could support both. Normally you could just go with a 2GB compatible layout (for the sake of argument, lets say RAM cached @ kernel VA 0x40000000 .. 0xBFFFFFFF, ignoring BEV overlays for now), but if less than 1GB is fitted then none of that RAM is outside of the user overlap range. > > > The only safe way for it to work would be to somehow disable or inhibit > > watchpoints before clearing EXL, and re-enable them after setting EXL, > > though you'd still get a loop of deferred watchpoints if it hit on the > > way out to user mode unless cleared at the last moment before ERET. > > Ah, I forgot about CP0.Cause.WP -- is it not enough to clear the bit to > have any pending exception cancelled? If so (and the architecture manual > is actually clear that it is) then it looks like we have a solution and we > don't have to place any code or data specially, although it'll have to be > carefully coded. > > > > Alternatively perhaps we could clobber > > > CP0.EntryHi.ASID, at least temporarily; that would be cheaper. > > > > Kernel mode still needs to access the user address space. > > Sure, that's why it would have to be temporary. Low-level exception > entry/exit code is not supposed to have a need to access user memory. Ah I see, really temporary. > > So we can put aside a certain ASID value, say 0 (for easy pasting with > INS from $0), and never use it for a real context. Then it can be cleared > right away at the general exception entry point if EVA is in use, say: > > <d>mfc0 $k0, $10 > <d>ins $k0, $zero, 0, 10 > <d>mtc0 $k0, $10 > > (there'll be a hazard here, but we can clear it later on if needed). > There is no neeed to save the old ASID as we can retrieve the original > from our data structures. > > Then we can proceed with the usual switch to the kernel mode, switching > stacks, saving registers, etc. We can then check CP0.Cause.WP and stash > it away for further processing if needed (though discarding it would I > think be the usual if not only choice) and clear, with a hazard barrier, > right before CP0.Status.EXL is cleared. > > Now that we're in the regular kernel mode, with ASID still set to 0, we > can check if process tracing has been enabled and if so, then iterate over > the watchpoints registers masking them all. At this point we can restore > the correct ASID in CP0.EntryHi and proceed with the exception handler. > > And then we'd do the reverse in the exception epilogue, only restoring > the ASID as the last instruction before final ERET. > > > Alternatively we could set WatchHi.ASID to a reserved one, and only > > clear/set the WatchHi.G bit (to bypass the ASID match) at the first/last > > moment while EXL=1. It still wouldn't protect against code watches > > around there exposing the kernel address of that code by the resulting > > hang though, so would need to move the ebase out of the overlap range > > too (which would have to be platform specific). > > You'd still have to iterate over all WatchHi registers, a variable number > up to 8 architecturally, which I think would be too expensive for the > common exception path. > > Poking at ASID as I described above is just a couple of instructions at > entry and exit, and the rest would only be done if tracing is active. > Plus you don't actually have to move anything away, except from the final > ERET, though likely not even that, owing to the delayed nature of an ASID > update. Probably, so long as you ignore QEMU. > > So can you find a flaw in my proposal so far? not a functional one. > We'll have to think about > the TLB refill handler yet though. A deferred watch from refill handler (e.g. page tables) would I think trigger an immediate watch exception on eret, and get cleared / ignored. It would probably make enough of a timing difference for userland to reliably detect (in order to probe where the process' page tables are in kernel virtual memory, to be able to mount a more successful attack given some other vulnerability). Cheers James [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 801 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 22:05 ` James Hogan @ 2017-01-24 22:05 ` James Hogan 2017-01-24 23:07 ` Maciej W. Rozycki 1 sibling, 0 replies; 16+ messages in thread From: James Hogan @ 2017-01-24 22:05 UTC (permalink / raw) To: Maciej W. Rozycki; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips [-- Attachment #1: Type: text/plain, Size: 5026 bytes --] On Tue, Jan 24, 2017 at 08:52:22PM +0000, Maciej W. Rozycki wrote: > On Tue, 24 Jan 2017, James Hogan wrote: > > > > Can't for example the low-level exception handling entry/exit code be > > > moved out of the way of the EVA overlap range and then all watchpoints > > > masked for the duration of kernel mode execution? This would be quite > > > expensive, however it could only be executed if a task flag indicates > > > watchpoints are being used. > > > > That doesn't cover data watches. RAM would still need accessing, e.g. to > > save/restore the watch state from the thread context, or even to read > > the task flag, and stack accesses in C code. > > All the critical data structures would have to be outside the EVA > overlap. This in itself is awkward. If a SoC supports multiple RAM sizes, e.g. up to 2GB, you might want a single EVA kernel that could support both. Normally you could just go with a 2GB compatible layout (for the sake of argument, lets say RAM cached @ kernel VA 0x40000000 .. 0xBFFFFFFF, ignoring BEV overlays for now), but if less than 1GB is fitted then none of that RAM is outside of the user overlap range. > > > The only safe way for it to work would be to somehow disable or inhibit > > watchpoints before clearing EXL, and re-enable them after setting EXL, > > though you'd still get a loop of deferred watchpoints if it hit on the > > way out to user mode unless cleared at the last moment before ERET. > > Ah, I forgot about CP0.Cause.WP -- is it not enough to clear the bit to > have any pending exception cancelled? If so (and the architecture manual > is actually clear that it is) then it looks like we have a solution and we > don't have to place any code or data specially, although it'll have to be > carefully coded. > > > > Alternatively perhaps we could clobber > > > CP0.EntryHi.ASID, at least temporarily; that would be cheaper. > > > > Kernel mode still needs to access the user address space. > > Sure, that's why it would have to be temporary. Low-level exception > entry/exit code is not supposed to have a need to access user memory. Ah I see, really temporary. > > So we can put aside a certain ASID value, say 0 (for easy pasting with > INS from $0), and never use it for a real context. Then it can be cleared > right away at the general exception entry point if EVA is in use, say: > > <d>mfc0 $k0, $10 > <d>ins $k0, $zero, 0, 10 > <d>mtc0 $k0, $10 > > (there'll be a hazard here, but we can clear it later on if needed). > There is no neeed to save the old ASID as we can retrieve the original > from our data structures. > > Then we can proceed with the usual switch to the kernel mode, switching > stacks, saving registers, etc. We can then check CP0.Cause.WP and stash > it away for further processing if needed (though discarding it would I > think be the usual if not only choice) and clear, with a hazard barrier, > right before CP0.Status.EXL is cleared. > > Now that we're in the regular kernel mode, with ASID still set to 0, we > can check if process tracing has been enabled and if so, then iterate over > the watchpoints registers masking them all. At this point we can restore > the correct ASID in CP0.EntryHi and proceed with the exception handler. > > And then we'd do the reverse in the exception epilogue, only restoring > the ASID as the last instruction before final ERET. > > > Alternatively we could set WatchHi.ASID to a reserved one, and only > > clear/set the WatchHi.G bit (to bypass the ASID match) at the first/last > > moment while EXL=1. It still wouldn't protect against code watches > > around there exposing the kernel address of that code by the resulting > > hang though, so would need to move the ebase out of the overlap range > > too (which would have to be platform specific). > > You'd still have to iterate over all WatchHi registers, a variable number > up to 8 architecturally, which I think would be too expensive for the > common exception path. > > Poking at ASID as I described above is just a couple of instructions at > entry and exit, and the rest would only be done if tracing is active. > Plus you don't actually have to move anything away, except from the final > ERET, though likely not even that, owing to the delayed nature of an ASID > update. Probably, so long as you ignore QEMU. > > So can you find a flaw in my proposal so far? not a functional one. > We'll have to think about > the TLB refill handler yet though. A deferred watch from refill handler (e.g. page tables) would I think trigger an immediate watch exception on eret, and get cleared / ignored. It would probably make enough of a timing difference for userland to reliably detect (in order to probe where the process' page tables are in kernel virtual memory, to be able to mount a more successful attack given some other vulnerability). Cheers James [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 801 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 22:05 ` James Hogan 2017-01-24 22:05 ` James Hogan @ 2017-01-24 23:07 ` Maciej W. Rozycki 2017-01-24 23:07 ` Maciej W. Rozycki 2017-01-25 14:39 ` Maciej W. Rozycki 1 sibling, 2 replies; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-24 23:07 UTC (permalink / raw) To: James Hogan; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips On Tue, 24 Jan 2017, James Hogan wrote: > > All the critical data structures would have to be outside the EVA > > overlap. > > This in itself is awkward. If a SoC supports multiple RAM sizes, e.g. > up to 2GB, you might want a single EVA kernel that could support both. > Normally you could just go with a 2GB compatible layout (for the sake of > argument, lets say RAM cached @ kernel VA 0x40000000 .. 0xBFFFFFFF, > ignoring BEV overlays for now), but if less than 1GB is fitted then none > of that RAM is outside of the user overlap range. Well, the kernel is in control of user mappings and can take a piece of the virtual address space away for internal use. At worst the kernel can map the necessary stuff in KSEG2 with a wired TLB entry. I agree this is far from being pretty though and do hope my other proposal turns out feasible. > > Poking at ASID as I described above is just a couple of instructions at > > entry and exit, and the rest would only be done if tracing is active. > > Plus you don't actually have to move anything away, except from the final > > ERET, though likely not even that, owing to the delayed nature of an ASID > > update. > > Probably, so long as you ignore QEMU. We can paper it over in QEMU I suppose -- we're not supposed to be interrupted at EXL and with a linear execution flow any sane hardware will have already fetched the following ERET by the time the immediately preceding MTC0 has been retired. We can cache-line-align the instruction pair to avoid surprises on real hardware too. > > > > So can you find a flaw in my proposal so far? > > not a functional one. Good! > > We'll have to think about > > the TLB refill handler yet though. > > A deferred watch from refill handler (e.g. page tables) would I think > trigger an immediate watch exception on eret, and get cleared / ignored. > It would probably make enough of a timing difference for userland to > reliably detect (in order to probe where the process' page tables are in > kernel virtual memory, to be able to mount a more successful attack > given some other vulnerability). I feel uneasy about it: if a watchpoint happens to be badly placed (not necessarily deliberately), then this could become a serious performance hit for the whole system (and if deliberately, then possibly a security concern). However I think we can get away quite easily again, by clearing CP0.Cause.WP unconditionally at the exit from the refill handler -- there's nothing of interest throughout the handler for watchpoints and we run at EXL until completion. Unfortunately some other writeable bits have been allocated in CP0.Cause, specifically DC and especially IV, so we can't just do: mtc0 $13, $zero However if we can prove that we won't need the IP[1:0] bits in scenarios that involve a TLB refill, then we could just quickly do a short sequence, say: lui $k0, 1 << 23 mtc0 $13, $k0 eret Otherwise we'll have to do a full RMW sequence; fortunately a single INS from $0 will do here again to clear CP0.Cause.WP and keep the remaining bits. Maybe we could do just the same in the regular exception epilogue to avoid the dependency on a hazard (and consequently an issue with QEMU). All these extra operations do cost some performance, but at least the latency is constant, so very predictable, which I believe is important in some use cases. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 23:07 ` Maciej W. Rozycki @ 2017-01-24 23:07 ` Maciej W. Rozycki 2017-01-25 14:39 ` Maciej W. Rozycki 1 sibling, 0 replies; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-24 23:07 UTC (permalink / raw) To: James Hogan; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips On Tue, 24 Jan 2017, James Hogan wrote: > > All the critical data structures would have to be outside the EVA > > overlap. > > This in itself is awkward. If a SoC supports multiple RAM sizes, e.g. > up to 2GB, you might want a single EVA kernel that could support both. > Normally you could just go with a 2GB compatible layout (for the sake of > argument, lets say RAM cached @ kernel VA 0x40000000 .. 0xBFFFFFFF, > ignoring BEV overlays for now), but if less than 1GB is fitted then none > of that RAM is outside of the user overlap range. Well, the kernel is in control of user mappings and can take a piece of the virtual address space away for internal use. At worst the kernel can map the necessary stuff in KSEG2 with a wired TLB entry. I agree this is far from being pretty though and do hope my other proposal turns out feasible. > > Poking at ASID as I described above is just a couple of instructions at > > entry and exit, and the rest would only be done if tracing is active. > > Plus you don't actually have to move anything away, except from the final > > ERET, though likely not even that, owing to the delayed nature of an ASID > > update. > > Probably, so long as you ignore QEMU. We can paper it over in QEMU I suppose -- we're not supposed to be interrupted at EXL and with a linear execution flow any sane hardware will have already fetched the following ERET by the time the immediately preceding MTC0 has been retired. We can cache-line-align the instruction pair to avoid surprises on real hardware too. > > > > So can you find a flaw in my proposal so far? > > not a functional one. Good! > > We'll have to think about > > the TLB refill handler yet though. > > A deferred watch from refill handler (e.g. page tables) would I think > trigger an immediate watch exception on eret, and get cleared / ignored. > It would probably make enough of a timing difference for userland to > reliably detect (in order to probe where the process' page tables are in > kernel virtual memory, to be able to mount a more successful attack > given some other vulnerability). I feel uneasy about it: if a watchpoint happens to be badly placed (not necessarily deliberately), then this could become a serious performance hit for the whole system (and if deliberately, then possibly a security concern). However I think we can get away quite easily again, by clearing CP0.Cause.WP unconditionally at the exit from the refill handler -- there's nothing of interest throughout the handler for watchpoints and we run at EXL until completion. Unfortunately some other writeable bits have been allocated in CP0.Cause, specifically DC and especially IV, so we can't just do: mtc0 $13, $zero However if we can prove that we won't need the IP[1:0] bits in scenarios that involve a TLB refill, then we could just quickly do a short sequence, say: lui $k0, 1 << 23 mtc0 $13, $k0 eret Otherwise we'll have to do a full RMW sequence; fortunately a single INS from $0 will do here again to clear CP0.Cause.WP and keep the remaining bits. Maybe we could do just the same in the regular exception epilogue to avoid the dependency on a hazard (and consequently an issue with QEMU). All these extra operations do cost some performance, but at least the latency is constant, so very predictable, which I believe is important in some use cases. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-24 23:07 ` Maciej W. Rozycki 2017-01-24 23:07 ` Maciej W. Rozycki @ 2017-01-25 14:39 ` Maciej W. Rozycki 2017-01-25 14:39 ` Maciej W. Rozycki 1 sibling, 1 reply; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-25 14:39 UTC (permalink / raw) To: James Hogan; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips On Tue, 24 Jan 2017, Maciej W. Rozycki wrote: > However if we can prove that we won't need the IP[1:0] bits in scenarios > that involve a TLB refill, then we could just quickly do a short sequence, > say: > > lui $k0, 1 << 23 Umm, thinko here, this obviously has to be: li $k0, 1 << 23 or alternatively: lui $k0, 1 << (23 - 16) (GAS will emit a single LUI instruction in either case). > mtc0 $13, $k0 > eret > > Otherwise we'll have to do a full RMW sequence; fortunately a single INS > from $0 will do here again to clear CP0.Cause.WP and keep the remaining > bits. Maybe we could do just the same in the regular exception epilogue > to avoid the dependency on a hazard (and consequently an issue with QEMU). Of course a similar hazard is still there, so the same precautions apply. Also I think we do need to clear CP0.Cause.WP in all cases before ERET, including the various exception fast paths, such as in the TLBL/TLBS/TLBM handlers, which also means we don't have to fiddle with CP0.EntryHi.ASID in handler execution paths that run at EXL entirely to completion. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space 2017-01-25 14:39 ` Maciej W. Rozycki @ 2017-01-25 14:39 ` Maciej W. Rozycki 0 siblings, 0 replies; 16+ messages in thread From: Maciej W. Rozycki @ 2017-01-25 14:39 UTC (permalink / raw) To: James Hogan; +Cc: Marcin Nowakowski, Ralf Baechle, linux-mips On Tue, 24 Jan 2017, Maciej W. Rozycki wrote: > However if we can prove that we won't need the IP[1:0] bits in scenarios > that involve a TLB refill, then we could just quickly do a short sequence, > say: > > lui $k0, 1 << 23 Umm, thinko here, this obviously has to be: li $k0, 1 << 23 or alternatively: lui $k0, 1 << (23 - 16) (GAS will emit a single LUI instruction in either case). > mtc0 $13, $k0 > eret > > Otherwise we'll have to do a full RMW sequence; fortunately a single INS > from $0 will do here again to clear CP0.Cause.WP and keep the remaining > bits. Maybe we could do just the same in the regular exception epilogue > to avoid the dependency on a hazard (and consequently an issue with QEMU). Of course a similar hazard is still there, so the same precautions apply. Also I think we do need to clear CP0.Cause.WP in all cases before ERET, including the various exception fast paths, such as in the TLBL/TLBS/TLBM handlers, which also means we don't have to fiddle with CP0.EntryHi.ASID in handler execution paths that run at EXL entirely to completion. Maciej ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2017-01-25 14:39 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-01-23 9:18 [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Marcin Nowakowski 2017-01-23 9:18 ` Marcin Nowakowski 2017-01-23 9:18 ` [PATCH v2 2/2] MIPS: ptrace: disable watchpoints if hit in kernel mode Marcin Nowakowski 2017-01-23 9:18 ` Marcin Nowakowski 2017-01-24 17:09 ` [PATCH v2 1/2] MIPS: ptrace: disallow setting watchpoints in kernel address space Maciej W. Rozycki 2017-01-24 17:09 ` Maciej W. Rozycki 2017-01-24 18:54 ` James Hogan 2017-01-24 18:54 ` James Hogan 2017-01-24 20:52 ` Maciej W. Rozycki 2017-01-24 20:52 ` Maciej W. Rozycki 2017-01-24 22:05 ` James Hogan 2017-01-24 22:05 ` James Hogan 2017-01-24 23:07 ` Maciej W. Rozycki 2017-01-24 23:07 ` Maciej W. Rozycki 2017-01-25 14:39 ` Maciej W. Rozycki 2017-01-25 14:39 ` Maciej W. Rozycki
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).