* KASAN: use-after-free Read in __dev_map_entry_free
@ 2018-06-20 15:19 syzbot
2019-04-02 20:03 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: syzbot @ 2018-06-20 15:19 UTC (permalink / raw)
To: ast, daniel, linux-kernel, netdev, syzkaller-bugs
Hello,
syzbot found the following crash on:
HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern..
git tree: bpf-next
console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000
kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1
dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com
==================================================================
BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365
[inline]
BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300
kernel/bpf/devmap.c:379
Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18
CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1b9/0x294 lib/dump_stack.c:113
print_address_description+0x6c/0x20b mm/kasan/report.c:256
kasan_report_error mm/kasan/report.c:354 [inline]
kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412
__asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
dev_map_flush_old kernel/bpf/devmap.c:365 [inline]
__dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379
__rcu_reclaim kernel/rcu/rcu.h:178 [inline]
rcu_do_batch kernel/rcu/tree.c:2558 [inline]
invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:2785 [inline]
rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802
__do_softirq+0x2e0/0xaf5 kernel/softirq.c:284
run_ksoftirqd+0x86/0x100 kernel/softirq.c:645
smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164
kthread+0x345/0x410 kernel/kthread.c:240
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
Allocated by task 6675:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553
kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620
kmalloc include/linux/slab.h:513 [inline]
kzalloc include/linux/slab.h:706 [inline]
dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102
find_and_alloc_map kernel/bpf/syscall.c:129 [inline]
map_create+0x393/0x1010 kernel/bpf/syscall.c:453
__do_sys_bpf kernel/bpf/syscall.c:2351 [inline]
__se_sys_bpf kernel/bpf/syscall.c:2328 [inline]
__x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328
do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
Freed by task 26:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
__kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521
kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
__cache_free mm/slab.c:3498 [inline]
kfree+0xd9/0x260 mm/slab.c:3813
dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191
bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262
process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153
worker_thread+0x181/0x13a0 kernel/workqueue.c:2296
kthread+0x345/0x410 kernel/kthread.c:240
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
The buggy address belongs to the object at ffff8801b8da37c0
which belongs to the cache kmalloc-512 of size 512
The buggy address is located 264 bytes inside of
512-byte region [ffff8801b8da37c0, ffff8801b8da39c0)
The buggy address belongs to the page:
page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940
index:0xffff8801b8da3540
flags: 0x2fffc0000000100(slab)
raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940
raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
==================================================================
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: KASAN: use-after-free Read in __dev_map_entry_free 2018-06-20 15:19 KASAN: use-after-free Read in __dev_map_entry_free syzbot @ 2019-04-02 20:03 ` Eric Dumazet 2019-04-04 3:59 ` Alexei Starovoitov 0 siblings, 1 reply; 8+ messages in thread From: Eric Dumazet @ 2019-04-02 20:03 UTC (permalink / raw) To: syzbot, ast, daniel, linux-kernel, netdev, syzkaller-bugs On 06/20/2018 08:19 AM, syzbot wrote: > Hello, > > syzbot found the following crash on: > > HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern.. > git tree: bpf-next > console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000 > kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1 > dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0 > compiler: gcc (GCC) 8.0.1 20180413 (experimental) > syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000 > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000 > > IMPORTANT: if you fix the bug, please add the following tag to the commit: > Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com > > ================================================================== > BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18 > > CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > Call Trace: > __dump_stack lib/dump_stack.c:77 [inline] > dump_stack+0x1b9/0x294 lib/dump_stack.c:113 > print_address_description+0x6c/0x20b mm/kasan/report.c:256 > kasan_report_error mm/kasan/report.c:354 [inline] > kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412 > __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 > dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > __rcu_reclaim kernel/rcu/rcu.h:178 [inline] > rcu_do_batch kernel/rcu/tree.c:2558 [inline] > invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline] > __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline] > rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802 > __do_softirq+0x2e0/0xaf5 kernel/softirq.c:284 > run_ksoftirqd+0x86/0x100 kernel/softirq.c:645 > smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164 > kthread+0x345/0x410 kernel/kthread.c:240 > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > > Allocated by task 6675: > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > set_track mm/kasan/kasan.c:460 [inline] > kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553 > kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620 > kmalloc include/linux/slab.h:513 [inline] > kzalloc include/linux/slab.h:706 [inline] > dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102 > find_and_alloc_map kernel/bpf/syscall.c:129 [inline] > map_create+0x393/0x1010 kernel/bpf/syscall.c:453 > __do_sys_bpf kernel/bpf/syscall.c:2351 [inline] > __se_sys_bpf kernel/bpf/syscall.c:2328 [inline] > __x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328 > do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290 > entry_SYSCALL_64_after_hwframe+0x49/0xbe > > Freed by task 26: > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > set_track mm/kasan/kasan.c:460 [inline] > __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 > kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 > __cache_free mm/slab.c:3498 [inline] > kfree+0xd9/0x260 mm/slab.c:3813 > dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191 > bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262 > process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153 > worker_thread+0x181/0x13a0 kernel/workqueue.c:2296 > kthread+0x345/0x410 kernel/kthread.c:240 > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > > The buggy address belongs to the object at ffff8801b8da37c0 > which belongs to the cache kmalloc-512 of size 512 > The buggy address is located 264 bytes inside of > 512-byte region [ffff8801b8da37c0, ffff8801b8da39c0) > The buggy address belongs to the page: > page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940 index:0xffff8801b8da3540 > flags: 0x2fffc0000000100(slab) > raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940 > raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000 > page dumped because: kasan: bad access detected > > Memory state around the buggy address: > ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb > ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ^ > ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc > ================================================================== > > > --- > This bug is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this bug report. See: > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. > syzbot can test patches for this bug, for details see: > https://goo.gl/tpsmEJ#testing-patches What about something like : diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 191b79948424f4b21b7aa120abc03801264bf0a6..1e525d70f83354e451b738ffb8e42d83b5fa932f 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -164,6 +164,9 @@ static void dev_map_free(struct bpf_map *map) bpf_clear_redirect_map(map); synchronize_rcu(); + /* Make sure prior __dev_map_entry_free() have completed. */ + rcu_barrier(); + /* To ensure all pending flush operations have completed wait for flush * bitmap to indicate all flush_needed bits to be zero on _all_ cpus. * Because the above synchronize_rcu() ensures the map is disconnected ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: KASAN: use-after-free Read in __dev_map_entry_free 2019-04-02 20:03 ` Eric Dumazet @ 2019-04-04 3:59 ` Alexei Starovoitov 2019-04-04 8:47 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 8+ messages in thread From: Alexei Starovoitov @ 2019-04-04 3:59 UTC (permalink / raw) To: Eric Dumazet, Jesper Dangaard Brouer Cc: syzbot, Alexei Starovoitov, Daniel Borkmann, LKML, Network Development, syzkaller-bugs On Tue, Apr 2, 2019 at 1:03 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > > > > On 06/20/2018 08:19 AM, syzbot wrote: > > Hello, > > > > syzbot found the following crash on: > > > > HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern.. > > git tree: bpf-next > > console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1 > > dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0 > > compiler: gcc (GCC) 8.0.1 20180413 (experimental) > > syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000 > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000 > > > > IMPORTANT: if you fix the bug, please add the following tag to the commit: > > Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com > > > > ================================================================== > > BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > > BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > > Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18 > > > > CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39 > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > Call Trace: > > __dump_stack lib/dump_stack.c:77 [inline] > > dump_stack+0x1b9/0x294 lib/dump_stack.c:113 > > print_address_description+0x6c/0x20b mm/kasan/report.c:256 > > kasan_report_error mm/kasan/report.c:354 [inline] > > kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412 > > __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 > > dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > > __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > > __rcu_reclaim kernel/rcu/rcu.h:178 [inline] > > rcu_do_batch kernel/rcu/tree.c:2558 [inline] > > invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline] > > __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline] > > rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802 > > __do_softirq+0x2e0/0xaf5 kernel/softirq.c:284 > > run_ksoftirqd+0x86/0x100 kernel/softirq.c:645 > > smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164 > > kthread+0x345/0x410 kernel/kthread.c:240 > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > > > > Allocated by task 6675: > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > > set_track mm/kasan/kasan.c:460 [inline] > > kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553 > > kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620 > > kmalloc include/linux/slab.h:513 [inline] > > kzalloc include/linux/slab.h:706 [inline] > > dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102 > > find_and_alloc_map kernel/bpf/syscall.c:129 [inline] > > map_create+0x393/0x1010 kernel/bpf/syscall.c:453 > > __do_sys_bpf kernel/bpf/syscall.c:2351 [inline] > > __se_sys_bpf kernel/bpf/syscall.c:2328 [inline] > > __x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328 > > do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290 > > entry_SYSCALL_64_after_hwframe+0x49/0xbe > > > > Freed by task 26: > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > > set_track mm/kasan/kasan.c:460 [inline] > > __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 > > kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 > > __cache_free mm/slab.c:3498 [inline] > > kfree+0xd9/0x260 mm/slab.c:3813 > > dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191 > > bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262 > > process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153 > > worker_thread+0x181/0x13a0 kernel/workqueue.c:2296 > > kthread+0x345/0x410 kernel/kthread.c:240 > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > > > > The buggy address belongs to the object at ffff8801b8da37c0 > > which belongs to the cache kmalloc-512 of size 512 > > The buggy address is located 264 bytes inside of > > 512-byte region [ffff8801b8da37c0, ffff8801b8da39c0) > > The buggy address belongs to the page: > > page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940 index:0xffff8801b8da3540 > > flags: 0x2fffc0000000100(slab) > > raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940 > > raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000 > > page dumped because: kasan: bad access detected > > > > Memory state around the buggy address: > > ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb > > ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > ^ > > ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc > > ================================================================== > > > > > > --- > > This bug is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this bug report. See: > > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. > > syzbot can test patches for this bug, for details see: > > https://goo.gl/tpsmEJ#testing-patches > > > What about something like : > > > > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > index 191b79948424f4b21b7aa120abc03801264bf0a6..1e525d70f83354e451b738ffb8e42d83b5fa932f 100644 > --- a/kernel/bpf/devmap.c > +++ b/kernel/bpf/devmap.c > @@ -164,6 +164,9 @@ static void dev_map_free(struct bpf_map *map) > bpf_clear_redirect_map(map); > synchronize_rcu(); > > + /* Make sure prior __dev_map_entry_free() have completed. */ > + rcu_barrier(); > + Eric, Thank you for looking at it. The fix makes sense to me. Jesper, thoughts? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: KASAN: use-after-free Read in __dev_map_entry_free 2019-04-04 3:59 ` Alexei Starovoitov @ 2019-04-04 8:47 ` Jesper Dangaard Brouer 2019-04-09 14:50 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 8+ messages in thread From: Jesper Dangaard Brouer @ 2019-04-04 8:47 UTC (permalink / raw) To: Alexei Starovoitov Cc: Eric Dumazet, syzbot, Alexei Starovoitov, Daniel Borkmann, LKML, Network Development, syzkaller-bugs, brouer, John Fastabend, Paul E. McKenney On Wed, 3 Apr 2019 20:59:24 -0700 Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > On Tue, Apr 2, 2019 at 1:03 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > > > > > > > > On 06/20/2018 08:19 AM, syzbot wrote: > > > Hello, > > > > > > syzbot found the following crash on: > > > > > > HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern.. > > > git tree: bpf-next > > > console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000 > > > kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1 > > > dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0 > > > compiler: gcc (GCC) 8.0.1 20180413 (experimental) > > > syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000 > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000 > > > > > > IMPORTANT: if you fix the bug, please add the following tag to the commit: > > > Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com > > > > > > ================================================================== > > > BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > > > BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > > > Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18 > > > > > > CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39 > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > > > Call Trace: > > > __dump_stack lib/dump_stack.c:77 [inline] > > > dump_stack+0x1b9/0x294 lib/dump_stack.c:113 > > > print_address_description+0x6c/0x20b mm/kasan/report.c:256 > > > kasan_report_error mm/kasan/report.c:354 [inline] > > > kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412 > > > __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 > > > dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > > > __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > > > __rcu_reclaim kernel/rcu/rcu.h:178 [inline] > > > rcu_do_batch kernel/rcu/tree.c:2558 [inline] > > > invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline] > > > __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline] > > > rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802 > > > __do_softirq+0x2e0/0xaf5 kernel/softirq.c:284 > > > run_ksoftirqd+0x86/0x100 kernel/softirq.c:645 > > > smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164 > > > kthread+0x345/0x410 kernel/kthread.c:240 > > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > > > > > > Allocated by task 6675: > > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > > > set_track mm/kasan/kasan.c:460 [inline] > > > kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553 > > > kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620 > > > kmalloc include/linux/slab.h:513 [inline] > > > kzalloc include/linux/slab.h:706 [inline] > > > dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102 > > > find_and_alloc_map kernel/bpf/syscall.c:129 [inline] > > > map_create+0x393/0x1010 kernel/bpf/syscall.c:453 > > > __do_sys_bpf kernel/bpf/syscall.c:2351 [inline] > > > __se_sys_bpf kernel/bpf/syscall.c:2328 [inline] > > > __x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328 > > > do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290 > > > entry_SYSCALL_64_after_hwframe+0x49/0xbe > > > > > > Freed by task 26: > > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > > > set_track mm/kasan/kasan.c:460 [inline] > > > __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 > > > kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 > > > __cache_free mm/slab.c:3498 [inline] > > > kfree+0xd9/0x260 mm/slab.c:3813 > > > dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191 > > > bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262 > > > process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153 > > > worker_thread+0x181/0x13a0 kernel/workqueue.c:2296 > > > kthread+0x345/0x410 kernel/kthread.c:240 > > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > > > > > > The buggy address belongs to the object at ffff8801b8da37c0 > > > which belongs to the cache kmalloc-512 of size 512 > > > The buggy address is located 264 bytes inside of > > > 512-byte region [ffff8801b8da37c0, ffff8801b8da39c0) > > > The buggy address belongs to the page: > > > page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940 index:0xffff8801b8da3540 > > > flags: 0x2fffc0000000100(slab) > > > raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940 > > > raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000 > > > page dumped because: kasan: bad access detected > > > > > > Memory state around the buggy address: > > > ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb > > > ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > >> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > ^ > > > ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc > > > ================================================================== > > > > > > > > > --- > > > This bug is generated by a bot. It may contain errors. > > > See https://goo.gl/tpsmEJ for more information about syzbot. > > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > > > syzbot will keep track of this bug report. See: > > > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. > > > syzbot can test patches for this bug, for details see: > > > https://goo.gl/tpsmEJ#testing-patches > > > > > > What about something like : > > > > > > > > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > > index 191b79948424f4b21b7aa120abc03801264bf0a6..1e525d70f83354e451b738ffb8e42d83b5fa932f 100644 > > --- a/kernel/bpf/devmap.c > > +++ b/kernel/bpf/devmap.c > > @@ -164,6 +164,9 @@ static void dev_map_free(struct bpf_map *map) > > bpf_clear_redirect_map(map); > > synchronize_rcu(); > > > > + /* Make sure prior __dev_map_entry_free() have completed. */ > > + rcu_barrier(); > > + > > Eric, Thank you for looking at it. The fix makes sense to me. > > Jesper, thoughts? First I though it looked strange to have a synchronize_rcu() followed by a rcu_barrier(). But I think the fix is actually correct. We do need the rcu_barrier() call as it states "Wait until all in-flight call_rcu() callbacks complete". I wonder if we also/still need the synchronize_rcu(), but I think so as it functions as a memory-barrier across all CPUs (as in bpf_clear_redirect_map we visit all CPUs and clear any left-over per_cpu_ptr redirect_info). Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: KASAN: use-after-free Read in __dev_map_entry_free 2019-04-04 8:47 ` Jesper Dangaard Brouer @ 2019-04-09 14:50 ` Toke Høiland-Jørgensen 2019-04-09 14:59 ` Paul E. McKenney 2019-04-09 15:34 ` Eric Dumazet 0 siblings, 2 replies; 8+ messages in thread From: Toke Høiland-Jørgensen @ 2019-04-09 14:50 UTC (permalink / raw) To: Jesper Dangaard Brouer, Alexei Starovoitov Cc: Eric Dumazet, syzbot, Alexei Starovoitov, Daniel Borkmann, LKML, Network Development, syzkaller-bugs, brouer, John Fastabend, Paul E. McKenney Jesper Dangaard Brouer <brouer@redhat.com> writes: > On Wed, 3 Apr 2019 20:59:24 -0700 > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > >> On Tue, Apr 2, 2019 at 1:03 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: >> > >> > >> > >> > On 06/20/2018 08:19 AM, syzbot wrote: >> > > Hello, >> > > >> > > syzbot found the following crash on: >> > > >> > > HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern.. >> > > git tree: bpf-next >> > > console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000 >> > > kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1 >> > > dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0 >> > > compiler: gcc (GCC) 8.0.1 20180413 (experimental) >> > > syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000 >> > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000 >> > > >> > > IMPORTANT: if you fix the bug, please add the following tag to the commit: >> > > Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com >> > > >> > > ================================================================== >> > > BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365 [inline] >> > > BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 >> > > Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18 >> > > >> > > CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39 >> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >> > > Call Trace: >> > > __dump_stack lib/dump_stack.c:77 [inline] >> > > dump_stack+0x1b9/0x294 lib/dump_stack.c:113 >> > > print_address_description+0x6c/0x20b mm/kasan/report.c:256 >> > > kasan_report_error mm/kasan/report.c:354 [inline] >> > > kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412 >> > > __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 >> > > dev_map_flush_old kernel/bpf/devmap.c:365 [inline] >> > > __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 >> > > __rcu_reclaim kernel/rcu/rcu.h:178 [inline] >> > > rcu_do_batch kernel/rcu/tree.c:2558 [inline] >> > > invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline] >> > > __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline] >> > > rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802 >> > > __do_softirq+0x2e0/0xaf5 kernel/softirq.c:284 >> > > run_ksoftirqd+0x86/0x100 kernel/softirq.c:645 >> > > smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164 >> > > kthread+0x345/0x410 kernel/kthread.c:240 >> > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 >> > > >> > > Allocated by task 6675: >> > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 >> > > set_track mm/kasan/kasan.c:460 [inline] >> > > kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553 >> > > kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620 >> > > kmalloc include/linux/slab.h:513 [inline] >> > > kzalloc include/linux/slab.h:706 [inline] >> > > dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102 >> > > find_and_alloc_map kernel/bpf/syscall.c:129 [inline] >> > > map_create+0x393/0x1010 kernel/bpf/syscall.c:453 >> > > __do_sys_bpf kernel/bpf/syscall.c:2351 [inline] >> > > __se_sys_bpf kernel/bpf/syscall.c:2328 [inline] >> > > __x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328 >> > > do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290 >> > > entry_SYSCALL_64_after_hwframe+0x49/0xbe >> > > >> > > Freed by task 26: >> > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 >> > > set_track mm/kasan/kasan.c:460 [inline] >> > > __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 >> > > kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 >> > > __cache_free mm/slab.c:3498 [inline] >> > > kfree+0xd9/0x260 mm/slab.c:3813 >> > > dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191 >> > > bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262 >> > > process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153 >> > > worker_thread+0x181/0x13a0 kernel/workqueue.c:2296 >> > > kthread+0x345/0x410 kernel/kthread.c:240 >> > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 >> > > >> > > The buggy address belongs to the object at ffff8801b8da37c0 >> > > which belongs to the cache kmalloc-512 of size 512 >> > > The buggy address is located 264 bytes inside of >> > > 512-byte region [ffff8801b8da37c0, ffff8801b8da39c0) >> > > The buggy address belongs to the page: >> > > page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940 index:0xffff8801b8da3540 >> > > flags: 0x2fffc0000000100(slab) >> > > raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940 >> > > raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000 >> > > page dumped because: kasan: bad access detected >> > > >> > > Memory state around the buggy address: >> > > ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb >> > > ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >> > >> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >> > > ^ >> > > ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >> > > ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc >> > > ================================================================== >> > > >> > > >> > > --- >> > > This bug is generated by a bot. It may contain errors. >> > > See https://goo.gl/tpsmEJ for more information about syzbot. >> > > syzbot engineers can be reached at syzkaller@googlegroups.com. >> > > >> > > syzbot will keep track of this bug report. See: >> > > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. >> > > syzbot can test patches for this bug, for details see: >> > > https://goo.gl/tpsmEJ#testing-patches >> > >> > >> > What about something like : >> > >> > >> > >> > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c >> > index 191b79948424f4b21b7aa120abc03801264bf0a6..1e525d70f83354e451b738ffb8e42d83b5fa932f 100644 >> > --- a/kernel/bpf/devmap.c >> > +++ b/kernel/bpf/devmap.c >> > @@ -164,6 +164,9 @@ static void dev_map_free(struct bpf_map *map) >> > bpf_clear_redirect_map(map); >> > synchronize_rcu(); >> > >> > + /* Make sure prior __dev_map_entry_free() have completed. */ >> > + rcu_barrier(); >> > + >> >> Eric, Thank you for looking at it. The fix makes sense to me. >> >> Jesper, thoughts? > > First I though it looked strange to have a synchronize_rcu() followed > by a rcu_barrier(). But I think the fix is actually correct. We do > need the rcu_barrier() call as it states "Wait until all in-flight > call_rcu() callbacks complete". I wonder if we also/still need the > synchronize_rcu(), but I think so as it functions as a memory-barrier > across all CPUs (as in bpf_clear_redirect_map we visit all CPUs and > clear any left-over per_cpu_ptr redirect_info). > > Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Eric, are you going to submit a proper patch for this? :) -Toke ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: KASAN: use-after-free Read in __dev_map_entry_free 2019-04-09 14:50 ` Toke Høiland-Jørgensen @ 2019-04-09 14:59 ` Paul E. McKenney 2019-04-09 15:34 ` Eric Dumazet 1 sibling, 0 replies; 8+ messages in thread From: Paul E. McKenney @ 2019-04-09 14:59 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Jesper Dangaard Brouer, Alexei Starovoitov, Eric Dumazet, syzbot, Alexei Starovoitov, Daniel Borkmann, LKML, Network Development, syzkaller-bugs, John Fastabend On Tue, Apr 09, 2019 at 04:50:37PM +0200, Toke Høiland-Jørgensen wrote: > Jesper Dangaard Brouer <brouer@redhat.com> writes: > > > On Wed, 3 Apr 2019 20:59:24 -0700 > > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > > >> On Tue, Apr 2, 2019 at 1:03 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: > >> > > >> > > >> > > >> > On 06/20/2018 08:19 AM, syzbot wrote: > >> > > Hello, > >> > > > >> > > syzbot found the following crash on: > >> > > > >> > > HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern.. > >> > > git tree: bpf-next > >> > > console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000 > >> > > kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1 > >> > > dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0 > >> > > compiler: gcc (GCC) 8.0.1 20180413 (experimental) > >> > > syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000 > >> > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000 > >> > > > >> > > IMPORTANT: if you fix the bug, please add the following tag to the commit: > >> > > Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com > >> > > > >> > > ================================================================== > >> > > BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > >> > > BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > >> > > Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18 > >> > > > >> > > CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39 > >> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > >> > > Call Trace: > >> > > __dump_stack lib/dump_stack.c:77 [inline] > >> > > dump_stack+0x1b9/0x294 lib/dump_stack.c:113 > >> > > print_address_description+0x6c/0x20b mm/kasan/report.c:256 > >> > > kasan_report_error mm/kasan/report.c:354 [inline] > >> > > kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412 > >> > > __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 > >> > > dev_map_flush_old kernel/bpf/devmap.c:365 [inline] > >> > > __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 > >> > > __rcu_reclaim kernel/rcu/rcu.h:178 [inline] > >> > > rcu_do_batch kernel/rcu/tree.c:2558 [inline] > >> > > invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline] > >> > > __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline] > >> > > rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802 > >> > > __do_softirq+0x2e0/0xaf5 kernel/softirq.c:284 > >> > > run_ksoftirqd+0x86/0x100 kernel/softirq.c:645 > >> > > smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164 > >> > > kthread+0x345/0x410 kernel/kthread.c:240 > >> > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > >> > > > >> > > Allocated by task 6675: > >> > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > >> > > set_track mm/kasan/kasan.c:460 [inline] > >> > > kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553 > >> > > kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620 > >> > > kmalloc include/linux/slab.h:513 [inline] > >> > > kzalloc include/linux/slab.h:706 [inline] > >> > > dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102 > >> > > find_and_alloc_map kernel/bpf/syscall.c:129 [inline] > >> > > map_create+0x393/0x1010 kernel/bpf/syscall.c:453 > >> > > __do_sys_bpf kernel/bpf/syscall.c:2351 [inline] > >> > > __se_sys_bpf kernel/bpf/syscall.c:2328 [inline] > >> > > __x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328 > >> > > do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290 > >> > > entry_SYSCALL_64_after_hwframe+0x49/0xbe > >> > > > >> > > Freed by task 26: > >> > > save_stack+0x43/0xd0 mm/kasan/kasan.c:448 > >> > > set_track mm/kasan/kasan.c:460 [inline] > >> > > __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 > >> > > kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 > >> > > __cache_free mm/slab.c:3498 [inline] > >> > > kfree+0xd9/0x260 mm/slab.c:3813 > >> > > dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191 > >> > > bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262 > >> > > process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153 > >> > > worker_thread+0x181/0x13a0 kernel/workqueue.c:2296 > >> > > kthread+0x345/0x410 kernel/kthread.c:240 > >> > > ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 > >> > > > >> > > The buggy address belongs to the object at ffff8801b8da37c0 > >> > > which belongs to the cache kmalloc-512 of size 512 > >> > > The buggy address is located 264 bytes inside of > >> > > 512-byte region [ffff8801b8da37c0, ffff8801b8da39c0) > >> > > The buggy address belongs to the page: > >> > > page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940 index:0xffff8801b8da3540 > >> > > flags: 0x2fffc0000000100(slab) > >> > > raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940 > >> > > raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000 > >> > > page dumped because: kasan: bad access detected > >> > > > >> > > Memory state around the buggy address: > >> > > ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb > >> > > ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> > >> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> > > ^ > >> > > ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> > > ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc > >> > > ================================================================== > >> > > > >> > > > >> > > --- > >> > > This bug is generated by a bot. It may contain errors. > >> > > See https://goo.gl/tpsmEJ for more information about syzbot. > >> > > syzbot engineers can be reached at syzkaller@googlegroups.com. > >> > > > >> > > syzbot will keep track of this bug report. See: > >> > > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. > >> > > syzbot can test patches for this bug, for details see: > >> > > https://goo.gl/tpsmEJ#testing-patches > >> > > >> > > >> > What about something like : > >> > > >> > > >> > > >> > diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c > >> > index 191b79948424f4b21b7aa120abc03801264bf0a6..1e525d70f83354e451b738ffb8e42d83b5fa932f 100644 > >> > --- a/kernel/bpf/devmap.c > >> > +++ b/kernel/bpf/devmap.c > >> > @@ -164,6 +164,9 @@ static void dev_map_free(struct bpf_map *map) > >> > bpf_clear_redirect_map(map); > >> > synchronize_rcu(); > >> > > >> > + /* Make sure prior __dev_map_entry_free() have completed. */ > >> > + rcu_barrier(); > >> > + > >> > >> Eric, Thank you for looking at it. The fix makes sense to me. > >> > >> Jesper, thoughts? > > > > First I though it looked strange to have a synchronize_rcu() followed > > by a rcu_barrier(). But I think the fix is actually correct. We do > > need the rcu_barrier() call as it states "Wait until all in-flight > > call_rcu() callbacks complete". I wonder if we also/still need the > > synchronize_rcu(), but I think so as it functions as a memory-barrier > > across all CPUs (as in bpf_clear_redirect_map we visit all CPUs and > > clear any left-over per_cpu_ptr redirect_info). In addition, an rcu_barrier() might return immediately if there happen to be no callbacks posted at the time. So if you need to wait for readers, you need synchronize_rcu() -- rcu_barrier() is -not- a substitute. So it really is OK to have rcu_barrier() followed by synchronize_rcu() and vice versa. ;-) Thanx, Paul > > Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> > > Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> > > Eric, are you going to submit a proper patch for this? :) > > -Toke > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: KASAN: use-after-free Read in __dev_map_entry_free 2019-04-09 14:50 ` Toke Høiland-Jørgensen 2019-04-09 14:59 ` Paul E. McKenney @ 2019-04-09 15:34 ` Eric Dumazet 2019-04-09 20:22 ` Toke Høiland-Jørgensen 1 sibling, 1 reply; 8+ messages in thread From: Eric Dumazet @ 2019-04-09 15:34 UTC (permalink / raw) To: Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Alexei Starovoitov Cc: syzbot, Alexei Starovoitov, Daniel Borkmann, LKML, Network Development, syzkaller-bugs, John Fastabend, Paul E. McKenney On 04/09/2019 07:50 AM, Toke Høiland-Jørgensen wrote: > Jesper Dangaard Brouer <brouer@redhat.com> writes: > >> On Wed, 3 Apr 2019 20:59:24 -0700 >> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: >> >>> On Tue, Apr 2, 2019 at 1:03 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: >>>> >>>> >>>> >>>> On 06/20/2018 08:19 AM, syzbot wrote: >>>>> Hello, >>>>> >>>>> syzbot found the following crash on: >>>>> >>>>> HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern.. >>>>> git tree: bpf-next >>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000 >>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1 >>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0 >>>>> compiler: gcc (GCC) 8.0.1 20180413 (experimental) >>>>> syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000 >>>>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000 >>>>> >>>>> IMPORTANT: if you fix the bug, please add the following tag to the commit: >>>>> Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com >>>>> >>>>> ================================================================== >>>>> BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365 [inline] >>>>> BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 >>>>> Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18 >>>>> >>>>> CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39 >>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>> Call Trace: >>>>> __dump_stack lib/dump_stack.c:77 [inline] >>>>> dump_stack+0x1b9/0x294 lib/dump_stack.c:113 >>>>> print_address_description+0x6c/0x20b mm/kasan/report.c:256 >>>>> kasan_report_error mm/kasan/report.c:354 [inline] >>>>> kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412 >>>>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 >>>>> dev_map_flush_old kernel/bpf/devmap.c:365 [inline] >>>>> __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 >>>>> __rcu_reclaim kernel/rcu/rcu.h:178 [inline] >>>>> rcu_do_batch kernel/rcu/tree.c:2558 [inline] >>>>> invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline] >>>>> __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline] >>>>> rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802 >>>>> __do_softirq+0x2e0/0xaf5 kernel/softirq.c:284 >>>>> run_ksoftirqd+0x86/0x100 kernel/softirq.c:645 >>>>> smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164 >>>>> kthread+0x345/0x410 kernel/kthread.c:240 >>>>> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 >>>>> >>>>> Allocated by task 6675: >>>>> save_stack+0x43/0xd0 mm/kasan/kasan.c:448 >>>>> set_track mm/kasan/kasan.c:460 [inline] >>>>> kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553 >>>>> kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620 >>>>> kmalloc include/linux/slab.h:513 [inline] >>>>> kzalloc include/linux/slab.h:706 [inline] >>>>> dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102 >>>>> find_and_alloc_map kernel/bpf/syscall.c:129 [inline] >>>>> map_create+0x393/0x1010 kernel/bpf/syscall.c:453 >>>>> __do_sys_bpf kernel/bpf/syscall.c:2351 [inline] >>>>> __se_sys_bpf kernel/bpf/syscall.c:2328 [inline] >>>>> __x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328 >>>>> do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290 >>>>> entry_SYSCALL_64_after_hwframe+0x49/0xbe >>>>> >>>>> Freed by task 26: >>>>> save_stack+0x43/0xd0 mm/kasan/kasan.c:448 >>>>> set_track mm/kasan/kasan.c:460 [inline] >>>>> __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 >>>>> kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 >>>>> __cache_free mm/slab.c:3498 [inline] >>>>> kfree+0xd9/0x260 mm/slab.c:3813 >>>>> dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191 >>>>> bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262 >>>>> process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153 >>>>> worker_thread+0x181/0x13a0 kernel/workqueue.c:2296 >>>>> kthread+0x345/0x410 kernel/kthread.c:240 >>>>> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 >>>>> >>>>> The buggy address belongs to the object at ffff8801b8da37c0 >>>>> which belongs to the cache kmalloc-512 of size 512 >>>>> The buggy address is located 264 bytes inside of >>>>> 512-byte region [ffff8801b8da37c0, ffff8801b8da39c0) >>>>> The buggy address belongs to the page: >>>>> page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940 index:0xffff8801b8da3540 >>>>> flags: 0x2fffc0000000100(slab) >>>>> raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940 >>>>> raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000 >>>>> page dumped because: kasan: bad access detected >>>>> >>>>> Memory state around the buggy address: >>>>> ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb >>>>> ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >>>>>> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >>>>> ^ >>>>> ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >>>>> ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc >>>>> ================================================================== >>>>> >>>>> >>>>> --- >>>>> This bug is generated by a bot. It may contain errors. >>>>> See https://goo.gl/tpsmEJ for more information about syzbot. >>>>> syzbot engineers can be reached at syzkaller@googlegroups.com. >>>>> >>>>> syzbot will keep track of this bug report. See: >>>>> https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. >>>>> syzbot can test patches for this bug, for details see: >>>>> https://goo.gl/tpsmEJ#testing-patches >>>> >>>> >>>> What about something like : >>>> >>>> >>>> >>>> diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c >>>> index 191b79948424f4b21b7aa120abc03801264bf0a6..1e525d70f83354e451b738ffb8e42d83b5fa932f 100644 >>>> --- a/kernel/bpf/devmap.c >>>> +++ b/kernel/bpf/devmap.c >>>> @@ -164,6 +164,9 @@ static void dev_map_free(struct bpf_map *map) >>>> bpf_clear_redirect_map(map); >>>> synchronize_rcu(); >>>> >>>> + /* Make sure prior __dev_map_entry_free() have completed. */ >>>> + rcu_barrier(); >>>> + >>> >>> Eric, Thank you for looking at it. The fix makes sense to me. >>> >>> Jesper, thoughts? >> >> First I though it looked strange to have a synchronize_rcu() followed >> by a rcu_barrier(). But I think the fix is actually correct. We do >> need the rcu_barrier() call as it states "Wait until all in-flight >> call_rcu() callbacks complete". I wonder if we also/still need the >> synchronize_rcu(), but I think so as it functions as a memory-barrier >> across all CPUs (as in bpf_clear_redirect_map we visit all CPUs and >> clear any left-over per_cpu_ptr redirect_info). >> >> Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> > > Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> > > Eric, are you going to submit a proper patch for this? :) Yes I will, I am back from vacation. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: KASAN: use-after-free Read in __dev_map_entry_free 2019-04-09 15:34 ` Eric Dumazet @ 2019-04-09 20:22 ` Toke Høiland-Jørgensen 0 siblings, 0 replies; 8+ messages in thread From: Toke Høiland-Jørgensen @ 2019-04-09 20:22 UTC (permalink / raw) To: Eric Dumazet, Jesper Dangaard Brouer, Alexei Starovoitov Cc: syzbot, Alexei Starovoitov, Daniel Borkmann, LKML, Network Development, syzkaller-bugs, John Fastabend, Paul E. McKenney Eric Dumazet <eric.dumazet@gmail.com> writes: > On 04/09/2019 07:50 AM, Toke Høiland-Jørgensen wrote: >> Jesper Dangaard Brouer <brouer@redhat.com> writes: >> >>> On Wed, 3 Apr 2019 20:59:24 -0700 >>> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: >>> >>>> On Tue, Apr 2, 2019 at 1:03 PM Eric Dumazet <eric.dumazet@gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> On 06/20/2018 08:19 AM, syzbot wrote: >>>>>> Hello, >>>>>> >>>>>> syzbot found the following crash on: >>>>>> >>>>>> HEAD commit: f0dc7f9c6dd9 Merge git://git.kernel.org/pub/scm/linux/kern.. >>>>>> git tree: bpf-next >>>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=15ad7d10400000 >>>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=fa9c20c48788d1c1 >>>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=457d3e2ffbcf31aee5c0 >>>>>> compiler: gcc (GCC) 8.0.1 20180413 (experimental) >>>>>> syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1195225f800000 >>>>>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=171a7ce4400000 >>>>>> >>>>>> IMPORTANT: if you fix the bug, please add the following tag to the commit: >>>>>> Reported-by: syzbot+457d3e2ffbcf31aee5c0@syzkaller.appspotmail.com >>>>>> >>>>>> ================================================================== >>>>>> BUG: KASAN: use-after-free in dev_map_flush_old kernel/bpf/devmap.c:365 [inline] >>>>>> BUG: KASAN: use-after-free in __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 >>>>>> Read of size 8 at addr ffff8801b8da38c8 by task ksoftirqd/1/18 >>>>>> >>>>>> CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.17.0+ #39 >>>>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 >>>>>> Call Trace: >>>>>> __dump_stack lib/dump_stack.c:77 [inline] >>>>>> dump_stack+0x1b9/0x294 lib/dump_stack.c:113 >>>>>> print_address_description+0x6c/0x20b mm/kasan/report.c:256 >>>>>> kasan_report_error mm/kasan/report.c:354 [inline] >>>>>> kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412 >>>>>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 >>>>>> dev_map_flush_old kernel/bpf/devmap.c:365 [inline] >>>>>> __dev_map_entry_free+0x2a8/0x300 kernel/bpf/devmap.c:379 >>>>>> __rcu_reclaim kernel/rcu/rcu.h:178 [inline] >>>>>> rcu_do_batch kernel/rcu/tree.c:2558 [inline] >>>>>> invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline] >>>>>> __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline] >>>>>> rcu_process_callbacks+0xe9d/0x1760 kernel/rcu/tree.c:2802 >>>>>> __do_softirq+0x2e0/0xaf5 kernel/softirq.c:284 >>>>>> run_ksoftirqd+0x86/0x100 kernel/softirq.c:645 >>>>>> smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164 >>>>>> kthread+0x345/0x410 kernel/kthread.c:240 >>>>>> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 >>>>>> >>>>>> Allocated by task 6675: >>>>>> save_stack+0x43/0xd0 mm/kasan/kasan.c:448 >>>>>> set_track mm/kasan/kasan.c:460 [inline] >>>>>> kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553 >>>>>> kmem_cache_alloc_trace+0x152/0x780 mm/slab.c:3620 >>>>>> kmalloc include/linux/slab.h:513 [inline] >>>>>> kzalloc include/linux/slab.h:706 [inline] >>>>>> dev_map_alloc+0x208/0x7f0 kernel/bpf/devmap.c:102 >>>>>> find_and_alloc_map kernel/bpf/syscall.c:129 [inline] >>>>>> map_create+0x393/0x1010 kernel/bpf/syscall.c:453 >>>>>> __do_sys_bpf kernel/bpf/syscall.c:2351 [inline] >>>>>> __se_sys_bpf kernel/bpf/syscall.c:2328 [inline] >>>>>> __x64_sys_bpf+0x303/0x510 kernel/bpf/syscall.c:2328 >>>>>> do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:290 >>>>>> entry_SYSCALL_64_after_hwframe+0x49/0xbe >>>>>> >>>>>> Freed by task 26: >>>>>> save_stack+0x43/0xd0 mm/kasan/kasan.c:448 >>>>>> set_track mm/kasan/kasan.c:460 [inline] >>>>>> __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 >>>>>> kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 >>>>>> __cache_free mm/slab.c:3498 [inline] >>>>>> kfree+0xd9/0x260 mm/slab.c:3813 >>>>>> dev_map_free+0x4fa/0x670 kernel/bpf/devmap.c:191 >>>>>> bpf_map_free_deferred+0xba/0xf0 kernel/bpf/syscall.c:262 >>>>>> process_one_work+0xc64/0x1b70 kernel/workqueue.c:2153 >>>>>> worker_thread+0x181/0x13a0 kernel/workqueue.c:2296 >>>>>> kthread+0x345/0x410 kernel/kthread.c:240 >>>>>> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412 >>>>>> >>>>>> The buggy address belongs to the object at ffff8801b8da37c0 >>>>>> which belongs to the cache kmalloc-512 of size 512 >>>>>> The buggy address is located 264 bytes inside of >>>>>> 512-byte region [ffff8801b8da37c0, ffff8801b8da39c0) >>>>>> The buggy address belongs to the page: >>>>>> page:ffffea0006e368c0 count:1 mapcount:0 mapping:ffff8801da800940 index:0xffff8801b8da3540 >>>>>> flags: 0x2fffc0000000100(slab) >>>>>> raw: 02fffc0000000100 ffffea0007217b88 ffffea0006e30cc8 ffff8801da800940 >>>>>> raw: ffff8801b8da3540 ffff8801b8da3040 0000000100000004 0000000000000000 >>>>>> page dumped because: kasan: bad access detected >>>>>> >>>>>> Memory state around the buggy address: >>>>>> ffff8801b8da3780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb >>>>>> ffff8801b8da3800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >>>>>>> ffff8801b8da3880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >>>>>> ^ >>>>>> ffff8801b8da3900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >>>>>> ffff8801b8da3980: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc >>>>>> ================================================================== >>>>>> >>>>>> >>>>>> --- >>>>>> This bug is generated by a bot. It may contain errors. >>>>>> See https://goo.gl/tpsmEJ for more information about syzbot. >>>>>> syzbot engineers can be reached at syzkaller@googlegroups.com. >>>>>> >>>>>> syzbot will keep track of this bug report. See: >>>>>> https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. >>>>>> syzbot can test patches for this bug, for details see: >>>>>> https://goo.gl/tpsmEJ#testing-patches >>>>> >>>>> >>>>> What about something like : >>>>> >>>>> >>>>> >>>>> diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c >>>>> index 191b79948424f4b21b7aa120abc03801264bf0a6..1e525d70f83354e451b738ffb8e42d83b5fa932f 100644 >>>>> --- a/kernel/bpf/devmap.c >>>>> +++ b/kernel/bpf/devmap.c >>>>> @@ -164,6 +164,9 @@ static void dev_map_free(struct bpf_map *map) >>>>> bpf_clear_redirect_map(map); >>>>> synchronize_rcu(); >>>>> >>>>> + /* Make sure prior __dev_map_entry_free() have completed. */ >>>>> + rcu_barrier(); >>>>> + >>>> >>>> Eric, Thank you for looking at it. The fix makes sense to me. >>>> >>>> Jesper, thoughts? >>> >>> First I though it looked strange to have a synchronize_rcu() followed >>> by a rcu_barrier(). But I think the fix is actually correct. We do >>> need the rcu_barrier() call as it states "Wait until all in-flight >>> call_rcu() callbacks complete". I wonder if we also/still need the >>> synchronize_rcu(), but I think so as it functions as a memory-barrier >>> across all CPUs (as in bpf_clear_redirect_map we visit all CPUs and >>> clear any left-over per_cpu_ptr redirect_info). >>> >>> Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> >> >> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> >> >> Eric, are you going to submit a proper patch for this? :) > > Yes I will, I am back from vacation. Great, thanks! And welcome back :) -Toke ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-04-09 20:23 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-06-20 15:19 KASAN: use-after-free Read in __dev_map_entry_free syzbot 2019-04-02 20:03 ` Eric Dumazet 2019-04-04 3:59 ` Alexei Starovoitov 2019-04-04 8:47 ` Jesper Dangaard Brouer 2019-04-09 14:50 ` Toke Høiland-Jørgensen 2019-04-09 14:59 ` Paul E. McKenney 2019-04-09 15:34 ` Eric Dumazet 2019-04-09 20:22 ` Toke Høiland-Jørgensen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).