From: Dmitry Vyukov <dvyukov@google.com> To: Johannes Berg <johannes@sipsolutions.net> Cc: Patricia Alfonso <trishalfonso@google.com>, Jeff Dike <jdike@addtoit.com>, Richard Weinberger <richard@nod.at>, anton.ivanov@cambridgegreys.com, Andrey Ryabinin <aryabinin@virtuozzo.com>, Brendan Higgins <brendanhiggins@google.com>, David Gow <davidgow@google.com>, linux-um@lists.infradead.org, LKML <linux-kernel@vger.kernel.org>, kasan-dev <kasan-dev@googlegroups.com> Subject: Re: [PATCH] UML: add support for KASAN under x86_64 Date: Fri, 20 Mar 2020 16:18:30 +0100 [thread overview] Message-ID: <CACT4Y+YzM5bwvJ=yryrz1_y=uh=NX+2PNu4pLFaqQ2BMS39Fdg@mail.gmail.com> (raw) In-Reply-To: <ded22d68e623d2663c96a0e1c81d660b9da747bc.camel@sipsolutions.net> On Fri, Mar 20, 2020 at 2:39 PM Johannes Berg <johannes@sipsolutions.net> wrote: > > On Wed, 2020-03-11 at 18:34 +0100, Dmitry Vyukov wrote: > > > > $ gdb -p ... > > > (gdb) p/x task_size > > > $1 = 0x7fc0000000 > > > (gdb) p/x __end_of_fixed_addresses > > > $2 = 0x0 > > > (gdb) p/x end_iomem > > > $3 = 0x70000000 > > > (gdb) p/x __va_space > > > > > > #define TASK_SIZE (task_size) > > > #define FIXADDR_TOP (TASK_SIZE - 2 * PAGE_SIZE) > > > > > > #define FIXADDR_START (FIXADDR_TOP - FIXADDR_SIZE) > > > #define FIXADDR_SIZE (__end_of_fixed_addresses << PAGE_SHIFT) > > > > > > #define VMALLOC_END (FIXADDR_START-2*PAGE_SIZE) > > > > > > #define MODULES_VADDR VMALLOC_START > > > #define MODULES_END VMALLOC_END > > > #define VMALLOC_START ((end_iomem + VMALLOC_OFFSET) & ~(VMALLOC_OFFSET-1)) > > > #define VMALLOC_OFFSET (__va_space) > > > #define __va_space (8*1024*1024) > > > > > > > > > So from that, it would look like the UML vmalloc area is from > > > 0x 70800000 all the way to > > > 0x7fbfffc000, which obviously clashes with the KASAN_SHADOW_OFFSET being > > > just 0x7fff8000. > > > > > > > > > I'm guessing that basically the module loading overwrote the kasan > > > shadow then? > > > > Well, ok, this is definitely not going to fly :) > > Yeah, not with vmalloc/modules at least, but you can't really prevent > vmalloc :) > > > I don't know if it's easy to move modules to a different location. > > We'd have to not just move modules, but also vmalloc space. They're one > and the same in UML. > > > It > > would be nice because 0x7fbfffc000 is the shadow start that's used in > > userspace asan and it allows to faster instrumentation (if offset is > > within first 2 gigs, the instruction encoding is much more compact, > > for >2gigs it will require several instructions). > > Wait ... Now you say 0x7fbfffc000, but that is almost fine? I think you > confused the values - because I see, on userspace, the following: Oh, sorry, I copy-pasted wrong number. I meant 0x7fff8000. Here is the user-space mapping that uses it: https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/asan/asan_mapping.h#L25 > || `[0x10007fff8000, 0x7fffffffffff]` || HighMem || > || `[0x02008fff7000, 0x10007fff7fff]` || HighShadow || > || `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap || > || `[0x00007fff8000, 0x00008fff6fff]` || LowShadow || > || `[0x000000000000, 0x00007fff7fff]` || LowMem || > > > Now, I also don't really understand what UML is doing here - > os_get_top_address() determines some sort of "top address"? But all that > is only on 32-bit, on 64-bit, that's always 0x7fc0000000. Then I would expect 0x1000 0000 0000 to work, but you say it doesn't... > So basically that means it's just _slightly_ higher than what you > suggested as the KASAN_SHADOW_OFFSET now (even if erroneously?), and > shouldn't actually clash (and we can just change the top address value > to be slightly lower anyway to prevent clashing). > > > But if it's not really easy, I guess we go with a large shadow start > > (at least initially). A slower but working KASAN is better than fast > > non-working KASAN :) > > Indeed, but I can't even get it to work regardless of the offset. > > Note that I have lockdep enabled, and at least some crashes appear to be > because of the stack unwinding code that is called by lockdep in various > situations... This is something new, right? The previous stacks you posted did not mention lockdep. > > > I tried changing it > > > > > > config KASAN_SHADOW_OFFSET > > > hex > > > depends on KASAN > > > - default 0x7fff8000 > > > + default 0x8000000000 > > > > > > > > > and also put a check in like this: > > > > > > +++ b/arch/um/kernel/um_arch.c > > > @@ -13,6 +13,7 @@ > > > #include <linux/sched.h> > > > #include <linux/sched/task.h> > > > #include <linux/kmsg_dump.h> > > > +#include <linux/kasan.h> > > > > > > #include <asm/pgtable.h> > > > #include <asm/processor.h> > > > @@ -267,9 +268,11 @@ int __init linux_main(int argc, char **argv) > > > /* > > > * TASK_SIZE needs to be PGDIR_SIZE aligned or else exit_mmap craps > > > * out > > > */ > > > task_size = host_task_size & PGDIR_MASK; > > > > > > + if (task_size > KASAN_SHADOW_OFFSET) > > > + panic("KASAN shadow offset must be bigger than task size"); > > > > > > > > > but now I just crash accessing the shadow even though it was mapped fine? > > > > Yes, this is puzzling. > > I noticed that RIP is the same in both cases and it relates to vmap code. > > A support for shadow for vmalloced-memory was added to KASAN recently > > and I suspect it may conflict with UML. > > This can't be it - HAVE_ARCH_KASAN_VMALLOC isn't selected, so > KASAN_VMALLOC isn't set. > > > What does pte-manipulation code even do under UML? > > No idea. > > > Looking at the code around, kasan_mem_notifier may be a problem too, > > or at least excessive and confusing. We already have shadow for > > everything, we don't need _any_ of dynamic/lazy shadow mapping. > > CONFIG_MEMORY_HOTPLUG is also not supported in ARCH=um, or at least not > used in my config. Ack. Maybe if you dump /proc/self/maps for the process, it will shed some light. Or is it possible to run it under strace? If we get all mmap/munmap/mprotect, we will maybe see the offender that messes the shadow...
WARNING: multiple messages have this Message-ID (diff)
From: Dmitry Vyukov <dvyukov@google.com> To: Johannes Berg <johannes@sipsolutions.net> Cc: Patricia Alfonso <trishalfonso@google.com>, Richard Weinberger <richard@nod.at>, Jeff Dike <jdike@addtoit.com>, Brendan Higgins <brendanhiggins@google.com>, LKML <linux-kernel@vger.kernel.org>, kasan-dev <kasan-dev@googlegroups.com>, linux-um@lists.infradead.org, David Gow <davidgow@google.com>, Andrey Ryabinin <aryabinin@virtuozzo.com>, anton.ivanov@cambridgegreys.com Subject: Re: [PATCH] UML: add support for KASAN under x86_64 Date: Fri, 20 Mar 2020 16:18:30 +0100 [thread overview] Message-ID: <CACT4Y+YzM5bwvJ=yryrz1_y=uh=NX+2PNu4pLFaqQ2BMS39Fdg@mail.gmail.com> (raw) In-Reply-To: <ded22d68e623d2663c96a0e1c81d660b9da747bc.camel@sipsolutions.net> On Fri, Mar 20, 2020 at 2:39 PM Johannes Berg <johannes@sipsolutions.net> wrote: > > On Wed, 2020-03-11 at 18:34 +0100, Dmitry Vyukov wrote: > > > > $ gdb -p ... > > > (gdb) p/x task_size > > > $1 = 0x7fc0000000 > > > (gdb) p/x __end_of_fixed_addresses > > > $2 = 0x0 > > > (gdb) p/x end_iomem > > > $3 = 0x70000000 > > > (gdb) p/x __va_space > > > > > > #define TASK_SIZE (task_size) > > > #define FIXADDR_TOP (TASK_SIZE - 2 * PAGE_SIZE) > > > > > > #define FIXADDR_START (FIXADDR_TOP - FIXADDR_SIZE) > > > #define FIXADDR_SIZE (__end_of_fixed_addresses << PAGE_SHIFT) > > > > > > #define VMALLOC_END (FIXADDR_START-2*PAGE_SIZE) > > > > > > #define MODULES_VADDR VMALLOC_START > > > #define MODULES_END VMALLOC_END > > > #define VMALLOC_START ((end_iomem + VMALLOC_OFFSET) & ~(VMALLOC_OFFSET-1)) > > > #define VMALLOC_OFFSET (__va_space) > > > #define __va_space (8*1024*1024) > > > > > > > > > So from that, it would look like the UML vmalloc area is from > > > 0x 70800000 all the way to > > > 0x7fbfffc000, which obviously clashes with the KASAN_SHADOW_OFFSET being > > > just 0x7fff8000. > > > > > > > > > I'm guessing that basically the module loading overwrote the kasan > > > shadow then? > > > > Well, ok, this is definitely not going to fly :) > > Yeah, not with vmalloc/modules at least, but you can't really prevent > vmalloc :) > > > I don't know if it's easy to move modules to a different location. > > We'd have to not just move modules, but also vmalloc space. They're one > and the same in UML. > > > It > > would be nice because 0x7fbfffc000 is the shadow start that's used in > > userspace asan and it allows to faster instrumentation (if offset is > > within first 2 gigs, the instruction encoding is much more compact, > > for >2gigs it will require several instructions). > > Wait ... Now you say 0x7fbfffc000, but that is almost fine? I think you > confused the values - because I see, on userspace, the following: Oh, sorry, I copy-pasted wrong number. I meant 0x7fff8000. Here is the user-space mapping that uses it: https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/asan/asan_mapping.h#L25 > || `[0x10007fff8000, 0x7fffffffffff]` || HighMem || > || `[0x02008fff7000, 0x10007fff7fff]` || HighShadow || > || `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap || > || `[0x00007fff8000, 0x00008fff6fff]` || LowShadow || > || `[0x000000000000, 0x00007fff7fff]` || LowMem || > > > Now, I also don't really understand what UML is doing here - > os_get_top_address() determines some sort of "top address"? But all that > is only on 32-bit, on 64-bit, that's always 0x7fc0000000. Then I would expect 0x1000 0000 0000 to work, but you say it doesn't... > So basically that means it's just _slightly_ higher than what you > suggested as the KASAN_SHADOW_OFFSET now (even if erroneously?), and > shouldn't actually clash (and we can just change the top address value > to be slightly lower anyway to prevent clashing). > > > But if it's not really easy, I guess we go with a large shadow start > > (at least initially). A slower but working KASAN is better than fast > > non-working KASAN :) > > Indeed, but I can't even get it to work regardless of the offset. > > Note that I have lockdep enabled, and at least some crashes appear to be > because of the stack unwinding code that is called by lockdep in various > situations... This is something new, right? The previous stacks you posted did not mention lockdep. > > > I tried changing it > > > > > > config KASAN_SHADOW_OFFSET > > > hex > > > depends on KASAN > > > - default 0x7fff8000 > > > + default 0x8000000000 > > > > > > > > > and also put a check in like this: > > > > > > +++ b/arch/um/kernel/um_arch.c > > > @@ -13,6 +13,7 @@ > > > #include <linux/sched.h> > > > #include <linux/sched/task.h> > > > #include <linux/kmsg_dump.h> > > > +#include <linux/kasan.h> > > > > > > #include <asm/pgtable.h> > > > #include <asm/processor.h> > > > @@ -267,9 +268,11 @@ int __init linux_main(int argc, char **argv) > > > /* > > > * TASK_SIZE needs to be PGDIR_SIZE aligned or else exit_mmap craps > > > * out > > > */ > > > task_size = host_task_size & PGDIR_MASK; > > > > > > + if (task_size > KASAN_SHADOW_OFFSET) > > > + panic("KASAN shadow offset must be bigger than task size"); > > > > > > > > > but now I just crash accessing the shadow even though it was mapped fine? > > > > Yes, this is puzzling. > > I noticed that RIP is the same in both cases and it relates to vmap code. > > A support for shadow for vmalloced-memory was added to KASAN recently > > and I suspect it may conflict with UML. > > This can't be it - HAVE_ARCH_KASAN_VMALLOC isn't selected, so > KASAN_VMALLOC isn't set. > > > What does pte-manipulation code even do under UML? > > No idea. > > > Looking at the code around, kasan_mem_notifier may be a problem too, > > or at least excessive and confusing. We already have shadow for > > everything, we don't need _any_ of dynamic/lazy shadow mapping. > > CONFIG_MEMORY_HOTPLUG is also not supported in ARCH=um, or at least not > used in my config. Ack. Maybe if you dump /proc/self/maps for the process, it will shed some light. Or is it possible to run it under strace? If we get all mmap/munmap/mprotect, we will maybe see the offender that messes the shadow... _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um
next prev parent reply other threads:[~2020-03-20 15:18 UTC|newest] Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-02-26 0:46 [PATCH] UML: add support for KASAN under x86_64 Patricia Alfonso 2020-02-26 0:46 ` Patricia Alfonso 2020-02-26 1:19 ` Brendan Higgins 2020-02-26 1:19 ` Brendan Higgins 2020-02-26 15:24 ` Dmitry Vyukov 2020-02-26 15:24 ` Dmitry Vyukov 2020-03-06 0:03 ` Patricia Alfonso 2020-03-06 0:03 ` Patricia Alfonso 2020-03-11 10:32 ` Johannes Berg 2020-03-11 10:32 ` Johannes Berg 2020-03-11 10:46 ` Dmitry Vyukov 2020-03-11 10:46 ` Dmitry Vyukov 2020-03-11 11:18 ` Johannes Berg 2020-03-11 11:18 ` Johannes Berg 2020-03-11 11:40 ` Johannes Berg 2020-03-11 11:40 ` Johannes Berg 2020-03-11 17:34 ` Dmitry Vyukov 2020-03-11 17:34 ` Dmitry Vyukov 2020-03-20 13:39 ` Johannes Berg 2020-03-20 13:39 ` Johannes Berg 2020-03-20 15:18 ` Dmitry Vyukov [this message] 2020-03-20 15:18 ` Dmitry Vyukov 2020-03-30 7:43 ` Johannes Berg 2020-03-30 7:43 ` Johannes Berg 2020-03-30 8:38 ` Dmitry Vyukov 2020-03-30 8:38 ` Dmitry Vyukov 2020-03-30 8:41 ` Johannes Berg 2020-03-30 8:41 ` Johannes Berg 2020-03-31 6:14 ` David Gow 2020-03-31 6:14 ` David Gow 2020-03-31 7:43 ` Johannes Berg 2020-03-31 7:43 ` Johannes Berg 2020-03-31 16:39 ` Patricia Alfonso 2020-03-31 16:39 ` Patricia Alfonso 2020-03-31 16:54 ` Richard Weinberger 2020-03-11 22:32 ` Patricia Alfonso 2020-03-11 22:32 ` Patricia Alfonso 2020-03-11 22:44 ` Johannes Berg 2020-03-11 22:44 ` Johannes Berg 2022-05-24 10:34 ` Vincent Whitchurch 2022-05-24 10:34 ` Vincent Whitchurch 2022-05-24 10:45 ` Johannes Berg 2022-05-24 10:45 ` Johannes Berg 2022-05-24 19:35 ` David Gow 2022-05-24 19:35 ` David Gow 2022-05-25 11:17 ` Vincent Whitchurch 2022-05-25 11:17 ` Vincent Whitchurch 2022-05-26 1:01 ` [RFC PATCH v3] " David Gow 2022-05-26 1:01 ` David Gow 2022-05-26 9:29 ` Johannes Berg 2022-05-26 9:29 ` Johannes Berg 2022-05-27 5:31 ` Dmitry Vyukov 2022-05-27 5:31 ` Dmitry Vyukov 2022-05-27 7:32 ` Johannes Berg 2022-05-27 7:32 ` Johannes Berg 2022-05-27 10:36 ` Johannes Berg 2022-05-27 10:36 ` Johannes Berg 2022-05-27 13:05 ` Johannes Berg 2022-05-27 13:05 ` Johannes Berg 2022-05-27 13:09 ` Dmitry Vyukov 2022-05-27 13:09 ` Dmitry Vyukov 2022-05-27 13:15 ` Johannes Berg 2022-05-27 13:15 ` Johannes Berg 2022-05-27 13:18 ` Dmitry Vyukov 2022-05-27 13:18 ` Dmitry Vyukov 2022-05-27 13:27 ` Johannes Berg 2022-05-27 13:27 ` Johannes Berg 2022-05-27 13:52 ` Dmitry Vyukov 2022-05-27 13:52 ` Dmitry Vyukov 2022-05-27 14:27 ` Johannes Berg 2022-05-27 14:27 ` Johannes Berg 2022-05-27 15:46 ` Dmitry Vyukov 2022-05-27 15:46 ` Dmitry Vyukov 2020-03-29 19:06 ` [PATCH] " Richard Weinberger 2020-03-29 19:06 ` Richard Weinberger
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CACT4Y+YzM5bwvJ=yryrz1_y=uh=NX+2PNu4pLFaqQ2BMS39Fdg@mail.gmail.com' \ --to=dvyukov@google.com \ --cc=anton.ivanov@cambridgegreys.com \ --cc=aryabinin@virtuozzo.com \ --cc=brendanhiggins@google.com \ --cc=davidgow@google.com \ --cc=jdike@addtoit.com \ --cc=johannes@sipsolutions.net \ --cc=kasan-dev@googlegroups.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-um@lists.infradead.org \ --cc=richard@nod.at \ --cc=trishalfonso@google.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.