From: Andrey Konovalov <andreyknvl@google.com> To: Andrew Morton <akpm@linux-foundation.org>, Catalin Marinas <catalin.marinas@arm.com>, Vincenzo Frascino <vincenzo.frascino@arm.com>, Dmitry Vyukov <dvyukov@google.com>, Alexander Potapenko <glider@google.com>, Marco Elver <elver@google.com> Cc: Will Deacon <will.deacon@arm.com>, Andrey Ryabinin <aryabinin@virtuozzo.com>, Peter Collingbourne <pcc@google.com>, Evgenii Stepanov <eugenis@google.com>, Branislav Rankov <Branislav.Rankov@arm.com>, Kevin Brodsky <kevin.brodsky@arm.com>, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov <andreyknvl@google.com> Subject: [PATCH v2 08/12] kasan, mm: optimize krealloc poisoning Date: Fri, 5 Feb 2021 16:39:09 +0100 [thread overview] Message-ID: <1cf400f36ab1fd3c83e7626c3797cb11ebf9ef7f.1612538932.git.andreyknvl@google.com> (raw) In-Reply-To: <cover.1612538932.git.andreyknvl@google.com> Currently, krealloc() always calls ksize(), which unpoisons the whole object including the redzone. This is inefficient, as kasan_krealloc() repoisons the redzone for objects that fit into the same buffer. This patch changes krealloc() instrumentation to use uninstrumented __ksize() that doesn't unpoison the memory. Instead, kasan_kreallos() is changed to unpoison the memory excluding the redzone. For objects that don't fit into the old allocation, this patch disables KASAN accessibility checks when copying memory into a new object instead of unpoisoning it. Signed-off-by: Andrey Konovalov <andreyknvl@google.com> --- mm/kasan/common.c | 12 ++++++++++-- mm/slab_common.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 7ea643f7e69c..a8a67dca5e55 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -476,7 +476,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* * The object has already been unpoisoned by kasan_slab_alloc() for - * kmalloc() or by ksize() for krealloc(). + * kmalloc() or by kasan_krealloc() for krealloc(). */ /* @@ -526,7 +526,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, /* * The object has already been unpoisoned by kasan_alloc_pages() for - * alloc_pages() or by ksize() for krealloc(). + * alloc_pages() or by kasan_krealloc() for krealloc(). */ /* @@ -554,8 +554,16 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(object == ZERO_SIZE_PTR)) return (void *)object; + /* + * Unpoison the object's data. + * Part of it might already have been unpoisoned, but it's unknown + * how big that part is. + */ + kasan_unpoison(object, size); + page = virt_to_head_page(object); + /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else diff --git a/mm/slab_common.c b/mm/slab_common.c index dad70239b54c..60a2f49df6ce 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,19 +1140,27 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; - if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) - return NULL; - - ks = ksize(p); + /* Don't use instrumented ksize to allow precise KASAN poisoning. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks = kfence_ksize(p) ?: __ksize(p); + } else + ks = 0; + /* If the object still fits, repoison it precisely. */ if (ks >= new_size) { p = kasan_krealloc((void *)p, new_size, flags); return (void *)p; } ret = kmalloc_track_caller(new_size, flags); - if (ret && p) - memcpy(ret, p, ks); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } return ret; } -- 2.30.0.365.g02bc693789-goog
WARNING: multiple messages have this Message-ID (diff)
From: Andrey Konovalov <andreyknvl@google.com> To: Andrew Morton <akpm@linux-foundation.org>, Catalin Marinas <catalin.marinas@arm.com>, Vincenzo Frascino <vincenzo.frascino@arm.com>, Dmitry Vyukov <dvyukov@google.com>, Alexander Potapenko <glider@google.com>, Marco Elver <elver@google.com> Cc: Branislav Rankov <Branislav.Rankov@arm.com>, Andrey Konovalov <andreyknvl@google.com>, Kevin Brodsky <kevin.brodsky@arm.com>, Will Deacon <will.deacon@arm.com>, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin <aryabinin@virtuozzo.com>, Peter Collingbourne <pcc@google.com>, Evgenii Stepanov <eugenis@google.com> Subject: [PATCH v2 08/12] kasan, mm: optimize krealloc poisoning Date: Fri, 5 Feb 2021 16:39:09 +0100 [thread overview] Message-ID: <1cf400f36ab1fd3c83e7626c3797cb11ebf9ef7f.1612538932.git.andreyknvl@google.com> (raw) In-Reply-To: <cover.1612538932.git.andreyknvl@google.com> Currently, krealloc() always calls ksize(), which unpoisons the whole object including the redzone. This is inefficient, as kasan_krealloc() repoisons the redzone for objects that fit into the same buffer. This patch changes krealloc() instrumentation to use uninstrumented __ksize() that doesn't unpoison the memory. Instead, kasan_kreallos() is changed to unpoison the memory excluding the redzone. For objects that don't fit into the old allocation, this patch disables KASAN accessibility checks when copying memory into a new object instead of unpoisoning it. Signed-off-by: Andrey Konovalov <andreyknvl@google.com> --- mm/kasan/common.c | 12 ++++++++++-- mm/slab_common.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 7ea643f7e69c..a8a67dca5e55 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -476,7 +476,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* * The object has already been unpoisoned by kasan_slab_alloc() for - * kmalloc() or by ksize() for krealloc(). + * kmalloc() or by kasan_krealloc() for krealloc(). */ /* @@ -526,7 +526,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, /* * The object has already been unpoisoned by kasan_alloc_pages() for - * alloc_pages() or by ksize() for krealloc(). + * alloc_pages() or by kasan_krealloc() for krealloc(). */ /* @@ -554,8 +554,16 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(object == ZERO_SIZE_PTR)) return (void *)object; + /* + * Unpoison the object's data. + * Part of it might already have been unpoisoned, but it's unknown + * how big that part is. + */ + kasan_unpoison(object, size); + page = virt_to_head_page(object); + /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else diff --git a/mm/slab_common.c b/mm/slab_common.c index dad70239b54c..60a2f49df6ce 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,19 +1140,27 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; - if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) - return NULL; - - ks = ksize(p); + /* Don't use instrumented ksize to allow precise KASAN poisoning. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks = kfence_ksize(p) ?: __ksize(p); + } else + ks = 0; + /* If the object still fits, repoison it precisely. */ if (ks >= new_size) { p = kasan_krealloc((void *)p, new_size, flags); return (void *)p; } ret = kmalloc_track_caller(new_size, flags); - if (ret && p) - memcpy(ret, p, ks); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } return ret; } -- 2.30.0.365.g02bc693789-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-02-05 16:07 UTC|newest] Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-02-05 15:39 [PATCH v2 00/12] kasan: optimizations and fixes for HW_TAGS Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 01/12] kasan, mm: don't save alloc stacks twice Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 02/12] kasan, mm: optimize kmalloc poisoning Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 03/12] kasan: optimize large " Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 04/12] kasan: clean up setting free info in kasan_slab_free Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 05/12] kasan: unify large kfree checks Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 06/12] kasan: rework krealloc tests Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 07/12] kasan, mm: fail krealloc on freed objects Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov [this message] 2021-02-05 15:39 ` [PATCH v2 08/12] kasan, mm: optimize krealloc poisoning Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 09/12] kasan: ensure poisoning size alignment Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 10/12] arm64: kasan: simplify and inline MTE functions Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 11/12] kasan: inline HW_TAGS helper functions Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` [PATCH v2 12/12] arm64: kasan: export MTE symbols for KASAN tests Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-05 15:39 ` Andrey Konovalov 2021-02-08 17:42 ` Christoph Hellwig 2021-02-08 17:42 ` Christoph Hellwig 2021-02-09 15:18 ` Andrey Konovalov 2021-02-09 15:18 ` Andrey Konovalov 2021-02-09 15:18 ` Andrey Konovalov 2021-02-05 16:25 ` [PATCH v2 00/12] kasan: optimizations and fixes for HW_TAGS Andrey Konovalov 2021-02-05 16:25 ` Andrey Konovalov 2021-02-05 16:25 ` Andrey Konovalov 2021-02-05 16:53 ` Andrey Konovalov 2021-02-05 16:53 ` Andrey Konovalov 2021-02-05 16:53 ` Andrey Konovalov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1cf400f36ab1fd3c83e7626c3797cb11ebf9ef7f.1612538932.git.andreyknvl@google.com \ --to=andreyknvl@google.com \ --cc=Branislav.Rankov@arm.com \ --cc=akpm@linux-foundation.org \ --cc=aryabinin@virtuozzo.com \ --cc=catalin.marinas@arm.com \ --cc=dvyukov@google.com \ --cc=elver@google.com \ --cc=eugenis@google.com \ --cc=glider@google.com \ --cc=kasan-dev@googlegroups.com \ --cc=kevin.brodsky@arm.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=pcc@google.com \ --cc=vincenzo.frascino@arm.com \ --cc=will.deacon@arm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.