* [patch 01/60] kasan: drop unnecessary GPL text from comment headers
2020-12-22 19:58 incoming Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 02/60] kasan: KASAN_VMALLOC depends on KASAN_GENERIC Andrew Morton
` (59 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: drop unnecessary GPL text from comment headers
Patch series "kasan: add hardware tag-based mode for arm64", v11.
This patchset adds a new hardware tag-based mode to KASAN [1]. The new
mode is similar to the existing software tag-based KASAN, but relies on
arm64 Memory Tagging Extension (MTE) [2] to perform memory and pointer
tagging (instead of shadow memory and compiler instrumentation).
This patchset is co-developed and tested by
Vincenzo Frascino <vincenzo.frascino@arm.com>.
This patchset is available here:
https://github.com/xairy/linux/tree/up-kasan-mte-v11
For testing in QEMU hardware tag-based KASAN requires:
1. QEMU built from master [4] (use "-machine virt,mte=on -cpu max" arguments
to run).
2. GCC version 10.
[1] https://www.kernel.org/doc/html/latest/dev-tools/kasan.html
[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux for-next/mte
[4] https://github.com/qemu/qemu
====== Overview
The underlying ideas of the approach used by hardware tag-based KASAN are:
1. By relying on the Top Byte Ignore (TBI) arm64 CPU feature, pointer tags
are stored in the top byte of each kernel pointer.
2. With the Memory Tagging Extension (MTE) arm64 CPU feature, memory tags
for kernel memory allocations are stored in a dedicated memory not
accessible via normal instuctions.
3. On each memory allocation, a random tag is generated, embedded it into
the returned pointer, and the corresponding memory is tagged with the
same tag value.
4. With MTE the CPU performs a check on each memory access to make sure
that the pointer tag matches the memory tag.
5. On a tag mismatch the CPU generates a tag fault, and a KASAN report is
printed.
Same as other KASAN modes, hardware tag-based KASAN is intended as a
debugging feature at this point.
====== Rationale
There are two main reasons for this new hardware tag-based mode:
1. Previously implemented software tag-based KASAN is being successfully
used on dogfood testing devices due to its low memory overhead (as
initially planned). The new hardware mode keeps the same low memory
overhead, and is expected to have significantly lower performance
impact, due to the tag checks being performed by the hardware.
Therefore the new mode can be used as a better alternative in dogfood
testing for hardware that supports MTE.
2. The new mode lays the groundwork for the planned in-kernel MTE-based
memory corruption mitigation to be used in production.
====== Technical details
Considering the implementation perspective, hardware tag-based KASAN is
almost identical to the software mode. The key difference is using MTE
for assigning and checking tags.
Compared to the software mode, the hardware mode uses 4 bits per tag, as
dictated by MTE. Pointer tags are stored in bits [56:60), the top 4 bits
have the normal value 0xF. Having less distict tags increases the
probablity of false negatives (from ~1/256 to ~1/16) in certain cases.
Only synchronous exceptions are set up and used by hardware tag-based KASAN.
====== Benchmarks
Note: all measurements have been performed with software emulation of Memory
Tagging Extension, performance numbers for hardware tag-based KASAN on the
actual hardware are expected to be better.
Boot time [1]:
* 2.8 sec for clean kernel
* 5.7 sec for hardware tag-based KASAN
* 11.8 sec for software tag-based KASAN
* 11.6 sec for generic KASAN
Slab memory usage after boot [2]:
* 7.0 kb for clean kernel
* 9.7 kb for hardware tag-based KASAN
* 9.7 kb for software tag-based KASAN
* 41.3 kb for generic KASAN
Measurements have been performed with:
* defconfig-based configs
* Manually built QEMU master
* QEMU arguments: -machine virt,mte=on -cpu max
* CONFIG_KASAN_STACK_ENABLE disabled
* CONFIG_KASAN_INLINE enabled
* clang-10 as the compiler and gcc-10 as the assembler
[1] Time before the ext4 driver is initialized.
[2] Measured as `cat /proc/meminfo | grep Slab`.
====== Notes
The cover letter for software tag-based KASAN patchset can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0116523cfffa62aeb5aa3b85ce7419f3dae0c1b8
===== Tags
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
This patch (of 41):
Don't mention "GNU General Public License version 2" text explicitly, as
it's already covered by the SPDX-License-Identifier.
Link: https://lkml.kernel.org/r/cover.1606161801.git.andreyknvl@google.com
Link: https://lkml.kernel.org/r/6ea9f5f4aa9dbbffa0d0c0a780b37699a4531034.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 5 -----
mm/kasan/generic.c | 5 -----
mm/kasan/generic_report.c | 5 -----
mm/kasan/init.c | 5 -----
mm/kasan/quarantine.c | 10 ----------
mm/kasan/report.c | 5 -----
mm/kasan/tags.c | 5 -----
mm/kasan/tags_report.c | 5 -----
8 files changed, 45 deletions(-)
--- a/mm/kasan/common.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/common.c
@@ -7,11 +7,6 @@
*
* Some code borrowed from https://github.com/xairy/kasan-prototype by
* Andrey Konovalov <andreyknvl@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#include <linux/export.h>
--- a/mm/kasan/generic.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/generic.c
@@ -7,11 +7,6 @@
*
* Some code borrowed from https://github.com/xairy/kasan-prototype by
* Andrey Konovalov <andreyknvl@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
--- a/mm/kasan/generic_report.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/generic_report.c
@@ -7,11 +7,6 @@
*
* Some code borrowed from https://github.com/xairy/kasan-prototype by
* Andrey Konovalov <andreyknvl@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#include <linux/bitops.h>
--- a/mm/kasan/init.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/init.c
@@ -4,11 +4,6 @@
*
* Copyright (c) 2015 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#include <linux/memblock.h>
--- a/mm/kasan/quarantine.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/quarantine.c
@@ -6,16 +6,6 @@
* Copyright (C) 2016 Google, Inc.
*
* Based on code by Dmitry Chernenkov.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
*/
#include <linux/gfp.h>
--- a/mm/kasan/report.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/report.c
@@ -7,11 +7,6 @@
*
* Some code borrowed from https://github.com/xairy/kasan-prototype by
* Andrey Konovalov <andreyknvl@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#include <linux/bitops.h>
--- a/mm/kasan/tags.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/tags.c
@@ -4,11 +4,6 @@
*
* Copyright (c) 2018 Google, Inc.
* Author: Andrey Konovalov <andreyknvl@google.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
--- a/mm/kasan/tags_report.c~kasan-drop-unnecessary-gpl-text-from-comment-headers
+++ a/mm/kasan/tags_report.c
@@ -7,11 +7,6 @@
*
* Some code borrowed from https://github.com/xairy/kasan-prototype by
* Andrey Konovalov <andreyknvl@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#include <linux/bitops.h>
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 02/60] kasan: KASAN_VMALLOC depends on KASAN_GENERIC
2020-12-22 19:58 incoming Andrew Morton
2020-12-22 20:00 ` [patch 01/60] kasan: drop unnecessary GPL text from comment headers Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 03/60] kasan: group vmalloc code Andrew Morton
` (58 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: KASAN_VMALLOC depends on KASAN_GENERIC
Currently only generic KASAN mode supports vmalloc, reflect that in the
config.
Link: https://lkml.kernel.org/r/0c493d3a065ad95b04313d00244e884a7e2498ff.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
lib/Kconfig.kasan | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/lib/Kconfig.kasan~kasan-kasan_vmalloc-depends-on-kasan_generic
+++ a/lib/Kconfig.kasan
@@ -146,7 +146,7 @@ config KASAN_SW_TAGS_IDENTIFY
config KASAN_VMALLOC
bool "Back mappings in vmalloc space with real shadow memory"
- depends on HAVE_ARCH_KASAN_VMALLOC
+ depends on KASAN_GENERIC && HAVE_ARCH_KASAN_VMALLOC
help
By default, the shadow region for vmalloc space is the read-only
zero page. This means that KASAN cannot detect errors involving
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 03/60] kasan: group vmalloc code
2020-12-22 19:58 incoming Andrew Morton
2020-12-22 20:00 ` [patch 01/60] kasan: drop unnecessary GPL text from comment headers Andrew Morton
2020-12-22 20:00 ` [patch 02/60] kasan: KASAN_VMALLOC depends on KASAN_GENERIC Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 04/60] kasan: shadow declarations only for software modes Andrew Morton
` (57 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: group vmalloc code
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Group all vmalloc-related function declarations in include/linux/kasan.h,
and their implementations in mm/kasan/common.c.
No functional changes.
Link: https://lkml.kernel.org/r/80a6fdd29b039962843bd6cf22ce2643a7c8904e.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 41 +++++++++++----------
mm/kasan/common.c | 78 ++++++++++++++++++++--------------------
2 files changed, 63 insertions(+), 56 deletions(-)
--- a/include/linux/kasan.h~kasan-group-vmalloc-code
+++ a/include/linux/kasan.h
@@ -75,19 +75,6 @@ struct kasan_cache {
int free_meta_offset;
};
-/*
- * These functions provide a special case to support backing module
- * allocations with real shadow memory. With KASAN vmalloc, the special
- * case is unnecessary, as the work is handled in the generic case.
- */
-#ifndef CONFIG_KASAN_VMALLOC
-int kasan_module_alloc(void *addr, size_t size);
-void kasan_free_shadow(const struct vm_struct *vm);
-#else
-static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
-static inline void kasan_free_shadow(const struct vm_struct *vm) {}
-#endif
-
int kasan_add_zero_shadow(void *start, unsigned long size);
void kasan_remove_zero_shadow(void *start, unsigned long size);
@@ -156,9 +143,6 @@ static inline bool kasan_slab_free(struc
return false;
}
-static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
-static inline void kasan_free_shadow(const struct vm_struct *vm) {}
-
static inline int kasan_add_zero_shadow(void *start, unsigned long size)
{
return 0;
@@ -211,13 +195,16 @@ static inline void *kasan_reset_tag(cons
#endif /* CONFIG_KASAN_SW_TAGS */
#ifdef CONFIG_KASAN_VMALLOC
+
int kasan_populate_vmalloc(unsigned long addr, unsigned long size);
void kasan_poison_vmalloc(const void *start, unsigned long size);
void kasan_unpoison_vmalloc(const void *start, unsigned long size);
void kasan_release_vmalloc(unsigned long start, unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end);
-#else
+
+#else /* CONFIG_KASAN_VMALLOC */
+
static inline int kasan_populate_vmalloc(unsigned long start,
unsigned long size)
{
@@ -232,7 +219,25 @@ static inline void kasan_release_vmalloc
unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end) {}
-#endif
+
+#endif /* CONFIG_KASAN_VMALLOC */
+
+#if defined(CONFIG_KASAN) && !defined(CONFIG_KASAN_VMALLOC)
+
+/*
+ * These functions provide a special case to support backing module
+ * allocations with real shadow memory. With KASAN vmalloc, the special
+ * case is unnecessary, as the work is handled in the generic case.
+ */
+int kasan_module_alloc(void *addr, size_t size);
+void kasan_free_shadow(const struct vm_struct *vm);
+
+#else /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
+
+static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
+static inline void kasan_free_shadow(const struct vm_struct *vm) {}
+
+#endif /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
#ifdef CONFIG_KASAN_INLINE
void kasan_non_canonical_hook(unsigned long addr);
--- a/mm/kasan/common.c~kasan-group-vmalloc-code
+++ a/mm/kasan/common.c
@@ -536,44 +536,6 @@ void kasan_kfree_large(void *ptr, unsign
/* The object will be poisoned by page_alloc. */
}
-#ifndef CONFIG_KASAN_VMALLOC
-int kasan_module_alloc(void *addr, size_t size)
-{
- void *ret;
- size_t scaled_size;
- size_t shadow_size;
- unsigned long shadow_start;
-
- shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
- scaled_size = (size + KASAN_SHADOW_MASK) >> KASAN_SHADOW_SCALE_SHIFT;
- shadow_size = round_up(scaled_size, PAGE_SIZE);
-
- if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
- return -EINVAL;
-
- ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
- shadow_start + shadow_size,
- GFP_KERNEL,
- PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
- __builtin_return_address(0));
-
- if (ret) {
- __memset(ret, KASAN_SHADOW_INIT, shadow_size);
- find_vm_area(addr)->flags |= VM_KASAN;
- kmemleak_ignore(ret);
- return 0;
- }
-
- return -ENOMEM;
-}
-
-void kasan_free_shadow(const struct vm_struct *vm)
-{
- if (vm->flags & VM_KASAN)
- vfree(kasan_mem_to_shadow(vm->addr));
-}
-#endif
-
#ifdef CONFIG_MEMORY_HOTPLUG
static bool shadow_mapped(unsigned long addr)
{
@@ -685,6 +647,7 @@ core_initcall(kasan_memhotplug_init);
#endif
#ifdef CONFIG_KASAN_VMALLOC
+
static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
void *unused)
{
@@ -923,4 +886,43 @@ void kasan_release_vmalloc(unsigned long
(unsigned long)shadow_end);
}
}
+
+#else /* CONFIG_KASAN_VMALLOC */
+
+int kasan_module_alloc(void *addr, size_t size)
+{
+ void *ret;
+ size_t scaled_size;
+ size_t shadow_size;
+ unsigned long shadow_start;
+
+ shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
+ scaled_size = (size + KASAN_SHADOW_MASK) >> KASAN_SHADOW_SCALE_SHIFT;
+ shadow_size = round_up(scaled_size, PAGE_SIZE);
+
+ if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
+ return -EINVAL;
+
+ ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
+ shadow_start + shadow_size,
+ GFP_KERNEL,
+ PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
+ __builtin_return_address(0));
+
+ if (ret) {
+ __memset(ret, KASAN_SHADOW_INIT, shadow_size);
+ find_vm_area(addr)->flags |= VM_KASAN;
+ kmemleak_ignore(ret);
+ return 0;
+ }
+
+ return -ENOMEM;
+}
+
+void kasan_free_shadow(const struct vm_struct *vm)
+{
+ if (vm->flags & VM_KASAN)
+ vfree(kasan_mem_to_shadow(vm->addr));
+}
+
#endif
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 04/60] kasan: shadow declarations only for software modes
2020-12-22 19:58 incoming Andrew Morton
` (2 preceding siblings ...)
2020-12-22 20:00 ` [patch 03/60] kasan: group vmalloc code Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 05/60] kasan: rename (un)poison_shadow to (un)poison_range Andrew Morton
` (56 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, valdis.kletnieks, vincenzo.frascino,
will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: shadow declarations only for software modes
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Group shadow-related KASAN function declarations and only define them for
the two existing software modes.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/e88d94eff94db883a65dca52e1736d80d28dd9bc.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Valdis Kletnieks <valdis.kletnieks@vt.edu>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
[valdis.kletnieks@vt.edu: fix build issue with asmlinkage]
Link: https://lkml.kernel.org/r/35126.1606402815@turing-police
Link: https://lore.kernel.org/linux-arm-kernel/24105.1606397102@turing-police/
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 48 ++++++++++++++++++++++++++--------------
1 file changed, 32 insertions(+), 16 deletions(-)
--- a/include/linux/kasan.h~kasan-shadow-declarations-only-for-software-modes
+++ a/include/linux/kasan.h
@@ -11,7 +11,7 @@ struct task_struct;
#ifdef CONFIG_KASAN
-#include <linux/pgtable.h>
+#include <linux/linkage.h>
#include <asm/kasan.h>
/* kasan_data struct is used in KUnit tests for KASAN expected failures */
@@ -20,6 +20,20 @@ struct kunit_kasan_expectation {
bool report_found;
};
+#endif
+
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+
+#include <linux/pgtable.h>
+
+/* Software KASAN implementations use shadow memory. */
+
+#ifdef CONFIG_KASAN_SW_TAGS
+#define KASAN_SHADOW_INIT 0xFF
+#else
+#define KASAN_SHADOW_INIT 0
+#endif
+
extern unsigned char kasan_early_shadow_page[PAGE_SIZE];
extern pte_t kasan_early_shadow_pte[PTRS_PER_PTE];
extern pmd_t kasan_early_shadow_pmd[PTRS_PER_PMD];
@@ -35,6 +49,23 @@ static inline void *kasan_mem_to_shadow(
+ KASAN_SHADOW_OFFSET;
}
+int kasan_add_zero_shadow(void *start, unsigned long size);
+void kasan_remove_zero_shadow(void *start, unsigned long size);
+
+#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+static inline int kasan_add_zero_shadow(void *start, unsigned long size)
+{
+ return 0;
+}
+static inline void kasan_remove_zero_shadow(void *start,
+ unsigned long size)
+{}
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+#ifdef CONFIG_KASAN
+
/* Enable reporting bugs after kasan_disable_current() */
extern void kasan_enable_current(void);
@@ -75,9 +106,6 @@ struct kasan_cache {
int free_meta_offset;
};
-int kasan_add_zero_shadow(void *start, unsigned long size);
-void kasan_remove_zero_shadow(void *start, unsigned long size);
-
size_t __ksize(const void *);
static inline void kasan_unpoison_slab(const void *ptr)
{
@@ -143,14 +171,6 @@ static inline bool kasan_slab_free(struc
return false;
}
-static inline int kasan_add_zero_shadow(void *start, unsigned long size)
-{
- return 0;
-}
-static inline void kasan_remove_zero_shadow(void *start,
- unsigned long size)
-{}
-
static inline void kasan_unpoison_slab(const void *ptr) { }
static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
@@ -158,8 +178,6 @@ static inline size_t kasan_metadata_size
#ifdef CONFIG_KASAN_GENERIC
-#define KASAN_SHADOW_INIT 0
-
void kasan_cache_shrink(struct kmem_cache *cache);
void kasan_cache_shutdown(struct kmem_cache *cache);
void kasan_record_aux_stack(void *ptr);
@@ -174,8 +192,6 @@ static inline void kasan_record_aux_stac
#ifdef CONFIG_KASAN_SW_TAGS
-#define KASAN_SHADOW_INIT 0xFF
-
void kasan_init_tags(void);
void *kasan_reset_tag(const void *addr);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 05/60] kasan: rename (un)poison_shadow to (un)poison_range
2020-12-22 19:58 incoming Andrew Morton
` (3 preceding siblings ...)
2020-12-22 20:00 ` [patch 04/60] kasan: shadow declarations only for software modes Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 06/60] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_* Andrew Morton
` (55 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: rename (un)poison_shadow to (un)poison_range
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory. Rename external annotation
kasan_unpoison_shadow() to kasan_unpoison_range(), and introduce internal
functions (un)poison_range() (without kasan_ prefix).
Co-developed-by: Marco Elver <elver@google.com>
Link: https://lkml.kernel.org/r/fccdcaa13dc6b2211bf363d6c6d499279a54fe3a.1606161801.git.andreyknvl@google.com
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 6 ++--
kernel/fork.c | 4 +--
mm/kasan/common.c | 49 ++++++++++++++++++++++------------------
mm/kasan/generic.c | 23 ++++++++----------
mm/kasan/kasan.h | 3 +-
mm/kasan/tags.c | 2 -
mm/slab_common.c | 2 -
7 files changed, 47 insertions(+), 42 deletions(-)
--- a/include/linux/kasan.h~kasan-rename-unpoison_shadow-to-unpoison_range
+++ a/include/linux/kasan.h
@@ -72,7 +72,7 @@ extern void kasan_enable_current(void);
/* Disable reporting bugs for current task */
extern void kasan_disable_current(void);
-void kasan_unpoison_shadow(const void *address, size_t size);
+void kasan_unpoison_range(const void *address, size_t size);
void kasan_unpoison_task_stack(struct task_struct *task);
@@ -109,7 +109,7 @@ struct kasan_cache {
size_t __ksize(const void *);
static inline void kasan_unpoison_slab(const void *ptr)
{
- kasan_unpoison_shadow(ptr, __ksize(ptr));
+ kasan_unpoison_range(ptr, __ksize(ptr));
}
size_t kasan_metadata_size(struct kmem_cache *cache);
@@ -118,7 +118,7 @@ void kasan_restore_multi_shot(bool enabl
#else /* CONFIG_KASAN */
-static inline void kasan_unpoison_shadow(const void *address, size_t size) {}
+static inline void kasan_unpoison_range(const void *address, size_t size) {}
static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
--- a/kernel/fork.c~kasan-rename-unpoison_shadow-to-unpoison_range
+++ a/kernel/fork.c
@@ -225,8 +225,8 @@ static unsigned long *alloc_thread_stack
if (!s)
continue;
- /* Clear the KASAN shadow of the stack. */
- kasan_unpoison_shadow(s->addr, THREAD_SIZE);
+ /* Mark stack accessible for KASAN. */
+ kasan_unpoison_range(s->addr, THREAD_SIZE);
/* Clear stale pointers from reused stack. */
memset(s->addr, 0, THREAD_SIZE);
--- a/mm/kasan/common.c~kasan-rename-unpoison_shadow-to-unpoison_range
+++ a/mm/kasan/common.c
@@ -108,7 +108,7 @@ void *memcpy(void *dest, const void *src
* Poisons the shadow memory for 'size' bytes starting from 'addr'.
* Memory addresses should be aligned to KASAN_SHADOW_SCALE_SIZE.
*/
-void kasan_poison_shadow(const void *address, size_t size, u8 value)
+void poison_range(const void *address, size_t size, u8 value)
{
void *shadow_start, *shadow_end;
@@ -125,7 +125,7 @@ void kasan_poison_shadow(const void *add
__memset(shadow_start, value, shadow_end - shadow_start);
}
-void kasan_unpoison_shadow(const void *address, size_t size)
+void unpoison_range(const void *address, size_t size)
{
u8 tag = get_tag(address);
@@ -136,7 +136,7 @@ void kasan_unpoison_shadow(const void *a
*/
address = reset_tag(address);
- kasan_poison_shadow(address, size, tag);
+ poison_range(address, size, tag);
if (size & KASAN_SHADOW_MASK) {
u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
@@ -148,12 +148,17 @@ void kasan_unpoison_shadow(const void *a
}
}
+void kasan_unpoison_range(const void *address, size_t size)
+{
+ unpoison_range(address, size);
+}
+
static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
{
void *base = task_stack_page(task);
size_t size = sp - base;
- kasan_unpoison_shadow(base, size);
+ unpoison_range(base, size);
}
/* Unpoison the entire stack for a task. */
@@ -172,7 +177,7 @@ asmlinkage void kasan_unpoison_task_stac
*/
void *base = (void *)((unsigned long)watermark & ~(THREAD_SIZE - 1));
- kasan_unpoison_shadow(base, watermark - base);
+ unpoison_range(base, watermark - base);
}
void kasan_alloc_pages(struct page *page, unsigned int order)
@@ -186,13 +191,13 @@ void kasan_alloc_pages(struct page *page
tag = random_tag();
for (i = 0; i < (1 << order); i++)
page_kasan_tag_set(page + i, tag);
- kasan_unpoison_shadow(page_address(page), PAGE_SIZE << order);
+ unpoison_range(page_address(page), PAGE_SIZE << order);
}
void kasan_free_pages(struct page *page, unsigned int order)
{
if (likely(!PageHighMem(page)))
- kasan_poison_shadow(page_address(page),
+ poison_range(page_address(page),
PAGE_SIZE << order,
KASAN_FREE_PAGE);
}
@@ -284,18 +289,18 @@ void kasan_poison_slab(struct page *page
for (i = 0; i < compound_nr(page); i++)
page_kasan_tag_reset(page + i);
- kasan_poison_shadow(page_address(page), page_size(page),
- KASAN_KMALLOC_REDZONE);
+ poison_range(page_address(page), page_size(page),
+ KASAN_KMALLOC_REDZONE);
}
void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
{
- kasan_unpoison_shadow(object, cache->object_size);
+ unpoison_range(object, cache->object_size);
}
void kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
- kasan_poison_shadow(object,
+ poison_range(object,
round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE),
KASAN_KMALLOC_REDZONE);
}
@@ -408,7 +413,7 @@ static bool __kasan_slab_free(struct kme
}
rounded_up_size = round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE);
- kasan_poison_shadow(object, rounded_up_size, KASAN_KMALLOC_FREE);
+ poison_range(object, rounded_up_size, KASAN_KMALLOC_FREE);
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
unlikely(!(cache->flags & SLAB_KASAN)))
@@ -448,9 +453,9 @@ static void *__kasan_kmalloc(struct kmem
tag = assign_tag(cache, object, false, keep_tag);
/* Tag is ignored in set_tag without CONFIG_KASAN_SW_TAGS */
- kasan_unpoison_shadow(set_tag(object, tag), size);
- kasan_poison_shadow((void *)redzone_start, redzone_end - redzone_start,
- KASAN_KMALLOC_REDZONE);
+ unpoison_range(set_tag(object, tag), size);
+ poison_range((void *)redzone_start, redzone_end - redzone_start,
+ KASAN_KMALLOC_REDZONE);
if (cache->flags & SLAB_KASAN)
kasan_set_track(&get_alloc_info(cache, object)->alloc_track, flags);
@@ -489,9 +494,9 @@ void * __must_check kasan_kmalloc_large(
KASAN_SHADOW_SCALE_SIZE);
redzone_end = (unsigned long)ptr + page_size(page);
- kasan_unpoison_shadow(ptr, size);
- kasan_poison_shadow((void *)redzone_start, redzone_end - redzone_start,
- KASAN_PAGE_REDZONE);
+ unpoison_range(ptr, size);
+ poison_range((void *)redzone_start, redzone_end - redzone_start,
+ KASAN_PAGE_REDZONE);
return (void *)ptr;
}
@@ -523,7 +528,7 @@ void kasan_poison_kfree(void *ptr, unsig
kasan_report_invalid_free(ptr, ip);
return;
}
- kasan_poison_shadow(ptr, page_size(page), KASAN_FREE_PAGE);
+ poison_range(ptr, page_size(page), KASAN_FREE_PAGE);
} else {
__kasan_slab_free(page->slab_cache, ptr, ip, false);
}
@@ -709,7 +714,7 @@ int kasan_populate_vmalloc(unsigned long
* // vmalloc() allocates memory
* // let a = area->addr
* // we reach kasan_populate_vmalloc
- * // and call kasan_unpoison_shadow:
+ * // and call unpoison_range:
* STORE shadow(a), unpoison_val
* ...
* STORE shadow(a+99), unpoison_val x = LOAD p
@@ -744,7 +749,7 @@ void kasan_poison_vmalloc(const void *st
return;
size = round_up(size, KASAN_SHADOW_SCALE_SIZE);
- kasan_poison_shadow(start, size, KASAN_VMALLOC_INVALID);
+ poison_range(start, size, KASAN_VMALLOC_INVALID);
}
void kasan_unpoison_vmalloc(const void *start, unsigned long size)
@@ -752,7 +757,7 @@ void kasan_unpoison_vmalloc(const void *
if (!is_vmalloc_or_module_addr(start))
return;
- kasan_unpoison_shadow(start, size);
+ unpoison_range(start, size);
}
static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,
--- a/mm/kasan/generic.c~kasan-rename-unpoison_shadow-to-unpoison_range
+++ a/mm/kasan/generic.c
@@ -202,11 +202,11 @@ static void register_global(struct kasan
{
size_t aligned_size = round_up(global->size, KASAN_SHADOW_SCALE_SIZE);
- kasan_unpoison_shadow(global->beg, global->size);
+ unpoison_range(global->beg, global->size);
- kasan_poison_shadow(global->beg + aligned_size,
- global->size_with_redzone - aligned_size,
- KASAN_GLOBAL_REDZONE);
+ poison_range(global->beg + aligned_size,
+ global->size_with_redzone - aligned_size,
+ KASAN_GLOBAL_REDZONE);
}
void __asan_register_globals(struct kasan_global *globals, size_t size)
@@ -285,13 +285,12 @@ void __asan_alloca_poison(unsigned long
WARN_ON(!IS_ALIGNED(addr, KASAN_ALLOCA_REDZONE_SIZE));
- kasan_unpoison_shadow((const void *)(addr + rounded_down_size),
- size - rounded_down_size);
- kasan_poison_shadow(left_redzone, KASAN_ALLOCA_REDZONE_SIZE,
- KASAN_ALLOCA_LEFT);
- kasan_poison_shadow(right_redzone,
- padding_size + KASAN_ALLOCA_REDZONE_SIZE,
- KASAN_ALLOCA_RIGHT);
+ unpoison_range((const void *)(addr + rounded_down_size),
+ size - rounded_down_size);
+ poison_range(left_redzone, KASAN_ALLOCA_REDZONE_SIZE,
+ KASAN_ALLOCA_LEFT);
+ poison_range(right_redzone, padding_size + KASAN_ALLOCA_REDZONE_SIZE,
+ KASAN_ALLOCA_RIGHT);
}
EXPORT_SYMBOL(__asan_alloca_poison);
@@ -301,7 +300,7 @@ void __asan_allocas_unpoison(const void
if (unlikely(!stack_top || stack_top > stack_bottom))
return;
- kasan_unpoison_shadow(stack_top, stack_bottom - stack_top);
+ unpoison_range(stack_top, stack_bottom - stack_top);
}
EXPORT_SYMBOL(__asan_allocas_unpoison);
--- a/mm/kasan/kasan.h~kasan-rename-unpoison_shadow-to-unpoison_range
+++ a/mm/kasan/kasan.h
@@ -150,7 +150,8 @@ static inline bool addr_has_shadow(const
return (addr >= kasan_shadow_to_mem((void *)KASAN_SHADOW_START));
}
-void kasan_poison_shadow(const void *address, size_t size, u8 value);
+void poison_range(const void *address, size_t size, u8 value);
+void unpoison_range(const void *address, size_t size);
/**
* check_memory_region - Check memory region, and report if invalid access.
--- a/mm/kasan/tags.c~kasan-rename-unpoison_shadow-to-unpoison_range
+++ a/mm/kasan/tags.c
@@ -153,7 +153,7 @@ EXPORT_SYMBOL(__hwasan_storeN_noabort);
void __hwasan_tag_memory(unsigned long addr, u8 tag, unsigned long size)
{
- kasan_poison_shadow((void *)addr, size, tag);
+ poison_range((void *)addr, size, tag);
}
EXPORT_SYMBOL(__hwasan_tag_memory);
--- a/mm/slab_common.c~kasan-rename-unpoison_shadow-to-unpoison_range
+++ a/mm/slab_common.c
@@ -1176,7 +1176,7 @@ size_t ksize(const void *objp)
* We assume that ksize callers could use whole allocated area,
* so we need to unpoison this area.
*/
- kasan_unpoison_shadow(objp, size);
+ kasan_unpoison_range(objp, size);
return size;
}
EXPORT_SYMBOL(ksize);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 06/60] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_*
2020-12-22 19:58 incoming Andrew Morton
` (4 preceding siblings ...)
2020-12-22 20:00 ` [patch 05/60] kasan: rename (un)poison_shadow to (un)poison_range Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 07/60] kasan: only build init.c for software modes Andrew Morton
` (54 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_*
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory, but will still use the concept
of memory granules. Each memory granule maps to a single metadata entry:
8 bytes per one shadow byte for generic mode, 16 bytes per one shadow byte
for software tag-based mode, and 16 bytes per one allocation tag for
hardware tag-based mode.
Rename KASAN_SHADOW_SCALE_SIZE to KASAN_GRANULE_SIZE, and
KASAN_SHADOW_MASK to KASAN_GRANULE_MASK.
Also use MASK when used as a mask, otherwise use SIZE.
No functional changes.
Link: https://lkml.kernel.org/r/939b5754e47f528a6e6a6f28ffc5815d8d128033.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
Documentation/dev-tools/kasan.rst | 2 -
lib/test_kasan.c | 2 -
lib/test_kasan_module.c | 2 -
mm/kasan/common.c | 39 ++++++++++++++--------------
mm/kasan/generic.c | 14 +++++-----
mm/kasan/generic_report.c | 8 ++---
mm/kasan/init.c | 8 ++---
mm/kasan/kasan.h | 4 +-
mm/kasan/report.c | 10 +++----
mm/kasan/tags_report.c | 2 -
10 files changed, 46 insertions(+), 45 deletions(-)
--- a/Documentation/dev-tools/kasan.rst~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/Documentation/dev-tools/kasan.rst
@@ -265,7 +265,7 @@ Most mappings in vmalloc space are small
page of shadow space. Allocating a full shadow page per mapping would
therefore be wasteful. Furthermore, to ensure that different mappings
use different shadow pages, mappings would have to be aligned to
-``KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE``.
+``KASAN_GRANULE_SIZE * PAGE_SIZE``.
Instead, we share backing space across multiple mappings. We allocate
a backing page when a mapping in vmalloc space uses a particular page
--- a/lib/test_kasan.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/lib/test_kasan.c
@@ -25,7 +25,7 @@
#include "../mm/kasan/kasan.h"
-#define OOB_TAG_OFF (IS_ENABLED(CONFIG_KASAN_GENERIC) ? 0 : KASAN_SHADOW_SCALE_SIZE)
+#define OOB_TAG_OFF (IS_ENABLED(CONFIG_KASAN_GENERIC) ? 0 : KASAN_GRANULE_SIZE)
/*
* We assign some test results to these globals to make sure the tests
--- a/lib/test_kasan_module.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/lib/test_kasan_module.c
@@ -15,7 +15,7 @@
#include "../mm/kasan/kasan.h"
-#define OOB_TAG_OFF (IS_ENABLED(CONFIG_KASAN_GENERIC) ? 0 : KASAN_SHADOW_SCALE_SIZE)
+#define OOB_TAG_OFF (IS_ENABLED(CONFIG_KASAN_GENERIC) ? 0 : KASAN_GRANULE_SIZE)
static noinline void __init copy_user_test(void)
{
--- a/mm/kasan/common.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/mm/kasan/common.c
@@ -106,7 +106,7 @@ void *memcpy(void *dest, const void *src
/*
* Poisons the shadow memory for 'size' bytes starting from 'addr'.
- * Memory addresses should be aligned to KASAN_SHADOW_SCALE_SIZE.
+ * Memory addresses should be aligned to KASAN_GRANULE_SIZE.
*/
void poison_range(const void *address, size_t size, u8 value)
{
@@ -138,13 +138,13 @@ void unpoison_range(const void *address,
poison_range(address, size, tag);
- if (size & KASAN_SHADOW_MASK) {
+ if (size & KASAN_GRANULE_MASK) {
u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
*shadow = tag;
else
- *shadow = size & KASAN_SHADOW_MASK;
+ *shadow = size & KASAN_GRANULE_MASK;
}
}
@@ -301,7 +301,7 @@ void kasan_unpoison_object_data(struct k
void kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
poison_range(object,
- round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE),
+ round_up(cache->object_size, KASAN_GRANULE_SIZE),
KASAN_KMALLOC_REDZONE);
}
@@ -373,7 +373,7 @@ static inline bool shadow_invalid(u8 tag
{
if (IS_ENABLED(CONFIG_KASAN_GENERIC))
return shadow_byte < 0 ||
- shadow_byte >= KASAN_SHADOW_SCALE_SIZE;
+ shadow_byte >= KASAN_GRANULE_SIZE;
/* else CONFIG_KASAN_SW_TAGS: */
if ((u8)shadow_byte == KASAN_TAG_INVALID)
@@ -412,7 +412,7 @@ static bool __kasan_slab_free(struct kme
return true;
}
- rounded_up_size = round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE);
+ rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
poison_range(object, rounded_up_size, KASAN_KMALLOC_FREE);
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
@@ -445,9 +445,9 @@ static void *__kasan_kmalloc(struct kmem
return NULL;
redzone_start = round_up((unsigned long)(object + size),
- KASAN_SHADOW_SCALE_SIZE);
+ KASAN_GRANULE_SIZE);
redzone_end = round_up((unsigned long)object + cache->object_size,
- KASAN_SHADOW_SCALE_SIZE);
+ KASAN_GRANULE_SIZE);
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
tag = assign_tag(cache, object, false, keep_tag);
@@ -491,7 +491,7 @@ void * __must_check kasan_kmalloc_large(
page = virt_to_page(ptr);
redzone_start = round_up((unsigned long)(ptr + size),
- KASAN_SHADOW_SCALE_SIZE);
+ KASAN_GRANULE_SIZE);
redzone_end = (unsigned long)ptr + page_size(page);
unpoison_range(ptr, size);
@@ -589,8 +589,8 @@ static int __meminit kasan_mem_notifier(
shadow_size = nr_shadow_pages << PAGE_SHIFT;
shadow_end = shadow_start + shadow_size;
- if (WARN_ON(mem_data->nr_pages % KASAN_SHADOW_SCALE_SIZE) ||
- WARN_ON(start_kaddr % (KASAN_SHADOW_SCALE_SIZE << PAGE_SHIFT)))
+ if (WARN_ON(mem_data->nr_pages % KASAN_GRANULE_SIZE) ||
+ WARN_ON(start_kaddr % (KASAN_GRANULE_SIZE << PAGE_SHIFT)))
return NOTIFY_BAD;
switch (action) {
@@ -748,7 +748,7 @@ void kasan_poison_vmalloc(const void *st
if (!is_vmalloc_or_module_addr(start))
return;
- size = round_up(size, KASAN_SHADOW_SCALE_SIZE);
+ size = round_up(size, KASAN_GRANULE_SIZE);
poison_range(start, size, KASAN_VMALLOC_INVALID);
}
@@ -861,22 +861,22 @@ void kasan_release_vmalloc(unsigned long
unsigned long region_start, region_end;
unsigned long size;
- region_start = ALIGN(start, PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
- region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
+ region_start = ALIGN(start, PAGE_SIZE * KASAN_GRANULE_SIZE);
+ region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_GRANULE_SIZE);
free_region_start = ALIGN(free_region_start,
- PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
if (start != region_start &&
free_region_start < region_start)
- region_start -= PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE;
+ region_start -= PAGE_SIZE * KASAN_GRANULE_SIZE;
free_region_end = ALIGN_DOWN(free_region_end,
- PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
if (end != region_end &&
free_region_end > region_end)
- region_end += PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE;
+ region_end += PAGE_SIZE * KASAN_GRANULE_SIZE;
shadow_start = kasan_mem_to_shadow((void *)region_start);
shadow_end = kasan_mem_to_shadow((void *)region_end);
@@ -902,7 +902,8 @@ int kasan_module_alloc(void *addr, size_
unsigned long shadow_start;
shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
- scaled_size = (size + KASAN_SHADOW_MASK) >> KASAN_SHADOW_SCALE_SHIFT;
+ scaled_size = (size + KASAN_GRANULE_SIZE - 1) >>
+ KASAN_SHADOW_SCALE_SHIFT;
shadow_size = round_up(scaled_size, PAGE_SIZE);
if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
--- a/mm/kasan/generic.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/mm/kasan/generic.c
@@ -46,7 +46,7 @@ static __always_inline bool memory_is_po
s8 shadow_value = *(s8 *)kasan_mem_to_shadow((void *)addr);
if (unlikely(shadow_value)) {
- s8 last_accessible_byte = addr & KASAN_SHADOW_MASK;
+ s8 last_accessible_byte = addr & KASAN_GRANULE_MASK;
return unlikely(last_accessible_byte >= shadow_value);
}
@@ -62,7 +62,7 @@ static __always_inline bool memory_is_po
* Access crosses 8(shadow size)-byte boundary. Such access maps
* into 2 shadow bytes, so we need to check them both.
*/
- if (unlikely(((addr + size - 1) & KASAN_SHADOW_MASK) < size - 1))
+ if (unlikely(((addr + size - 1) & KASAN_GRANULE_MASK) < size - 1))
return *shadow_addr || memory_is_poisoned_1(addr + size - 1);
return memory_is_poisoned_1(addr + size - 1);
@@ -73,7 +73,7 @@ static __always_inline bool memory_is_po
u16 *shadow_addr = (u16 *)kasan_mem_to_shadow((void *)addr);
/* Unaligned 16-bytes access maps into 3 shadow bytes. */
- if (unlikely(!IS_ALIGNED(addr, KASAN_SHADOW_SCALE_SIZE)))
+ if (unlikely(!IS_ALIGNED(addr, KASAN_GRANULE_SIZE)))
return *shadow_addr || memory_is_poisoned_1(addr + 15);
return *shadow_addr;
@@ -134,7 +134,7 @@ static __always_inline bool memory_is_po
s8 *last_shadow = (s8 *)kasan_mem_to_shadow((void *)last_byte);
if (unlikely(ret != (unsigned long)last_shadow ||
- ((long)(last_byte & KASAN_SHADOW_MASK) >= *last_shadow)))
+ ((long)(last_byte & KASAN_GRANULE_MASK) >= *last_shadow)))
return true;
}
return false;
@@ -200,7 +200,7 @@ void kasan_cache_shutdown(struct kmem_ca
static void register_global(struct kasan_global *global)
{
- size_t aligned_size = round_up(global->size, KASAN_SHADOW_SCALE_SIZE);
+ size_t aligned_size = round_up(global->size, KASAN_GRANULE_SIZE);
unpoison_range(global->beg, global->size);
@@ -274,10 +274,10 @@ EXPORT_SYMBOL(__asan_handle_no_return);
/* Emitted by compiler to poison alloca()ed objects. */
void __asan_alloca_poison(unsigned long addr, size_t size)
{
- size_t rounded_up_size = round_up(size, KASAN_SHADOW_SCALE_SIZE);
+ size_t rounded_up_size = round_up(size, KASAN_GRANULE_SIZE);
size_t padding_size = round_up(size, KASAN_ALLOCA_REDZONE_SIZE) -
rounded_up_size;
- size_t rounded_down_size = round_down(size, KASAN_SHADOW_SCALE_SIZE);
+ size_t rounded_down_size = round_down(size, KASAN_GRANULE_SIZE);
const void *left_redzone = (const void *)(addr -
KASAN_ALLOCA_REDZONE_SIZE);
--- a/mm/kasan/generic_report.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/mm/kasan/generic_report.c
@@ -34,7 +34,7 @@ void *find_first_bad_addr(void *addr, si
void *p = addr;
while (p < addr + size && !(*(u8 *)kasan_mem_to_shadow(p)))
- p += KASAN_SHADOW_SCALE_SIZE;
+ p += KASAN_GRANULE_SIZE;
return p;
}
@@ -46,14 +46,14 @@ static const char *get_shadow_bug_type(s
shadow_addr = (u8 *)kasan_mem_to_shadow(info->first_bad_addr);
/*
- * If shadow byte value is in [0, KASAN_SHADOW_SCALE_SIZE) we can look
+ * If shadow byte value is in [0, KASAN_GRANULE_SIZE) we can look
* at the next shadow byte to determine the type of the bad access.
*/
- if (*shadow_addr > 0 && *shadow_addr <= KASAN_SHADOW_SCALE_SIZE - 1)
+ if (*shadow_addr > 0 && *shadow_addr <= KASAN_GRANULE_SIZE - 1)
shadow_addr++;
switch (*shadow_addr) {
- case 0 ... KASAN_SHADOW_SCALE_SIZE - 1:
+ case 0 ... KASAN_GRANULE_SIZE - 1:
/*
* In theory it's still possible to see these shadow values
* due to a data race in the kernel code.
--- a/mm/kasan/init.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/mm/kasan/init.c
@@ -442,8 +442,8 @@ void kasan_remove_zero_shadow(void *star
end = addr + (size >> KASAN_SHADOW_SCALE_SHIFT);
if (WARN_ON((unsigned long)start %
- (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)) ||
- WARN_ON(size % (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)))
+ (KASAN_GRANULE_SIZE * PAGE_SIZE)) ||
+ WARN_ON(size % (KASAN_GRANULE_SIZE * PAGE_SIZE)))
return;
for (; addr < end; addr = next) {
@@ -477,8 +477,8 @@ int kasan_add_zero_shadow(void *start, u
shadow_end = shadow_start + (size >> KASAN_SHADOW_SCALE_SHIFT);
if (WARN_ON((unsigned long)start %
- (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)) ||
- WARN_ON(size % (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)))
+ (KASAN_GRANULE_SIZE * PAGE_SIZE)) ||
+ WARN_ON(size % (KASAN_GRANULE_SIZE * PAGE_SIZE)))
return -EINVAL;
ret = kasan_populate_early_shadow(shadow_start, shadow_end);
--- a/mm/kasan/kasan.h~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/mm/kasan/kasan.h
@@ -5,8 +5,8 @@
#include <linux/kasan.h>
#include <linux/stackdepot.h>
-#define KASAN_SHADOW_SCALE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
-#define KASAN_SHADOW_MASK (KASAN_SHADOW_SCALE_SIZE - 1)
+#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
+#define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
#define KASAN_TAG_INVALID 0xFE /* inaccessible memory tag */
--- a/mm/kasan/report.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/mm/kasan/report.c
@@ -314,24 +314,24 @@ static bool __must_check get_address_sta
return false;
aligned_addr = round_down((unsigned long)addr, sizeof(long));
- mem_ptr = round_down(aligned_addr, KASAN_SHADOW_SCALE_SIZE);
+ mem_ptr = round_down(aligned_addr, KASAN_GRANULE_SIZE);
shadow_ptr = kasan_mem_to_shadow((void *)aligned_addr);
shadow_bottom = kasan_mem_to_shadow(end_of_stack(current));
while (shadow_ptr >= shadow_bottom && *shadow_ptr != KASAN_STACK_LEFT) {
shadow_ptr--;
- mem_ptr -= KASAN_SHADOW_SCALE_SIZE;
+ mem_ptr -= KASAN_GRANULE_SIZE;
}
while (shadow_ptr >= shadow_bottom && *shadow_ptr == KASAN_STACK_LEFT) {
shadow_ptr--;
- mem_ptr -= KASAN_SHADOW_SCALE_SIZE;
+ mem_ptr -= KASAN_GRANULE_SIZE;
}
if (shadow_ptr < shadow_bottom)
return false;
- frame = (const unsigned long *)(mem_ptr + KASAN_SHADOW_SCALE_SIZE);
+ frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",
frame[0]);
@@ -599,6 +599,6 @@ void kasan_non_canonical_hook(unsigned l
else
bug_type = "maybe wild-memory-access";
pr_alert("KASAN: %s in range [0x%016lx-0x%016lx]\n", bug_type,
- orig_addr, orig_addr + KASAN_SHADOW_MASK);
+ orig_addr, orig_addr + KASAN_GRANULE_SIZE - 1);
}
#endif
--- a/mm/kasan/tags_report.c~kasan-rename-kasan_shadow_-to-kasan_granule_
+++ a/mm/kasan/tags_report.c
@@ -76,7 +76,7 @@ void *find_first_bad_addr(void *addr, si
void *end = p + size;
while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p))
- p += KASAN_SHADOW_SCALE_SIZE;
+ p += KASAN_GRANULE_SIZE;
return p;
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 07/60] kasan: only build init.c for software modes
2020-12-22 19:58 incoming Andrew Morton
` (5 preceding siblings ...)
2020-12-22 20:00 ` [patch 06/60] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_* Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 08/60] kasan: split out shadow.c from common.c Andrew Morton
` (53 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: only build init.c for software modes
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory, so only build init.c that
contains shadow initialization code for software modes.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/bae0a6a35b7a9b1a443803c1a55e6e3fecc311c9.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/Makefile | 6 +++---
mm/kasan/init.c | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
--- a/mm/kasan/init.c~kasan-only-build-initc-for-software-modes
+++ a/mm/kasan/init.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains some kasan initialization code.
+ * This file contains KASAN shadow initialization code.
*
* Copyright (c) 2015 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
--- a/mm/kasan/Makefile~kasan-only-build-initc-for-software-modes
+++ a/mm/kasan/Makefile
@@ -29,6 +29,6 @@ CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNT
CFLAGS_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags_report.o := $(CC_FLAGS_KASAN_RUNTIME)
-obj-$(CONFIG_KASAN) := common.o init.o report.o
-obj-$(CONFIG_KASAN_GENERIC) += generic.o generic_report.o quarantine.o
-obj-$(CONFIG_KASAN_SW_TAGS) += tags.o tags_report.o
+obj-$(CONFIG_KASAN) := common.o report.o
+obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o quarantine.o
+obj-$(CONFIG_KASAN_SW_TAGS) += init.o tags.o tags_report.o
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 08/60] kasan: split out shadow.c from common.c
2020-12-22 19:58 incoming Andrew Morton
` (6 preceding siblings ...)
2020-12-22 20:00 ` [patch 07/60] kasan: only build init.c for software modes Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 09/60] kasan: define KASAN_MEMORY_PER_SHADOW_PAGE Andrew Morton
` (52 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: split out shadow.c from common.c
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory. Move all shadow-related code
to shadow.c, which is only enabled for software KASAN modes that use
shadow memory.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/17d95cfa7d5cf9c4fcd9bf415f2a8dea911668df.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/Makefile | 6
mm/kasan/common.c | 486 ------------------------------------------
mm/kasan/shadow.c | 505 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 510 insertions(+), 487 deletions(-)
--- a/mm/kasan/common.c~kasan-split-out-shadowc-from-commonc
+++ a/mm/kasan/common.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains common generic and tag-based KASAN code.
+ * This file contains common KASAN code.
*
* Copyright (c) 2014 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
@@ -13,7 +13,6 @@
#include <linux/init.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
-#include <linux/kmemleak.h>
#include <linux/linkage.h>
#include <linux/memblock.h>
#include <linux/memory.h>
@@ -26,12 +25,8 @@
#include <linux/stacktrace.h>
#include <linux/string.h>
#include <linux/types.h>
-#include <linux/vmalloc.h>
#include <linux/bug.h>
-#include <asm/cacheflush.h>
-#include <asm/tlbflush.h>
-
#include "kasan.h"
#include "../slab.h"
@@ -61,93 +56,6 @@ void kasan_disable_current(void)
current->kasan_depth--;
}
-bool __kasan_check_read(const volatile void *p, unsigned int size)
-{
- return check_memory_region((unsigned long)p, size, false, _RET_IP_);
-}
-EXPORT_SYMBOL(__kasan_check_read);
-
-bool __kasan_check_write(const volatile void *p, unsigned int size)
-{
- return check_memory_region((unsigned long)p, size, true, _RET_IP_);
-}
-EXPORT_SYMBOL(__kasan_check_write);
-
-#undef memset
-void *memset(void *addr, int c, size_t len)
-{
- if (!check_memory_region((unsigned long)addr, len, true, _RET_IP_))
- return NULL;
-
- return __memset(addr, c, len);
-}
-
-#ifdef __HAVE_ARCH_MEMMOVE
-#undef memmove
-void *memmove(void *dest, const void *src, size_t len)
-{
- if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
- !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
- return NULL;
-
- return __memmove(dest, src, len);
-}
-#endif
-
-#undef memcpy
-void *memcpy(void *dest, const void *src, size_t len)
-{
- if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
- !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
- return NULL;
-
- return __memcpy(dest, src, len);
-}
-
-/*
- * Poisons the shadow memory for 'size' bytes starting from 'addr'.
- * Memory addresses should be aligned to KASAN_GRANULE_SIZE.
- */
-void poison_range(const void *address, size_t size, u8 value)
-{
- void *shadow_start, *shadow_end;
-
- /*
- * Perform shadow offset calculation based on untagged address, as
- * some of the callers (e.g. kasan_poison_object_data) pass tagged
- * addresses to this function.
- */
- address = reset_tag(address);
-
- shadow_start = kasan_mem_to_shadow(address);
- shadow_end = kasan_mem_to_shadow(address + size);
-
- __memset(shadow_start, value, shadow_end - shadow_start);
-}
-
-void unpoison_range(const void *address, size_t size)
-{
- u8 tag = get_tag(address);
-
- /*
- * Perform shadow offset calculation based on untagged address, as
- * some of the callers (e.g. kasan_unpoison_object_data) pass tagged
- * addresses to this function.
- */
- address = reset_tag(address);
-
- poison_range(address, size, tag);
-
- if (size & KASAN_GRANULE_MASK) {
- u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
-
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
- *shadow = tag;
- else
- *shadow = size & KASAN_GRANULE_MASK;
- }
-}
-
void kasan_unpoison_range(const void *address, size_t size)
{
unpoison_range(address, size);
@@ -540,395 +448,3 @@ void kasan_kfree_large(void *ptr, unsign
kasan_report_invalid_free(ptr, ip);
/* The object will be poisoned by page_alloc. */
}
-
-#ifdef CONFIG_MEMORY_HOTPLUG
-static bool shadow_mapped(unsigned long addr)
-{
- pgd_t *pgd = pgd_offset_k(addr);
- p4d_t *p4d;
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
-
- if (pgd_none(*pgd))
- return false;
- p4d = p4d_offset(pgd, addr);
- if (p4d_none(*p4d))
- return false;
- pud = pud_offset(p4d, addr);
- if (pud_none(*pud))
- return false;
-
- /*
- * We can't use pud_large() or pud_huge(), the first one is
- * arch-specific, the last one depends on HUGETLB_PAGE. So let's abuse
- * pud_bad(), if pud is bad then it's bad because it's huge.
- */
- if (pud_bad(*pud))
- return true;
- pmd = pmd_offset(pud, addr);
- if (pmd_none(*pmd))
- return false;
-
- if (pmd_bad(*pmd))
- return true;
- pte = pte_offset_kernel(pmd, addr);
- return !pte_none(*pte);
-}
-
-static int __meminit kasan_mem_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- struct memory_notify *mem_data = data;
- unsigned long nr_shadow_pages, start_kaddr, shadow_start;
- unsigned long shadow_end, shadow_size;
-
- nr_shadow_pages = mem_data->nr_pages >> KASAN_SHADOW_SCALE_SHIFT;
- start_kaddr = (unsigned long)pfn_to_kaddr(mem_data->start_pfn);
- shadow_start = (unsigned long)kasan_mem_to_shadow((void *)start_kaddr);
- shadow_size = nr_shadow_pages << PAGE_SHIFT;
- shadow_end = shadow_start + shadow_size;
-
- if (WARN_ON(mem_data->nr_pages % KASAN_GRANULE_SIZE) ||
- WARN_ON(start_kaddr % (KASAN_GRANULE_SIZE << PAGE_SHIFT)))
- return NOTIFY_BAD;
-
- switch (action) {
- case MEM_GOING_ONLINE: {
- void *ret;
-
- /*
- * If shadow is mapped already than it must have been mapped
- * during the boot. This could happen if we onlining previously
- * offlined memory.
- */
- if (shadow_mapped(shadow_start))
- return NOTIFY_OK;
-
- ret = __vmalloc_node_range(shadow_size, PAGE_SIZE, shadow_start,
- shadow_end, GFP_KERNEL,
- PAGE_KERNEL, VM_NO_GUARD,
- pfn_to_nid(mem_data->start_pfn),
- __builtin_return_address(0));
- if (!ret)
- return NOTIFY_BAD;
-
- kmemleak_ignore(ret);
- return NOTIFY_OK;
- }
- case MEM_CANCEL_ONLINE:
- case MEM_OFFLINE: {
- struct vm_struct *vm;
-
- /*
- * shadow_start was either mapped during boot by kasan_init()
- * or during memory online by __vmalloc_node_range().
- * In the latter case we can use vfree() to free shadow.
- * Non-NULL result of the find_vm_area() will tell us if
- * that was the second case.
- *
- * Currently it's not possible to free shadow mapped
- * during boot by kasan_init(). It's because the code
- * to do that hasn't been written yet. So we'll just
- * leak the memory.
- */
- vm = find_vm_area((void *)shadow_start);
- if (vm)
- vfree((void *)shadow_start);
- }
- }
-
- return NOTIFY_OK;
-}
-
-static int __init kasan_memhotplug_init(void)
-{
- hotplug_memory_notifier(kasan_mem_notifier, 0);
-
- return 0;
-}
-
-core_initcall(kasan_memhotplug_init);
-#endif
-
-#ifdef CONFIG_KASAN_VMALLOC
-
-static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
- void *unused)
-{
- unsigned long page;
- pte_t pte;
-
- if (likely(!pte_none(*ptep)))
- return 0;
-
- page = __get_free_page(GFP_KERNEL);
- if (!page)
- return -ENOMEM;
-
- memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
- pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
-
- spin_lock(&init_mm.page_table_lock);
- if (likely(pte_none(*ptep))) {
- set_pte_at(&init_mm, addr, ptep, pte);
- page = 0;
- }
- spin_unlock(&init_mm.page_table_lock);
- if (page)
- free_page(page);
- return 0;
-}
-
-int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
-{
- unsigned long shadow_start, shadow_end;
- int ret;
-
- if (!is_vmalloc_or_module_addr((void *)addr))
- return 0;
-
- shadow_start = (unsigned long)kasan_mem_to_shadow((void *)addr);
- shadow_start = ALIGN_DOWN(shadow_start, PAGE_SIZE);
- shadow_end = (unsigned long)kasan_mem_to_shadow((void *)addr + size);
- shadow_end = ALIGN(shadow_end, PAGE_SIZE);
-
- ret = apply_to_page_range(&init_mm, shadow_start,
- shadow_end - shadow_start,
- kasan_populate_vmalloc_pte, NULL);
- if (ret)
- return ret;
-
- flush_cache_vmap(shadow_start, shadow_end);
-
- /*
- * We need to be careful about inter-cpu effects here. Consider:
- *
- * CPU#0 CPU#1
- * WRITE_ONCE(p, vmalloc(100)); while (x = READ_ONCE(p)) ;
- * p[99] = 1;
- *
- * With compiler instrumentation, that ends up looking like this:
- *
- * CPU#0 CPU#1
- * // vmalloc() allocates memory
- * // let a = area->addr
- * // we reach kasan_populate_vmalloc
- * // and call unpoison_range:
- * STORE shadow(a), unpoison_val
- * ...
- * STORE shadow(a+99), unpoison_val x = LOAD p
- * // rest of vmalloc process <data dependency>
- * STORE p, a LOAD shadow(x+99)
- *
- * If there is no barrier between the end of unpoisioning the shadow
- * and the store of the result to p, the stores could be committed
- * in a different order by CPU#0, and CPU#1 could erroneously observe
- * poison in the shadow.
- *
- * We need some sort of barrier between the stores.
- *
- * In the vmalloc() case, this is provided by a smp_wmb() in
- * clear_vm_uninitialized_flag(). In the per-cpu allocator and in
- * get_vm_area() and friends, the caller gets shadow allocated but
- * doesn't have any pages mapped into the virtual address space that
- * has been reserved. Mapping those pages in will involve taking and
- * releasing a page-table lock, which will provide the barrier.
- */
-
- return 0;
-}
-
-/*
- * Poison the shadow for a vmalloc region. Called as part of the
- * freeing process at the time the region is freed.
- */
-void kasan_poison_vmalloc(const void *start, unsigned long size)
-{
- if (!is_vmalloc_or_module_addr(start))
- return;
-
- size = round_up(size, KASAN_GRANULE_SIZE);
- poison_range(start, size, KASAN_VMALLOC_INVALID);
-}
-
-void kasan_unpoison_vmalloc(const void *start, unsigned long size)
-{
- if (!is_vmalloc_or_module_addr(start))
- return;
-
- unpoison_range(start, size);
-}
-
-static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,
- void *unused)
-{
- unsigned long page;
-
- page = (unsigned long)__va(pte_pfn(*ptep) << PAGE_SHIFT);
-
- spin_lock(&init_mm.page_table_lock);
-
- if (likely(!pte_none(*ptep))) {
- pte_clear(&init_mm, addr, ptep);
- free_page(page);
- }
- spin_unlock(&init_mm.page_table_lock);
-
- return 0;
-}
-
-/*
- * Release the backing for the vmalloc region [start, end), which
- * lies within the free region [free_region_start, free_region_end).
- *
- * This can be run lazily, long after the region was freed. It runs
- * under vmap_area_lock, so it's not safe to interact with the vmalloc/vmap
- * infrastructure.
- *
- * How does this work?
- * -------------------
- *
- * We have a region that is page aligned, labelled as A.
- * That might not map onto the shadow in a way that is page-aligned:
- *
- * start end
- * v v
- * |????????|????????|AAAAAAAA|AA....AA|AAAAAAAA|????????| < vmalloc
- * -------- -------- -------- -------- --------
- * | | | | |
- * | | | /-------/ |
- * \-------\|/------/ |/---------------/
- * ||| ||
- * |??AAAAAA|AAAAAAAA|AA??????| < shadow
- * (1) (2) (3)
- *
- * First we align the start upwards and the end downwards, so that the
- * shadow of the region aligns with shadow page boundaries. In the
- * example, this gives us the shadow page (2). This is the shadow entirely
- * covered by this allocation.
- *
- * Then we have the tricky bits. We want to know if we can free the
- * partially covered shadow pages - (1) and (3) in the example. For this,
- * we are given the start and end of the free region that contains this
- * allocation. Extending our previous example, we could have:
- *
- * free_region_start free_region_end
- * | start end |
- * v v v v
- * |FFFFFFFF|FFFFFFFF|AAAAAAAA|AA....AA|AAAAAAAA|FFFFFFFF| < vmalloc
- * -------- -------- -------- -------- --------
- * | | | | |
- * | | | /-------/ |
- * \-------\|/------/ |/---------------/
- * ||| ||
- * |FFAAAAAA|AAAAAAAA|AAF?????| < shadow
- * (1) (2) (3)
- *
- * Once again, we align the start of the free region up, and the end of
- * the free region down so that the shadow is page aligned. So we can free
- * page (1) - we know no allocation currently uses anything in that page,
- * because all of it is in the vmalloc free region. But we cannot free
- * page (3), because we can't be sure that the rest of it is unused.
- *
- * We only consider pages that contain part of the original region for
- * freeing: we don't try to free other pages from the free region or we'd
- * end up trying to free huge chunks of virtual address space.
- *
- * Concurrency
- * -----------
- *
- * How do we know that we're not freeing a page that is simultaneously
- * being used for a fresh allocation in kasan_populate_vmalloc(_pte)?
- *
- * We _can_ have kasan_release_vmalloc and kasan_populate_vmalloc running
- * at the same time. While we run under free_vmap_area_lock, the population
- * code does not.
- *
- * free_vmap_area_lock instead operates to ensure that the larger range
- * [free_region_start, free_region_end) is safe: because __alloc_vmap_area and
- * the per-cpu region-finding algorithm both run under free_vmap_area_lock,
- * no space identified as free will become used while we are running. This
- * means that so long as we are careful with alignment and only free shadow
- * pages entirely covered by the free region, we will not run in to any
- * trouble - any simultaneous allocations will be for disjoint regions.
- */
-void kasan_release_vmalloc(unsigned long start, unsigned long end,
- unsigned long free_region_start,
- unsigned long free_region_end)
-{
- void *shadow_start, *shadow_end;
- unsigned long region_start, region_end;
- unsigned long size;
-
- region_start = ALIGN(start, PAGE_SIZE * KASAN_GRANULE_SIZE);
- region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_GRANULE_SIZE);
-
- free_region_start = ALIGN(free_region_start,
- PAGE_SIZE * KASAN_GRANULE_SIZE);
-
- if (start != region_start &&
- free_region_start < region_start)
- region_start -= PAGE_SIZE * KASAN_GRANULE_SIZE;
-
- free_region_end = ALIGN_DOWN(free_region_end,
- PAGE_SIZE * KASAN_GRANULE_SIZE);
-
- if (end != region_end &&
- free_region_end > region_end)
- region_end += PAGE_SIZE * KASAN_GRANULE_SIZE;
-
- shadow_start = kasan_mem_to_shadow((void *)region_start);
- shadow_end = kasan_mem_to_shadow((void *)region_end);
-
- if (shadow_end > shadow_start) {
- size = shadow_end - shadow_start;
- apply_to_existing_page_range(&init_mm,
- (unsigned long)shadow_start,
- size, kasan_depopulate_vmalloc_pte,
- NULL);
- flush_tlb_kernel_range((unsigned long)shadow_start,
- (unsigned long)shadow_end);
- }
-}
-
-#else /* CONFIG_KASAN_VMALLOC */
-
-int kasan_module_alloc(void *addr, size_t size)
-{
- void *ret;
- size_t scaled_size;
- size_t shadow_size;
- unsigned long shadow_start;
-
- shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
- scaled_size = (size + KASAN_GRANULE_SIZE - 1) >>
- KASAN_SHADOW_SCALE_SHIFT;
- shadow_size = round_up(scaled_size, PAGE_SIZE);
-
- if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
- return -EINVAL;
-
- ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
- shadow_start + shadow_size,
- GFP_KERNEL,
- PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
- __builtin_return_address(0));
-
- if (ret) {
- __memset(ret, KASAN_SHADOW_INIT, shadow_size);
- find_vm_area(addr)->flags |= VM_KASAN;
- kmemleak_ignore(ret);
- return 0;
- }
-
- return -ENOMEM;
-}
-
-void kasan_free_shadow(const struct vm_struct *vm)
-{
- if (vm->flags & VM_KASAN)
- vfree(kasan_mem_to_shadow(vm->addr));
-}
-
-#endif
--- a/mm/kasan/Makefile~kasan-split-out-shadowc-from-commonc
+++ a/mm/kasan/Makefile
@@ -10,6 +10,7 @@ CFLAGS_REMOVE_generic_report.o = $(CC_FL
CFLAGS_REMOVE_init.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_quarantine.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_shadow.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_tags_report.o = $(CC_FLAGS_FTRACE)
@@ -26,9 +27,10 @@ CFLAGS_generic_report.o := $(CC_FLAGS_KA
CFLAGS_init.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_quarantine.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_shadow.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags_report.o := $(CC_FLAGS_KASAN_RUNTIME)
obj-$(CONFIG_KASAN) := common.o report.o
-obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o quarantine.o
-obj-$(CONFIG_KASAN_SW_TAGS) += init.o tags.o tags_report.o
+obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o shadow.o quarantine.o
+obj-$(CONFIG_KASAN_SW_TAGS) += init.o shadow.o tags.o tags_report.o
--- /dev/null
+++ a/mm/kasan/shadow.c
@@ -0,0 +1,505 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file contains KASAN runtime code that manages shadow memory for
+ * generic and software tag-based KASAN modes.
+ *
+ * Copyright (c) 2014 Samsung Electronics Co., Ltd.
+ * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
+ *
+ * Some code borrowed from https://github.com/xairy/kasan-prototype by
+ * Andrey Konovalov <andreyknvl@gmail.com>
+ */
+
+#include <linux/init.h>
+#include <linux/kasan.h>
+#include <linux/kernel.h>
+#include <linux/kmemleak.h>
+#include <linux/memory.h>
+#include <linux/mm.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/vmalloc.h>
+
+#include <asm/cacheflush.h>
+#include <asm/tlbflush.h>
+
+#include "kasan.h"
+
+bool __kasan_check_read(const volatile void *p, unsigned int size)
+{
+ return check_memory_region((unsigned long)p, size, false, _RET_IP_);
+}
+EXPORT_SYMBOL(__kasan_check_read);
+
+bool __kasan_check_write(const volatile void *p, unsigned int size)
+{
+ return check_memory_region((unsigned long)p, size, true, _RET_IP_);
+}
+EXPORT_SYMBOL(__kasan_check_write);
+
+#undef memset
+void *memset(void *addr, int c, size_t len)
+{
+ if (!check_memory_region((unsigned long)addr, len, true, _RET_IP_))
+ return NULL;
+
+ return __memset(addr, c, len);
+}
+
+#ifdef __HAVE_ARCH_MEMMOVE
+#undef memmove
+void *memmove(void *dest, const void *src, size_t len)
+{
+ if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
+ !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
+ return NULL;
+
+ return __memmove(dest, src, len);
+}
+#endif
+
+#undef memcpy
+void *memcpy(void *dest, const void *src, size_t len)
+{
+ if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
+ !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
+ return NULL;
+
+ return __memcpy(dest, src, len);
+}
+
+/*
+ * Poisons the shadow memory for 'size' bytes starting from 'addr'.
+ * Memory addresses should be aligned to KASAN_GRANULE_SIZE.
+ */
+void poison_range(const void *address, size_t size, u8 value)
+{
+ void *shadow_start, *shadow_end;
+
+ /*
+ * Perform shadow offset calculation based on untagged address, as
+ * some of the callers (e.g. kasan_poison_object_data) pass tagged
+ * addresses to this function.
+ */
+ address = reset_tag(address);
+
+ shadow_start = kasan_mem_to_shadow(address);
+ shadow_end = kasan_mem_to_shadow(address + size);
+
+ __memset(shadow_start, value, shadow_end - shadow_start);
+}
+
+void unpoison_range(const void *address, size_t size)
+{
+ u8 tag = get_tag(address);
+
+ /*
+ * Perform shadow offset calculation based on untagged address, as
+ * some of the callers (e.g. kasan_unpoison_object_data) pass tagged
+ * addresses to this function.
+ */
+ address = reset_tag(address);
+
+ poison_range(address, size, tag);
+
+ if (size & KASAN_GRANULE_MASK) {
+ u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
+
+ if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ *shadow = tag;
+ else
+ *shadow = size & KASAN_GRANULE_MASK;
+ }
+}
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static bool shadow_mapped(unsigned long addr)
+{
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+ pte_t *pte;
+
+ if (pgd_none(*pgd))
+ return false;
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(*p4d))
+ return false;
+ pud = pud_offset(p4d, addr);
+ if (pud_none(*pud))
+ return false;
+
+ /*
+ * We can't use pud_large() or pud_huge(), the first one is
+ * arch-specific, the last one depends on HUGETLB_PAGE. So let's abuse
+ * pud_bad(), if pud is bad then it's bad because it's huge.
+ */
+ if (pud_bad(*pud))
+ return true;
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return false;
+
+ if (pmd_bad(*pmd))
+ return true;
+ pte = pte_offset_kernel(pmd, addr);
+ return !pte_none(*pte);
+}
+
+static int __meminit kasan_mem_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ struct memory_notify *mem_data = data;
+ unsigned long nr_shadow_pages, start_kaddr, shadow_start;
+ unsigned long shadow_end, shadow_size;
+
+ nr_shadow_pages = mem_data->nr_pages >> KASAN_SHADOW_SCALE_SHIFT;
+ start_kaddr = (unsigned long)pfn_to_kaddr(mem_data->start_pfn);
+ shadow_start = (unsigned long)kasan_mem_to_shadow((void *)start_kaddr);
+ shadow_size = nr_shadow_pages << PAGE_SHIFT;
+ shadow_end = shadow_start + shadow_size;
+
+ if (WARN_ON(mem_data->nr_pages % KASAN_GRANULE_SIZE) ||
+ WARN_ON(start_kaddr % (KASAN_GRANULE_SIZE << PAGE_SHIFT)))
+ return NOTIFY_BAD;
+
+ switch (action) {
+ case MEM_GOING_ONLINE: {
+ void *ret;
+
+ /*
+ * If shadow is mapped already than it must have been mapped
+ * during the boot. This could happen if we onlining previously
+ * offlined memory.
+ */
+ if (shadow_mapped(shadow_start))
+ return NOTIFY_OK;
+
+ ret = __vmalloc_node_range(shadow_size, PAGE_SIZE, shadow_start,
+ shadow_end, GFP_KERNEL,
+ PAGE_KERNEL, VM_NO_GUARD,
+ pfn_to_nid(mem_data->start_pfn),
+ __builtin_return_address(0));
+ if (!ret)
+ return NOTIFY_BAD;
+
+ kmemleak_ignore(ret);
+ return NOTIFY_OK;
+ }
+ case MEM_CANCEL_ONLINE:
+ case MEM_OFFLINE: {
+ struct vm_struct *vm;
+
+ /*
+ * shadow_start was either mapped during boot by kasan_init()
+ * or during memory online by __vmalloc_node_range().
+ * In the latter case we can use vfree() to free shadow.
+ * Non-NULL result of the find_vm_area() will tell us if
+ * that was the second case.
+ *
+ * Currently it's not possible to free shadow mapped
+ * during boot by kasan_init(). It's because the code
+ * to do that hasn't been written yet. So we'll just
+ * leak the memory.
+ */
+ vm = find_vm_area((void *)shadow_start);
+ if (vm)
+ vfree((void *)shadow_start);
+ }
+ }
+
+ return NOTIFY_OK;
+}
+
+static int __init kasan_memhotplug_init(void)
+{
+ hotplug_memory_notifier(kasan_mem_notifier, 0);
+
+ return 0;
+}
+
+core_initcall(kasan_memhotplug_init);
+#endif
+
+#ifdef CONFIG_KASAN_VMALLOC
+
+static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
+ void *unused)
+{
+ unsigned long page;
+ pte_t pte;
+
+ if (likely(!pte_none(*ptep)))
+ return 0;
+
+ page = __get_free_page(GFP_KERNEL);
+ if (!page)
+ return -ENOMEM;
+
+ memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
+ pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
+
+ spin_lock(&init_mm.page_table_lock);
+ if (likely(pte_none(*ptep))) {
+ set_pte_at(&init_mm, addr, ptep, pte);
+ page = 0;
+ }
+ spin_unlock(&init_mm.page_table_lock);
+ if (page)
+ free_page(page);
+ return 0;
+}
+
+int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
+{
+ unsigned long shadow_start, shadow_end;
+ int ret;
+
+ if (!is_vmalloc_or_module_addr((void *)addr))
+ return 0;
+
+ shadow_start = (unsigned long)kasan_mem_to_shadow((void *)addr);
+ shadow_start = ALIGN_DOWN(shadow_start, PAGE_SIZE);
+ shadow_end = (unsigned long)kasan_mem_to_shadow((void *)addr + size);
+ shadow_end = ALIGN(shadow_end, PAGE_SIZE);
+
+ ret = apply_to_page_range(&init_mm, shadow_start,
+ shadow_end - shadow_start,
+ kasan_populate_vmalloc_pte, NULL);
+ if (ret)
+ return ret;
+
+ flush_cache_vmap(shadow_start, shadow_end);
+
+ /*
+ * We need to be careful about inter-cpu effects here. Consider:
+ *
+ * CPU#0 CPU#1
+ * WRITE_ONCE(p, vmalloc(100)); while (x = READ_ONCE(p)) ;
+ * p[99] = 1;
+ *
+ * With compiler instrumentation, that ends up looking like this:
+ *
+ * CPU#0 CPU#1
+ * // vmalloc() allocates memory
+ * // let a = area->addr
+ * // we reach kasan_populate_vmalloc
+ * // and call unpoison_range:
+ * STORE shadow(a), unpoison_val
+ * ...
+ * STORE shadow(a+99), unpoison_val x = LOAD p
+ * // rest of vmalloc process <data dependency>
+ * STORE p, a LOAD shadow(x+99)
+ *
+ * If there is no barrier between the end of unpoisioning the shadow
+ * and the store of the result to p, the stores could be committed
+ * in a different order by CPU#0, and CPU#1 could erroneously observe
+ * poison in the shadow.
+ *
+ * We need some sort of barrier between the stores.
+ *
+ * In the vmalloc() case, this is provided by a smp_wmb() in
+ * clear_vm_uninitialized_flag(). In the per-cpu allocator and in
+ * get_vm_area() and friends, the caller gets shadow allocated but
+ * doesn't have any pages mapped into the virtual address space that
+ * has been reserved. Mapping those pages in will involve taking and
+ * releasing a page-table lock, which will provide the barrier.
+ */
+
+ return 0;
+}
+
+/*
+ * Poison the shadow for a vmalloc region. Called as part of the
+ * freeing process at the time the region is freed.
+ */
+void kasan_poison_vmalloc(const void *start, unsigned long size)
+{
+ if (!is_vmalloc_or_module_addr(start))
+ return;
+
+ size = round_up(size, KASAN_GRANULE_SIZE);
+ poison_range(start, size, KASAN_VMALLOC_INVALID);
+}
+
+void kasan_unpoison_vmalloc(const void *start, unsigned long size)
+{
+ if (!is_vmalloc_or_module_addr(start))
+ return;
+
+ unpoison_range(start, size);
+}
+
+static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,
+ void *unused)
+{
+ unsigned long page;
+
+ page = (unsigned long)__va(pte_pfn(*ptep) << PAGE_SHIFT);
+
+ spin_lock(&init_mm.page_table_lock);
+
+ if (likely(!pte_none(*ptep))) {
+ pte_clear(&init_mm, addr, ptep);
+ free_page(page);
+ }
+ spin_unlock(&init_mm.page_table_lock);
+
+ return 0;
+}
+
+/*
+ * Release the backing for the vmalloc region [start, end), which
+ * lies within the free region [free_region_start, free_region_end).
+ *
+ * This can be run lazily, long after the region was freed. It runs
+ * under vmap_area_lock, so it's not safe to interact with the vmalloc/vmap
+ * infrastructure.
+ *
+ * How does this work?
+ * -------------------
+ *
+ * We have a region that is page aligned, labelled as A.
+ * That might not map onto the shadow in a way that is page-aligned:
+ *
+ * start end
+ * v v
+ * |????????|????????|AAAAAAAA|AA....AA|AAAAAAAA|????????| < vmalloc
+ * -------- -------- -------- -------- --------
+ * | | | | |
+ * | | | /-------/ |
+ * \-------\|/------/ |/---------------/
+ * ||| ||
+ * |??AAAAAA|AAAAAAAA|AA??????| < shadow
+ * (1) (2) (3)
+ *
+ * First we align the start upwards and the end downwards, so that the
+ * shadow of the region aligns with shadow page boundaries. In the
+ * example, this gives us the shadow page (2). This is the shadow entirely
+ * covered by this allocation.
+ *
+ * Then we have the tricky bits. We want to know if we can free the
+ * partially covered shadow pages - (1) and (3) in the example. For this,
+ * we are given the start and end of the free region that contains this
+ * allocation. Extending our previous example, we could have:
+ *
+ * free_region_start free_region_end
+ * | start end |
+ * v v v v
+ * |FFFFFFFF|FFFFFFFF|AAAAAAAA|AA....AA|AAAAAAAA|FFFFFFFF| < vmalloc
+ * -------- -------- -------- -------- --------
+ * | | | | |
+ * | | | /-------/ |
+ * \-------\|/------/ |/---------------/
+ * ||| ||
+ * |FFAAAAAA|AAAAAAAA|AAF?????| < shadow
+ * (1) (2) (3)
+ *
+ * Once again, we align the start of the free region up, and the end of
+ * the free region down so that the shadow is page aligned. So we can free
+ * page (1) - we know no allocation currently uses anything in that page,
+ * because all of it is in the vmalloc free region. But we cannot free
+ * page (3), because we can't be sure that the rest of it is unused.
+ *
+ * We only consider pages that contain part of the original region for
+ * freeing: we don't try to free other pages from the free region or we'd
+ * end up trying to free huge chunks of virtual address space.
+ *
+ * Concurrency
+ * -----------
+ *
+ * How do we know that we're not freeing a page that is simultaneously
+ * being used for a fresh allocation in kasan_populate_vmalloc(_pte)?
+ *
+ * We _can_ have kasan_release_vmalloc and kasan_populate_vmalloc running
+ * at the same time. While we run under free_vmap_area_lock, the population
+ * code does not.
+ *
+ * free_vmap_area_lock instead operates to ensure that the larger range
+ * [free_region_start, free_region_end) is safe: because __alloc_vmap_area and
+ * the per-cpu region-finding algorithm both run under free_vmap_area_lock,
+ * no space identified as free will become used while we are running. This
+ * means that so long as we are careful with alignment and only free shadow
+ * pages entirely covered by the free region, we will not run in to any
+ * trouble - any simultaneous allocations will be for disjoint regions.
+ */
+void kasan_release_vmalloc(unsigned long start, unsigned long end,
+ unsigned long free_region_start,
+ unsigned long free_region_end)
+{
+ void *shadow_start, *shadow_end;
+ unsigned long region_start, region_end;
+ unsigned long size;
+
+ region_start = ALIGN(start, PAGE_SIZE * KASAN_GRANULE_SIZE);
+ region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_GRANULE_SIZE);
+
+ free_region_start = ALIGN(free_region_start,
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
+
+ if (start != region_start &&
+ free_region_start < region_start)
+ region_start -= PAGE_SIZE * KASAN_GRANULE_SIZE;
+
+ free_region_end = ALIGN_DOWN(free_region_end,
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
+
+ if (end != region_end &&
+ free_region_end > region_end)
+ region_end += PAGE_SIZE * KASAN_GRANULE_SIZE;
+
+ shadow_start = kasan_mem_to_shadow((void *)region_start);
+ shadow_end = kasan_mem_to_shadow((void *)region_end);
+
+ if (shadow_end > shadow_start) {
+ size = shadow_end - shadow_start;
+ apply_to_existing_page_range(&init_mm,
+ (unsigned long)shadow_start,
+ size, kasan_depopulate_vmalloc_pte,
+ NULL);
+ flush_tlb_kernel_range((unsigned long)shadow_start,
+ (unsigned long)shadow_end);
+ }
+}
+
+#else /* CONFIG_KASAN_VMALLOC */
+
+int kasan_module_alloc(void *addr, size_t size)
+{
+ void *ret;
+ size_t scaled_size;
+ size_t shadow_size;
+ unsigned long shadow_start;
+
+ shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
+ scaled_size = (size + KASAN_GRANULE_SIZE - 1) >>
+ KASAN_SHADOW_SCALE_SHIFT;
+ shadow_size = round_up(scaled_size, PAGE_SIZE);
+
+ if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
+ return -EINVAL;
+
+ ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
+ shadow_start + shadow_size,
+ GFP_KERNEL,
+ PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
+ __builtin_return_address(0));
+
+ if (ret) {
+ __memset(ret, KASAN_SHADOW_INIT, shadow_size);
+ find_vm_area(addr)->flags |= VM_KASAN;
+ kmemleak_ignore(ret);
+ return 0;
+ }
+
+ return -ENOMEM;
+}
+
+void kasan_free_shadow(const struct vm_struct *vm)
+{
+ if (vm->flags & VM_KASAN)
+ vfree(kasan_mem_to_shadow(vm->addr));
+}
+
+#endif
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 09/60] kasan: define KASAN_MEMORY_PER_SHADOW_PAGE
2020-12-22 19:58 incoming Andrew Morton
` (7 preceding siblings ...)
2020-12-22 20:00 ` [patch 08/60] kasan: split out shadow.c from common.c Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 10/60] kasan: rename report and tags files Andrew Morton
` (51 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: define KASAN_MEMORY_PER_SHADOW_PAGE
Define KASAN_MEMORY_PER_SHADOW_PAGE as (KASAN_GRANULE_SIZE << PAGE_SHIFT),
which is the same as (KASAN_GRANULE_SIZE * PAGE_SIZE) for software modes
that use shadow memory, and use it across KASAN code to simplify it.
Link: https://lkml.kernel.org/r/8329391cfe14b5cffd3decf3b5c535b6ce21eef6.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/init.c | 10 ++++------
mm/kasan/kasan.h | 2 ++
mm/kasan/shadow.c | 16 +++++++---------
3 files changed, 13 insertions(+), 15 deletions(-)
--- a/mm/kasan/init.c~kasan-define-kasan_memory_per_shadow_page
+++ a/mm/kasan/init.c
@@ -441,9 +441,8 @@ void kasan_remove_zero_shadow(void *star
addr = (unsigned long)kasan_mem_to_shadow(start);
end = addr + (size >> KASAN_SHADOW_SCALE_SHIFT);
- if (WARN_ON((unsigned long)start %
- (KASAN_GRANULE_SIZE * PAGE_SIZE)) ||
- WARN_ON(size % (KASAN_GRANULE_SIZE * PAGE_SIZE)))
+ if (WARN_ON((unsigned long)start % KASAN_MEMORY_PER_SHADOW_PAGE) ||
+ WARN_ON(size % KASAN_MEMORY_PER_SHADOW_PAGE))
return;
for (; addr < end; addr = next) {
@@ -476,9 +475,8 @@ int kasan_add_zero_shadow(void *start, u
shadow_start = kasan_mem_to_shadow(start);
shadow_end = shadow_start + (size >> KASAN_SHADOW_SCALE_SHIFT);
- if (WARN_ON((unsigned long)start %
- (KASAN_GRANULE_SIZE * PAGE_SIZE)) ||
- WARN_ON(size % (KASAN_GRANULE_SIZE * PAGE_SIZE)))
+ if (WARN_ON((unsigned long)start % KASAN_MEMORY_PER_SHADOW_PAGE) ||
+ WARN_ON(size % KASAN_MEMORY_PER_SHADOW_PAGE))
return -EINVAL;
ret = kasan_populate_early_shadow(shadow_start, shadow_end);
--- a/mm/kasan/kasan.h~kasan-define-kasan_memory_per_shadow_page
+++ a/mm/kasan/kasan.h
@@ -8,6 +8,8 @@
#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
#define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
+#define KASAN_MEMORY_PER_SHADOW_PAGE (KASAN_GRANULE_SIZE << PAGE_SHIFT)
+
#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
#define KASAN_TAG_INVALID 0xFE /* inaccessible memory tag */
#define KASAN_TAG_MAX 0xFD /* maximum value for random tags */
--- a/mm/kasan/shadow.c~kasan-define-kasan_memory_per_shadow_page
+++ a/mm/kasan/shadow.c
@@ -161,7 +161,7 @@ static int __meminit kasan_mem_notifier(
shadow_end = shadow_start + shadow_size;
if (WARN_ON(mem_data->nr_pages % KASAN_GRANULE_SIZE) ||
- WARN_ON(start_kaddr % (KASAN_GRANULE_SIZE << PAGE_SHIFT)))
+ WARN_ON(start_kaddr % KASAN_MEMORY_PER_SHADOW_PAGE))
return NOTIFY_BAD;
switch (action) {
@@ -432,22 +432,20 @@ void kasan_release_vmalloc(unsigned long
unsigned long region_start, region_end;
unsigned long size;
- region_start = ALIGN(start, PAGE_SIZE * KASAN_GRANULE_SIZE);
- region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_GRANULE_SIZE);
+ region_start = ALIGN(start, KASAN_MEMORY_PER_SHADOW_PAGE);
+ region_end = ALIGN_DOWN(end, KASAN_MEMORY_PER_SHADOW_PAGE);
- free_region_start = ALIGN(free_region_start,
- PAGE_SIZE * KASAN_GRANULE_SIZE);
+ free_region_start = ALIGN(free_region_start, KASAN_MEMORY_PER_SHADOW_PAGE);
if (start != region_start &&
free_region_start < region_start)
- region_start -= PAGE_SIZE * KASAN_GRANULE_SIZE;
+ region_start -= KASAN_MEMORY_PER_SHADOW_PAGE;
- free_region_end = ALIGN_DOWN(free_region_end,
- PAGE_SIZE * KASAN_GRANULE_SIZE);
+ free_region_end = ALIGN_DOWN(free_region_end, KASAN_MEMORY_PER_SHADOW_PAGE);
if (end != region_end &&
free_region_end > region_end)
- region_end += PAGE_SIZE * KASAN_GRANULE_SIZE;
+ region_end += KASAN_MEMORY_PER_SHADOW_PAGE;
shadow_start = kasan_mem_to_shadow((void *)region_start);
shadow_end = kasan_mem_to_shadow((void *)region_end);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 10/60] kasan: rename report and tags files
2020-12-22 19:58 incoming Andrew Morton
` (8 preceding siblings ...)
2020-12-22 20:00 ` [patch 09/60] kasan: define KASAN_MEMORY_PER_SHADOW_PAGE Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 11/60] kasan: don't duplicate config dependencies Andrew Morton
` (50 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: rename report and tags files
Rename generic_report.c to report_generic.c and tags_report.c to
report_sw_tags.c, as their content is more relevant to report.c file.
Also rename tags.c to sw_tags.c to better reflect that this file contains
code for software tag-based mode.
No functional changes.
Link: https://lkml.kernel.org/r/a6105d416da97d389580015afed66c4c3cfd4c08.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/Makefile | 16 +-
mm/kasan/generic_report.c | 160 -----------------------------
mm/kasan/report.c | 2
mm/kasan/report_generic.c | 160 +++++++++++++++++++++++++++++
mm/kasan/report_sw_tags.c | 88 ++++++++++++++++
mm/kasan/sw_tags.c | 195 ++++++++++++++++++++++++++++++++++++
mm/kasan/tags.c | 195 ------------------------------------
mm/kasan/tags_report.c | 88 ----------------
8 files changed, 452 insertions(+), 452 deletions(-)
--- a/mm/kasan/Makefile~kasan-rename-report-and-tags-files
+++ a/mm/kasan/Makefile
@@ -6,13 +6,13 @@ KCOV_INSTRUMENT := n
# Disable ftrace to avoid recursion.
CFLAGS_REMOVE_common.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_generic.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_generic_report.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_init.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_quarantine.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_report_generic.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_report_sw_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_shadow.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_tags.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_tags_report.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_sw_tags.o = $(CC_FLAGS_FTRACE)
# Function splitter causes unnecessary splits in __asan_load1/__asan_store1
# see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63533
@@ -23,14 +23,14 @@ CC_FLAGS_KASAN_RUNTIME += -DDISABLE_BRAN
CFLAGS_common.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_generic.o := $(CC_FLAGS_KASAN_RUNTIME)
-CFLAGS_generic_report.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_init.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_quarantine.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_report_generic.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_report_sw_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_shadow.o := $(CC_FLAGS_KASAN_RUNTIME)
-CFLAGS_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
-CFLAGS_tags_report.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_sw_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
obj-$(CONFIG_KASAN) := common.o report.o
-obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o shadow.o quarantine.o
-obj-$(CONFIG_KASAN_SW_TAGS) += init.o shadow.o tags.o tags_report.o
+obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o report_generic.o shadow.o quarantine.o
+obj-$(CONFIG_KASAN_SW_TAGS) += init.o report_sw_tags.o shadow.o sw_tags.o
--- a/mm/kasan/report.c~kasan-rename-report-and-tags-files
+++ a/mm/kasan/report.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains common generic and tag-based KASAN error reporting code.
+ * This file contains common KASAN error reporting code.
*
* Copyright (c) 2014 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
diff --git a/mm/kasan/generic_report.c b/mm/kasan/report_generic.c
similarity index 100%
rename from mm/kasan/generic_report.c
rename to mm/kasan/report_generic.c
diff --git a/mm/kasan/tags_report.c b/mm/kasan/report_sw_tags.c
similarity index 100%
rename from mm/kasan/tags_report.c
rename to mm/kasan/report_sw_tags.c
diff --git a/mm/kasan/tags.c b/mm/kasan/sw_tags.c
similarity index 100%
rename from mm/kasan/tags.c
rename to mm/kasan/sw_tags.c
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 11/60] kasan: don't duplicate config dependencies
2020-12-22 19:58 incoming Andrew Morton
` (9 preceding siblings ...)
2020-12-22 20:00 ` [patch 10/60] kasan: rename report and tags files Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 12/60] kasan: hide invalid free check implementation Andrew Morton
` (49 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: don't duplicate config dependencies
Both KASAN_GENERIC and KASAN_SW_TAGS have common dependencies, move those
to KASAN.
Link: https://lkml.kernel.org/r/c1cc0d562608a318c607afe22db5ec2a7af72e47.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
lib/Kconfig.kasan | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
--- a/lib/Kconfig.kasan~kasan-dont-duplicate-config-dependencies
+++ a/lib/Kconfig.kasan
@@ -24,6 +24,8 @@ menuconfig KASAN
(HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS)
depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
depends on CC_HAS_WORKING_NOSANITIZE_ADDRESS
+ select CONSTRUCTORS
+ select STACKDEPOT
help
Enables KASAN (KernelAddressSANitizer) - runtime memory debugger,
designed to find out-of-bounds accesses and use-after-free bugs.
@@ -46,10 +48,7 @@ choice
config KASAN_GENERIC
bool "Generic mode"
depends on HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC
- depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
select SLUB_DEBUG if SLUB
- select CONSTRUCTORS
- select STACKDEPOT
help
Enables generic KASAN mode.
@@ -70,10 +69,7 @@ config KASAN_GENERIC
config KASAN_SW_TAGS
bool "Software tag-based mode"
depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
- depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
select SLUB_DEBUG if SLUB
- select CONSTRUCTORS
- select STACKDEPOT
help
Enables software tag-based KASAN mode.
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 12/60] kasan: hide invalid free check implementation
2020-12-22 19:58 incoming Andrew Morton
` (10 preceding siblings ...)
2020-12-22 20:00 ` [patch 11/60] kasan: don't duplicate config dependencies Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 13/60] kasan: decode stack frame only with KASAN_STACK_ENABLE Andrew Morton
` (48 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: hide invalid free check implementation
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
For software KASAN modes the check is based on the value in the shadow
memory. Hardware tag-based KASAN won't be using shadow, so hide the
implementation of the check in check_invalid_free().
Also simplify the code for software tag-based mode.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/d01534a4b977f97d87515dc590e6348e1406de81.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 19 +------------------
mm/kasan/generic.c | 7 +++++++
mm/kasan/kasan.h | 2 ++
mm/kasan/sw_tags.c | 9 +++++++++
4 files changed, 19 insertions(+), 18 deletions(-)
--- a/mm/kasan/common.c~kasan-hide-invalid-free-check-implementation
+++ a/mm/kasan/common.c
@@ -277,25 +277,9 @@ void * __must_check kasan_init_slab_obj(
return (void *)object;
}
-static inline bool shadow_invalid(u8 tag, s8 shadow_byte)
-{
- if (IS_ENABLED(CONFIG_KASAN_GENERIC))
- return shadow_byte < 0 ||
- shadow_byte >= KASAN_GRANULE_SIZE;
-
- /* else CONFIG_KASAN_SW_TAGS: */
- if ((u8)shadow_byte == KASAN_TAG_INVALID)
- return true;
- if ((tag != KASAN_TAG_KERNEL) && (tag != (u8)shadow_byte))
- return true;
-
- return false;
-}
-
static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
unsigned long ip, bool quarantine)
{
- s8 shadow_byte;
u8 tag;
void *tagged_object;
unsigned long rounded_up_size;
@@ -314,8 +298,7 @@ static bool __kasan_slab_free(struct kme
if (unlikely(cache->flags & SLAB_TYPESAFE_BY_RCU))
return false;
- shadow_byte = READ_ONCE(*(s8 *)kasan_mem_to_shadow(object));
- if (shadow_invalid(tag, shadow_byte)) {
+ if (check_invalid_free(tagged_object)) {
kasan_report_invalid_free(tagged_object, ip);
return true;
}
--- a/mm/kasan/generic.c~kasan-hide-invalid-free-check-implementation
+++ a/mm/kasan/generic.c
@@ -187,6 +187,13 @@ bool check_memory_region(unsigned long a
return check_memory_region_inline(addr, size, write, ret_ip);
}
+bool check_invalid_free(void *addr)
+{
+ s8 shadow_byte = READ_ONCE(*(s8 *)kasan_mem_to_shadow(addr));
+
+ return shadow_byte < 0 || shadow_byte >= KASAN_GRANULE_SIZE;
+}
+
void kasan_cache_shrink(struct kmem_cache *cache)
{
quarantine_remove_cache(cache);
--- a/mm/kasan/kasan.h~kasan-hide-invalid-free-check-implementation
+++ a/mm/kasan/kasan.h
@@ -166,6 +166,8 @@ void unpoison_range(const void *address,
bool check_memory_region(unsigned long addr, size_t size, bool write,
unsigned long ret_ip);
+bool check_invalid_free(void *addr);
+
void *find_first_bad_addr(void *addr, size_t size);
const char *get_bug_type(struct kasan_access_info *info);
--- a/mm/kasan/sw_tags.c~kasan-hide-invalid-free-check-implementation
+++ a/mm/kasan/sw_tags.c
@@ -121,6 +121,15 @@ bool check_memory_region(unsigned long a
return true;
}
+bool check_invalid_free(void *addr)
+{
+ u8 tag = get_tag(addr);
+ u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(reset_tag(addr)));
+
+ return (shadow_byte == KASAN_TAG_INVALID) ||
+ (tag != KASAN_TAG_KERNEL && tag != shadow_byte);
+}
+
#define DEFINE_HWASAN_LOAD_STORE(size) \
void __hwasan_load##size##_noabort(unsigned long addr) \
{ \
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 13/60] kasan: decode stack frame only with KASAN_STACK_ENABLE
2020-12-22 19:58 incoming Andrew Morton
` (11 preceding siblings ...)
2020-12-22 20:00 ` [patch 12/60] kasan: hide invalid free check implementation Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 14/60] kasan, arm64: only init shadow for software modes Andrew Morton
` (47 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: decode stack frame only with KASAN_STACK_ENABLE
Decoding routines aren't needed when CONFIG_KASAN_STACK_ENABLE is not
enabled. Currently only generic KASAN mode implements stack error
reporting.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/05a24db36f5ec876af876a299bbea98c29468ebd.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/kasan.h | 6 +
mm/kasan/report.c | 162 ------------------------------------
mm/kasan/report_generic.c | 162 ++++++++++++++++++++++++++++++++++++
3 files changed, 168 insertions(+), 162 deletions(-)
--- a/mm/kasan/kasan.h~kasan-decode-stack-frame-only-with-kasan_stack_enable
+++ a/mm/kasan/kasan.h
@@ -171,6 +171,12 @@ bool check_invalid_free(void *addr);
void *find_first_bad_addr(void *addr, size_t size);
const char *get_bug_type(struct kasan_access_info *info);
+#if defined(CONFIG_KASAN_GENERIC) && CONFIG_KASAN_STACK
+void print_address_stack_frame(const void *addr);
+#else
+static inline void print_address_stack_frame(const void *addr) { }
+#endif
+
bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
void kasan_report_invalid_free(void *object, unsigned long ip);
--- a/mm/kasan/report.c~kasan-decode-stack-frame-only-with-kasan_stack_enable
+++ a/mm/kasan/report.c
@@ -211,168 +211,6 @@ static inline bool init_task_stack_addr(
sizeof(init_thread_union.stack));
}
-static bool __must_check tokenize_frame_descr(const char **frame_descr,
- char *token, size_t max_tok_len,
- unsigned long *value)
-{
- const char *sep = strchr(*frame_descr, ' ');
-
- if (sep == NULL)
- sep = *frame_descr + strlen(*frame_descr);
-
- if (token != NULL) {
- const size_t tok_len = sep - *frame_descr;
-
- if (tok_len + 1 > max_tok_len) {
- pr_err("KASAN internal error: frame description too long: %s\n",
- *frame_descr);
- return false;
- }
-
- /* Copy token (+ 1 byte for '\0'). */
- strlcpy(token, *frame_descr, tok_len + 1);
- }
-
- /* Advance frame_descr past separator. */
- *frame_descr = sep + 1;
-
- if (value != NULL && kstrtoul(token, 10, value)) {
- pr_err("KASAN internal error: not a valid number: %s\n", token);
- return false;
- }
-
- return true;
-}
-
-static void print_decoded_frame_descr(const char *frame_descr)
-{
- /*
- * We need to parse the following string:
- * "n alloc_1 alloc_2 ... alloc_n"
- * where alloc_i looks like
- * "offset size len name"
- * or "offset size len name:line".
- */
-
- char token[64];
- unsigned long num_objects;
-
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- &num_objects))
- return;
-
- pr_err("\n");
- pr_err("this frame has %lu %s:\n", num_objects,
- num_objects == 1 ? "object" : "objects");
-
- while (num_objects--) {
- unsigned long offset;
- unsigned long size;
-
- /* access offset */
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- &offset))
- return;
- /* access size */
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- &size))
- return;
- /* name length (unused) */
- if (!tokenize_frame_descr(&frame_descr, NULL, 0, NULL))
- return;
- /* object name */
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- NULL))
- return;
-
- /* Strip line number; without filename it's not very helpful. */
- strreplace(token, ':', '\0');
-
- /* Finally, print object information. */
- pr_err(" [%lu, %lu) '%s'", offset, offset + size, token);
- }
-}
-
-static bool __must_check get_address_stack_frame_info(const void *addr,
- unsigned long *offset,
- const char **frame_descr,
- const void **frame_pc)
-{
- unsigned long aligned_addr;
- unsigned long mem_ptr;
- const u8 *shadow_bottom;
- const u8 *shadow_ptr;
- const unsigned long *frame;
-
- BUILD_BUG_ON(IS_ENABLED(CONFIG_STACK_GROWSUP));
-
- /*
- * NOTE: We currently only support printing frame information for
- * accesses to the task's own stack.
- */
- if (!object_is_on_stack(addr))
- return false;
-
- aligned_addr = round_down((unsigned long)addr, sizeof(long));
- mem_ptr = round_down(aligned_addr, KASAN_GRANULE_SIZE);
- shadow_ptr = kasan_mem_to_shadow((void *)aligned_addr);
- shadow_bottom = kasan_mem_to_shadow(end_of_stack(current));
-
- while (shadow_ptr >= shadow_bottom && *shadow_ptr != KASAN_STACK_LEFT) {
- shadow_ptr--;
- mem_ptr -= KASAN_GRANULE_SIZE;
- }
-
- while (shadow_ptr >= shadow_bottom && *shadow_ptr == KASAN_STACK_LEFT) {
- shadow_ptr--;
- mem_ptr -= KASAN_GRANULE_SIZE;
- }
-
- if (shadow_ptr < shadow_bottom)
- return false;
-
- frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
- if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
- pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",
- frame[0]);
- return false;
- }
-
- *offset = (unsigned long)addr - (unsigned long)frame;
- *frame_descr = (const char *)frame[1];
- *frame_pc = (void *)frame[2];
-
- return true;
-}
-
-static void print_address_stack_frame(const void *addr)
-{
- unsigned long offset;
- const char *frame_descr;
- const void *frame_pc;
-
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
- return;
-
- if (!get_address_stack_frame_info(addr, &offset, &frame_descr,
- &frame_pc))
- return;
-
- /*
- * get_address_stack_frame_info only returns true if the given addr is
- * on the current task's stack.
- */
- pr_err("\n");
- pr_err("addr %px is located in stack of task %s/%d at offset %lu in frame:\n",
- addr, current->comm, task_pid_nr(current), offset);
- pr_err(" %pS\n", frame_pc);
-
- if (!frame_descr)
- return;
-
- print_decoded_frame_descr(frame_descr);
-}
-
static void print_address_description(void *addr, u8 tag)
{
struct page *page = kasan_addr_to_page(addr);
--- a/mm/kasan/report_generic.c~kasan-decode-stack-frame-only-with-kasan_stack_enable
+++ a/mm/kasan/report_generic.c
@@ -16,6 +16,7 @@
#include <linux/mm.h>
#include <linux/printk.h>
#include <linux/sched.h>
+#include <linux/sched/task_stack.h>
#include <linux/slab.h>
#include <linux/stackdepot.h>
#include <linux/stacktrace.h>
@@ -122,6 +123,167 @@ const char *get_bug_type(struct kasan_ac
return get_wild_bug_type(info);
}
+#if CONFIG_KASAN_STACK
+static bool __must_check tokenize_frame_descr(const char **frame_descr,
+ char *token, size_t max_tok_len,
+ unsigned long *value)
+{
+ const char *sep = strchr(*frame_descr, ' ');
+
+ if (sep == NULL)
+ sep = *frame_descr + strlen(*frame_descr);
+
+ if (token != NULL) {
+ const size_t tok_len = sep - *frame_descr;
+
+ if (tok_len + 1 > max_tok_len) {
+ pr_err("KASAN internal error: frame description too long: %s\n",
+ *frame_descr);
+ return false;
+ }
+
+ /* Copy token (+ 1 byte for '\0'). */
+ strlcpy(token, *frame_descr, tok_len + 1);
+ }
+
+ /* Advance frame_descr past separator. */
+ *frame_descr = sep + 1;
+
+ if (value != NULL && kstrtoul(token, 10, value)) {
+ pr_err("KASAN internal error: not a valid number: %s\n", token);
+ return false;
+ }
+
+ return true;
+}
+
+static void print_decoded_frame_descr(const char *frame_descr)
+{
+ /*
+ * We need to parse the following string:
+ * "n alloc_1 alloc_2 ... alloc_n"
+ * where alloc_i looks like
+ * "offset size len name"
+ * or "offset size len name:line".
+ */
+
+ char token[64];
+ unsigned long num_objects;
+
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ &num_objects))
+ return;
+
+ pr_err("\n");
+ pr_err("this frame has %lu %s:\n", num_objects,
+ num_objects == 1 ? "object" : "objects");
+
+ while (num_objects--) {
+ unsigned long offset;
+ unsigned long size;
+
+ /* access offset */
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ &offset))
+ return;
+ /* access size */
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ &size))
+ return;
+ /* name length (unused) */
+ if (!tokenize_frame_descr(&frame_descr, NULL, 0, NULL))
+ return;
+ /* object name */
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ NULL))
+ return;
+
+ /* Strip line number; without filename it's not very helpful. */
+ strreplace(token, ':', '\0');
+
+ /* Finally, print object information. */
+ pr_err(" [%lu, %lu) '%s'", offset, offset + size, token);
+ }
+}
+
+static bool __must_check get_address_stack_frame_info(const void *addr,
+ unsigned long *offset,
+ const char **frame_descr,
+ const void **frame_pc)
+{
+ unsigned long aligned_addr;
+ unsigned long mem_ptr;
+ const u8 *shadow_bottom;
+ const u8 *shadow_ptr;
+ const unsigned long *frame;
+
+ BUILD_BUG_ON(IS_ENABLED(CONFIG_STACK_GROWSUP));
+
+ /*
+ * NOTE: We currently only support printing frame information for
+ * accesses to the task's own stack.
+ */
+ if (!object_is_on_stack(addr))
+ return false;
+
+ aligned_addr = round_down((unsigned long)addr, sizeof(long));
+ mem_ptr = round_down(aligned_addr, KASAN_GRANULE_SIZE);
+ shadow_ptr = kasan_mem_to_shadow((void *)aligned_addr);
+ shadow_bottom = kasan_mem_to_shadow(end_of_stack(current));
+
+ while (shadow_ptr >= shadow_bottom && *shadow_ptr != KASAN_STACK_LEFT) {
+ shadow_ptr--;
+ mem_ptr -= KASAN_GRANULE_SIZE;
+ }
+
+ while (shadow_ptr >= shadow_bottom && *shadow_ptr == KASAN_STACK_LEFT) {
+ shadow_ptr--;
+ mem_ptr -= KASAN_GRANULE_SIZE;
+ }
+
+ if (shadow_ptr < shadow_bottom)
+ return false;
+
+ frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
+ if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
+ pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",
+ frame[0]);
+ return false;
+ }
+
+ *offset = (unsigned long)addr - (unsigned long)frame;
+ *frame_descr = (const char *)frame[1];
+ *frame_pc = (void *)frame[2];
+
+ return true;
+}
+
+void print_address_stack_frame(const void *addr)
+{
+ unsigned long offset;
+ const char *frame_descr;
+ const void *frame_pc;
+
+ if (!get_address_stack_frame_info(addr, &offset, &frame_descr,
+ &frame_pc))
+ return;
+
+ /*
+ * get_address_stack_frame_info only returns true if the given addr is
+ * on the current task's stack.
+ */
+ pr_err("\n");
+ pr_err("addr %px is located in stack of task %s/%d at offset %lu in frame:\n",
+ addr, current->comm, task_pid_nr(current), offset);
+ pr_err(" %pS\n", frame_pc);
+
+ if (!frame_descr)
+ return;
+
+ print_decoded_frame_descr(frame_descr);
+}
+#endif /* CONFIG_KASAN_STACK */
+
#define DEFINE_ASAN_REPORT_LOAD(size) \
void __asan_report_load##size##_noabort(unsigned long addr) \
{ \
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 14/60] kasan, arm64: only init shadow for software modes
2020-12-22 19:58 incoming Andrew Morton
` (12 preceding siblings ...)
2020-12-22 20:00 ` [patch 13/60] kasan: decode stack frame only with KASAN_STACK_ENABLE Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:00 ` [patch 15/60] kasan, arm64: only use kasan_depth " Andrew Morton
` (46 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: only init shadow for software modes
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory. Only initialize it
when one of the software KASAN modes are enabled.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/d1742eea2cd728d150d49b144e49b6433405c7ba.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/kasan.h | 8 ++++++--
arch/arm64/mm/kasan_init.c | 15 ++++++++++++++-
2 files changed, 20 insertions(+), 3 deletions(-)
--- a/arch/arm64/include/asm/kasan.h~kasan-arm64-only-init-shadow-for-software-modes
+++ a/arch/arm64/include/asm/kasan.h
@@ -13,6 +13,12 @@
#define arch_kasan_get_tag(addr) __tag_get(addr)
#ifdef CONFIG_KASAN
+void kasan_init(void);
+#else
+static inline void kasan_init(void) { }
+#endif
+
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
/*
* KASAN_SHADOW_START: beginning of the kernel virtual addresses.
@@ -33,12 +39,10 @@
#define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT)))
#define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual)
-void kasan_init(void);
void kasan_copy_shadow(pgd_t *pgdir);
asmlinkage void kasan_early_init(void);
#else
-static inline void kasan_init(void) { }
static inline void kasan_copy_shadow(pgd_t *pgdir) { }
#endif
--- a/arch/arm64/mm/kasan_init.c~kasan-arm64-only-init-shadow-for-software-modes
+++ a/arch/arm64/mm/kasan_init.c
@@ -21,6 +21,8 @@
#include <asm/sections.h>
#include <asm/tlbflush.h>
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+
static pgd_t tmp_pg_dir[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE);
/*
@@ -208,7 +210,7 @@ static void __init clear_pgds(unsigned l
set_pgd(pgd_offset_k(start), __pgd(0));
}
-void __init kasan_init(void)
+static void __init kasan_init_shadow(void)
{
u64 kimg_shadow_start, kimg_shadow_end;
u64 mod_shadow_start, mod_shadow_end;
@@ -269,6 +271,17 @@ void __init kasan_init(void)
memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE);
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
+}
+
+#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) */
+
+static inline void __init kasan_init_shadow(void) { }
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+void __init kasan_init(void)
+{
+ kasan_init_shadow();
/* At this point kasan is fully initialized. Enable error messages */
init_task.kasan_depth = 0;
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 15/60] kasan, arm64: only use kasan_depth for software modes
2020-12-22 19:58 incoming Andrew Morton
` (13 preceding siblings ...)
2020-12-22 20:00 ` [patch 14/60] kasan, arm64: only init shadow for software modes Andrew Morton
@ 2020-12-22 20:00 ` Andrew Morton
2020-12-22 20:01 ` [patch 16/60] kasan, arm64: move initialization message Andrew Morton
` (45 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:00 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: only use kasan_depth for software modes
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't use kasan_depth. Only define and use it
when one of the software KASAN modes are enabled.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/e16f15aeda90bc7fb4dfc2e243a14b74cc5c8219.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/mm/kasan_init.c | 11 ++++++++---
include/linux/kasan.h | 18 +++++++++---------
include/linux/sched.h | 2 +-
init/init_task.c | 2 +-
mm/kasan/common.c | 2 ++
mm/kasan/report.c | 2 ++
6 files changed, 23 insertions(+), 14 deletions(-)
--- a/arch/arm64/mm/kasan_init.c~kasan-arm64-only-use-kasan_depth-for-software-modes
+++ a/arch/arm64/mm/kasan_init.c
@@ -273,17 +273,22 @@ static void __init kasan_init_shadow(voi
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
}
+static void __init kasan_init_depth(void)
+{
+ init_task.kasan_depth = 0;
+}
+
#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) */
static inline void __init kasan_init_shadow(void) { }
+static inline void __init kasan_init_depth(void) { }
+
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
void __init kasan_init(void)
{
kasan_init_shadow();
-
- /* At this point kasan is fully initialized. Enable error messages */
- init_task.kasan_depth = 0;
+ kasan_init_depth();
pr_info("KernelAddressSanitizer initialized\n");
}
--- a/include/linux/kasan.h~kasan-arm64-only-use-kasan_depth-for-software-modes
+++ a/include/linux/kasan.h
@@ -52,6 +52,12 @@ static inline void *kasan_mem_to_shadow(
int kasan_add_zero_shadow(void *start, unsigned long size);
void kasan_remove_zero_shadow(void *start, unsigned long size);
+/* Enable reporting bugs after kasan_disable_current() */
+extern void kasan_enable_current(void);
+
+/* Disable reporting bugs for current task */
+extern void kasan_disable_current(void);
+
#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
static inline int kasan_add_zero_shadow(void *start, unsigned long size)
@@ -62,16 +68,13 @@ static inline void kasan_remove_zero_sha
unsigned long size)
{}
+static inline void kasan_enable_current(void) {}
+static inline void kasan_disable_current(void) {}
+
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
#ifdef CONFIG_KASAN
-/* Enable reporting bugs after kasan_disable_current() */
-extern void kasan_enable_current(void);
-
-/* Disable reporting bugs for current task */
-extern void kasan_disable_current(void);
-
void kasan_unpoison_range(const void *address, size_t size);
void kasan_unpoison_task_stack(struct task_struct *task);
@@ -122,9 +125,6 @@ static inline void kasan_unpoison_range(
static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
-static inline void kasan_enable_current(void) {}
-static inline void kasan_disable_current(void) {}
-
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}
--- a/include/linux/sched.h~kasan-arm64-only-use-kasan_depth-for-software-modes
+++ a/include/linux/sched.h
@@ -1234,7 +1234,7 @@ struct task_struct {
u64 timer_slack_ns;
u64 default_timer_slack_ns;
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
unsigned int kasan_depth;
#endif
--- a/init/init_task.c~kasan-arm64-only-use-kasan_depth-for-software-modes
+++ a/init/init_task.c
@@ -176,7 +176,7 @@ struct task_struct init_task
.numa_group = NULL,
.numa_faults = NULL,
#endif
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
.kasan_depth = 1,
#endif
#ifdef CONFIG_KCSAN
--- a/mm/kasan/common.c~kasan-arm64-only-use-kasan_depth-for-software-modes
+++ a/mm/kasan/common.c
@@ -46,6 +46,7 @@ void kasan_set_track(struct kasan_track
track->stack = kasan_save_stack(flags);
}
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
void kasan_enable_current(void)
{
current->kasan_depth++;
@@ -55,6 +56,7 @@ void kasan_disable_current(void)
{
current->kasan_depth--;
}
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
void kasan_unpoison_range(const void *address, size_t size)
{
--- a/mm/kasan/report.c~kasan-arm64-only-use-kasan_depth-for-software-modes
+++ a/mm/kasan/report.c
@@ -292,8 +292,10 @@ static void print_shadow_for_address(con
static bool report_enabled(void)
{
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
if (current->kasan_depth)
return false;
+#endif
if (test_bit(KASAN_BIT_MULTI_SHOT, &kasan_flags))
return true;
return !test_and_set_bit(KASAN_BIT_REPORTED, &kasan_flags);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 16/60] kasan, arm64: move initialization message
2020-12-22 19:58 incoming Andrew Morton
` (14 preceding siblings ...)
2020-12-22 20:00 ` [patch 15/60] kasan, arm64: only use kasan_depth " Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 17/60] kasan, arm64: rename kasan_init_tags and mark as __init Andrew Morton
` (44 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: move initialization message
Software tag-based KASAN mode is fully initialized with kasan_init_tags(),
while the generic mode only requires kasan_init(). Move the
initialization message for tag-based mode into kasan_init_tags().
Also fix pr_fmt() usage for KASAN code: generic.c doesn't need it as it
doesn't use any printing functions; tag-based mode should use "kasan:"
instead of KBUILD_MODNAME (which stands for file name).
Link: https://lkml.kernel.org/r/29a30ea4e1750450dd1f693d25b7b6cb05913ecf.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/kasan.h | 9 +++------
arch/arm64/mm/kasan_init.c | 13 +++++--------
mm/kasan/generic.c | 2 --
mm/kasan/sw_tags.c | 4 +++-
4 files changed, 11 insertions(+), 17 deletions(-)
--- a/arch/arm64/include/asm/kasan.h~kasan-arm64-move-initialization-message
+++ a/arch/arm64/include/asm/kasan.h
@@ -12,14 +12,10 @@
#define arch_kasan_reset_tag(addr) __tag_reset(addr)
#define arch_kasan_get_tag(addr) __tag_get(addr)
-#ifdef CONFIG_KASAN
-void kasan_init(void);
-#else
-static inline void kasan_init(void) { }
-#endif
-
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+void kasan_init(void);
+
/*
* KASAN_SHADOW_START: beginning of the kernel virtual addresses.
* KASAN_SHADOW_END: KASAN_SHADOW_START + 1/N of kernel virtual addresses,
@@ -43,6 +39,7 @@ void kasan_copy_shadow(pgd_t *pgdir);
asmlinkage void kasan_early_init(void);
#else
+static inline void kasan_init(void) { }
static inline void kasan_copy_shadow(pgd_t *pgdir) { }
#endif
--- a/arch/arm64/mm/kasan_init.c~kasan-arm64-move-initialization-message
+++ a/arch/arm64/mm/kasan_init.c
@@ -278,17 +278,14 @@ static void __init kasan_init_depth(void
init_task.kasan_depth = 0;
}
-#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) */
-
-static inline void __init kasan_init_shadow(void) { }
-
-static inline void __init kasan_init_depth(void) { }
-
-#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
-
void __init kasan_init(void)
{
kasan_init_shadow();
kasan_init_depth();
+#if defined(CONFIG_KASAN_GENERIC)
+ /* CONFIG_KASAN_SW_TAGS also requires kasan_init_tags(). */
pr_info("KernelAddressSanitizer initialized\n");
+#endif
}
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
--- a/mm/kasan/generic.c~kasan-arm64-move-initialization-message
+++ a/mm/kasan/generic.c
@@ -9,8 +9,6 @@
* Andrey Konovalov <andreyknvl@gmail.com>
*/
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
#include <linux/export.h>
#include <linux/interrupt.h>
#include <linux/init.h>
--- a/mm/kasan/sw_tags.c~kasan-arm64-move-initialization-message
+++ a/mm/kasan/sw_tags.c
@@ -6,7 +6,7 @@
* Author: Andrey Konovalov <andreyknvl@google.com>
*/
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#define pr_fmt(fmt) "kasan: " fmt
#include <linux/export.h>
#include <linux/interrupt.h>
@@ -41,6 +41,8 @@ void kasan_init_tags(void)
for_each_possible_cpu(cpu)
per_cpu(prng_state, cpu) = (u32)get_cycles();
+
+ pr_info("KernelAddressSanitizer initialized\n");
}
/*
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 17/60] kasan, arm64: rename kasan_init_tags and mark as __init
2020-12-22 19:58 incoming Andrew Morton
` (15 preceding siblings ...)
2020-12-22 20:01 ` [patch 16/60] kasan, arm64: move initialization message Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 18/60] kasan: rename addr_has_shadow to addr_has_metadata Andrew Morton
` (43 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: rename kasan_init_tags and mark as __init
Rename kasan_init_tags() to kasan_init_sw_tags() as the upcoming hardware
tag-based KASAN mode will have its own initialization routine. Also
similarly to kasan_init() mark kasan_init_tags() as __init.
Link: https://lkml.kernel.org/r/71e52af72a09f4b50c8042f16101c60e50649fbb.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/kernel/setup.c | 2 +-
arch/arm64/mm/kasan_init.c | 2 +-
include/linux/kasan.h | 4 ++--
mm/kasan/sw_tags.c | 2 +-
4 files changed, 5 insertions(+), 5 deletions(-)
--- a/arch/arm64/kernel/setup.c~kasan-arm64-rename-kasan_init_tags-and-mark-as-__init
+++ a/arch/arm64/kernel/setup.c
@@ -358,7 +358,7 @@ void __init __no_sanitize_address setup_
smp_build_mpidr_hash();
/* Init percpu seeds for random tags after cpus are set up. */
- kasan_init_tags();
+ kasan_init_sw_tags();
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
/*
--- a/arch/arm64/mm/kasan_init.c~kasan-arm64-rename-kasan_init_tags-and-mark-as-__init
+++ a/arch/arm64/mm/kasan_init.c
@@ -283,7 +283,7 @@ void __init kasan_init(void)
kasan_init_shadow();
kasan_init_depth();
#if defined(CONFIG_KASAN_GENERIC)
- /* CONFIG_KASAN_SW_TAGS also requires kasan_init_tags(). */
+ /* CONFIG_KASAN_SW_TAGS also requires kasan_init_sw_tags(). */
pr_info("KernelAddressSanitizer initialized\n");
#endif
}
--- a/include/linux/kasan.h~kasan-arm64-rename-kasan_init_tags-and-mark-as-__init
+++ a/include/linux/kasan.h
@@ -192,7 +192,7 @@ static inline void kasan_record_aux_stac
#ifdef CONFIG_KASAN_SW_TAGS
-void kasan_init_tags(void);
+void __init kasan_init_sw_tags(void);
void *kasan_reset_tag(const void *addr);
@@ -201,7 +201,7 @@ bool kasan_report(unsigned long addr, si
#else /* CONFIG_KASAN_SW_TAGS */
-static inline void kasan_init_tags(void) { }
+static inline void kasan_init_sw_tags(void) { }
static inline void *kasan_reset_tag(const void *addr)
{
--- a/mm/kasan/sw_tags.c~kasan-arm64-rename-kasan_init_tags-and-mark-as-__init
+++ a/mm/kasan/sw_tags.c
@@ -35,7 +35,7 @@
static DEFINE_PER_CPU(u32, prng_state);
-void kasan_init_tags(void)
+void __init kasan_init_sw_tags(void)
{
int cpu;
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 18/60] kasan: rename addr_has_shadow to addr_has_metadata
2020-12-22 19:58 incoming Andrew Morton
` (16 preceding siblings ...)
2020-12-22 20:01 ` [patch 17/60] kasan, arm64: rename kasan_init_tags and mark as __init Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 19/60] kasan: rename print_shadow_for_address to print_memory_metadata Andrew Morton
` (42 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: rename addr_has_shadow to addr_has_metadata
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory, but will reuse this
function. Rename "shadow" to implementation-neutral "metadata".
No functional changes.
Link: https://lkml.kernel.org/r/370466fba590a4596b55ffd38adfd990f8886db4.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/kasan.h | 2 +-
mm/kasan/report.c | 6 +++---
mm/kasan/report_generic.c | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
--- a/mm/kasan/kasan.h~kasan-rename-addr_has_shadow-to-addr_has_metadata
+++ a/mm/kasan/kasan.h
@@ -147,7 +147,7 @@ static inline const void *kasan_shadow_t
<< KASAN_SHADOW_SCALE_SHIFT);
}
-static inline bool addr_has_shadow(const void *addr)
+static inline bool addr_has_metadata(const void *addr)
{
return (addr >= kasan_shadow_to_mem((void *)KASAN_SHADOW_START));
}
--- a/mm/kasan/report.c~kasan-rename-addr_has_shadow-to-addr_has_metadata
+++ a/mm/kasan/report.c
@@ -361,7 +361,7 @@ static void __kasan_report(unsigned long
untagged_addr = reset_tag(tagged_addr);
info.access_addr = tagged_addr;
- if (addr_has_shadow(untagged_addr))
+ if (addr_has_metadata(untagged_addr))
info.first_bad_addr = find_first_bad_addr(tagged_addr, size);
else
info.first_bad_addr = untagged_addr;
@@ -372,11 +372,11 @@ static void __kasan_report(unsigned long
start_report(&flags);
print_error_description(&info);
- if (addr_has_shadow(untagged_addr))
+ if (addr_has_metadata(untagged_addr))
print_tags(get_tag(tagged_addr), info.first_bad_addr);
pr_err("\n");
- if (addr_has_shadow(untagged_addr)) {
+ if (addr_has_metadata(untagged_addr)) {
print_address_description(untagged_addr, get_tag(tagged_addr));
pr_err("\n");
print_shadow_for_address(info.first_bad_addr);
--- a/mm/kasan/report_generic.c~kasan-rename-addr_has_shadow-to-addr_has_metadata
+++ a/mm/kasan/report_generic.c
@@ -118,7 +118,7 @@ const char *get_bug_type(struct kasan_ac
if (info->access_addr + info->access_size < info->access_addr)
return "out-of-bounds";
- if (addr_has_shadow(info->access_addr))
+ if (addr_has_metadata(info->access_addr))
return get_shadow_bug_type(info);
return get_wild_bug_type(info);
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 19/60] kasan: rename print_shadow_for_address to print_memory_metadata
2020-12-22 19:58 incoming Andrew Morton
` (17 preceding siblings ...)
2020-12-22 20:01 ` [patch 18/60] kasan: rename addr_has_shadow to addr_has_metadata Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 20/60] kasan: rename SHADOW layout macros to META Andrew Morton
` (41 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: rename print_shadow_for_address to print_memory_metadata
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory, but will reuse this
function. Rename "shadow" to implementation-neutral "metadata".
No functional changes.
Link: https://lkml.kernel.org/r/dd955c5aadaee16aef451a6189d19172166a23f5.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/report.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--- a/mm/kasan/report.c~kasan-rename-print_shadow_for_address-to-print_memory_metadata
+++ a/mm/kasan/report.c
@@ -252,7 +252,7 @@ static int shadow_pointer_offset(const v
(shadow - row) / SHADOW_BYTES_PER_BLOCK + 1;
}
-static void print_shadow_for_address(const void *addr)
+static void print_memory_metadata(const void *addr)
{
int i;
const void *shadow = kasan_mem_to_shadow(addr);
@@ -338,7 +338,7 @@ void kasan_report_invalid_free(void *obj
pr_err("\n");
print_address_description(object, tag);
pr_err("\n");
- print_shadow_for_address(object);
+ print_memory_metadata(object);
end_report(&flags);
}
@@ -379,7 +379,7 @@ static void __kasan_report(unsigned long
if (addr_has_metadata(untagged_addr)) {
print_address_description(untagged_addr, get_tag(tagged_addr));
pr_err("\n");
- print_shadow_for_address(info.first_bad_addr);
+ print_memory_metadata(info.first_bad_addr);
} else {
dump_stack();
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 20/60] kasan: rename SHADOW layout macros to META
2020-12-22 19:58 incoming Andrew Morton
` (18 preceding siblings ...)
2020-12-22 20:01 ` [patch 19/60] kasan: rename print_shadow_for_address to print_memory_metadata Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 21/60] kasan: separate metadata_fetch_row for each mode Andrew Morton
` (40 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: rename SHADOW layout macros to META
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory, but will reuse
these macros. Rename "SHADOW" to implementation-neutral "META".
No functional changes.
Link: https://lkml.kernel.org/r/f96244ec59dc17db35173ec352c5592b14aefaf8.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/report.c | 30 +++++++++++++++---------------
1 file changed, 15 insertions(+), 15 deletions(-)
--- a/mm/kasan/report.c~kasan-rename-shadow-layout-macros-to-meta
+++ a/mm/kasan/report.c
@@ -33,11 +33,11 @@
#include "kasan.h"
#include "../slab.h"
-/* Shadow layout customization. */
-#define SHADOW_BYTES_PER_BLOCK 1
-#define SHADOW_BLOCKS_PER_ROW 16
-#define SHADOW_BYTES_PER_ROW (SHADOW_BLOCKS_PER_ROW * SHADOW_BYTES_PER_BLOCK)
-#define SHADOW_ROWS_AROUND_ADDR 2
+/* Metadata layout customization. */
+#define META_BYTES_PER_BLOCK 1
+#define META_BLOCKS_PER_ROW 16
+#define META_BYTES_PER_ROW (META_BLOCKS_PER_ROW * META_BYTES_PER_BLOCK)
+#define META_ROWS_AROUND_ADDR 2
static unsigned long kasan_flags;
@@ -240,7 +240,7 @@ static void print_address_description(vo
static bool row_is_guilty(const void *row, const void *guilty)
{
- return (row <= guilty) && (guilty < row + SHADOW_BYTES_PER_ROW);
+ return (row <= guilty) && (guilty < row + META_BYTES_PER_ROW);
}
static int shadow_pointer_offset(const void *row, const void *shadow)
@@ -249,7 +249,7 @@ static int shadow_pointer_offset(const v
* 3 + (BITS_PER_LONG/8)*2 chars.
*/
return 3 + (BITS_PER_LONG/8)*2 + (shadow - row)*2 +
- (shadow - row) / SHADOW_BYTES_PER_BLOCK + 1;
+ (shadow - row) / META_BYTES_PER_BLOCK + 1;
}
static void print_memory_metadata(const void *addr)
@@ -259,15 +259,15 @@ static void print_memory_metadata(const
const void *shadow_row;
shadow_row = (void *)round_down((unsigned long)shadow,
- SHADOW_BYTES_PER_ROW)
- - SHADOW_ROWS_AROUND_ADDR * SHADOW_BYTES_PER_ROW;
+ META_BYTES_PER_ROW)
+ - META_ROWS_AROUND_ADDR * META_BYTES_PER_ROW;
pr_err("Memory state around the buggy address:\n");
- for (i = -SHADOW_ROWS_AROUND_ADDR; i <= SHADOW_ROWS_AROUND_ADDR; i++) {
+ for (i = -META_ROWS_AROUND_ADDR; i <= META_ROWS_AROUND_ADDR; i++) {
const void *kaddr = kasan_shadow_to_mem(shadow_row);
char buffer[4 + (BITS_PER_LONG/8)*2];
- char shadow_buf[SHADOW_BYTES_PER_ROW];
+ char shadow_buf[META_BYTES_PER_ROW];
snprintf(buffer, sizeof(buffer),
(i == 0) ? ">%px: " : " %px: ", kaddr);
@@ -276,17 +276,17 @@ static void print_memory_metadata(const
* function, because generic functions may try to
* access kasan mapping for the passed address.
*/
- memcpy(shadow_buf, shadow_row, SHADOW_BYTES_PER_ROW);
+ memcpy(shadow_buf, shadow_row, META_BYTES_PER_ROW);
print_hex_dump(KERN_ERR, buffer,
- DUMP_PREFIX_NONE, SHADOW_BYTES_PER_ROW, 1,
- shadow_buf, SHADOW_BYTES_PER_ROW, 0);
+ DUMP_PREFIX_NONE, META_BYTES_PER_ROW, 1,
+ shadow_buf, META_BYTES_PER_ROW, 0);
if (row_is_guilty(shadow_row, shadow))
pr_err("%*c\n",
shadow_pointer_offset(shadow_row, shadow),
'^');
- shadow_row += SHADOW_BYTES_PER_ROW;
+ shadow_row += META_BYTES_PER_ROW;
}
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 21/60] kasan: separate metadata_fetch_row for each mode
2020-12-22 19:58 incoming Andrew Morton
` (19 preceding siblings ...)
2020-12-22 20:01 ` [patch 20/60] kasan: rename SHADOW layout macros to META Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 22/60] kasan: introduce CONFIG_KASAN_HW_TAGS Andrew Morton
` (39 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: separate metadata_fetch_row for each mode
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Rework print_memory_metadata() to make it agnostic with regard to the way
metadata is stored. Allow providing a separate metadata_fetch_row()
implementation for each KASAN mode. Hardware tag-based KASAN will provide
its own implementation that doesn't use shadow memory.
No functional changes for software modes.
Link: https://lkml.kernel.org/r/5fb1ec0152bb1f521505017800387ec3e36ffe18.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/kasan.h | 8 +++++
mm/kasan/report.c | 56 +++++++++++++++++-------------------
mm/kasan/report_generic.c | 5 +++
mm/kasan/report_sw_tags.c | 5 +++
4 files changed, 45 insertions(+), 29 deletions(-)
--- a/mm/kasan/kasan.h~kasan-separate-metadata_fetch_row-for-each-mode
+++ a/mm/kasan/kasan.h
@@ -58,6 +58,13 @@
#define KASAN_ABI_VERSION 1
#endif
+/* Metadata layout customization. */
+#define META_BYTES_PER_BLOCK 1
+#define META_BLOCKS_PER_ROW 16
+#define META_BYTES_PER_ROW (META_BLOCKS_PER_ROW * META_BYTES_PER_BLOCK)
+#define META_MEM_BYTES_PER_ROW (META_BYTES_PER_ROW * KASAN_GRANULE_SIZE)
+#define META_ROWS_AROUND_ADDR 2
+
struct kasan_access_info {
const void *access_addr;
const void *first_bad_addr;
@@ -170,6 +177,7 @@ bool check_invalid_free(void *addr);
void *find_first_bad_addr(void *addr, size_t size);
const char *get_bug_type(struct kasan_access_info *info);
+void metadata_fetch_row(char *buffer, void *row);
#if defined(CONFIG_KASAN_GENERIC) && CONFIG_KASAN_STACK
void print_address_stack_frame(const void *addr);
--- a/mm/kasan/report.c~kasan-separate-metadata_fetch_row-for-each-mode
+++ a/mm/kasan/report.c
@@ -33,12 +33,6 @@
#include "kasan.h"
#include "../slab.h"
-/* Metadata layout customization. */
-#define META_BYTES_PER_BLOCK 1
-#define META_BLOCKS_PER_ROW 16
-#define META_BYTES_PER_ROW (META_BLOCKS_PER_ROW * META_BYTES_PER_BLOCK)
-#define META_ROWS_AROUND_ADDR 2
-
static unsigned long kasan_flags;
#define KASAN_BIT_REPORTED 0
@@ -238,55 +232,59 @@ static void print_address_description(vo
print_address_stack_frame(addr);
}
-static bool row_is_guilty(const void *row, const void *guilty)
+static bool meta_row_is_guilty(const void *row, const void *addr)
{
- return (row <= guilty) && (guilty < row + META_BYTES_PER_ROW);
+ return (row <= addr) && (addr < row + META_MEM_BYTES_PER_ROW);
}
-static int shadow_pointer_offset(const void *row, const void *shadow)
+static int meta_pointer_offset(const void *row, const void *addr)
{
- /* The length of ">ff00ff00ff00ff00: " is
- * 3 + (BITS_PER_LONG/8)*2 chars.
+ /*
+ * Memory state around the buggy address:
+ * ff00ff00ff00ff00: 00 00 00 05 fe fe fe fe fe fe fe fe fe fe fe fe
+ * ...
+ *
+ * The length of ">ff00ff00ff00ff00: " is
+ * 3 + (BITS_PER_LONG / 8) * 2 chars.
+ * The length of each granule metadata is 2 bytes
+ * plus 1 byte for space.
*/
- return 3 + (BITS_PER_LONG/8)*2 + (shadow - row)*2 +
- (shadow - row) / META_BYTES_PER_BLOCK + 1;
+ return 3 + (BITS_PER_LONG / 8) * 2 +
+ (addr - row) / KASAN_GRANULE_SIZE * 3 + 1;
}
static void print_memory_metadata(const void *addr)
{
int i;
- const void *shadow = kasan_mem_to_shadow(addr);
- const void *shadow_row;
+ void *row;
- shadow_row = (void *)round_down((unsigned long)shadow,
- META_BYTES_PER_ROW)
- - META_ROWS_AROUND_ADDR * META_BYTES_PER_ROW;
+ row = (void *)round_down((unsigned long)addr, META_MEM_BYTES_PER_ROW)
+ - META_ROWS_AROUND_ADDR * META_MEM_BYTES_PER_ROW;
pr_err("Memory state around the buggy address:\n");
for (i = -META_ROWS_AROUND_ADDR; i <= META_ROWS_AROUND_ADDR; i++) {
- const void *kaddr = kasan_shadow_to_mem(shadow_row);
- char buffer[4 + (BITS_PER_LONG/8)*2];
- char shadow_buf[META_BYTES_PER_ROW];
+ char buffer[4 + (BITS_PER_LONG / 8) * 2];
+ char metadata[META_BYTES_PER_ROW];
snprintf(buffer, sizeof(buffer),
- (i == 0) ? ">%px: " : " %px: ", kaddr);
+ (i == 0) ? ">%px: " : " %px: ", row);
+
/*
* We should not pass a shadow pointer to generic
* function, because generic functions may try to
* access kasan mapping for the passed address.
*/
- memcpy(shadow_buf, shadow_row, META_BYTES_PER_ROW);
+ metadata_fetch_row(&metadata[0], row);
+
print_hex_dump(KERN_ERR, buffer,
DUMP_PREFIX_NONE, META_BYTES_PER_ROW, 1,
- shadow_buf, META_BYTES_PER_ROW, 0);
+ metadata, META_BYTES_PER_ROW, 0);
- if (row_is_guilty(shadow_row, shadow))
- pr_err("%*c\n",
- shadow_pointer_offset(shadow_row, shadow),
- '^');
+ if (meta_row_is_guilty(row, addr))
+ pr_err("%*c\n", meta_pointer_offset(row, addr), '^');
- shadow_row += META_BYTES_PER_ROW;
+ row += META_MEM_BYTES_PER_ROW;
}
}
--- a/mm/kasan/report_generic.c~kasan-separate-metadata_fetch_row-for-each-mode
+++ a/mm/kasan/report_generic.c
@@ -123,6 +123,11 @@ const char *get_bug_type(struct kasan_ac
return get_wild_bug_type(info);
}
+void metadata_fetch_row(char *buffer, void *row)
+{
+ memcpy(buffer, kasan_mem_to_shadow(row), META_BYTES_PER_ROW);
+}
+
#if CONFIG_KASAN_STACK
static bool __must_check tokenize_frame_descr(const char **frame_descr,
char *token, size_t max_tok_len,
--- a/mm/kasan/report_sw_tags.c~kasan-separate-metadata_fetch_row-for-each-mode
+++ a/mm/kasan/report_sw_tags.c
@@ -80,6 +80,11 @@ void *find_first_bad_addr(void *addr, si
return p;
}
+void metadata_fetch_row(char *buffer, void *row)
+{
+ memcpy(buffer, kasan_mem_to_shadow(row), META_BYTES_PER_ROW);
+}
+
void print_tags(u8 addr_tag, const void *addr)
{
u8 *shadow = (u8 *)kasan_mem_to_shadow(addr);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 22/60] kasan: introduce CONFIG_KASAN_HW_TAGS
2020-12-22 19:58 incoming Andrew Morton
` (20 preceding siblings ...)
2020-12-22 20:01 ` [patch 21/60] kasan: separate metadata_fetch_row for each mode Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 23/60] arm64: enable armv8.5-a asm-arch option Andrew Morton
` (38 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: introduce CONFIG_KASAN_HW_TAGS
This patch adds a configuration option for a new KASAN mode called
hardware tag-based KASAN. This mode uses the memory tagging approach like
the software tag-based mode, but relies on arm64 Memory Tagging Extension
feature for tag management and access checking.
Link: https://lkml.kernel.org/r/44906a209d3a44f9c6f5a21841e90988e365601e.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Co-developed-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
lib/Kconfig.kasan | 61 +++++++++++++++++++++++++++++++-------------
1 file changed, 44 insertions(+), 17 deletions(-)
--- a/lib/Kconfig.kasan~kasan-introduce-config_kasan_hw_tags
+++ a/lib/Kconfig.kasan
@@ -6,7 +6,10 @@ config HAVE_ARCH_KASAN
config HAVE_ARCH_KASAN_SW_TAGS
bool
-config HAVE_ARCH_KASAN_VMALLOC
+config HAVE_ARCH_KASAN_HW_TAGS
+ bool
+
+config HAVE_ARCH_KASAN_VMALLOC
bool
config CC_HAS_KASAN_GENERIC
@@ -15,16 +18,19 @@ config CC_HAS_KASAN_GENERIC
config CC_HAS_KASAN_SW_TAGS
def_bool $(cc-option, -fsanitize=kernel-hwaddress)
+# This option is only required for software KASAN modes.
+# Old GCC versions don't have proper support for no_sanitize_address.
+# See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89124 for details.
config CC_HAS_WORKING_NOSANITIZE_ADDRESS
def_bool !CC_IS_GCC || GCC_VERSION >= 80300
menuconfig KASAN
bool "KASAN: runtime memory debugger"
- depends on (HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC) || \
- (HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS)
+ depends on (((HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC) || \
+ (HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS)) && \
+ CC_HAS_WORKING_NOSANITIZE_ADDRESS) || \
+ HAVE_ARCH_KASAN_HW_TAGS
depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
- depends on CC_HAS_WORKING_NOSANITIZE_ADDRESS
- select CONSTRUCTORS
select STACKDEPOT
help
Enables KASAN (KernelAddressSANitizer) - runtime memory debugger,
@@ -37,18 +43,24 @@ choice
prompt "KASAN mode"
default KASAN_GENERIC
help
- KASAN has two modes: generic KASAN (similar to userspace ASan,
- x86_64/arm64/xtensa, enabled with CONFIG_KASAN_GENERIC) and
- software tag-based KASAN (a version based on software memory
- tagging, arm64 only, similar to userspace HWASan, enabled with
- CONFIG_KASAN_SW_TAGS).
+ KASAN has three modes:
+ 1. generic KASAN (similar to userspace ASan,
+ x86_64/arm64/xtensa, enabled with CONFIG_KASAN_GENERIC),
+ 2. software tag-based KASAN (arm64 only, based on software
+ memory tagging (similar to userspace HWASan), enabled with
+ CONFIG_KASAN_SW_TAGS), and
+ 3. hardware tag-based KASAN (arm64 only, based on hardware
+ memory tagging, enabled with CONFIG_KASAN_HW_TAGS).
+
+ All KASAN modes are strictly debugging features.
- Both generic and tag-based KASAN are strictly debugging features.
+ For better error reports enable CONFIG_STACKTRACE.
config KASAN_GENERIC
bool "Generic mode"
depends on HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC
select SLUB_DEBUG if SLUB
+ select CONSTRUCTORS
help
Enables generic KASAN mode.
@@ -61,8 +73,6 @@ config KASAN_GENERIC
and introduces an overhead of ~x1.5 for the rest of the allocations.
The performance slowdown is ~x3.
- For better error detection enable CONFIG_STACKTRACE.
-
Currently CONFIG_KASAN_GENERIC doesn't work with CONFIG_DEBUG_SLAB
(the resulting kernel does not boot).
@@ -70,11 +80,15 @@ config KASAN_SW_TAGS
bool "Software tag-based mode"
depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
select SLUB_DEBUG if SLUB
+ select CONSTRUCTORS
help
Enables software tag-based KASAN mode.
- This mode requires Top Byte Ignore support by the CPU and therefore
- is only supported for arm64. This mode requires Clang.
+ This mode require software memory tagging support in the form of
+ HWASan-like compiler instrumentation.
+
+ Currently this mode is only implemented for arm64 CPUs and relies on
+ Top Byte Ignore. This mode requires Clang.
This mode consumes about 1/16th of available memory at kernel start
and introduces an overhead of ~20% for the rest of the allocations.
@@ -82,15 +96,27 @@ config KASAN_SW_TAGS
casting and comparison, as it embeds tags into the top byte of each
pointer.
- For better error detection enable CONFIG_STACKTRACE.
-
Currently CONFIG_KASAN_SW_TAGS doesn't work with CONFIG_DEBUG_SLAB
(the resulting kernel does not boot).
+config KASAN_HW_TAGS
+ bool "Hardware tag-based mode"
+ depends on HAVE_ARCH_KASAN_HW_TAGS
+ depends on SLUB
+ help
+ Enables hardware tag-based KASAN mode.
+
+ This mode requires hardware memory tagging support, and can be used
+ by any architecture that provides it.
+
+ Currently this mode is only implemented for arm64 CPUs starting from
+ ARMv8.5 and relies on Memory Tagging Extension and Top Byte Ignore.
+
endchoice
choice
prompt "Instrumentation type"
+ depends on KASAN_GENERIC || KASAN_SW_TAGS
default KASAN_OUTLINE
config KASAN_OUTLINE
@@ -114,6 +140,7 @@ endchoice
config KASAN_STACK_ENABLE
bool "Enable stack instrumentation (unsafe)" if CC_IS_CLANG && !COMPILE_TEST
+ depends on KASAN_GENERIC || KASAN_SW_TAGS
help
The LLVM stack address sanitizer has a know problem that
causes excessive stack usage in a lot of functions, see
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 23/60] arm64: enable armv8.5-a asm-arch option
2020-12-22 19:58 incoming Andrew Morton
` (21 preceding siblings ...)
2020-12-22 20:01 ` [patch 22/60] kasan: introduce CONFIG_KASAN_HW_TAGS Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 24/60] arm64: mte: add in-kernel MTE helpers Andrew Morton
` (37 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: arm64: enable armv8.5-a asm-arch option
Hardware tag-based KASAN relies on Memory Tagging Extension (MTE) which is
an armv8.5-a architecture extension.
Enable the correct asm option when the compiler supports it in order to
allow the usage of ALTERNATIVE()s with MTE instructions.
Link: https://lkml.kernel.org/r/d03d1157124ea3532eaeb77507988733f5734986.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/Kconfig | 4 ++++
arch/arm64/Makefile | 5 +++++
2 files changed, 9 insertions(+)
--- a/arch/arm64/Kconfig~arm64-enable-armv85-a-asm-arch-option
+++ a/arch/arm64/Kconfig
@@ -1571,6 +1571,9 @@ endmenu
menu "ARMv8.5 architectural features"
+config AS_HAS_ARMV8_5
+ def_bool $(cc-option,-Wa$(comma)-march=armv8.5-a)
+
config ARM64_BTI
bool "Branch Target Identification support"
default y
@@ -1645,6 +1648,7 @@ config ARM64_MTE
bool "Memory Tagging Extension support"
default y
depends on ARM64_AS_HAS_MTE && ARM64_TAGGED_ADDR_ABI
+ depends on AS_HAS_ARMV8_5
select ARCH_USES_HIGH_VMA_FLAGS
help
Memory Tagging (part of the ARMv8.5 Extensions) provides
--- a/arch/arm64/Makefile~arm64-enable-armv85-a-asm-arch-option
+++ a/arch/arm64/Makefile
@@ -96,6 +96,11 @@ ifeq ($(CONFIG_AS_HAS_ARMV8_4), y)
asm-arch := armv8.4-a
endif
+ifeq ($(CONFIG_AS_HAS_ARMV8_5), y)
+# make sure to pass the newest target architecture to -march.
+asm-arch := armv8.5-a
+endif
+
ifdef asm-arch
KBUILD_CFLAGS += -Wa,-march=$(asm-arch) \
-DARM64_ASM_ARCH='"$(asm-arch)"'
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 24/60] arm64: mte: add in-kernel MTE helpers
2020-12-22 19:58 incoming Andrew Morton
` (22 preceding siblings ...)
2020-12-22 20:01 ` [patch 23/60] arm64: enable armv8.5-a asm-arch option Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 25/60] arm64: mte: reset the page tag in page->flags Andrew Morton
` (36 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: arm64: mte: add in-kernel MTE helpers
Provide helper functions to manipulate allocation and pointer tags for
kernel addresses.
Low-level helper functions (mte_assign_*, written in assembly) operate tag
values from the [0x0, 0xF] range. High-level helper functions
(mte_get/set_*) use the [0xF0, 0xFF] range to preserve compatibility with
normal kernel pointers that have 0xFF in their top byte.
MTE_GRANULE_SIZE and related definitions are moved to mte-def.h header
that doesn't have any dependencies and is safe to include into any
low-level header.
Link: https://lkml.kernel.org/r/c31bf759b4411b2d98cdd801eb928e241584fd1f.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Co-developed-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/esr.h | 1
arch/arm64/include/asm/mte-def.h | 15 +++++++
arch/arm64/include/asm/mte-kasan.h | 56 +++++++++++++++++++++++++++
arch/arm64/include/asm/mte.h | 20 ++++++---
arch/arm64/kernel/mte.c | 48 +++++++++++++++++++++++
arch/arm64/lib/mte.S | 16 +++++++
6 files changed, 150 insertions(+), 6 deletions(-)
--- a/arch/arm64/include/asm/esr.h~arm64-mte-add-in-kernel-mte-helpers
+++ a/arch/arm64/include/asm/esr.h
@@ -106,6 +106,7 @@
#define ESR_ELx_FSC_TYPE (0x3C)
#define ESR_ELx_FSC_LEVEL (0x03)
#define ESR_ELx_FSC_EXTABT (0x10)
+#define ESR_ELx_FSC_MTE (0x11)
#define ESR_ELx_FSC_SERROR (0x11)
#define ESR_ELx_FSC_ACCESS (0x08)
#define ESR_ELx_FSC_FAULT (0x04)
--- /dev/null
+++ a/arch/arm64/include/asm/mte-def.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 ARM Ltd.
+ */
+#ifndef __ASM_MTE_DEF_H
+#define __ASM_MTE_DEF_H
+
+#define MTE_GRANULE_SIZE UL(16)
+#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
+#define MTE_TAG_SHIFT 56
+#define MTE_TAG_SIZE 4
+#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
+#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
+
+#endif /* __ASM_MTE_DEF_H */
--- a/arch/arm64/include/asm/mte.h~arm64-mte-add-in-kernel-mte-helpers
+++ a/arch/arm64/include/asm/mte.h
@@ -5,14 +5,16 @@
#ifndef __ASM_MTE_H
#define __ASM_MTE_H
-#define MTE_GRANULE_SIZE UL(16)
-#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
-#define MTE_TAG_SHIFT 56
-#define MTE_TAG_SIZE 4
+#include <asm/compiler.h>
+#include <asm/mte-def.h>
+
+#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n"
#ifndef __ASSEMBLY__
+#include <linux/bitfield.h>
#include <linux/page-flags.h>
+#include <linux/types.h>
#include <asm/pgtable-types.h>
@@ -45,7 +47,9 @@ long get_mte_ctrl(struct task_struct *ta
int mte_ptrace_copy_tags(struct task_struct *child, long request,
unsigned long addr, unsigned long data);
-#else
+void mte_assign_mem_tag_range(void *addr, size_t size);
+
+#else /* CONFIG_ARM64_MTE */
/* unused if !CONFIG_ARM64_MTE, silence the compiler */
#define PG_mte_tagged 0
@@ -80,7 +84,11 @@ static inline int mte_ptrace_copy_tags(s
return -EIO;
}
-#endif
+static inline void mte_assign_mem_tag_range(void *addr, size_t size)
+{
+}
+
+#endif /* CONFIG_ARM64_MTE */
#endif /* __ASSEMBLY__ */
#endif /* __ASM_MTE_H */
--- /dev/null
+++ a/arch/arm64/include/asm/mte-kasan.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 ARM Ltd.
+ */
+#ifndef __ASM_MTE_KASAN_H
+#define __ASM_MTE_KASAN_H
+
+#include <asm/mte-def.h>
+
+#ifndef __ASSEMBLY__
+
+#include <linux/types.h>
+
+/*
+ * The functions below are meant to be used only for the
+ * KASAN_HW_TAGS interface defined in asm/memory.h.
+ */
+#ifdef CONFIG_ARM64_MTE
+
+static inline u8 mte_get_ptr_tag(void *ptr)
+{
+ /* Note: The format of KASAN tags is 0xF<x> */
+ u8 tag = 0xF0 | (u8)(((u64)(ptr)) >> MTE_TAG_SHIFT);
+
+ return tag;
+}
+
+u8 mte_get_mem_tag(void *addr);
+u8 mte_get_random_tag(void);
+void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag);
+
+#else /* CONFIG_ARM64_MTE */
+
+static inline u8 mte_get_ptr_tag(void *ptr)
+{
+ return 0xFF;
+}
+
+static inline u8 mte_get_mem_tag(void *addr)
+{
+ return 0xFF;
+}
+static inline u8 mte_get_random_tag(void)
+{
+ return 0xFF;
+}
+static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
+{
+ return addr;
+}
+
+#endif /* CONFIG_ARM64_MTE */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_MTE_KASAN_H */
--- a/arch/arm64/kernel/mte.c~arm64-mte-add-in-kernel-mte-helpers
+++ a/arch/arm64/kernel/mte.c
@@ -13,10 +13,13 @@
#include <linux/swap.h>
#include <linux/swapops.h>
#include <linux/thread_info.h>
+#include <linux/types.h>
#include <linux/uio.h>
+#include <asm/barrier.h>
#include <asm/cpufeature.h>
#include <asm/mte.h>
+#include <asm/mte-kasan.h>
#include <asm/ptrace.h>
#include <asm/sysreg.h>
@@ -72,6 +75,51 @@ int memcmp_pages(struct page *page1, str
return ret;
}
+u8 mte_get_mem_tag(void *addr)
+{
+ if (!system_supports_mte())
+ return 0xFF;
+
+ asm(__MTE_PREAMBLE "ldg %0, [%0]"
+ : "+r" (addr));
+
+ return mte_get_ptr_tag(addr);
+}
+
+u8 mte_get_random_tag(void)
+{
+ void *addr;
+
+ if (!system_supports_mte())
+ return 0xFF;
+
+ asm(__MTE_PREAMBLE "irg %0, %0"
+ : "+r" (addr));
+
+ return mte_get_ptr_tag(addr);
+}
+
+void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
+{
+ void *ptr = addr;
+
+ if ((!system_supports_mte()) || (size == 0))
+ return addr;
+
+ /* Make sure that size is MTE granule aligned. */
+ WARN_ON(size & (MTE_GRANULE_SIZE - 1));
+
+ /* Make sure that the address is MTE granule aligned. */
+ WARN_ON((u64)addr & (MTE_GRANULE_SIZE - 1));
+
+ tag = 0xF0 | tag;
+ ptr = (void *)__tag_set(ptr, tag);
+
+ mte_assign_mem_tag_range(ptr, size);
+
+ return ptr;
+}
+
static void update_sctlr_el1_tcf0(u64 tcf0)
{
/* ISB required for the kernel uaccess routines */
--- a/arch/arm64/lib/mte.S~arm64-mte-add-in-kernel-mte-helpers
+++ a/arch/arm64/lib/mte.S
@@ -149,3 +149,19 @@ SYM_FUNC_START(mte_restore_page_tags)
ret
SYM_FUNC_END(mte_restore_page_tags)
+
+/*
+ * Assign allocation tags for a region of memory based on the pointer tag
+ * x0 - source pointer
+ * x1 - size
+ *
+ * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and
+ * size must be non-zero and MTE_GRANULE_SIZE aligned.
+ */
+SYM_FUNC_START(mte_assign_mem_tag_range)
+1: stg x0, [x0]
+ add x0, x0, #MTE_GRANULE_SIZE
+ subs x1, x1, #MTE_GRANULE_SIZE
+ b.gt 1b
+ ret
+SYM_FUNC_END(mte_assign_mem_tag_range)
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 25/60] arm64: mte: reset the page tag in page->flags
2020-12-22 19:58 incoming Andrew Morton
` (23 preceding siblings ...)
2020-12-22 20:01 ` [patch 24/60] arm64: mte: add in-kernel MTE helpers Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 26/60] arm64: mte: add in-kernel tag fault handler Andrew Morton
` (35 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: arm64: mte: reset the page tag in page->flags
The hardware tag-based KASAN for compatibility with the other modes stores
the tag associated to a page in page->flags. Due to this the kernel
faults on access when it allocates a page with an initial tag and the user
changes the tags.
Reset the tag associated by the kernel to a page in all the meaningful
places to prevent kernel faults on access.
Note: An alternative to this approach could be to modify page_to_virt().
This though could end up being racy, in fact if a CPU checks the
PG_mte_tagged bit and decides that the page is not tagged but another CPU
maps the same with PROT_MTE and becomes tagged the subsequent kernel
access would fail.
Link: https://lkml.kernel.org/r/9073d4e973747a6f78d5bdd7ebe17f290d087096.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/kernel/hibernate.c | 5 +++++
arch/arm64/kernel/mte.c | 9 +++++++++
arch/arm64/mm/copypage.c | 9 +++++++++
arch/arm64/mm/mteswap.c | 9 +++++++++
4 files changed, 32 insertions(+)
--- a/arch/arm64/kernel/hibernate.c~arm64-mte-reset-the-page-tag-in-page-flags
+++ a/arch/arm64/kernel/hibernate.c
@@ -371,6 +371,11 @@ static void swsusp_mte_restore_tags(void
unsigned long pfn = xa_state.xa_index;
struct page *page = pfn_to_online_page(pfn);
+ /*
+ * It is not required to invoke page_kasan_tag_reset(page)
+ * at this point since the tags stored in page->flags are
+ * already restored.
+ */
mte_restore_page_tags(page_address(page), tags);
mte_free_tag_storage(tags);
--- a/arch/arm64/kernel/mte.c~arm64-mte-reset-the-page-tag-in-page-flags
+++ a/arch/arm64/kernel/mte.c
@@ -34,6 +34,15 @@ static void mte_sync_page_tags(struct pa
return;
}
+ page_kasan_tag_reset(page);
+ /*
+ * We need smp_wmb() in between setting the flags and clearing the
+ * tags because if another thread reads page->flags and builds a
+ * tagged address out of it, there is an actual dependency to the
+ * memory access, but on the current thread we do not guarantee that
+ * the new page->flags are visible before the tags were updated.
+ */
+ smp_wmb();
mte_clear_page_tags(page_address(page));
}
--- a/arch/arm64/mm/copypage.c~arm64-mte-reset-the-page-tag-in-page-flags
+++ a/arch/arm64/mm/copypage.c
@@ -23,6 +23,15 @@ void copy_highpage(struct page *to, stru
if (system_supports_mte() && test_bit(PG_mte_tagged, &from->flags)) {
set_bit(PG_mte_tagged, &to->flags);
+ page_kasan_tag_reset(to);
+ /*
+ * We need smp_wmb() in between setting the flags and clearing the
+ * tags because if another thread reads page->flags and builds a
+ * tagged address out of it, there is an actual dependency to the
+ * memory access, but on the current thread we do not guarantee that
+ * the new page->flags are visible before the tags were updated.
+ */
+ smp_wmb();
mte_copy_page_tags(kto, kfrom);
}
}
--- a/arch/arm64/mm/mteswap.c~arm64-mte-reset-the-page-tag-in-page-flags
+++ a/arch/arm64/mm/mteswap.c
@@ -53,6 +53,15 @@ bool mte_restore_tags(swp_entry_t entry,
if (!tags)
return false;
+ page_kasan_tag_reset(page);
+ /*
+ * We need smp_wmb() in between setting the flags and clearing the
+ * tags because if another thread reads page->flags and builds a
+ * tagged address out of it, there is an actual dependency to the
+ * memory access, but on the current thread we do not guarantee that
+ * the new page->flags are visible before the tags were updated.
+ */
+ smp_wmb();
mte_restore_page_tags(page_address(page), tags);
return true;
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 26/60] arm64: mte: add in-kernel tag fault handler
2020-12-22 19:58 incoming Andrew Morton
` (24 preceding siblings ...)
2020-12-22 20:01 ` [patch 25/60] arm64: mte: reset the page tag in page->flags Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 27/60] arm64: kasan: allow enabling in-kernel MTE Andrew Morton
` (34 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: arm64: mte: add in-kernel tag fault handler
Add the implementation of the in-kernel fault handler.
When a tag fault happens on a kernel address:
* MTE is disabled on the current CPU,
* the execution continues.
When a tag fault happens on a user address:
* the kernel executes do_bad_area() and panics.
The tag fault handler for kernel addresses is currently empty and will be
filled in by a future commit.
Link: https://lkml.kernel.org/r/ad31529b073e22840b7a2246172c2b67747ed7c4.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Co-developed-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
[catalin.marinas@arm.com: ensure CONFIG_ARM64_PAN is enabled with MTE]
Link: https://lkml.kernel.org/r/20201203102628.GB2224@gaia
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/Kconfig | 2 +
arch/arm64/include/asm/uaccess.h | 23 ++++++++++++++
arch/arm64/mm/fault.c | 45 +++++++++++++++++++++++++++++
3 files changed, 70 insertions(+)
--- a/arch/arm64/include/asm/uaccess.h~arm64-mte-add-in-kernel-tag-fault-handler
+++ a/arch/arm64/include/asm/uaccess.h
@@ -159,8 +159,28 @@ static inline void __uaccess_enable_hw_p
CONFIG_ARM64_PAN));
}
+/*
+ * The Tag Check Flag (TCF) mode for MTE is per EL, hence TCF0
+ * affects EL0 and TCF affects EL1 irrespective of which TTBR is
+ * used.
+ * The kernel accesses TTBR0 usually with LDTR/STTR instructions
+ * when UAO is available, so these would act as EL0 accesses using
+ * TCF0.
+ * However futex.h code uses exclusives which would be executed as
+ * EL1, this can potentially cause a tag check fault even if the
+ * user disables TCF0.
+ *
+ * To address the problem we set the PSTATE.TCO bit in uaccess_enable()
+ * and reset it in uaccess_disable().
+ *
+ * The Tag check override (TCO) bit disables temporarily the tag checking
+ * preventing the issue.
+ */
static inline void uaccess_disable_privileged(void)
{
+ asm volatile(ALTERNATIVE("nop", SET_PSTATE_TCO(0),
+ ARM64_MTE, CONFIG_KASAN_HW_TAGS));
+
if (uaccess_ttbr0_disable())
return;
@@ -169,6 +189,9 @@ static inline void uaccess_disable_privi
static inline void uaccess_enable_privileged(void)
{
+ asm volatile(ALTERNATIVE("nop", SET_PSTATE_TCO(1),
+ ARM64_MTE, CONFIG_KASAN_HW_TAGS));
+
if (uaccess_ttbr0_enable())
return;
--- a/arch/arm64/Kconfig~arm64-mte-add-in-kernel-tag-fault-handler
+++ a/arch/arm64/Kconfig
@@ -1649,6 +1649,8 @@ config ARM64_MTE
default y
depends on ARM64_AS_HAS_MTE && ARM64_TAGGED_ADDR_ABI
depends on AS_HAS_ARMV8_5
+ # Required for tag checking in the uaccess routines
+ depends on ARM64_PAN
select ARCH_USES_HIGH_VMA_FLAGS
help
Memory Tagging (part of the ARMv8.5 Extensions) provides
--- a/arch/arm64/mm/fault.c~arm64-mte-add-in-kernel-tag-fault-handler
+++ a/arch/arm64/mm/fault.c
@@ -33,6 +33,7 @@
#include <asm/debug-monitors.h>
#include <asm/esr.h>
#include <asm/kprobes.h>
+#include <asm/mte.h>
#include <asm/processor.h>
#include <asm/sysreg.h>
#include <asm/system_misc.h>
@@ -296,6 +297,44 @@ static void die_kernel_fault(const char
do_exit(SIGKILL);
}
+static void report_tag_fault(unsigned long addr, unsigned int esr,
+ struct pt_regs *regs)
+{
+}
+
+static void do_tag_recovery(unsigned long addr, unsigned int esr,
+ struct pt_regs *regs)
+{
+ static bool reported;
+
+ if (!READ_ONCE(reported)) {
+ report_tag_fault(addr, esr, regs);
+ WRITE_ONCE(reported, true);
+ }
+
+ /*
+ * Disable MTE Tag Checking on the local CPU for the current EL.
+ * It will be done lazily on the other CPUs when they will hit a
+ * tag fault.
+ */
+ sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_NONE);
+ isb();
+}
+
+static bool is_el1_mte_sync_tag_check_fault(unsigned int esr)
+{
+ unsigned int ec = ESR_ELx_EC(esr);
+ unsigned int fsc = esr & ESR_ELx_FSC;
+
+ if (ec != ESR_ELx_EC_DABT_CUR)
+ return false;
+
+ if (fsc == ESR_ELx_FSC_MTE)
+ return true;
+
+ return false;
+}
+
static void __do_kernel_fault(unsigned long addr, unsigned int esr,
struct pt_regs *regs)
{
@@ -312,6 +351,12 @@ static void __do_kernel_fault(unsigned l
"Ignoring spurious kernel translation fault at virtual address %016lx\n", addr))
return;
+ if (is_el1_mte_sync_tag_check_fault(esr)) {
+ do_tag_recovery(addr, esr, regs);
+
+ return;
+ }
+
if (is_el1_permission_fault(addr, esr, regs)) {
if (esr & ESR_ELx_WNR)
msg = "write to read-only memory";
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 27/60] arm64: kasan: allow enabling in-kernel MTE
2020-12-22 19:58 incoming Andrew Morton
` (25 preceding siblings ...)
2020-12-22 20:01 ` [patch 26/60] arm64: mte: add in-kernel tag fault handler Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 28/60] arm64: mte: convert gcr_user into an exclude mask Andrew Morton
` (33 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: arm64: kasan: allow enabling in-kernel MTE
Hardware tag-based KASAN relies on Memory Tagging Extension (MTE) feature
and requires it to be enabled. MTE supports
This patch adds a new mte_enable_kernel() helper, that enables MTE in
Synchronous mode in EL1 and is intended to be called from KASAN runtime
during initialization.
The Tag Checking operation causes a synchronous data abort as a
consequence of a tag check fault when MTE is configured in synchronous
mode.
As part of this change enable match-all tag for EL1 to allow the kernel to
access user pages without faulting. This is required because the kernel
does not have knowledge of the tags set by the user in a page.
Note: For MTE, the TCF bit field in SCTLR_EL1 affects only EL1 in a
similar way as TCF0 affects EL0.
MTE that is built on top of the Top Byte Ignore (TBI) feature hence we
enable it as part of this patch as well.
Link: https://lkml.kernel.org/r/7352b0a0899af65c2785416c8ca6bf3845b66fa1.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Co-developed-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/mte-kasan.h | 6 ++++++
arch/arm64/kernel/mte.c | 7 +++++++
arch/arm64/mm/proc.S | 23 ++++++++++++++++++++---
3 files changed, 33 insertions(+), 3 deletions(-)
--- a/arch/arm64/include/asm/mte-kasan.h~arm64-kasan-allow-enabling-in-kernel-mte
+++ a/arch/arm64/include/asm/mte-kasan.h
@@ -29,6 +29,8 @@ u8 mte_get_mem_tag(void *addr);
u8 mte_get_random_tag(void);
void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag);
+void mte_enable_kernel(void);
+
#else /* CONFIG_ARM64_MTE */
static inline u8 mte_get_ptr_tag(void *ptr)
@@ -49,6 +51,10 @@ static inline void *mte_set_mem_tag_rang
return addr;
}
+static inline void mte_enable_kernel(void)
+{
+}
+
#endif /* CONFIG_ARM64_MTE */
#endif /* __ASSEMBLY__ */
--- a/arch/arm64/kernel/mte.c~arm64-kasan-allow-enabling-in-kernel-mte
+++ a/arch/arm64/kernel/mte.c
@@ -129,6 +129,13 @@ void *mte_set_mem_tag_range(void *addr,
return ptr;
}
+void mte_enable_kernel(void)
+{
+ /* Enable MTE Sync Mode for EL1. */
+ sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC);
+ isb();
+}
+
static void update_sctlr_el1_tcf0(u64 tcf0)
{
/* ISB required for the kernel uaccess routines */
--- a/arch/arm64/mm/proc.S~arm64-kasan-allow-enabling-in-kernel-mte
+++ a/arch/arm64/mm/proc.S
@@ -40,9 +40,15 @@
#define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
#ifdef CONFIG_KASAN_SW_TAGS
-#define TCR_KASAN_FLAGS TCR_TBI1 | TCR_TBID1
+#define TCR_KASAN_SW_FLAGS TCR_TBI1 | TCR_TBID1
#else
-#define TCR_KASAN_FLAGS 0
+#define TCR_KASAN_SW_FLAGS 0
+#endif
+
+#ifdef CONFIG_KASAN_HW_TAGS
+#define TCR_KASAN_HW_FLAGS SYS_TCR_EL1_TCMA1 | TCR_TBI1
+#else
+#define TCR_KASAN_HW_FLAGS 0
#endif
/*
@@ -427,6 +433,10 @@ SYM_FUNC_START(__cpu_setup)
*/
mov_q x5, MAIR_EL1_SET
#ifdef CONFIG_ARM64_MTE
+ mte_tcr .req x20
+
+ mov mte_tcr, #0
+
/*
* Update MAIR_EL1, GCR_EL1 and TFSR*_EL1 if MTE is supported
* (ID_AA64PFR1_EL1[11:8] > 1).
@@ -447,6 +457,9 @@ SYM_FUNC_START(__cpu_setup)
/* clear any pending tag check faults in TFSR*_EL1 */
msr_s SYS_TFSR_EL1, xzr
msr_s SYS_TFSRE0_EL1, xzr
+
+ /* set the TCR_EL1 bits */
+ mov_q mte_tcr, TCR_KASAN_HW_FLAGS
1:
#endif
msr mair_el1, x5
@@ -456,7 +469,11 @@ SYM_FUNC_START(__cpu_setup)
*/
mov_q x10, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \
TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \
- TCR_TBI0 | TCR_A1 | TCR_KASAN_FLAGS
+ TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS
+#ifdef CONFIG_ARM64_MTE
+ orr x10, x10, mte_tcr
+ .unreq mte_tcr
+#endif
tcr_clear_errata_bits x10, x9, x5
#ifdef CONFIG_ARM64_VA_BITS_52
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 28/60] arm64: mte: convert gcr_user into an exclude mask
2020-12-22 19:58 incoming Andrew Morton
` (26 preceding siblings ...)
2020-12-22 20:01 ` [patch 27/60] arm64: kasan: allow enabling in-kernel MTE Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 29/60] arm64: mte: switch GCR_EL1 in kernel entry and exit Andrew Morton
` (32 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: arm64: mte: convert gcr_user into an exclude mask
The gcr_user mask is a per thread mask that represents the tags that are
excluded from random generation when the Memory Tagging Extension is
present and an 'irg' instruction is invoked.
gcr_user affects the behavior on EL0 only.
Currently that mask is an include mask and it is controlled by the user
via prctl() while GCR_EL1 accepts an exclude mask.
Convert the include mask into an exclude one to make it easier the
register setting.
Note: This change will affect gcr_kernel (for EL1) introduced with a
future patch.
Link: https://lkml.kernel.org/r/946dd31be833b660334c4f93410acf6d6c4cf3c4.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/processor.h | 2 -
arch/arm64/kernel/mte.c | 29 +++++++++++++--------------
2 files changed, 16 insertions(+), 15 deletions(-)
--- a/arch/arm64/include/asm/processor.h~arm64-mte-convert-gcr_user-into-an-exclude-mask
+++ a/arch/arm64/include/asm/processor.h
@@ -152,7 +152,7 @@ struct thread_struct {
#endif
#ifdef CONFIG_ARM64_MTE
u64 sctlr_tcf0;
- u64 gcr_user_incl;
+ u64 gcr_user_excl;
#endif
};
--- a/arch/arm64/kernel/mte.c~arm64-mte-convert-gcr_user-into-an-exclude-mask
+++ a/arch/arm64/kernel/mte.c
@@ -156,23 +156,22 @@ static void set_sctlr_el1_tcf0(u64 tcf0)
preempt_enable();
}
-static void update_gcr_el1_excl(u64 incl)
+static void update_gcr_el1_excl(u64 excl)
{
- u64 excl = ~incl & SYS_GCR_EL1_EXCL_MASK;
/*
- * Note that 'incl' is an include mask (controlled by the user via
- * prctl()) while GCR_EL1 accepts an exclude mask.
+ * Note that the mask controlled by the user via prctl() is an
+ * include while GCR_EL1 accepts an exclude mask.
* No need for ISB since this only affects EL0 currently, implicit
* with ERET.
*/
sysreg_clear_set_s(SYS_GCR_EL1, SYS_GCR_EL1_EXCL_MASK, excl);
}
-static void set_gcr_el1_excl(u64 incl)
+static void set_gcr_el1_excl(u64 excl)
{
- current->thread.gcr_user_incl = incl;
- update_gcr_el1_excl(incl);
+ current->thread.gcr_user_excl = excl;
+ update_gcr_el1_excl(excl);
}
void flush_mte_state(void)
@@ -187,7 +186,7 @@ void flush_mte_state(void)
/* disable tag checking */
set_sctlr_el1_tcf0(SCTLR_EL1_TCF0_NONE);
/* reset tag generation mask */
- set_gcr_el1_excl(0);
+ set_gcr_el1_excl(SYS_GCR_EL1_EXCL_MASK);
}
void mte_thread_switch(struct task_struct *next)
@@ -198,7 +197,7 @@ void mte_thread_switch(struct task_struc
/* avoid expensive SCTLR_EL1 accesses if no change */
if (current->thread.sctlr_tcf0 != next->thread.sctlr_tcf0)
update_sctlr_el1_tcf0(next->thread.sctlr_tcf0);
- update_gcr_el1_excl(next->thread.gcr_user_incl);
+ update_gcr_el1_excl(next->thread.gcr_user_excl);
}
void mte_suspend_exit(void)
@@ -206,13 +205,14 @@ void mte_suspend_exit(void)
if (!system_supports_mte())
return;
- update_gcr_el1_excl(current->thread.gcr_user_incl);
+ update_gcr_el1_excl(current->thread.gcr_user_excl);
}
long set_mte_ctrl(struct task_struct *task, unsigned long arg)
{
u64 tcf0;
- u64 gcr_incl = (arg & PR_MTE_TAG_MASK) >> PR_MTE_TAG_SHIFT;
+ u64 gcr_excl = ~((arg & PR_MTE_TAG_MASK) >> PR_MTE_TAG_SHIFT) &
+ SYS_GCR_EL1_EXCL_MASK;
if (!system_supports_mte())
return 0;
@@ -233,10 +233,10 @@ long set_mte_ctrl(struct task_struct *ta
if (task != current) {
task->thread.sctlr_tcf0 = tcf0;
- task->thread.gcr_user_incl = gcr_incl;
+ task->thread.gcr_user_excl = gcr_excl;
} else {
set_sctlr_el1_tcf0(tcf0);
- set_gcr_el1_excl(gcr_incl);
+ set_gcr_el1_excl(gcr_excl);
}
return 0;
@@ -245,11 +245,12 @@ long set_mte_ctrl(struct task_struct *ta
long get_mte_ctrl(struct task_struct *task)
{
unsigned long ret;
+ u64 incl = ~task->thread.gcr_user_excl & SYS_GCR_EL1_EXCL_MASK;
if (!system_supports_mte())
return 0;
- ret = task->thread.gcr_user_incl << PR_MTE_TAG_SHIFT;
+ ret = incl << PR_MTE_TAG_SHIFT;
switch (task->thread.sctlr_tcf0) {
case SCTLR_EL1_TCF0_NONE:
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 29/60] arm64: mte: switch GCR_EL1 in kernel entry and exit
2020-12-22 19:58 incoming Andrew Morton
` (27 preceding siblings ...)
2020-12-22 20:01 ` [patch 28/60] arm64: mte: convert gcr_user into an exclude mask Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 30/60] kasan, mm: untag page address in free_reserved_area Andrew Morton
` (31 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: arm64: mte: switch GCR_EL1 in kernel entry and exit
When MTE is present, the GCR_EL1 register contains the tags mask that
allows to exclude tags from the random generation via the IRG instruction.
With the introduction of the new Tag-Based KASAN API that provides a
mechanism to reserve tags for special reasons, the MTE implementation has
to make sure that the GCR_EL1 setting for the kernel does not affect the
userspace processes and viceversa.
Save and restore the kernel/user mask in GCR_EL1 in kernel entry and exit.
Link: https://lkml.kernel.org/r/578b03294708cc7258fad0dc9c2a2e809e5a8214.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Co-developed-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/mte-def.h | 1
arch/arm64/include/asm/mte-kasan.h | 5 +++
arch/arm64/include/asm/mte.h | 2 +
arch/arm64/kernel/asm-offsets.c | 3 +
arch/arm64/kernel/entry.S | 41 +++++++++++++++++++++++++++
arch/arm64/kernel/mte.c | 31 ++++++++++++++++++--
6 files changed, 79 insertions(+), 4 deletions(-)
--- a/arch/arm64/include/asm/mte-def.h~arm64-mte-switch-gcr_el1-in-kernel-entry-and-exit
+++ a/arch/arm64/include/asm/mte-def.h
@@ -10,6 +10,5 @@
#define MTE_TAG_SHIFT 56
#define MTE_TAG_SIZE 4
#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
-#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
#endif /* __ASM_MTE_DEF_H */
--- a/arch/arm64/include/asm/mte.h~arm64-mte-switch-gcr_el1-in-kernel-entry-and-exit
+++ a/arch/arm64/include/asm/mte.h
@@ -18,6 +18,8 @@
#include <asm/pgtable-types.h>
+extern u64 gcr_kernel_excl;
+
void mte_clear_page_tags(void *addr);
unsigned long mte_copy_tags_from_user(void *to, const void __user *from,
unsigned long n);
--- a/arch/arm64/include/asm/mte-kasan.h~arm64-mte-switch-gcr_el1-in-kernel-entry-and-exit
+++ a/arch/arm64/include/asm/mte-kasan.h
@@ -30,6 +30,7 @@ u8 mte_get_random_tag(void);
void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag);
void mte_enable_kernel(void);
+void mte_init_tags(u64 max_tag);
#else /* CONFIG_ARM64_MTE */
@@ -55,6 +56,10 @@ static inline void mte_enable_kernel(voi
{
}
+static inline void mte_init_tags(u64 max_tag)
+{
+}
+
#endif /* CONFIG_ARM64_MTE */
#endif /* __ASSEMBLY__ */
--- a/arch/arm64/kernel/asm-offsets.c~arm64-mte-switch-gcr_el1-in-kernel-entry-and-exit
+++ a/arch/arm64/kernel/asm-offsets.c
@@ -47,6 +47,9 @@ int main(void)
DEFINE(THREAD_KEYS_USER, offsetof(struct task_struct, thread.keys_user));
DEFINE(THREAD_KEYS_KERNEL, offsetof(struct task_struct, thread.keys_kernel));
#endif
+#ifdef CONFIG_ARM64_MTE
+ DEFINE(THREAD_GCR_EL1_USER, offsetof(struct task_struct, thread.gcr_user_excl));
+#endif
BLANK();
DEFINE(S_X0, offsetof(struct pt_regs, regs[0]));
DEFINE(S_X2, offsetof(struct pt_regs, regs[2]));
--- a/arch/arm64/kernel/entry.S~arm64-mte-switch-gcr_el1-in-kernel-entry-and-exit
+++ a/arch/arm64/kernel/entry.S
@@ -173,6 +173,43 @@ alternative_else_nop_endif
#endif
.endm
+ .macro mte_set_gcr, tmp, tmp2
+#ifdef CONFIG_ARM64_MTE
+ /*
+ * Calculate and set the exclude mask preserving
+ * the RRND (bit[16]) setting.
+ */
+ mrs_s \tmp2, SYS_GCR_EL1
+ bfi \tmp2, \tmp, #0, #16
+ msr_s SYS_GCR_EL1, \tmp2
+ isb
+#endif
+ .endm
+
+ .macro mte_set_kernel_gcr, tmp, tmp2
+#ifdef CONFIG_KASAN_HW_TAGS
+alternative_if_not ARM64_MTE
+ b 1f
+alternative_else_nop_endif
+ ldr_l \tmp, gcr_kernel_excl
+
+ mte_set_gcr \tmp, \tmp2
+1:
+#endif
+ .endm
+
+ .macro mte_set_user_gcr, tsk, tmp, tmp2
+#ifdef CONFIG_ARM64_MTE
+alternative_if_not ARM64_MTE
+ b 1f
+alternative_else_nop_endif
+ ldr \tmp, [\tsk, #THREAD_GCR_EL1_USER]
+
+ mte_set_gcr \tmp, \tmp2
+1:
+#endif
+ .endm
+
.macro kernel_entry, el, regsize = 64
.if \regsize == 32
mov w0, w0 // zero upper 32 bits of x0
@@ -212,6 +249,8 @@ alternative_else_nop_endif
ptrauth_keys_install_kernel tsk, x20, x22, x23
+ mte_set_kernel_gcr x22, x23
+
scs_load tsk, x20
.else
add x21, sp, #S_FRAME_SIZE
@@ -315,6 +354,8 @@ alternative_else_nop_endif
/* No kernel C function calls after this as user keys are set. */
ptrauth_keys_install_user tsk, x0, x1, x2
+ mte_set_user_gcr tsk, x0, x1
+
apply_ssbd 0, x0, x1
.endif
--- a/arch/arm64/kernel/mte.c~arm64-mte-switch-gcr_el1-in-kernel-entry-and-exit
+++ a/arch/arm64/kernel/mte.c
@@ -23,6 +23,8 @@
#include <asm/ptrace.h>
#include <asm/sysreg.h>
+u64 gcr_kernel_excl __ro_after_init;
+
static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap)
{
pte_t old_pte = READ_ONCE(*ptep);
@@ -129,6 +131,26 @@ void *mte_set_mem_tag_range(void *addr,
return ptr;
}
+void mte_init_tags(u64 max_tag)
+{
+ static bool gcr_kernel_excl_initialized;
+
+ if (!gcr_kernel_excl_initialized) {
+ /*
+ * The format of the tags in KASAN is 0xFF and in MTE is 0xF.
+ * This conversion extracts an MTE tag from a KASAN tag.
+ */
+ u64 incl = GENMASK(FIELD_GET(MTE_TAG_MASK >> MTE_TAG_SHIFT,
+ max_tag), 0);
+
+ gcr_kernel_excl = ~incl & SYS_GCR_EL1_EXCL_MASK;
+ gcr_kernel_excl_initialized = true;
+ }
+
+ /* Enable the kernel exclude mask for random tags generation. */
+ write_sysreg_s(SYS_GCR_EL1_RRND | gcr_kernel_excl, SYS_GCR_EL1);
+}
+
void mte_enable_kernel(void)
{
/* Enable MTE Sync Mode for EL1. */
@@ -171,7 +193,11 @@ static void update_gcr_el1_excl(u64 excl
static void set_gcr_el1_excl(u64 excl)
{
current->thread.gcr_user_excl = excl;
- update_gcr_el1_excl(excl);
+
+ /*
+ * SYS_GCR_EL1 will be set to current->thread.gcr_user_excl value
+ * by mte_set_user_gcr() in kernel_exit,
+ */
}
void flush_mte_state(void)
@@ -197,7 +223,6 @@ void mte_thread_switch(struct task_struc
/* avoid expensive SCTLR_EL1 accesses if no change */
if (current->thread.sctlr_tcf0 != next->thread.sctlr_tcf0)
update_sctlr_el1_tcf0(next->thread.sctlr_tcf0);
- update_gcr_el1_excl(next->thread.gcr_user_excl);
}
void mte_suspend_exit(void)
@@ -205,7 +230,7 @@ void mte_suspend_exit(void)
if (!system_supports_mte())
return;
- update_gcr_el1_excl(current->thread.gcr_user_excl);
+ update_gcr_el1_excl(gcr_kernel_excl);
}
long set_mte_ctrl(struct task_struct *task, unsigned long arg)
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 30/60] kasan, mm: untag page address in free_reserved_area
2020-12-22 19:58 incoming Andrew Morton
` (28 preceding siblings ...)
2020-12-22 20:01 ` [patch 29/60] arm64: mte: switch GCR_EL1 in kernel entry and exit Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 31/60] arm64: kasan: align allocations for HW_TAGS Andrew Morton
` (30 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: kasan, mm: untag page address in free_reserved_area
free_reserved_area() memsets the pages belonging to a given memory area.
As that memory hasn't been allocated via page_alloc, the KASAN tags that
those pages have are 0x00. As the result the memset might result in a tag
mismatch.
Untag the address to avoid spurious faults.
Link: https://lkml.kernel.org/r/ebef6425f4468d063e2f09c1b62ccbb2236b71d3.1606161801.git.andreyknvl@google.com
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/page_alloc.c | 5 +++++
1 file changed, 5 insertions(+)
--- a/mm/page_alloc.c~kasan-mm-untag-page-address-in-free_reserved_area
+++ a/mm/page_alloc.c
@@ -7671,6 +7671,11 @@ unsigned long free_reserved_area(void *s
* alias for the memset().
*/
direct_map_addr = page_address(page);
+ /*
+ * Perform a kasan-unchecked memset() since this memory
+ * has not been initialized.
+ */
+ direct_map_addr = kasan_reset_tag(direct_map_addr);
if ((unsigned int)poison <= 0xFF)
memset(direct_map_addr, poison, PAGE_SIZE);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 31/60] arm64: kasan: align allocations for HW_TAGS
2020-12-22 19:58 incoming Andrew Morton
` (29 preceding siblings ...)
2020-12-22 20:01 ` [patch 30/60] kasan, mm: untag page address in free_reserved_area Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 32/60] arm64: kasan: add arch layer for memory tagging helpers Andrew Morton
` (29 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: arm64: kasan: align allocations for HW_TAGS
Hardware tag-based KASAN uses the memory tagging approach, which requires
all allocations to be aligned to the memory granule size. Align the
allocations to MTE_GRANULE_SIZE via ARCH_SLAB_MINALIGN when
CONFIG_KASAN_HW_TAGS is enabled.
Link: https://lkml.kernel.org/r/fe64131606b1c2aabfd34ae99554c0d9df18eb19.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/cache.h | 3 +++
1 file changed, 3 insertions(+)
--- a/arch/arm64/include/asm/cache.h~arm64-kasan-align-allocations-for-hw_tags
+++ a/arch/arm64/include/asm/cache.h
@@ -6,6 +6,7 @@
#define __ASM_CACHE_H
#include <asm/cputype.h>
+#include <asm/mte-kasan.h>
#define CTR_L1IP_SHIFT 14
#define CTR_L1IP_MASK 3
@@ -51,6 +52,8 @@
#ifdef CONFIG_KASAN_SW_TAGS
#define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT)
+#elif defined(CONFIG_KASAN_HW_TAGS)
+#define ARCH_SLAB_MINALIGN MTE_GRANULE_SIZE
#endif
#ifndef __ASSEMBLY__
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 32/60] arm64: kasan: add arch layer for memory tagging helpers
2020-12-22 19:58 incoming Andrew Morton
` (30 preceding siblings ...)
2020-12-22 20:01 ` [patch 31/60] arm64: kasan: align allocations for HW_TAGS Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:01 ` [patch 33/60] kasan: define KASAN_GRANULE_SIZE for HW_TAGS Andrew Morton
` (28 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: arm64: kasan: add arch layer for memory tagging helpers
This patch add a set of arch_*() memory tagging helpers currently only
defined for arm64 when hardware tag-based KASAN is enabled. These helpers
will be used by KASAN runtime to implement the hardware tag-based mode.
The arch-level indirection level is introduced to simplify adding hardware
tag-based KASAN support for other architectures in the future by defining
the appropriate arch_*() macros.
Link: https://lkml.kernel.org/r/fc9e5bb71201c03131a2fc00a74125723568dda9.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Co-developed-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/memory.h | 9 +++++++++
mm/kasan/kasan.h | 26 ++++++++++++++++++++++++++
2 files changed, 35 insertions(+)
--- a/arch/arm64/include/asm/memory.h~arm64-kasan-add-arch-layer-for-memory-tagging-helpers
+++ a/arch/arm64/include/asm/memory.h
@@ -230,6 +230,15 @@ static inline const void *__tag_set(cons
return (const void *)(__addr | __tag_shifted(tag));
}
+#ifdef CONFIG_KASAN_HW_TAGS
+#define arch_enable_tagging() mte_enable_kernel()
+#define arch_init_tags(max_tag) mte_init_tags(max_tag)
+#define arch_get_random_tag() mte_get_random_tag()
+#define arch_get_mem_tag(addr) mte_get_mem_tag(addr)
+#define arch_set_mem_tag_range(addr, size, tag) \
+ mte_set_mem_tag_range((addr), (size), (tag))
+#endif /* CONFIG_KASAN_HW_TAGS */
+
/*
* Physical vs virtual RAM address space conversion. These are
* private definitions which should NOT be used outside memory.h
--- a/mm/kasan/kasan.h~arm64-kasan-add-arch-layer-for-memory-tagging-helpers
+++ a/mm/kasan/kasan.h
@@ -243,6 +243,32 @@ static inline const void *arch_kasan_set
#define reset_tag(addr) ((void *)arch_kasan_reset_tag(addr))
#define get_tag(addr) arch_kasan_get_tag(addr)
+#ifdef CONFIG_KASAN_HW_TAGS
+
+#ifndef arch_enable_tagging
+#define arch_enable_tagging()
+#endif
+#ifndef arch_init_tags
+#define arch_init_tags(max_tag)
+#endif
+#ifndef arch_get_random_tag
+#define arch_get_random_tag() (0xFF)
+#endif
+#ifndef arch_get_mem_tag
+#define arch_get_mem_tag(addr) (0xFF)
+#endif
+#ifndef arch_set_mem_tag_range
+#define arch_set_mem_tag_range(addr, size, tag) ((void *)(addr))
+#endif
+
+#define hw_enable_tagging() arch_enable_tagging()
+#define hw_init_tags(max_tag) arch_init_tags(max_tag)
+#define hw_get_random_tag() arch_get_random_tag()
+#define hw_get_mem_tag(addr) arch_get_mem_tag(addr)
+#define hw_set_mem_tag_range(addr, size, tag) arch_set_mem_tag_range((addr), (size), (tag))
+
+#endif /* CONFIG_KASAN_HW_TAGS */
+
/*
* Exported functions for interfaces called from assembly or from generated
* code. Declarations here to avoid warning about missing declarations.
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 33/60] kasan: define KASAN_GRANULE_SIZE for HW_TAGS
2020-12-22 19:58 incoming Andrew Morton
` (31 preceding siblings ...)
2020-12-22 20:01 ` [patch 32/60] arm64: kasan: add arch layer for memory tagging helpers Andrew Morton
@ 2020-12-22 20:01 ` Andrew Morton
2020-12-22 20:02 ` [patch 34/60] kasan, x86, s390: update undef CONFIG_KASAN Andrew Morton
` (27 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:01 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: define KASAN_GRANULE_SIZE for HW_TAGS
Hardware tag-based KASAN has granules of MTE_GRANULE_SIZE. Define
KASAN_GRANULE_SIZE to MTE_GRANULE_SIZE for CONFIG_KASAN_HW_TAGS.
Link: https://lkml.kernel.org/r/3d15794b3d1b27447fd7fdf862c073192ba657bd.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/kasan.h | 6 ++++++
1 file changed, 6 insertions(+)
--- a/mm/kasan/kasan.h~kasan-define-kasan_granule_size-for-hw_tags
+++ a/mm/kasan/kasan.h
@@ -5,7 +5,13 @@
#include <linux/kasan.h>
#include <linux/stackdepot.h>
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
+#else
+#include <asm/mte-kasan.h>
+#define KASAN_GRANULE_SIZE MTE_GRANULE_SIZE
+#endif
+
#define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
#define KASAN_MEMORY_PER_SHADOW_PAGE (KASAN_GRANULE_SIZE << PAGE_SHIFT)
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 34/60] kasan, x86, s390: update undef CONFIG_KASAN
2020-12-22 19:58 incoming Andrew Morton
` (32 preceding siblings ...)
2020-12-22 20:01 ` [patch 33/60] kasan: define KASAN_GRANULE_SIZE for HW_TAGS Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 35/60] kasan, arm64: expand CONFIG_KASAN checks Andrew Morton
` (26 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, x86, s390: update undef CONFIG_KASAN
With the intoduction of hardware tag-based KASAN some kernel checks of
this kind:
ifdef CONFIG_KASAN
will be updated to:
if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
x86 and s390 use a trick to #undef CONFIG_KASAN for some of the code
that isn't linked with KASAN runtime and shouldn't have any KASAN
annotations.
Also #undef CONFIG_KASAN_GENERIC with CONFIG_KASAN.
Link: https://lkml.kernel.org/r/9d84bfaaf8fabe0fc89f913c9e420a30bd31a260.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Acked-by: Vasily Gorbik <gor@linux.ibm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/s390/boot/string.c | 1 +
arch/x86/boot/compressed/misc.h | 1 +
2 files changed, 2 insertions(+)
--- a/arch/s390/boot/string.c~kasan-x86-s390-update-undef-config_kasan
+++ a/arch/s390/boot/string.c
@@ -3,6 +3,7 @@
#include <linux/kernel.h>
#include <linux/errno.h>
#undef CONFIG_KASAN
+#undef CONFIG_KASAN_GENERIC
#include "../lib/string.c"
int strncmp(const char *cs, const char *ct, size_t count)
--- a/arch/x86/boot/compressed/misc.h~kasan-x86-s390-update-undef-config_kasan
+++ a/arch/x86/boot/compressed/misc.h
@@ -12,6 +12,7 @@
#undef CONFIG_PARAVIRT_XXL
#undef CONFIG_PARAVIRT_SPINLOCKS
#undef CONFIG_KASAN
+#undef CONFIG_KASAN_GENERIC
/* cpu_feature_enabled() cannot be used this early */
#define USE_EARLY_PGTABLE_L5
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 35/60] kasan, arm64: expand CONFIG_KASAN checks
2020-12-22 19:58 incoming Andrew Morton
` (33 preceding siblings ...)
2020-12-22 20:02 ` [patch 34/60] kasan, x86, s390: update undef CONFIG_KASAN Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 36/60] kasan, arm64: implement HW_TAGS runtime Andrew Morton
` (25 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: expand CONFIG_KASAN checks
Some #ifdef CONFIG_KASAN checks are only relevant for software KASAN modes
(either related to shadow memory or compiler instrumentation). Expand
those into CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS.
Link: https://lkml.kernel.org/r/e6971e432dbd72bb897ff14134ebb7e169bdcf0c.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/Kconfig | 2 +-
arch/arm64/Makefile | 2 +-
arch/arm64/include/asm/assembler.h | 2 +-
arch/arm64/include/asm/memory.h | 2 +-
arch/arm64/include/asm/string.h | 5 +++--
arch/arm64/kernel/head.S | 2 +-
arch/arm64/kernel/image-vars.h | 2 +-
arch/arm64/kernel/kaslr.c | 3 ++-
arch/arm64/kernel/module.c | 6 ++++--
arch/arm64/mm/ptdump.c | 6 +++---
include/linux/kasan-checks.h | 2 +-
include/linux/kasan.h | 7 ++++---
include/linux/moduleloader.h | 3 ++-
include/linux/string.h | 2 +-
mm/ptdump.c | 13 ++++++++-----
scripts/Makefile.lib | 2 ++
16 files changed, 36 insertions(+), 25 deletions(-)
--- a/arch/arm64/include/asm/assembler.h~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/include/asm/assembler.h
@@ -473,7 +473,7 @@ USER(\label, ic ivau, \tmp2) // invali
#define NOKPROBE(x)
#endif
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define EXPORT_SYMBOL_NOKASAN(name)
#else
#define EXPORT_SYMBOL_NOKASAN(name) EXPORT_SYMBOL(name)
--- a/arch/arm64/include/asm/memory.h~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/include/asm/memory.h
@@ -72,7 +72,7 @@
* address space for the shadow region respectively. They can bloat the stack
* significantly, so double the (minimum) stack size when they are in use.
*/
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
#define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) \
+ KASAN_SHADOW_OFFSET)
--- a/arch/arm64/include/asm/string.h~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/include/asm/string.h
@@ -5,7 +5,7 @@
#ifndef __ASM_STRING_H
#define __ASM_STRING_H
-#ifndef CONFIG_KASAN
+#if !(defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
#define __HAVE_ARCH_STRRCHR
extern char *strrchr(const char *, int c);
@@ -48,7 +48,8 @@ extern void *__memset(void *, int, __ker
void memcpy_flushcache(void *dst, const void *src, size_t cnt);
#endif
-#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__)
+#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
+ !defined(__SANITIZE_ADDRESS__)
/*
* For files that are not instrumented (e.g. mm/slub.c) we
--- a/arch/arm64/Kconfig~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/Kconfig
@@ -334,7 +334,7 @@ config BROKEN_GAS_INST
config KASAN_SHADOW_OFFSET
hex
- depends on KASAN
+ depends on KASAN_GENERIC || KASAN_SW_TAGS
default 0xdfff800000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && !KASAN_SW_TAGS
default 0xdfffc00000000000 if ARM64_VA_BITS_47 && !KASAN_SW_TAGS
default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS
--- a/arch/arm64/kernel/head.S~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/kernel/head.S
@@ -433,7 +433,7 @@ SYM_FUNC_START_LOCAL(__primary_switched)
bl __pi_memset
dsb ishst // Make zero page visible to PTW
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
bl kasan_early_init
#endif
#ifdef CONFIG_RANDOMIZE_BASE
--- a/arch/arm64/kernel/image-vars.h~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/kernel/image-vars.h
@@ -37,7 +37,7 @@ __efistub_strncmp = __pi_strncmp;
__efistub_strrchr = __pi_strrchr;
__efistub___clean_dcache_area_poc = __pi___clean_dcache_area_poc;
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
__efistub___memcpy = __pi_memcpy;
__efistub___memmove = __pi_memmove;
__efistub___memset = __pi_memset;
--- a/arch/arm64/kernel/kaslr.c~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/kernel/kaslr.c
@@ -161,7 +161,8 @@ u64 __init kaslr_early_init(u64 dt_phys)
/* use the top 16 bits to randomize the linear region */
memstart_offset_seed = seed >> 48;
- if (IS_ENABLED(CONFIG_KASAN))
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC) ||
+ IS_ENABLED(CONFIG_KASAN_SW_TAGS))
/*
* KASAN does not expect the module region to intersect the
* vmalloc region, since shadow memory is allocated for each
--- a/arch/arm64/kernel/module.c~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/kernel/module.c
@@ -30,7 +30,8 @@ void *module_alloc(unsigned long size)
if (IS_ENABLED(CONFIG_ARM64_MODULE_PLTS))
gfp_mask |= __GFP_NOWARN;
- if (IS_ENABLED(CONFIG_KASAN))
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC) ||
+ IS_ENABLED(CONFIG_KASAN_SW_TAGS))
/* don't exceed the static module region - see below */
module_alloc_end = MODULES_END;
@@ -39,7 +40,8 @@ void *module_alloc(unsigned long size)
NUMA_NO_NODE, __builtin_return_address(0));
if (!p && IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) &&
- !IS_ENABLED(CONFIG_KASAN))
+ !IS_ENABLED(CONFIG_KASAN_GENERIC) &&
+ !IS_ENABLED(CONFIG_KASAN_SW_TAGS))
/*
* KASAN can only deal with module allocations being served
* from the reserved module region, since the remainder of
--- a/arch/arm64/Makefile~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/Makefile
@@ -137,7 +137,7 @@ head-y := arch/arm64/kernel/head.o
ifeq ($(CONFIG_KASAN_SW_TAGS), y)
KASAN_SHADOW_SCALE_SHIFT := 4
-else
+else ifeq ($(CONFIG_KASAN_GENERIC), y)
KASAN_SHADOW_SCALE_SHIFT := 3
endif
--- a/arch/arm64/mm/ptdump.c~kasan-arm64-expand-config_kasan-checks
+++ a/arch/arm64/mm/ptdump.c
@@ -29,7 +29,7 @@
enum address_markers_idx {
PAGE_OFFSET_NR = 0,
PAGE_END_NR,
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
KASAN_START_NR,
#endif
};
@@ -37,7 +37,7 @@ enum address_markers_idx {
static struct addr_marker address_markers[] = {
{ PAGE_OFFSET, "Linear Mapping start" },
{ 0 /* PAGE_END */, "Linear Mapping end" },
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
{ 0 /* KASAN_SHADOW_START */, "Kasan shadow start" },
{ KASAN_SHADOW_END, "Kasan shadow end" },
#endif
@@ -383,7 +383,7 @@ void ptdump_check_wx(void)
static int ptdump_init(void)
{
address_markers[PAGE_END_NR].start_address = PAGE_END;
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
address_markers[KASAN_START_NR].start_address = KASAN_SHADOW_START;
#endif
ptdump_initialize();
--- a/include/linux/kasan-checks.h~kasan-arm64-expand-config_kasan-checks
+++ a/include/linux/kasan-checks.h
@@ -9,7 +9,7 @@
* even in compilation units that selectively disable KASAN, but must use KASAN
* to validate access to an address. Never use these in header files!
*/
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
bool __kasan_check_read(const volatile void *p, unsigned int size);
bool __kasan_check_write(const volatile void *p, unsigned int size);
#else
--- a/include/linux/kasan.h~kasan-arm64-expand-config_kasan-checks
+++ a/include/linux/kasan.h
@@ -238,7 +238,8 @@ static inline void kasan_release_vmalloc
#endif /* CONFIG_KASAN_VMALLOC */
-#if defined(CONFIG_KASAN) && !defined(CONFIG_KASAN_VMALLOC)
+#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
+ !defined(CONFIG_KASAN_VMALLOC)
/*
* These functions provide a special case to support backing module
@@ -248,12 +249,12 @@ static inline void kasan_release_vmalloc
int kasan_module_alloc(void *addr, size_t size);
void kasan_free_shadow(const struct vm_struct *vm);
-#else /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
+#else /* (CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) && !CONFIG_KASAN_VMALLOC */
static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
static inline void kasan_free_shadow(const struct vm_struct *vm) {}
-#endif /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
+#endif /* (CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) && !CONFIG_KASAN_VMALLOC */
#ifdef CONFIG_KASAN_INLINE
void kasan_non_canonical_hook(unsigned long addr);
--- a/include/linux/moduleloader.h~kasan-arm64-expand-config_kasan-checks
+++ a/include/linux/moduleloader.h
@@ -96,7 +96,8 @@ void module_arch_cleanup(struct module *
/* Any cleanup before freeing mod->module_init */
void module_arch_freeing_init(struct module *mod);
-#if defined(CONFIG_KASAN) && !defined(CONFIG_KASAN_VMALLOC)
+#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
+ !defined(CONFIG_KASAN_VMALLOC)
#include <linux/kasan.h>
#define MODULE_ALIGN (PAGE_SIZE << KASAN_SHADOW_SCALE_SHIFT)
#else
--- a/include/linux/string.h~kasan-arm64-expand-config_kasan-checks
+++ a/include/linux/string.h
@@ -267,7 +267,7 @@ void __write_overflow(void) __compiletim
#if !defined(__NO_FORTIFY) && defined(__OPTIMIZE__) && defined(CONFIG_FORTIFY_SOURCE)
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
extern void *__underlying_memchr(const void *p, int c, __kernel_size_t size) __RENAME(memchr);
extern int __underlying_memcmp(const void *p, const void *q, __kernel_size_t size) __RENAME(memcmp);
extern void *__underlying_memcpy(void *p, const void *q, __kernel_size_t size) __RENAME(memcpy);
--- a/mm/ptdump.c~kasan-arm64-expand-config_kasan-checks
+++ a/mm/ptdump.c
@@ -4,7 +4,7 @@
#include <linux/ptdump.h>
#include <linux/kasan.h>
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
/*
* This is an optimization for KASAN=y case. Since all kasan page tables
* eventually point to the kasan_early_shadow_page we could call note_page()
@@ -31,7 +31,8 @@ static int ptdump_pgd_entry(pgd_t *pgd,
struct ptdump_state *st = walk->private;
pgd_t val = READ_ONCE(*pgd);
-#if CONFIG_PGTABLE_LEVELS > 4 && defined(CONFIG_KASAN)
+#if CONFIG_PGTABLE_LEVELS > 4 && \
+ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
if (pgd_page(val) == virt_to_page(lm_alias(kasan_early_shadow_p4d)))
return note_kasan_page_table(walk, addr);
#endif
@@ -51,7 +52,8 @@ static int ptdump_p4d_entry(p4d_t *p4d,
struct ptdump_state *st = walk->private;
p4d_t val = READ_ONCE(*p4d);
-#if CONFIG_PGTABLE_LEVELS > 3 && defined(CONFIG_KASAN)
+#if CONFIG_PGTABLE_LEVELS > 3 && \
+ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
if (p4d_page(val) == virt_to_page(lm_alias(kasan_early_shadow_pud)))
return note_kasan_page_table(walk, addr);
#endif
@@ -71,7 +73,8 @@ static int ptdump_pud_entry(pud_t *pud,
struct ptdump_state *st = walk->private;
pud_t val = READ_ONCE(*pud);
-#if CONFIG_PGTABLE_LEVELS > 2 && defined(CONFIG_KASAN)
+#if CONFIG_PGTABLE_LEVELS > 2 && \
+ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
if (pud_page(val) == virt_to_page(lm_alias(kasan_early_shadow_pmd)))
return note_kasan_page_table(walk, addr);
#endif
@@ -91,7 +94,7 @@ static int ptdump_pmd_entry(pmd_t *pmd,
struct ptdump_state *st = walk->private;
pmd_t val = READ_ONCE(*pmd);
-#if defined(CONFIG_KASAN)
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
if (pmd_page(val) == virt_to_page(lm_alias(kasan_early_shadow_pte)))
return note_kasan_page_table(walk, addr);
#endif
--- a/scripts/Makefile.lib~kasan-arm64-expand-config_kasan-checks
+++ a/scripts/Makefile.lib
@@ -148,10 +148,12 @@ endif
# we don't want to check (depends on variables KASAN_SANITIZE_obj.o, KASAN_SANITIZE)
#
ifeq ($(CONFIG_KASAN),y)
+ifneq ($(CONFIG_KASAN_HW_TAGS),y)
_c_flags += $(if $(patsubst n%,, \
$(KASAN_SANITIZE_$(basetarget).o)$(KASAN_SANITIZE)y), \
$(CFLAGS_KASAN), $(CFLAGS_KASAN_NOSANITIZE))
endif
+endif
ifeq ($(CONFIG_UBSAN),y)
_c_flags += $(if $(patsubst n%,, \
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 36/60] kasan, arm64: implement HW_TAGS runtime
2020-12-22 19:58 incoming Andrew Morton
` (34 preceding siblings ...)
2020-12-22 20:02 ` [patch 35/60] kasan, arm64: expand CONFIG_KASAN checks Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 37/60] kasan, arm64: print report from tag fault handler Andrew Morton
` (24 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: implement HW_TAGS runtime
Provide implementation of KASAN functions required for the hardware
tag-based mode. Those include core functions for memory and pointer
tagging (tags_hw.c) and bug reporting (report_tags_hw.c). Also adapt
common KASAN code to support the new mode.
Link: https://lkml.kernel.org/r/cfd0fbede579a6b66755c98c88c108e54f9c56bf.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/include/asm/memory.h | 4 -
arch/arm64/kernel/cpufeature.c | 3 +
arch/arm64/kernel/smp.c | 2
include/linux/kasan.h | 24 +++++---
include/linux/mm.h | 2
include/linux/page-flags-layout.h | 2
mm/kasan/Makefile | 5 +
mm/kasan/common.c | 15 ++---
mm/kasan/hw_tags.c | 80 ++++++++++++++++++++++++++++
mm/kasan/kasan.h | 19 +++++-
mm/kasan/report_hw_tags.c | 42 ++++++++++++++
mm/kasan/report_sw_tags.c | 2
mm/kasan/shadow.c | 2
mm/kasan/sw_tags.c | 2
14 files changed, 178 insertions(+), 26 deletions(-)
--- a/arch/arm64/include/asm/memory.h~kasan-arm64-implement-hw_tags-runtime
+++ a/arch/arm64/include/asm/memory.h
@@ -214,7 +214,7 @@ static inline unsigned long kaslr_offset
(__force __typeof__(addr))__addr; \
})
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
#define __tag_shifted(tag) ((u64)(tag) << 56)
#define __tag_reset(addr) __untagged_addr(addr)
#define __tag_get(addr) (__u8)((u64)(addr) >> 56)
@@ -222,7 +222,7 @@ static inline unsigned long kaslr_offset
#define __tag_shifted(tag) 0UL
#define __tag_reset(addr) (addr)
#define __tag_get(addr) 0
-#endif /* CONFIG_KASAN_SW_TAGS */
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
static inline const void *__tag_set(const void *addr, u8 tag)
{
--- a/arch/arm64/kernel/cpufeature.c~kasan-arm64-implement-hw_tags-runtime
+++ a/arch/arm64/kernel/cpufeature.c
@@ -70,6 +70,7 @@
#include <linux/types.h>
#include <linux/mm.h>
#include <linux/cpu.h>
+#include <linux/kasan.h>
#include <asm/cpu.h>
#include <asm/cpufeature.h>
#include <asm/cpu_ops.h>
@@ -1710,6 +1711,8 @@ static void cpu_enable_mte(struct arm64_
cleared_zero_page = true;
mte_clear_page_tags(lm_alias(empty_zero_page));
}
+
+ kasan_init_hw_tags_cpu();
}
#endif /* CONFIG_ARM64_MTE */
--- a/arch/arm64/kernel/smp.c~kasan-arm64-implement-hw_tags-runtime
+++ a/arch/arm64/kernel/smp.c
@@ -462,6 +462,8 @@ void __init smp_prepare_boot_cpu(void)
/* Conditionally switch to GIC PMR for interrupt masking */
if (system_uses_irq_prio_masking())
init_gic_priority_masking();
+
+ kasan_init_hw_tags();
}
static u64 __init of_get_cpu_mpidr(struct device_node *dn)
--- a/include/linux/kasan.h~kasan-arm64-implement-hw_tags-runtime
+++ a/include/linux/kasan.h
@@ -190,25 +190,35 @@ static inline void kasan_record_aux_stac
#endif /* CONFIG_KASAN_GENERIC */
-#ifdef CONFIG_KASAN_SW_TAGS
-
-void __init kasan_init_sw_tags(void);
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
void *kasan_reset_tag(const void *addr);
bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
-#else /* CONFIG_KASAN_SW_TAGS */
-
-static inline void kasan_init_sw_tags(void) { }
+#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
static inline void *kasan_reset_tag(const void *addr)
{
return (void *)addr;
}
-#endif /* CONFIG_KASAN_SW_TAGS */
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS*/
+
+#ifdef CONFIG_KASAN_SW_TAGS
+void __init kasan_init_sw_tags(void);
+#else
+static inline void kasan_init_sw_tags(void) { }
+#endif
+
+#ifdef CONFIG_KASAN_HW_TAGS
+void kasan_init_hw_tags_cpu(void);
+void __init kasan_init_hw_tags(void);
+#else
+static inline void kasan_init_hw_tags_cpu(void) { }
+static inline void kasan_init_hw_tags(void) { }
+#endif
#ifdef CONFIG_KASAN_VMALLOC
--- a/include/linux/mm.h~kasan-arm64-implement-hw_tags-runtime
+++ a/include/linux/mm.h
@@ -1421,7 +1421,7 @@ static inline bool cpupid_match_pid(stru
}
#endif /* CONFIG_NUMA_BALANCING */
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
static inline u8 page_kasan_tag(const struct page *page)
{
return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
--- a/include/linux/page-flags-layout.h~kasan-arm64-implement-hw_tags-runtime
+++ a/include/linux/page-flags-layout.h
@@ -77,7 +77,7 @@
#define LAST_CPUPID_SHIFT 0
#endif
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
#define KASAN_TAG_WIDTH 8
#else
#define KASAN_TAG_WIDTH 0
--- a/mm/kasan/common.c~kasan-arm64-implement-hw_tags-runtime
+++ a/mm/kasan/common.c
@@ -118,7 +118,7 @@ void kasan_free_pages(struct page *page,
*/
static inline unsigned int optimal_redzone(unsigned int object_size)
{
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ if (!IS_ENABLED(CONFIG_KASAN_GENERIC))
return 0;
return
@@ -183,14 +183,14 @@ size_t kasan_metadata_size(struct kmem_c
struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
const void *object)
{
- return (void *)object + cache->kasan_info.alloc_meta_offset;
+ return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}
struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
- return (void *)object + cache->kasan_info.free_meta_offset;
+ return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
}
void kasan_poison_slab(struct page *page)
@@ -272,9 +272,8 @@ void * __must_check kasan_init_slab_obj(
alloc_info = get_alloc_info(cache, object);
__memset(alloc_info, 0, sizeof(*alloc_info));
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
- object = set_tag(object,
- assign_tag(cache, object, true, false));
+ if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
+ object = set_tag(object, assign_tag(cache, object, true, false));
return (void *)object;
}
@@ -342,10 +341,10 @@ static void *__kasan_kmalloc(struct kmem
redzone_end = round_up((unsigned long)object + cache->object_size,
KASAN_GRANULE_SIZE);
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
tag = assign_tag(cache, object, false, keep_tag);
- /* Tag is ignored in set_tag without CONFIG_KASAN_SW_TAGS */
+ /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */
unpoison_range(set_tag(object, tag), size);
poison_range((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);
--- /dev/null
+++ a/mm/kasan/hw_tags.c
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file contains core hardware tag-based KASAN code.
+ *
+ * Copyright (c) 2020 Google, Inc.
+ * Author: Andrey Konovalov <andreyknvl@google.com>
+ */
+
+#define pr_fmt(fmt) "kasan: " fmt
+
+#include <linux/kasan.h>
+#include <linux/kernel.h>
+#include <linux/memory.h>
+#include <linux/mm.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+#include "kasan.h"
+
+/* kasan_init_hw_tags_cpu() is called for each CPU. */
+void kasan_init_hw_tags_cpu(void)
+{
+ hw_init_tags(KASAN_TAG_MAX);
+ hw_enable_tagging();
+}
+
+/* kasan_init_hw_tags() is called once on boot CPU. */
+void __init kasan_init_hw_tags(void)
+{
+ pr_info("KernelAddressSanitizer initialized\n");
+}
+
+void *kasan_reset_tag(const void *addr)
+{
+ return reset_tag(addr);
+}
+
+void poison_range(const void *address, size_t size, u8 value)
+{
+ hw_set_mem_tag_range(reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), value);
+}
+
+void unpoison_range(const void *address, size_t size)
+{
+ hw_set_mem_tag_range(reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
+}
+
+u8 random_tag(void)
+{
+ return hw_get_random_tag();
+}
+
+bool check_invalid_free(void *addr)
+{
+ u8 ptr_tag = get_tag(addr);
+ u8 mem_tag = hw_get_mem_tag(addr);
+
+ return (mem_tag == KASAN_TAG_INVALID) ||
+ (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);
+}
+
+void kasan_set_free_info(struct kmem_cache *cache,
+ void *object, u8 tag)
+{
+ struct kasan_alloc_meta *alloc_meta;
+
+ alloc_meta = get_alloc_info(cache, object);
+ kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
+}
+
+struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
+ void *object, u8 tag)
+{
+ struct kasan_alloc_meta *alloc_meta;
+
+ alloc_meta = get_alloc_info(cache, object);
+ return &alloc_meta->free_track[0];
+}
--- a/mm/kasan/kasan.h~kasan-arm64-implement-hw_tags-runtime
+++ a/mm/kasan/kasan.h
@@ -154,6 +154,11 @@ struct kasan_alloc_meta *get_alloc_info(
struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
const void *object);
+void poison_range(const void *address, size_t size, u8 value);
+void unpoison_range(const void *address, size_t size);
+
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+
static inline const void *kasan_shadow_to_mem(const void *shadow_addr)
{
return (void *)(((unsigned long)shadow_addr - KASAN_SHADOW_OFFSET)
@@ -165,9 +170,6 @@ static inline bool addr_has_metadata(con
return (addr >= kasan_shadow_to_mem((void *)KASAN_SHADOW_START));
}
-void poison_range(const void *address, size_t size, u8 value);
-void unpoison_range(const void *address, size_t size);
-
/**
* check_memory_region - Check memory region, and report if invalid access.
* @addr: the accessed address
@@ -179,6 +181,15 @@ void unpoison_range(const void *address,
bool check_memory_region(unsigned long addr, size_t size, bool write,
unsigned long ret_ip);
+#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+static inline bool addr_has_metadata(const void *addr)
+{
+ return true;
+}
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
bool check_invalid_free(void *addr);
void *find_first_bad_addr(void *addr, size_t size);
@@ -215,7 +226,7 @@ static inline void quarantine_reduce(voi
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
void print_tags(u8 addr_tag, const void *addr);
--- a/mm/kasan/Makefile~kasan-arm64-implement-hw_tags-runtime
+++ a/mm/kasan/Makefile
@@ -10,8 +10,10 @@ CFLAGS_REMOVE_init.o = $(CC_FLAGS_FTRACE
CFLAGS_REMOVE_quarantine.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report_generic.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_report_hw_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report_sw_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_shadow.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_hw_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_sw_tags.o = $(CC_FLAGS_FTRACE)
# Function splitter causes unnecessary splits in __asan_load1/__asan_store1
@@ -27,10 +29,13 @@ CFLAGS_init.o := $(CC_FLAGS_KASAN_RUNTIM
CFLAGS_quarantine.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report_generic.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_report_hw_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report_sw_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_shadow.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_hw_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_sw_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
obj-$(CONFIG_KASAN) := common.o report.o
obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o report_generic.o shadow.o quarantine.o
+obj-$(CONFIG_KASAN_HW_TAGS) += hw_tags.o report_hw_tags.o
obj-$(CONFIG_KASAN_SW_TAGS) += init.o report_sw_tags.o shadow.o sw_tags.o
--- /dev/null
+++ a/mm/kasan/report_hw_tags.c
@@ -0,0 +1,42 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file contains hardware tag-based KASAN specific error reporting code.
+ *
+ * Copyright (c) 2020 Google, Inc.
+ * Author: Andrey Konovalov <andreyknvl@google.com>
+ */
+
+#include <linux/kasan.h>
+#include <linux/kernel.h>
+#include <linux/memory.h>
+#include <linux/mm.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+#include "kasan.h"
+
+const char *get_bug_type(struct kasan_access_info *info)
+{
+ return "invalid-access";
+}
+
+void *find_first_bad_addr(void *addr, size_t size)
+{
+ return reset_tag(addr);
+}
+
+void metadata_fetch_row(char *buffer, void *row)
+{
+ int i;
+
+ for (i = 0; i < META_BYTES_PER_ROW; i++)
+ buffer[i] = hw_get_mem_tag(row + i * KASAN_GRANULE_SIZE);
+}
+
+void print_tags(u8 addr_tag, const void *addr)
+{
+ u8 memory_tag = hw_get_mem_tag((void *)addr);
+
+ pr_err("Pointer tag: [%02x], memory tag: [%02x]\n",
+ addr_tag, memory_tag);
+}
--- a/mm/kasan/report_sw_tags.c~kasan-arm64-implement-hw_tags-runtime
+++ a/mm/kasan/report_sw_tags.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains tag-based KASAN specific error reporting code.
+ * This file contains software tag-based KASAN specific error reporting code.
*
* Copyright (c) 2014 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
--- a/mm/kasan/shadow.c~kasan-arm64-implement-hw_tags-runtime
+++ a/mm/kasan/shadow.c
@@ -107,7 +107,7 @@ void unpoison_range(const void *address,
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
*shadow = tag;
- else
+ else /* CONFIG_KASAN_GENERIC */
*shadow = size & KASAN_GRANULE_MASK;
}
}
--- a/mm/kasan/sw_tags.c~kasan-arm64-implement-hw_tags-runtime
+++ a/mm/kasan/sw_tags.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains core tag-based KASAN code.
+ * This file contains core software tag-based KASAN code.
*
* Copyright (c) 2018 Google, Inc.
* Author: Andrey Konovalov <andreyknvl@google.com>
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 37/60] kasan, arm64: print report from tag fault handler
2020-12-22 19:58 incoming Andrew Morton
` (35 preceding siblings ...)
2020-12-22 20:02 ` [patch 36/60] kasan, arm64: implement HW_TAGS runtime Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 38/60] kasan, mm: reset tags when accessing metadata Andrew Morton
` (23 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: print report from tag fault handler
Add error reporting for hardware tag-based KASAN. When
CONFIG_KASAN_HW_TAGS is enabled, print KASAN report from the arm64 tag
fault handler.
SAS bits aren't set in ESR for all faults reported in EL1, so it's
impossible to find out the size of the access the caused the fault. Adapt
KASAN reporting code to handle this case.
Link: https://lkml.kernel.org/r/b559c82b6a969afedf53b4694b475f0234067a1a.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Co-developed-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/mm/fault.c | 14 ++++++++++++++
mm/kasan/report.c | 11 ++++++++---
2 files changed, 22 insertions(+), 3 deletions(-)
--- a/arch/arm64/mm/fault.c~kasan-arm64-print-report-from-tag-fault-handler
+++ a/arch/arm64/mm/fault.c
@@ -14,6 +14,7 @@
#include <linux/mm.h>
#include <linux/hardirq.h>
#include <linux/init.h>
+#include <linux/kasan.h>
#include <linux/kprobes.h>
#include <linux/uaccess.h>
#include <linux/page-flags.h>
@@ -297,10 +298,23 @@ static void die_kernel_fault(const char
do_exit(SIGKILL);
}
+#ifdef CONFIG_KASAN_HW_TAGS
static void report_tag_fault(unsigned long addr, unsigned int esr,
struct pt_regs *regs)
{
+ bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
+
+ /*
+ * SAS bits aren't set for all faults reported in EL1, so we can't
+ * find out access size.
+ */
+ kasan_report(addr, 0, is_write, regs->pc);
}
+#else
+/* Tag faults aren't enabled without CONFIG_KASAN_HW_TAGS. */
+static inline void report_tag_fault(unsigned long addr, unsigned int esr,
+ struct pt_regs *regs) { }
+#endif
static void do_tag_recovery(unsigned long addr, unsigned int esr,
struct pt_regs *regs)
--- a/mm/kasan/report.c~kasan-arm64-print-report-from-tag-fault-handler
+++ a/mm/kasan/report.c
@@ -62,9 +62,14 @@ static void print_error_description(stru
{
pr_err("BUG: KASAN: %s in %pS\n",
get_bug_type(info), (void *)info->ip);
- pr_err("%s of size %zu at addr %px by task %s/%d\n",
- info->is_write ? "Write" : "Read", info->access_size,
- info->access_addr, current->comm, task_pid_nr(current));
+ if (info->access_size)
+ pr_err("%s of size %zu at addr %px by task %s/%d\n",
+ info->is_write ? "Write" : "Read", info->access_size,
+ info->access_addr, current->comm, task_pid_nr(current));
+ else
+ pr_err("%s at addr %px by task %s/%d\n",
+ info->is_write ? "Write" : "Read",
+ info->access_addr, current->comm, task_pid_nr(current));
}
static DEFINE_SPINLOCK(report_lock);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 38/60] kasan, mm: reset tags when accessing metadata
2020-12-22 19:58 incoming Andrew Morton
` (36 preceding siblings ...)
2020-12-22 20:02 ` [patch 37/60] kasan, arm64: print report from tag fault handler Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 39/60] kasan, arm64: enable CONFIG_KASAN_HW_TAGS Andrew Morton
` (22 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, mm: reset tags when accessing metadata
Kernel allocator code accesses metadata for slab objects, that may lie
out-of-bounds of the object itself, or be accessed when an object is
freed. Such accesses trigger tag faults and lead to false-positive
reports with hardware tag-based KASAN.
Software KASAN modes disable instrumentation for allocator code via
KASAN_SANITIZE Makefile macro, and rely on kasan_enable/disable_current()
annotations which are used to ignore KASAN reports.
With hardware tag-based KASAN neither of those options are available, as
it doesn't use compiler instrumetation, no tag faults are ignored, and MTE
is disabled after the first one.
Instead, reset tags when accessing metadata (currently only for SLUB).
Link: https://lkml.kernel.org/r/a0f3cefbc49f34c843b664110842de4db28179d0.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Acked-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/page_alloc.c | 4 +++-
mm/page_poison.c | 2 +-
mm/slub.c | 29 ++++++++++++++++-------------
3 files changed, 20 insertions(+), 15 deletions(-)
--- a/mm/page_alloc.c~kasan-mm-reset-tags-when-accessing-metadata
+++ a/mm/page_alloc.c
@@ -1204,8 +1204,10 @@ static void kernel_init_free_pages(struc
/* s390's use of memset() could override KASAN redzones. */
kasan_disable_current();
- for (i = 0; i < numpages; i++)
+ for (i = 0; i < numpages; i++) {
+ page_kasan_tag_reset(page + i);
clear_highpage(page + i);
+ }
kasan_enable_current();
}
--- a/mm/page_poison.c~kasan-mm-reset-tags-when-accessing-metadata
+++ a/mm/page_poison.c
@@ -25,7 +25,7 @@ static void poison_page(struct page *pag
/* KASAN still think the page is in-use, so skip it. */
kasan_disable_current();
- memset(addr, PAGE_POISON, PAGE_SIZE);
+ memset(kasan_reset_tag(addr), PAGE_POISON, PAGE_SIZE);
kasan_enable_current();
kunmap_atomic(addr);
}
--- a/mm/slub.c~kasan-mm-reset-tags-when-accessing-metadata
+++ a/mm/slub.c
@@ -249,7 +249,7 @@ static inline void *freelist_ptr(const s
{
#ifdef CONFIG_SLAB_FREELIST_HARDENED
/*
- * When CONFIG_KASAN_SW_TAGS is enabled, ptr_addr might be tagged.
+ * When CONFIG_KASAN_SW/HW_TAGS is enabled, ptr_addr might be tagged.
* Normally, this doesn't cause any issues, as both set_freepointer()
* and get_freepointer() are called with a pointer with the same tag.
* However, there are some issues with CONFIG_SLUB_DEBUG code. For
@@ -275,6 +275,7 @@ static inline void *freelist_dereference
static inline void *get_freepointer(struct kmem_cache *s, void *object)
{
+ object = kasan_reset_tag(object);
return freelist_dereference(s, object + s->offset);
}
@@ -304,6 +305,7 @@ static inline void set_freepointer(struc
BUG_ON(object == fp); /* naive detection of double free or corruption */
#endif
+ freeptr_addr = (unsigned long)kasan_reset_tag((void *)freeptr_addr);
*(void **)freeptr_addr = freelist_ptr(s, fp, freeptr_addr);
}
@@ -538,8 +540,8 @@ static void print_section(char *level, c
unsigned int length)
{
metadata_access_enable();
- print_hex_dump(level, text, DUMP_PREFIX_ADDRESS, 16, 1, addr,
- length, 1);
+ print_hex_dump(level, kasan_reset_tag(text), DUMP_PREFIX_ADDRESS,
+ 16, 1, addr, length, 1);
metadata_access_disable();
}
@@ -570,7 +572,7 @@ static struct track *get_track(struct km
p = object + get_info_end(s);
- return p + alloc;
+ return kasan_reset_tag(p + alloc);
}
static void set_track(struct kmem_cache *s, void *object,
@@ -583,7 +585,8 @@ static void set_track(struct kmem_cache
unsigned int nr_entries;
metadata_access_enable();
- nr_entries = stack_trace_save(p->addrs, TRACK_ADDRS_COUNT, 3);
+ nr_entries = stack_trace_save(kasan_reset_tag(p->addrs),
+ TRACK_ADDRS_COUNT, 3);
metadata_access_disable();
if (nr_entries < TRACK_ADDRS_COUNT)
@@ -747,7 +750,7 @@ static __printf(3, 4) void slab_err(stru
static void init_object(struct kmem_cache *s, void *object, u8 val)
{
- u8 *p = object;
+ u8 *p = kasan_reset_tag(object);
if (s->flags & SLAB_RED_ZONE)
memset(p - s->red_left_pad, val, s->red_left_pad);
@@ -777,7 +780,7 @@ static int check_bytes_and_report(struct
u8 *addr = page_address(page);
metadata_access_enable();
- fault = memchr_inv(start, value, bytes);
+ fault = memchr_inv(kasan_reset_tag(start), value, bytes);
metadata_access_disable();
if (!fault)
return 1;
@@ -873,7 +876,7 @@ static int slab_pad_check(struct kmem_ca
pad = end - remainder;
metadata_access_enable();
- fault = memchr_inv(pad, POISON_INUSE, remainder);
+ fault = memchr_inv(kasan_reset_tag(pad), POISON_INUSE, remainder);
metadata_access_disable();
if (!fault)
return 1;
@@ -1118,7 +1121,7 @@ void setup_page_debug(struct kmem_cache
return;
metadata_access_enable();
- memset(addr, POISON_INUSE, page_size(page));
+ memset(kasan_reset_tag(addr), POISON_INUSE, page_size(page));
metadata_access_disable();
}
@@ -1566,10 +1569,10 @@ static inline bool slab_free_freelist_ho
* Clear the object and the metadata, but don't touch
* the redzone.
*/
- memset(object, 0, s->object_size);
+ memset(kasan_reset_tag(object), 0, s->object_size);
rsize = (s->flags & SLAB_RED_ZONE) ? s->red_left_pad
: 0;
- memset((char *)object + s->inuse, 0,
+ memset((char *)kasan_reset_tag(object) + s->inuse, 0,
s->size - s->inuse - rsize);
}
@@ -2881,10 +2884,10 @@ redo:
stat(s, ALLOC_FASTPATH);
}
- maybe_wipe_obj_freeptr(s, object);
+ maybe_wipe_obj_freeptr(s, kasan_reset_tag(object));
if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object)
- memset(object, 0, s->object_size);
+ memset(kasan_reset_tag(object), 0, s->object_size);
slab_post_alloc_hook(s, objcg, gfpflags, 1, &object);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 39/60] kasan, arm64: enable CONFIG_KASAN_HW_TAGS
2020-12-22 19:58 incoming Andrew Morton
` (37 preceding siblings ...)
2020-12-22 20:02 ` [patch 38/60] kasan, mm: reset tags when accessing metadata Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 40/60] kasan: add documentation for hardware tag-based mode Andrew Morton
` (21 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: enable CONFIG_KASAN_HW_TAGS
Hardware tag-based KASAN is now ready, enable the configuration option.
Link: https://lkml.kernel.org/r/a6fa50d3bb6b318e05c6389a44095be96442b8b0.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/Kconfig | 1 +
1 file changed, 1 insertion(+)
--- a/arch/arm64/Kconfig~kasan-arm64-enable-config_kasan_hw_tags
+++ a/arch/arm64/Kconfig
@@ -137,6 +137,7 @@ config ARM64
select HAVE_ARCH_JUMP_LABEL_RELATIVE
select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
+ select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE)
select HAVE_ARCH_KGDB
select HAVE_ARCH_MMAP_RND_BITS
select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 40/60] kasan: add documentation for hardware tag-based mode
2020-12-22 19:58 incoming Andrew Morton
` (38 preceding siblings ...)
2020-12-22 20:02 ` [patch 39/60] kasan, arm64: enable CONFIG_KASAN_HW_TAGS Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 41/60] kselftest/arm64: check GCR_EL1 after context switch Andrew Morton
` (20 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: add documentation for hardware tag-based mode
Add documentation for hardware tag-based KASAN mode and also add some
clarifications for software tag-based mode.
Link: https://lkml.kernel.org/r/20ed1d387685e89fc31be068f890f070ef9fd5d5.1606161801.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Alexander Potapenko <glider@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
Documentation/dev-tools/kasan.rst | 84 ++++++++++++++++++++--------
1 file changed, 61 insertions(+), 23 deletions(-)
--- a/Documentation/dev-tools/kasan.rst~kasan-add-documentation-for-hardware-tag-based-mode
+++ a/Documentation/dev-tools/kasan.rst
@@ -5,12 +5,14 @@ Overview
--------
KernelAddressSANitizer (KASAN) is a dynamic memory error detector designed to
-find out-of-bound and use-after-free bugs. KASAN has two modes: generic KASAN
-(similar to userspace ASan) and software tag-based KASAN (similar to userspace
-HWASan).
-
-KASAN uses compile-time instrumentation to insert validity checks before every
-memory access, and therefore requires a compiler version that supports that.
+find out-of-bound and use-after-free bugs. KASAN has three modes:
+1. generic KASAN (similar to userspace ASan),
+2. software tag-based KASAN (similar to userspace HWASan),
+3. hardware tag-based KASAN (based on hardware memory tagging).
+
+Software KASAN modes (1 and 2) use compile-time instrumentation to insert
+validity checks before every memory access, and therefore require a compiler
+version that supports that.
Generic KASAN is supported in both GCC and Clang. With GCC it requires version
8.3.0 or later. Any supported Clang version is compatible, but detection of
@@ -19,7 +21,7 @@ out-of-bounds accesses for global variab
Tag-based KASAN is only supported in Clang.
Currently generic KASAN is supported for the x86_64, arm64, xtensa, s390 and
-riscv architectures, and tag-based KASAN is supported only for arm64.
+and riscv architectures, and tag-based KASAN modes are supported only for arm64.
Usage
-----
@@ -28,14 +30,16 @@ To enable KASAN configure kernel with::
CONFIG_KASAN = y
-and choose between CONFIG_KASAN_GENERIC (to enable generic KASAN) and
-CONFIG_KASAN_SW_TAGS (to enable software tag-based KASAN).
-
-You also need to choose between CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE.
-Outline and inline are compiler instrumentation types. The former produces
-smaller binary while the latter is 1.1 - 2 times faster.
+and choose between CONFIG_KASAN_GENERIC (to enable generic KASAN),
+CONFIG_KASAN_SW_TAGS (to enable software tag-based KASAN), and
+CONFIG_KASAN_HW_TAGS (to enable hardware tag-based KASAN).
+
+For software modes, you also need to choose between CONFIG_KASAN_OUTLINE and
+CONFIG_KASAN_INLINE. Outline and inline are compiler instrumentation types.
+The former produces smaller binary while the latter is 1.1 - 2 times faster.
-Both KASAN modes work with both SLUB and SLAB memory allocators.
+Both software KASAN modes work with both SLUB and SLAB memory allocators,
+hardware tag-based KASAN currently only support SLUB.
For better bug detection and nicer reporting, enable CONFIG_STACKTRACE.
To augment reports with last allocation and freeing stack of the physical page,
@@ -197,17 +201,24 @@ call_rcu() and workqueue queuing.
Software tag-based KASAN
~~~~~~~~~~~~~~~~~~~~~~~~
-Tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64 CPUs to
-store a pointer tag in the top byte of kernel pointers. Like generic KASAN it
-uses shadow memory to store memory tags associated with each 16-byte memory
+Software tag-based KASAN requires software memory tagging support in the form
+of HWASan-like compiler instrumentation (see HWASan documentation for details).
+
+Software tag-based KASAN is currently only implemented for arm64 architecture.
+
+Software tag-based KASAN uses the Top Byte Ignore (TBI) feature of arm64 CPUs
+to store a pointer tag in the top byte of kernel pointers. Like generic KASAN
+it uses shadow memory to store memory tags associated with each 16-byte memory
cell (therefore it dedicates 1/16th of the kernel memory for shadow memory).
-On each memory allocation tag-based KASAN generates a random tag, tags the
-allocated memory with this tag, and embeds this tag into the returned pointer.
+On each memory allocation software tag-based KASAN generates a random tag, tags
+the allocated memory with this tag, and embeds this tag into the returned
+pointer.
+
Software tag-based KASAN uses compile-time instrumentation to insert checks
before each memory access. These checks make sure that tag of the memory that
is being accessed is equal to tag of the pointer that is used to access this
-memory. In case of a tag mismatch tag-based KASAN prints a bug report.
+memory. In case of a tag mismatch software tag-based KASAN prints a bug report.
Software tag-based KASAN also has two instrumentation modes (outline, that
emits callbacks to check memory accesses; and inline, that performs the shadow
@@ -216,9 +227,36 @@ simply printed from the function that pe
instrumentation a brk instruction is emitted by the compiler, and a dedicated
brk handler is used to print bug reports.
-A potential expansion of this mode is a hardware tag-based mode, which would
-use hardware memory tagging support instead of compiler instrumentation and
-manual shadow memory manipulation.
+Software tag-based KASAN uses 0xFF as a match-all pointer tag (accesses through
+pointers with 0xFF pointer tag aren't checked). The value 0xFE is currently
+reserved to tag freed memory regions.
+
+Software tag-based KASAN currently only supports tagging of
+kmem_cache_alloc/kmalloc and page_alloc memory.
+
+Hardware tag-based KASAN
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Hardware tag-based KASAN is similar to the software mode in concept, but uses
+hardware memory tagging support instead of compiler instrumentation and
+shadow memory.
+
+Hardware tag-based KASAN is currently only implemented for arm64 architecture
+and based on both arm64 Memory Tagging Extension (MTE) introduced in ARMv8.5
+Instruction Set Architecture, and Top Byte Ignore (TBI).
+
+Special arm64 instructions are used to assign memory tags for each allocation.
+Same tags are assigned to pointers to those allocations. On every memory
+access, hardware makes sure that tag of the memory that is being accessed is
+equal to tag of the pointer that is used to access this memory. In case of a
+tag mismatch a fault is generated and a report is printed.
+
+Hardware tag-based KASAN uses 0xFF as a match-all pointer tag (accesses through
+pointers with 0xFF pointer tag aren't checked). The value 0xFE is currently
+reserved to tag freed memory regions.
+
+Hardware tag-based KASAN currently only supports tagging of
+kmem_cache_alloc/kmalloc and page_alloc memory.
What memory accesses are sanitised by KASAN?
--------------------------------------------
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 41/60] kselftest/arm64: check GCR_EL1 after context switch
2020-12-22 19:58 incoming Andrew Morton
` (39 preceding siblings ...)
2020-12-22 20:02 ` [patch 40/60] kasan: add documentation for hardware tag-based mode Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 42/60] kasan: simplify quarantine_put call site Andrew Morton
` (19 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Subject: kselftest/arm64: check GCR_EL1 after context switch
This test is specific to MTE and verifies that the GCR_EL1 register is
context switched correctly.
It spawns 1024 processes and each process spawns 5 threads. Each thread
writes a random setting of GCR_EL1 through the prctl() system call and
reads it back verifying that it is the same. If the values are not the
same it reports a failure.
Note: The test has been extended to verify that even SYNC and ASYNC mode
setting is preserved correctly over context switching.
Link: https://lkml.kernel.org/r/b51a165426e906e7ec8a68d806ef3f8cd92581a6.1606161801.git.andreyknvl@google.com
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Marco Elver <elver@google.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
tools/testing/selftests/arm64/mte/Makefile | 2
tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c | 155 ++++++++++
2 files changed, 156 insertions(+), 1 deletion(-)
--- /dev/null
+++ a/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c
@@ -0,0 +1,155 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2020 ARM Limited
+
+#define _GNU_SOURCE
+
+#include <errno.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/auxv.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+
+#include "kselftest.h"
+#include "mte_common_util.h"
+
+#define PR_SET_TAGGED_ADDR_CTRL 55
+#define PR_GET_TAGGED_ADDR_CTRL 56
+# define PR_TAGGED_ADDR_ENABLE (1UL << 0)
+# define PR_MTE_TCF_SHIFT 1
+# define PR_MTE_TCF_NONE (0UL << PR_MTE_TCF_SHIFT)
+# define PR_MTE_TCF_SYNC (1UL << PR_MTE_TCF_SHIFT)
+# define PR_MTE_TCF_ASYNC (2UL << PR_MTE_TCF_SHIFT)
+# define PR_MTE_TCF_MASK (3UL << PR_MTE_TCF_SHIFT)
+# define PR_MTE_TAG_SHIFT 3
+# define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT)
+
+#include "mte_def.h"
+
+#define NUM_ITERATIONS 1024
+#define MAX_THREADS 5
+#define THREAD_ITERATIONS 1000
+
+void *execute_thread(void *x)
+{
+ pid_t pid = *((pid_t *)x);
+ pid_t tid = gettid();
+ uint64_t prctl_tag_mask;
+ uint64_t prctl_set;
+ uint64_t prctl_get;
+ uint64_t prctl_tcf;
+
+ srand(time(NULL) ^ (pid << 16) ^ (tid << 16));
+
+ prctl_tag_mask = rand() & 0xffff;
+
+ if (prctl_tag_mask % 2)
+ prctl_tcf = PR_MTE_TCF_SYNC;
+ else
+ prctl_tcf = PR_MTE_TCF_ASYNC;
+
+ prctl_set = PR_TAGGED_ADDR_ENABLE | prctl_tcf | (prctl_tag_mask << PR_MTE_TAG_SHIFT);
+
+ for (int j = 0; j < THREAD_ITERATIONS; j++) {
+ if (prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_set, 0, 0, 0)) {
+ perror("prctl() failed");
+ goto fail;
+ }
+
+ prctl_get = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0);
+
+ if (prctl_set != prctl_get) {
+ ksft_print_msg("Error: prctl_set: 0x%lx != prctl_get: 0x%lx\n",
+ prctl_set, prctl_get);
+ goto fail;
+ }
+ }
+
+ return (void *)KSFT_PASS;
+
+fail:
+ return (void *)KSFT_FAIL;
+}
+
+int execute_test(pid_t pid)
+{
+ pthread_t thread_id[MAX_THREADS];
+ int thread_data[MAX_THREADS];
+
+ for (int i = 0; i < MAX_THREADS; i++)
+ pthread_create(&thread_id[i], NULL,
+ execute_thread, (void *)&pid);
+
+ for (int i = 0; i < MAX_THREADS; i++)
+ pthread_join(thread_id[i], (void *)&thread_data[i]);
+
+ for (int i = 0; i < MAX_THREADS; i++)
+ if (thread_data[i] == KSFT_FAIL)
+ return KSFT_FAIL;
+
+ return KSFT_PASS;
+}
+
+int mte_gcr_fork_test(void)
+{
+ pid_t pid;
+ int results[NUM_ITERATIONS];
+ pid_t cpid;
+ int res;
+
+ for (int i = 0; i < NUM_ITERATIONS; i++) {
+ pid = fork();
+
+ if (pid < 0)
+ return KSFT_FAIL;
+
+ if (pid == 0) {
+ cpid = getpid();
+
+ res = execute_test(cpid);
+
+ exit(res);
+ }
+ }
+
+ for (int i = 0; i < NUM_ITERATIONS; i++) {
+ wait(&res);
+
+ if (WIFEXITED(res))
+ results[i] = WEXITSTATUS(res);
+ else
+ --i;
+ }
+
+ for (int i = 0; i < NUM_ITERATIONS; i++)
+ if (results[i] == KSFT_FAIL)
+ return KSFT_FAIL;
+
+ return KSFT_PASS;
+}
+
+int main(int argc, char *argv[])
+{
+ int err;
+
+ err = mte_default_setup();
+ if (err)
+ return err;
+
+ ksft_set_plan(1);
+
+ evaluate_test(mte_gcr_fork_test(),
+ "Verify that GCR_EL1 is set correctly on context switch\n");
+
+ mte_restore_setup();
+ ksft_print_cnts();
+
+ return ksft_get_fail_cnt() == 0 ? KSFT_PASS : KSFT_FAIL;
+}
+
--- a/tools/testing/selftests/arm64/mte/Makefile~kselftest-arm64-check-gcr_el1-after-context-switch
+++ a/tools/testing/selftests/arm64/mte/Makefile
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
# Copyright (C) 2020 ARM Limited
-CFLAGS += -std=gnu99 -I.
+CFLAGS += -std=gnu99 -I. -lpthread
SRCS := $(filter-out mte_common_util.c,$(wildcard *.c))
PROGS := $(patsubst %.c,%,$(SRCS))
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 42/60] kasan: simplify quarantine_put call site
2020-12-22 19:58 incoming Andrew Morton
` (40 preceding siblings ...)
2020-12-22 20:02 ` [patch 41/60] kselftest/arm64: check GCR_EL1 after context switch Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 43/60] kasan: rename get_alloc/free_info Andrew Morton
` (18 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: simplify quarantine_put call site
Patch series "kasan: boot parameters for hardware tag-based mode", v4.
=== Overview
Hardware tag-based KASAN mode [1] is intended to eventually be used in
production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.
This patchset adds a few boot parameters for hardware tag-based KASAN that
allow to disable or otherwise control particular KASAN features, as well
as provides some initial optimizations for running KASAN in production.
There's another planned patchset what will further optimize hardware
tag-based KASAN, provide proper benchmarking and tests, and will fully
enable tag-based KASAN for production use.
Hardware tag-based KASAN relies on arm64 Memory Tagging Extension (MTE)
[2] to perform memory and pointer tagging. Please see [3] and [4] for
detailed analysis of how MTE helps to fight memory safety problems.
The features that can be controlled are:
1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.
The patch titled "kasan: add and integrate kasan boot parameters" of this
series adds a few new boot parameters.
kasan.mode allows to choose one of three main modes:
- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled
The chosen mode provides default control values for the features mentioned
above. However it's also possible to override the default values by
providing:
- kasan.stacktrace=off/on - enable stacks collection
(default: on for mode=full, otherwise off)
- kasan.fault=report/panic - only report tag fault or also panic
(default: report)
If kasan.mode parameter is not provided, it defaults to full when
CONFIG_DEBUG_KERNEL is enabled, and to prod otherwise.
It is essential that switching between these modes doesn't require
rebuilding the kernel with different configs, as this is required by
the Android GKI (Generic Kernel Image) initiative.
=== Benchmarks
For now I've only performed a few simple benchmarks such as measuring
kernel boot time and slab memory usage after boot. There's an upcoming
patchset which will optimize KASAN further and include more detailed
benchmarking results.
The benchmarks were performed in QEMU and the results below exclude the
slowdown caused by QEMU memory tagging emulation (as it's different from
the slowdown that will be introduced by hardware and is therefore
irrelevant).
KASAN_HW_TAGS=y + kasan.mode=off introduces no performance or memory
impact compared to KASAN_HW_TAGS=n.
kasan.mode=prod (manually excluding tagging) introduces 3% of performance
and no memory impact (except memory used by hardware to store tags)
compared to kasan.mode=off.
kasan.mode=full has about 40% performance and 30% memory impact over
kasan.mode=prod. Both come from alloc/free stack collection.
=== Notes
This patchset is available here:
https://github.com/xairy/linux/tree/up-boot-mte-v4
This patchset is based on v11 of "kasan: add hardware tag-based mode for
arm64" patchset [1].
For testing in QEMU hardware tag-based KASAN requires:
1. QEMU built from master [6] (use "-machine virt,mte=on -cpu max" arguments
to run).
2. GCC version 10.
[1] https://lore.kernel.org/linux-arm-kernel/cover.1606161801.git.andreyknvl@google.com/T/#t
[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] https://arxiv.org/pdf/1802.09517.pdf
[4] https://github.com/microsoft/MSRC-Security-Research/blob/master/papers/2020/Security%20analysis%20of%20memory%20tagging.pdf
[5] https://source.android.com/devices/architecture/kernel/generic-kernel-image
[6] https://github.com/qemu/qemu
=== Tags
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
This patch (of 19):
Move get_free_info() call into quarantine_put() to simplify the call site.
No functional changes.
Link: https://lkml.kernel.org/r/cover.1606162397.git.andreyknvl@google.com
Link: https://lkml.kernel.org/r/312d0a3ef92cc6dc4fa5452cbc1714f9393ca239.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Iab0f04e7ebf8d83247024b7190c67c3c34c7940f
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 2 +-
mm/kasan/kasan.h | 5 ++---
mm/kasan/quarantine.c | 3 ++-
3 files changed, 5 insertions(+), 5 deletions(-)
--- a/mm/kasan/common.c~kasan-simplify-quarantine_put-call-site
+++ a/mm/kasan/common.c
@@ -313,7 +313,7 @@ static bool __kasan_slab_free(struct kme
kasan_set_free_info(cache, object, tag);
- quarantine_put(get_free_info(cache, object), cache);
+ quarantine_put(cache, object);
return IS_ENABLED(CONFIG_KASAN_GENERIC);
}
--- a/mm/kasan/kasan.h~kasan-simplify-quarantine_put-call-site
+++ a/mm/kasan/kasan.h
@@ -216,12 +216,11 @@ struct kasan_track *kasan_get_free_track
#if defined(CONFIG_KASAN_GENERIC) && \
(defined(CONFIG_SLAB) || defined(CONFIG_SLUB))
-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache);
+void quarantine_put(struct kmem_cache *cache, void *object);
void quarantine_reduce(void);
void quarantine_remove_cache(struct kmem_cache *cache);
#else
-static inline void quarantine_put(struct kasan_free_meta *info,
- struct kmem_cache *cache) { }
+static inline void quarantine_put(struct kmem_cache *cache, void *object) { }
static inline void quarantine_reduce(void) { }
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif
--- a/mm/kasan/quarantine.c~kasan-simplify-quarantine_put-call-site
+++ a/mm/kasan/quarantine.c
@@ -163,11 +163,12 @@ static void qlist_free_all(struct qlist_
qlist_init(q);
}
-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache)
+void quarantine_put(struct kmem_cache *cache, void *object)
{
unsigned long flags;
struct qlist_head *q;
struct qlist_head temp = QLIST_INIT;
+ struct kasan_free_meta *info = get_free_info(cache, object);
/*
* Note: irq must be disabled until after we move the batch to the
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 43/60] kasan: rename get_alloc/free_info
2020-12-22 19:58 incoming Andrew Morton
` (41 preceding siblings ...)
2020-12-22 20:02 ` [patch 42/60] kasan: simplify quarantine_put call site Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 44/60] kasan: introduce set_alloc_info Andrew Morton
` (17 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: rename get_alloc/free_info
Rename get_alloc_info() and get_free_info() to kasan_get_alloc_meta() and
kasan_get_free_meta() to better reflect what those do and avoid confusion
with kasan_set_free_info().
No functional changes.
Link: https://lkml.kernel.org/r/27b7c036b754af15a2839e945f6d8bfce32b4c2f.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Ib6e4ba61c8b12112b403d3479a9799ac8fff8de1
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 16 ++++++++--------
mm/kasan/generic.c | 12 ++++++------
mm/kasan/hw_tags.c | 4 ++--
mm/kasan/kasan.h | 8 ++++----
mm/kasan/quarantine.c | 4 ++--
mm/kasan/report.c | 12 ++++++------
mm/kasan/report_sw_tags.c | 2 +-
mm/kasan/sw_tags.c | 4 ++--
8 files changed, 31 insertions(+), 31 deletions(-)
--- a/mm/kasan/common.c~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/common.c
@@ -180,14 +180,14 @@ size_t kasan_metadata_size(struct kmem_c
sizeof(struct kasan_free_meta) : 0);
}
-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object)
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object)
{
return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}
-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object)
+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
+ const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
@@ -264,13 +264,13 @@ static u8 assign_tag(struct kmem_cache *
void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
const void *object)
{
- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;
if (!(cache->flags & SLAB_KASAN))
return (void *)object;
- alloc_info = get_alloc_info(cache, object);
- __memset(alloc_info, 0, sizeof(*alloc_info));
+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object, assign_tag(cache, object, true, false));
@@ -350,7 +350,7 @@ static void *__kasan_kmalloc(struct kmem
KASAN_KMALLOC_REDZONE);
if (cache->flags & SLAB_KASAN)
- kasan_set_track(&get_alloc_info(cache, object)->alloc_track, flags);
+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
return set_tag(object, tag);
}
--- a/mm/kasan/generic.c~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/generic.c
@@ -328,7 +328,7 @@ void kasan_record_aux_stack(void *addr)
{
struct page *page = kasan_addr_to_page(addr);
struct kmem_cache *cache;
- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;
void *object;
if (!(page && PageSlab(page)))
@@ -336,10 +336,10 @@ void kasan_record_aux_stack(void *addr)
cache = page->slab_cache;
object = nearest_obj(cache, page, addr);
- alloc_info = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
- alloc_info->aux_stack[1] = alloc_info->aux_stack[0];
- alloc_info->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
+ alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0];
+ alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
}
void kasan_set_free_info(struct kmem_cache *cache,
@@ -347,7 +347,7 @@ void kasan_set_free_info(struct kmem_cac
{
struct kasan_free_meta *free_meta;
- free_meta = get_free_info(cache, object);
+ free_meta = kasan_get_free_meta(cache, object);
kasan_set_track(&free_meta->free_track, GFP_NOWAIT);
/*
@@ -361,5 +361,5 @@ struct kasan_track *kasan_get_free_track
{
if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_KMALLOC_FREETRACK)
return NULL;
- return &get_free_info(cache, object)->free_track;
+ return &kasan_get_free_meta(cache, object)->free_track;
}
--- a/mm/kasan/hw_tags.c~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/hw_tags.c
@@ -66,7 +66,7 @@ void kasan_set_free_info(struct kmem_cac
{
struct kasan_alloc_meta *alloc_meta;
- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
}
@@ -75,6 +75,6 @@ struct kasan_track *kasan_get_free_track
{
struct kasan_alloc_meta *alloc_meta;
- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
return &alloc_meta->free_track[0];
}
--- a/mm/kasan/kasan.h~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/kasan.h
@@ -149,10 +149,10 @@ struct kasan_free_meta {
#endif
};
-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object);
-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object);
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object);
+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
+ const void *object);
void poison_range(const void *address, size_t size, u8 value);
void unpoison_range(const void *address, size_t size);
--- a/mm/kasan/quarantine.c~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/quarantine.c
@@ -168,7 +168,7 @@ void quarantine_put(struct kmem_cache *c
unsigned long flags;
struct qlist_head *q;
struct qlist_head temp = QLIST_INIT;
- struct kasan_free_meta *info = get_free_info(cache, object);
+ struct kasan_free_meta *meta = kasan_get_free_meta(cache, object);
/*
* Note: irq must be disabled until after we move the batch to the
@@ -185,7 +185,7 @@ void quarantine_put(struct kmem_cache *c
local_irq_restore(flags);
return;
}
- qlist_put(q, &info->quarantine_link, cache->size);
+ qlist_put(q, &meta->quarantine_link, cache->size);
if (unlikely(q->bytes > QUARANTINE_PERCPU_SIZE)) {
qlist_move_all(q, &temp);
--- a/mm/kasan/report.c~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/report.c
@@ -164,12 +164,12 @@ static void describe_object_addr(struct
static void describe_object(struct kmem_cache *cache, void *object,
const void *addr, u8 tag)
{
- struct kasan_alloc_meta *alloc_info = get_alloc_info(cache, object);
+ struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);
if (cache->flags & SLAB_KASAN) {
struct kasan_track *free_track;
- print_track(&alloc_info->alloc_track, "Allocated");
+ print_track(&alloc_meta->alloc_track, "Allocated");
pr_err("\n");
free_track = kasan_get_free_track(cache, object, tag);
if (free_track) {
@@ -178,14 +178,14 @@ static void describe_object(struct kmem_
}
#ifdef CONFIG_KASAN_GENERIC
- if (alloc_info->aux_stack[0]) {
+ if (alloc_meta->aux_stack[0]) {
pr_err("Last potentially related work creation:\n");
- print_stack(alloc_info->aux_stack[0]);
+ print_stack(alloc_meta->aux_stack[0]);
pr_err("\n");
}
- if (alloc_info->aux_stack[1]) {
+ if (alloc_meta->aux_stack[1]) {
pr_err("Second to last potentially related work creation:\n");
- print_stack(alloc_info->aux_stack[1]);
+ print_stack(alloc_meta->aux_stack[1]);
pr_err("\n");
}
#endif
--- a/mm/kasan/report_sw_tags.c~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/report_sw_tags.c
@@ -46,7 +46,7 @@ const char *get_bug_type(struct kasan_ac
if (page && PageSlab(page)) {
cache = page->slab_cache;
object = nearest_obj(cache, page, (void *)addr);
- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
for (i = 0; i < KASAN_NR_FREE_STACKS; i++)
if (alloc_meta->free_pointer_tag[i] == tag)
--- a/mm/kasan/sw_tags.c~kasan-rename-get_alloc-free_info
+++ a/mm/kasan/sw_tags.c
@@ -174,7 +174,7 @@ void kasan_set_free_info(struct kmem_cac
struct kasan_alloc_meta *alloc_meta;
u8 idx = 0;
- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
idx = alloc_meta->free_track_idx;
@@ -191,7 +191,7 @@ struct kasan_track *kasan_get_free_track
struct kasan_alloc_meta *alloc_meta;
int i = 0;
- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
for (i = 0; i < KASAN_NR_FREE_STACKS; i++) {
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 44/60] kasan: introduce set_alloc_info
2020-12-22 19:58 incoming Andrew Morton
` (42 preceding siblings ...)
2020-12-22 20:02 ` [patch 43/60] kasan: rename get_alloc/free_info Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 45/60] kasan, arm64: unpoison stack only with CONFIG_KASAN_STACK Andrew Morton
` (16 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: introduce set_alloc_info
Add set_alloc_info() helper and move kasan_set_track() into it. This will
simplify the code for one of the upcoming changes.
No functional changes.
Link: https://lkml.kernel.org/r/b2393e8f1e311a70fc3aaa2196461b6acdee7d21.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/I0316193cbb4ecc9b87b7c2eee0dd79f8ec908c1a
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
--- a/mm/kasan/common.c~kasan-introduce-set_alloc_info
+++ a/mm/kasan/common.c
@@ -323,6 +323,11 @@ bool kasan_slab_free(struct kmem_cache *
return __kasan_slab_free(cache, object, ip, true);
}
+static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
+{
+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
+}
+
static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
size_t size, gfp_t flags, bool keep_tag)
{
@@ -350,7 +355,7 @@ static void *__kasan_kmalloc(struct kmem
KASAN_KMALLOC_REDZONE);
if (cache->flags & SLAB_KASAN)
- kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
+ set_alloc_info(cache, (void *)object, flags);
return set_tag(object, tag);
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 45/60] kasan, arm64: unpoison stack only with CONFIG_KASAN_STACK
2020-12-22 19:58 incoming Andrew Morton
` (43 preceding siblings ...)
2020-12-22 20:02 ` [patch 44/60] kasan: introduce set_alloc_info Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 46/60] kasan: allow VMAP_STACK for HW_TAGS mode Andrew Morton
` (15 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, arm64: unpoison stack only with CONFIG_KASAN_STACK
There's a config option CONFIG_KASAN_STACK that has to be enabled for
KASAN to use stack instrumentation and perform validity checks for
stack variables.
There's no need to unpoison stack when CONFIG_KASAN_STACK is not enabled.
Only call kasan_unpoison_task_stack[_below]() when CONFIG_KASAN_STACK is
enabled.
Note, that CONFIG_KASAN_STACK is an option that is currently always
defined when CONFIG_KASAN is enabled, and therefore has to be tested
with #if instead of #ifdef.
Link: https://lkml.kernel.org/r/d09dd3f8abb388da397fd11598c5edeaa83fe559.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/If8a891e9fe01ea543e00b576852685afec0887e3
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/arm64/kernel/sleep.S | 2 +-
arch/x86/kernel/acpi/wakeup_64.S | 2 +-
include/linux/kasan.h | 10 ++++++----
mm/kasan/common.c | 2 ++
4 files changed, 10 insertions(+), 6 deletions(-)
--- a/arch/arm64/kernel/sleep.S~kasan-arm64-unpoison-stack-only-with-config_kasan_stack
+++ a/arch/arm64/kernel/sleep.S
@@ -133,7 +133,7 @@ SYM_FUNC_START(_cpu_resume)
*/
bl cpu_do_resume
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
mov x0, sp
bl kasan_unpoison_task_stack_below
#endif
--- a/arch/x86/kernel/acpi/wakeup_64.S~kasan-arm64-unpoison-stack-only-with-config_kasan_stack
+++ a/arch/x86/kernel/acpi/wakeup_64.S
@@ -112,7 +112,7 @@ SYM_FUNC_START(do_suspend_lowlevel)
movq pt_regs_r14(%rax), %r14
movq pt_regs_r15(%rax), %r15
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
/*
* The suspend path may have poisoned some areas deeper in the stack,
* which we now need to unpoison.
--- a/include/linux/kasan.h~kasan-arm64-unpoison-stack-only-with-config_kasan_stack
+++ a/include/linux/kasan.h
@@ -77,8 +77,6 @@ static inline void kasan_disable_current
void kasan_unpoison_range(const void *address, size_t size);
-void kasan_unpoison_task_stack(struct task_struct *task);
-
void kasan_alloc_pages(struct page *page, unsigned int order);
void kasan_free_pages(struct page *page, unsigned int order);
@@ -123,8 +121,6 @@ void kasan_restore_multi_shot(bool enabl
static inline void kasan_unpoison_range(const void *address, size_t size) {}
-static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
-
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}
@@ -176,6 +172,12 @@ static inline size_t kasan_metadata_size
#endif /* CONFIG_KASAN */
+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
+void kasan_unpoison_task_stack(struct task_struct *task);
+#else
+static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
+#endif
+
#ifdef CONFIG_KASAN_GENERIC
void kasan_cache_shrink(struct kmem_cache *cache);
--- a/mm/kasan/common.c~kasan-arm64-unpoison-stack-only-with-config_kasan_stack
+++ a/mm/kasan/common.c
@@ -63,6 +63,7 @@ void kasan_unpoison_range(const void *ad
unpoison_range(address, size);
}
+#if CONFIG_KASAN_STACK
static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
{
void *base = task_stack_page(task);
@@ -89,6 +90,7 @@ asmlinkage void kasan_unpoison_task_stac
unpoison_range(base, watermark - base);
}
+#endif /* CONFIG_KASAN_STACK */
void kasan_alloc_pages(struct page *page, unsigned int order)
{
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 46/60] kasan: allow VMAP_STACK for HW_TAGS mode
2020-12-22 19:58 incoming Andrew Morton
` (44 preceding siblings ...)
2020-12-22 20:02 ` [patch 45/60] kasan, arm64: unpoison stack only with CONFIG_KASAN_STACK Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 47/60] kasan: remove __kasan_unpoison_stack Andrew Morton
` (14 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: allow VMAP_STACK for HW_TAGS mode
Even though hardware tag-based mode currently doesn't support checking
vmalloc allocations, it doesn't use shadow memory and works with
VMAP_STACK as is. Change VMAP_STACK definition accordingly.
Link: https://lkml.kernel.org/r/ecdb2a1658ebd88eb276dee2493518ac0e82de41.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/I3552cbc12321dec82cd7372676e9372a2eb452ac
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
arch/Kconfig | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
--- a/arch/Kconfig~kasan-allow-vmap_stack-for-hw_tags-mode
+++ a/arch/Kconfig
@@ -976,16 +976,16 @@ config VMAP_STACK
default y
bool "Use a virtually-mapped stack"
depends on HAVE_ARCH_VMAP_STACK
- depends on !KASAN || KASAN_VMALLOC
+ depends on !KASAN || KASAN_HW_TAGS || KASAN_VMALLOC
help
Enable this if you want the use virtually-mapped kernel stacks
with guard pages. This causes kernel stack overflows to be
caught immediately rather than causing difficult-to-diagnose
corruption.
- To use this with KASAN, the architecture must support backing
- virtual mappings with real shadow memory, and KASAN_VMALLOC must
- be enabled.
+ To use this with software KASAN modes, the architecture must support
+ backing virtual mappings with real shadow memory, and KASAN_VMALLOC
+ must be enabled.
config ARCH_OPTIONAL_KERNEL_RWX
def_bool n
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 47/60] kasan: remove __kasan_unpoison_stack
2020-12-22 19:58 incoming Andrew Morton
` (45 preceding siblings ...)
2020-12-22 20:02 ` [patch 46/60] kasan: allow VMAP_STACK for HW_TAGS mode Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 48/60] kasan: inline kasan_reset_tag for tag-based modes Andrew Morton
` (13 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: remove __kasan_unpoison_stack
There's no need for __kasan_unpoison_stack() helper, as it's only
currently used in a single place. Removing it also removes unneeded
arithmetic.
No functional changes.
Link: https://lkml.kernel.org/r/93e78948704a42ea92f6248ff8a725613d721161.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Ie5ba549d445292fe629b4a96735e4034957bcc50
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)
--- a/mm/kasan/common.c~kasan-remove-__kasan_unpoison_stack
+++ a/mm/kasan/common.c
@@ -64,18 +64,12 @@ void kasan_unpoison_range(const void *ad
}
#if CONFIG_KASAN_STACK
-static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
-{
- void *base = task_stack_page(task);
- size_t size = sp - base;
-
- unpoison_range(base, size);
-}
-
/* Unpoison the entire stack for a task. */
void kasan_unpoison_task_stack(struct task_struct *task)
{
- __kasan_unpoison_stack(task, task_stack_page(task) + THREAD_SIZE);
+ void *base = task_stack_page(task);
+
+ unpoison_range(base, THREAD_SIZE);
}
/* Unpoison the stack for the current task beyond a watermark sp value. */
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 48/60] kasan: inline kasan_reset_tag for tag-based modes
2020-12-22 19:58 incoming Andrew Morton
` (46 preceding siblings ...)
2020-12-22 20:02 ` [patch 47/60] kasan: remove __kasan_unpoison_stack Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 49/60] kasan: inline random_tag for HW_TAGS Andrew Morton
` (12 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: inline kasan_reset_tag for tag-based modes
Using kasan_reset_tag() currently results in a function call. As it's
called quite often from the allocator code, this leads to a noticeable
slowdown. Move it to include/linux/kasan.h and turn it into a static
inline function. Also remove the now unneeded reset_tag() internal KASAN
macro and use kasan_reset_tag() instead.
Link: https://lkml.kernel.org/r/6940383a3a9dfb416134d338d8fac97a9ebb8686.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/I4d2061acfe91d480a75df00b07c22d8494ef14b5
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 5 ++++-
mm/kasan/common.c | 6 +++---
mm/kasan/hw_tags.c | 9 ++-------
mm/kasan/kasan.h | 4 ----
mm/kasan/report.c | 4 ++--
mm/kasan/report_hw_tags.c | 2 +-
mm/kasan/report_sw_tags.c | 4 ++--
mm/kasan/shadow.c | 4 ++--
mm/kasan/sw_tags.c | 9 ++-------
9 files changed, 18 insertions(+), 29 deletions(-)
--- a/include/linux/kasan.h~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/include/linux/kasan.h
@@ -194,7 +194,10 @@ static inline void kasan_record_aux_stac
#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
-void *kasan_reset_tag(const void *addr);
+static inline void *kasan_reset_tag(const void *addr)
+{
+ return (void *)arch_kasan_reset_tag(addr);
+}
bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
--- a/mm/kasan/common.c~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/common.c
@@ -179,14 +179,14 @@ size_t kasan_metadata_size(struct kmem_c
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{
- return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}
struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
- return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}
void kasan_poison_slab(struct page *page)
@@ -283,7 +283,7 @@ static bool __kasan_slab_free(struct kme
tag = get_tag(object);
tagged_object = object;
- object = reset_tag(object);
+ object = kasan_reset_tag(object);
if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) !=
object)) {
--- a/mm/kasan/hw_tags.c~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/hw_tags.c
@@ -30,20 +30,15 @@ void __init kasan_init_hw_tags(void)
pr_info("KernelAddressSanitizer initialized\n");
}
-void *kasan_reset_tag(const void *addr)
-{
- return reset_tag(addr);
-}
-
void poison_range(const void *address, size_t size, u8 value)
{
- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), value);
}
void unpoison_range(const void *address, size_t size)
{
- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}
--- a/mm/kasan/kasan.h~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/kasan.h
@@ -248,15 +248,11 @@ static inline const void *arch_kasan_set
return addr;
}
#endif
-#ifndef arch_kasan_reset_tag
-#define arch_kasan_reset_tag(addr) ((void *)(addr))
-#endif
#ifndef arch_kasan_get_tag
#define arch_kasan_get_tag(addr) 0
#endif
#define set_tag(addr, tag) ((void *)arch_kasan_set_tag((addr), (tag)))
-#define reset_tag(addr) ((void *)arch_kasan_reset_tag(addr))
#define get_tag(addr) arch_kasan_get_tag(addr)
#ifdef CONFIG_KASAN_HW_TAGS
--- a/mm/kasan/report.c~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/report.c
@@ -328,7 +328,7 @@ void kasan_report_invalid_free(void *obj
unsigned long flags;
u8 tag = get_tag(object);
- object = reset_tag(object);
+ object = kasan_reset_tag(object);
#if IS_ENABLED(CONFIG_KUNIT)
if (current->kunit_test)
@@ -361,7 +361,7 @@ static void __kasan_report(unsigned long
disable_trace_on_warning();
tagged_addr = (void *)addr;
- untagged_addr = reset_tag(tagged_addr);
+ untagged_addr = kasan_reset_tag(tagged_addr);
info.access_addr = tagged_addr;
if (addr_has_metadata(untagged_addr))
--- a/mm/kasan/report_hw_tags.c~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/report_hw_tags.c
@@ -22,7 +22,7 @@ const char *get_bug_type(struct kasan_ac
void *find_first_bad_addr(void *addr, size_t size)
{
- return reset_tag(addr);
+ return kasan_reset_tag(addr);
}
void metadata_fetch_row(char *buffer, void *row)
--- a/mm/kasan/report_sw_tags.c~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/report_sw_tags.c
@@ -41,7 +41,7 @@ const char *get_bug_type(struct kasan_ac
int i;
tag = get_tag(info->access_addr);
- addr = reset_tag(info->access_addr);
+ addr = kasan_reset_tag(info->access_addr);
page = kasan_addr_to_page(addr);
if (page && PageSlab(page)) {
cache = page->slab_cache;
@@ -72,7 +72,7 @@ const char *get_bug_type(struct kasan_ac
void *find_first_bad_addr(void *addr, size_t size)
{
u8 tag = get_tag(addr);
- void *p = reset_tag(addr);
+ void *p = kasan_reset_tag(addr);
void *end = p + size;
while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p))
--- a/mm/kasan/shadow.c~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/shadow.c
@@ -81,7 +81,7 @@ void poison_range(const void *address, s
* some of the callers (e.g. kasan_poison_object_data) pass tagged
* addresses to this function.
*/
- address = reset_tag(address);
+ address = kasan_reset_tag(address);
shadow_start = kasan_mem_to_shadow(address);
shadow_end = kasan_mem_to_shadow(address + size);
@@ -98,7 +98,7 @@ void unpoison_range(const void *address,
* some of the callers (e.g. kasan_unpoison_object_data) pass tagged
* addresses to this function.
*/
- address = reset_tag(address);
+ address = kasan_reset_tag(address);
poison_range(address, size, tag);
--- a/mm/kasan/sw_tags.c~kasan-inline-kasan_reset_tag-for-tag-based-modes
+++ a/mm/kasan/sw_tags.c
@@ -67,11 +67,6 @@ u8 random_tag(void)
return (u8)(state % (KASAN_TAG_MAX + 1));
}
-void *kasan_reset_tag(const void *addr)
-{
- return reset_tag(addr);
-}
-
bool check_memory_region(unsigned long addr, size_t size, bool write,
unsigned long ret_ip)
{
@@ -107,7 +102,7 @@ bool check_memory_region(unsigned long a
if (tag == KASAN_TAG_KERNEL)
return true;
- untagged_addr = reset_tag((const void *)addr);
+ untagged_addr = kasan_reset_tag((const void *)addr);
if (unlikely(untagged_addr <
kasan_shadow_to_mem((void *)KASAN_SHADOW_START))) {
return !kasan_report(addr, size, write, ret_ip);
@@ -126,7 +121,7 @@ bool check_memory_region(unsigned long a
bool check_invalid_free(void *addr)
{
u8 tag = get_tag(addr);
- u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(reset_tag(addr)));
+ u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(kasan_reset_tag(addr)));
return (shadow_byte == KASAN_TAG_INVALID) ||
(tag != KASAN_TAG_KERNEL && tag != shadow_byte);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 49/60] kasan: inline random_tag for HW_TAGS
2020-12-22 19:58 incoming Andrew Morton
` (47 preceding siblings ...)
2020-12-22 20:02 ` [patch 48/60] kasan: inline kasan_reset_tag for tag-based modes Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:02 ` [patch 50/60] kasan: open-code kasan_unpoison_slab Andrew Morton
` (11 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: inline random_tag for HW_TAGS
Using random_tag() currently results in a function call. Move its
definition to mm/kasan/kasan.h and turn it into a static inline function
for hardware tag-based mode to avoid uneeded function calls.
Link: https://lkml.kernel.org/r/be438471690e351e1d792e6bb432e8c03ccb15d3.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Iac5b2faf9a912900e16cca6834d621f5d4abf427
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/hw_tags.c | 5 -----
mm/kasan/kasan.h | 31 ++++++++++++++-----------------
2 files changed, 14 insertions(+), 22 deletions(-)
--- a/mm/kasan/hw_tags.c~kasan-inline-random_tag-for-hw_tags
+++ a/mm/kasan/hw_tags.c
@@ -42,11 +42,6 @@ void unpoison_range(const void *address,
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}
-u8 random_tag(void)
-{
- return hw_get_random_tag();
-}
-
bool check_invalid_free(void *addr)
{
u8 ptr_tag = get_tag(addr);
--- a/mm/kasan/kasan.h~kasan-inline-random_tag-for-hw_tags
+++ a/mm/kasan/kasan.h
@@ -190,6 +190,12 @@ static inline bool addr_has_metadata(con
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+void print_tags(u8 addr_tag, const void *addr);
+#else
+static inline void print_tags(u8 addr_tag, const void *addr) { }
+#endif
+
bool check_invalid_free(void *addr);
void *find_first_bad_addr(void *addr, size_t size);
@@ -225,23 +231,6 @@ static inline void quarantine_reduce(voi
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif
-#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
-
-void print_tags(u8 addr_tag, const void *addr);
-
-u8 random_tag(void);
-
-#else
-
-static inline void print_tags(u8 addr_tag, const void *addr) { }
-
-static inline u8 random_tag(void)
-{
- return 0;
-}
-
-#endif
-
#ifndef arch_kasan_set_tag
static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)
{
@@ -281,6 +270,14 @@ static inline const void *arch_kasan_set
#endif /* CONFIG_KASAN_HW_TAGS */
+#ifdef CONFIG_KASAN_SW_TAGS
+u8 random_tag(void);
+#elif defined(CONFIG_KASAN_HW_TAGS)
+static inline u8 random_tag(void) { return hw_get_random_tag(); }
+#else
+static inline u8 random_tag(void) { return 0; }
+#endif
+
/*
* Exported functions for interfaces called from assembly or from generated
* code. Declarations here to avoid warning about missing declarations.
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 50/60] kasan: open-code kasan_unpoison_slab
2020-12-22 19:58 incoming Andrew Morton
` (48 preceding siblings ...)
2020-12-22 20:02 ` [patch 49/60] kasan: inline random_tag for HW_TAGS Andrew Morton
@ 2020-12-22 20:02 ` Andrew Morton
2020-12-22 20:03 ` [patch 51/60] kasan: inline (un)poison_range and check_invalid_free Andrew Morton
` (10 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:02 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: open-code kasan_unpoison_slab
There's the external annotation kasan_unpoison_slab() that is currently
defined as static inline and uses kasan_unpoison_range(). Open-code this
function in mempool.c. Otherwise with an upcoming change this function
will result in an unnecessary function call.
Link: https://lkml.kernel.org/r/131a6694a978a9a8b150187e539eecc8bcbf759b.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Ia7c8b659f79209935cbaab3913bf7f082cc43a0e
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 6 ------
mm/mempool.c | 2 +-
2 files changed, 1 insertion(+), 7 deletions(-)
--- a/include/linux/kasan.h~kasan-open-code-kasan_unpoison_slab
+++ a/include/linux/kasan.h
@@ -107,11 +107,6 @@ struct kasan_cache {
int free_meta_offset;
};
-size_t __ksize(const void *);
-static inline void kasan_unpoison_slab(const void *ptr)
-{
- kasan_unpoison_range(ptr, __ksize(ptr));
-}
size_t kasan_metadata_size(struct kmem_cache *cache);
bool kasan_save_enable_multi_shot(void);
@@ -167,7 +162,6 @@ static inline bool kasan_slab_free(struc
return false;
}
-static inline void kasan_unpoison_slab(const void *ptr) { }
static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
#endif /* CONFIG_KASAN */
--- a/mm/mempool.c~kasan-open-code-kasan_unpoison_slab
+++ a/mm/mempool.c
@@ -112,7 +112,7 @@ static __always_inline void kasan_poison
static void kasan_unpoison_element(mempool_t *pool, void *element)
{
if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)
- kasan_unpoison_slab(element);
+ kasan_unpoison_range(element, __ksize(element));
else if (pool->alloc == mempool_alloc_pages)
kasan_alloc_pages(element, (unsigned long)pool->pool_data);
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 51/60] kasan: inline (un)poison_range and check_invalid_free
2020-12-22 19:58 incoming Andrew Morton
` (49 preceding siblings ...)
2020-12-22 20:02 ` [patch 50/60] kasan: open-code kasan_unpoison_slab Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 52/60] kasan: add and integrate kasan boot parameters Andrew Morton
` (9 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: inline (un)poison_range and check_invalid_free
Using (un)poison_range() or check_invalid_free() currently results in
function calls. Move their definitions to mm/kasan/kasan.h and turn them
into static inline functions for hardware tag-based mode to avoid
unneeded function calls.
Link: https://lkml.kernel.org/r/7007955b69eb31b5376a7dc1e0f4ac49138504f2.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Ia9d8191024a12d1374675b3d27197f10193f50bb
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/hw_tags.c | 21 ---------------------
mm/kasan/kasan.h | 36 +++++++++++++++++++++++++++++++-----
2 files changed, 31 insertions(+), 26 deletions(-)
--- a/mm/kasan/hw_tags.c~kasan-inline-unpoison_range-and-check_invalid_free
+++ a/mm/kasan/hw_tags.c
@@ -30,27 +30,6 @@ void __init kasan_init_hw_tags(void)
pr_info("KernelAddressSanitizer initialized\n");
}
-void poison_range(const void *address, size_t size, u8 value)
-{
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), value);
-}
-
-void unpoison_range(const void *address, size_t size)
-{
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
-}
-
-bool check_invalid_free(void *addr)
-{
- u8 ptr_tag = get_tag(addr);
- u8 mem_tag = hw_get_mem_tag(addr);
-
- return (mem_tag == KASAN_TAG_INVALID) ||
- (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);
-}
-
void kasan_set_free_info(struct kmem_cache *cache,
void *object, u8 tag)
{
--- a/mm/kasan/kasan.h~kasan-inline-unpoison_range-and-check_invalid_free
+++ a/mm/kasan/kasan.h
@@ -154,9 +154,6 @@ struct kasan_alloc_meta *kasan_get_alloc
struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object);
-void poison_range(const void *address, size_t size, u8 value);
-void unpoison_range(const void *address, size_t size);
-
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
static inline const void *kasan_shadow_to_mem(const void *shadow_addr)
@@ -196,8 +193,6 @@ void print_tags(u8 addr_tag, const void
static inline void print_tags(u8 addr_tag, const void *addr) { }
#endif
-bool check_invalid_free(void *addr);
-
void *find_first_bad_addr(void *addr, size_t size);
const char *get_bug_type(struct kasan_access_info *info);
void metadata_fetch_row(char *buffer, void *row);
@@ -278,6 +273,37 @@ static inline u8 random_tag(void) { retu
static inline u8 random_tag(void) { return 0; }
#endif
+#ifdef CONFIG_KASAN_HW_TAGS
+
+static inline void poison_range(const void *address, size_t size, u8 value)
+{
+ hw_set_mem_tag_range(kasan_reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), value);
+}
+
+static inline void unpoison_range(const void *address, size_t size)
+{
+ hw_set_mem_tag_range(kasan_reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
+}
+
+static inline bool check_invalid_free(void *addr)
+{
+ u8 ptr_tag = get_tag(addr);
+ u8 mem_tag = hw_get_mem_tag(addr);
+
+ return (mem_tag == KASAN_TAG_INVALID) ||
+ (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);
+}
+
+#else /* CONFIG_KASAN_HW_TAGS */
+
+void poison_range(const void *address, size_t size, u8 value);
+void unpoison_range(const void *address, size_t size);
+bool check_invalid_free(void *addr);
+
+#endif /* CONFIG_KASAN_HW_TAGS */
+
/*
* Exported functions for interfaces called from assembly or from generated
* code. Declarations here to avoid warning about missing declarations.
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 52/60] kasan: add and integrate kasan boot parameters
2020-12-22 19:58 incoming Andrew Morton
` (50 preceding siblings ...)
2020-12-22 20:03 ` [patch 51/60] kasan: inline (un)poison_range and check_invalid_free Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 53/60] kasan, mm: check kasan_enabled in annotations Andrew Morton
` (8 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: add and integrate kasan boot parameters
Hardware tag-based KASAN mode is intended to eventually be used in
production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.
This change adds a few boot parameters for hardware tag-based KASAN that
allow to disable or otherwise control particular KASAN features.
The features that can be controlled are:
1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.
With this change a new boot parameter kasan.mode allows to choose one of
three main modes:
- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled
The chosen mode provides default control values for the features mentioned
above. However it's also possible to override the default values by
providing:
- kasan.stacktrace=off/on - enable alloc/free stack collection
(default: on for mode=full, otherwise off)
- kasan.fault=report/panic - only report tag fault or also panic
(default: report)
If kasan.mode parameter is not provided, it defaults to full when
CONFIG_DEBUG_KERNEL is enabled, and to prod otherwise.
It is essential that switching between these modes doesn't require
rebuilding the kernel with different configs, as this is required by
the Android GKI (Generic Kernel Image) initiative [1].
[1] https://source.android.com/devices/architecture/kernel/generic-kernel-image
[andreyknvl@google.com: don't use read-only static keys]
Link: https://lkml.kernel.org/r/f2ded589eba1597f7360a972226083de9afd86e2.1607537948.git.andreyknvl@google.com
Link: https://lkml.kernel.org/r/cb093613879d8d8841173f090133eddeb4c35f1f.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/If7d37003875b2ed3e0935702c8015c223d6416a4
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 22 ++++--
mm/kasan/hw_tags.c | 151 +++++++++++++++++++++++++++++++++++++++++++
mm/kasan/kasan.h | 16 ++++
mm/kasan/report.c | 14 +++
4 files changed, 196 insertions(+), 7 deletions(-)
--- a/mm/kasan/common.c~kasan-add-and-integrate-kasan-boot-parameters
+++ a/mm/kasan/common.c
@@ -134,6 +134,11 @@ void kasan_cache_create(struct kmem_cach
unsigned int redzone_size;
int redzone_adjust;
+ if (!kasan_stack_collection_enabled()) {
+ *flags |= SLAB_KASAN;
+ return;
+ }
+
/* Add alloc meta. */
cache->kasan_info.alloc_meta_offset = *size;
*size += sizeof(struct kasan_alloc_meta);
@@ -170,6 +175,8 @@ void kasan_cache_create(struct kmem_cach
size_t kasan_metadata_size(struct kmem_cache *cache)
{
+ if (!kasan_stack_collection_enabled())
+ return 0;
return (cache->kasan_info.alloc_meta_offset ?
sizeof(struct kasan_alloc_meta) : 0) +
(cache->kasan_info.free_meta_offset ?
@@ -262,11 +269,13 @@ void * __must_check kasan_init_slab_obj(
{
struct kasan_alloc_meta *alloc_meta;
- if (!(cache->flags & SLAB_KASAN))
- return (void *)object;
+ if (kasan_stack_collection_enabled()) {
+ if (!(cache->flags & SLAB_KASAN))
+ return (void *)object;
- alloc_meta = kasan_get_alloc_meta(cache, object);
- __memset(alloc_meta, 0, sizeof(*alloc_meta));
+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));
+ }
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object, assign_tag(cache, object, true, false));
@@ -303,6 +312,9 @@ static bool __kasan_slab_free(struct kme
rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
poison_range(object, rounded_up_size, KASAN_KMALLOC_FREE);
+ if (!kasan_stack_collection_enabled())
+ return false;
+
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
unlikely(!(cache->flags & SLAB_KASAN)))
return false;
@@ -350,7 +362,7 @@ static void *__kasan_kmalloc(struct kmem
poison_range((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);
- if (cache->flags & SLAB_KASAN)
+ if (kasan_stack_collection_enabled() && (cache->flags & SLAB_KASAN))
set_alloc_info(cache, (void *)object, flags);
return set_tag(object, tag);
--- a/mm/kasan/hw_tags.c~kasan-add-and-integrate-kasan-boot-parameters
+++ a/mm/kasan/hw_tags.c
@@ -8,18 +8,115 @@
#define pr_fmt(fmt) "kasan: " fmt
+#include <linux/init.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
#include <linux/memory.h>
#include <linux/mm.h>
+#include <linux/static_key.h>
#include <linux/string.h>
#include <linux/types.h>
#include "kasan.h"
+enum kasan_arg_mode {
+ KASAN_ARG_MODE_DEFAULT,
+ KASAN_ARG_MODE_OFF,
+ KASAN_ARG_MODE_PROD,
+ KASAN_ARG_MODE_FULL,
+};
+
+enum kasan_arg_stacktrace {
+ KASAN_ARG_STACKTRACE_DEFAULT,
+ KASAN_ARG_STACKTRACE_OFF,
+ KASAN_ARG_STACKTRACE_ON,
+};
+
+enum kasan_arg_fault {
+ KASAN_ARG_FAULT_DEFAULT,
+ KASAN_ARG_FAULT_REPORT,
+ KASAN_ARG_FAULT_PANIC,
+};
+
+static enum kasan_arg_mode kasan_arg_mode __ro_after_init;
+static enum kasan_arg_stacktrace kasan_arg_stacktrace __ro_after_init;
+static enum kasan_arg_fault kasan_arg_fault __ro_after_init;
+
+/* Whether KASAN is enabled at all. */
+DEFINE_STATIC_KEY_FALSE(kasan_flag_enabled);
+EXPORT_SYMBOL(kasan_flag_enabled);
+
+/* Whether to collect alloc/free stack traces. */
+DEFINE_STATIC_KEY_FALSE(kasan_flag_stacktrace);
+
+/* Whether panic or disable tag checking on fault. */
+bool kasan_flag_panic __ro_after_init;
+
+/* kasan.mode=off/prod/full */
+static int __init early_kasan_mode(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "off"))
+ kasan_arg_mode = KASAN_ARG_MODE_OFF;
+ else if (!strcmp(arg, "prod"))
+ kasan_arg_mode = KASAN_ARG_MODE_PROD;
+ else if (!strcmp(arg, "full"))
+ kasan_arg_mode = KASAN_ARG_MODE_FULL;
+ else
+ return -EINVAL;
+
+ return 0;
+}
+early_param("kasan.mode", early_kasan_mode);
+
+/* kasan.stack=off/on */
+static int __init early_kasan_flag_stacktrace(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "off"))
+ kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_OFF;
+ else if (!strcmp(arg, "on"))
+ kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_ON;
+ else
+ return -EINVAL;
+
+ return 0;
+}
+early_param("kasan.stacktrace", early_kasan_flag_stacktrace);
+
+/* kasan.fault=report/panic */
+static int __init early_kasan_fault(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "report"))
+ kasan_arg_fault = KASAN_ARG_FAULT_REPORT;
+ else if (!strcmp(arg, "panic"))
+ kasan_arg_fault = KASAN_ARG_FAULT_PANIC;
+ else
+ return -EINVAL;
+
+ return 0;
+}
+early_param("kasan.fault", early_kasan_fault);
+
/* kasan_init_hw_tags_cpu() is called for each CPU. */
void kasan_init_hw_tags_cpu(void)
{
+ /*
+ * There's no need to check that the hardware is MTE-capable here,
+ * as this function is only called for MTE-capable hardware.
+ */
+
+ /* If KASAN is disabled, do nothing. */
+ if (kasan_arg_mode == KASAN_ARG_MODE_OFF)
+ return;
+
hw_init_tags(KASAN_TAG_MAX);
hw_enable_tagging();
}
@@ -27,6 +124,60 @@ void kasan_init_hw_tags_cpu(void)
/* kasan_init_hw_tags() is called once on boot CPU. */
void __init kasan_init_hw_tags(void)
{
+ /* If hardware doesn't support MTE, do nothing. */
+ if (!system_supports_mte())
+ return;
+
+ /* Choose KASAN mode if kasan boot parameter is not provided. */
+ if (kasan_arg_mode == KASAN_ARG_MODE_DEFAULT) {
+ if (IS_ENABLED(CONFIG_DEBUG_KERNEL))
+ kasan_arg_mode = KASAN_ARG_MODE_FULL;
+ else
+ kasan_arg_mode = KASAN_ARG_MODE_PROD;
+ }
+
+ /* Preset parameter values based on the mode. */
+ switch (kasan_arg_mode) {
+ case KASAN_ARG_MODE_DEFAULT:
+ /* Shouldn't happen as per the check above. */
+ WARN_ON(1);
+ return;
+ case KASAN_ARG_MODE_OFF:
+ /* If KASAN is disabled, do nothing. */
+ return;
+ case KASAN_ARG_MODE_PROD:
+ static_branch_enable(&kasan_flag_enabled);
+ break;
+ case KASAN_ARG_MODE_FULL:
+ static_branch_enable(&kasan_flag_enabled);
+ static_branch_enable(&kasan_flag_stacktrace);
+ break;
+ }
+
+ /* Now, optionally override the presets. */
+
+ switch (kasan_arg_stacktrace) {
+ case KASAN_ARG_STACKTRACE_DEFAULT:
+ break;
+ case KASAN_ARG_STACKTRACE_OFF:
+ static_branch_disable(&kasan_flag_stacktrace);
+ break;
+ case KASAN_ARG_STACKTRACE_ON:
+ static_branch_enable(&kasan_flag_stacktrace);
+ break;
+ }
+
+ switch (kasan_arg_fault) {
+ case KASAN_ARG_FAULT_DEFAULT:
+ break;
+ case KASAN_ARG_FAULT_REPORT:
+ kasan_flag_panic = false;
+ break;
+ case KASAN_ARG_FAULT_PANIC:
+ kasan_flag_panic = true;
+ break;
+ }
+
pr_info("KernelAddressSanitizer initialized\n");
}
--- a/mm/kasan/kasan.h~kasan-add-and-integrate-kasan-boot-parameters
+++ a/mm/kasan/kasan.h
@@ -5,6 +5,22 @@
#include <linux/kasan.h>
#include <linux/stackdepot.h>
+#ifdef CONFIG_KASAN_HW_TAGS
+#include <linux/static_key.h>
+DECLARE_STATIC_KEY_FALSE(kasan_flag_stacktrace);
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return static_branch_unlikely(&kasan_flag_stacktrace);
+}
+#else
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return true;
+}
+#endif
+
+extern bool kasan_flag_panic __ro_after_init;
+
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
#else
--- a/mm/kasan/report.c~kasan-add-and-integrate-kasan-boot-parameters
+++ a/mm/kasan/report.c
@@ -99,6 +99,10 @@ static void end_report(unsigned long *fl
panic_on_warn = 0;
panic("panic_on_warn set ...\n");
}
+#ifdef CONFIG_KASAN_HW_TAGS
+ if (kasan_flag_panic)
+ panic("kasan.fault=panic set ...\n");
+#endif
kasan_enable_current();
}
@@ -161,8 +165,8 @@ static void describe_object_addr(struct
(void *)(object_addr + cache->object_size));
}
-static void describe_object(struct kmem_cache *cache, void *object,
- const void *addr, u8 tag)
+static void describe_object_stacks(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
{
struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);
@@ -190,7 +194,13 @@ static void describe_object(struct kmem_
}
#endif
}
+}
+static void describe_object(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
+{
+ if (kasan_stack_collection_enabled())
+ describe_object_stacks(cache, object, addr, tag);
describe_object_addr(cache, object, addr);
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 53/60] kasan, mm: check kasan_enabled in annotations
2020-12-22 19:58 incoming Andrew Morton
` (51 preceding siblings ...)
2020-12-22 20:03 ` [patch 52/60] kasan: add and integrate kasan boot parameters Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 54/60] kasan, mm: rename kasan_poison_kfree Andrew Morton
` (7 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, Vincenzo.Frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, mm: check kasan_enabled in annotations
Declare the kasan_enabled static key in include/linux/kasan.h and in
include/linux/mm.h and check it in all kasan annotations. This allows to
avoid any slowdown caused by function calls when kasan_enabled is
disabled.
Link: https://lkml.kernel.org/r/9f90e3c0aa840dbb4833367c2335193299f69023.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/I2589451d3c96c97abbcbf714baabe6161c6f153e
Co-developed-by: Vincenzo Frascino <Vincenzo.Frascino@arm.com>
Signed-off-by: Vincenzo Frascino <Vincenzo.Frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 217 ++++++++++++++++++++++++++++++----------
include/linux/mm.h | 22 ++--
mm/kasan/common.c | 56 +++++-----
3 files changed, 212 insertions(+), 83 deletions(-)
--- a/include/linux/kasan.h~kasan-mm-check-kasan_enabled-in-annotations
+++ a/include/linux/kasan.h
@@ -2,6 +2,7 @@
#ifndef _LINUX_KASAN_H
#define _LINUX_KASAN_H
+#include <linux/static_key.h>
#include <linux/types.h>
struct kmem_cache;
@@ -75,54 +76,176 @@ static inline void kasan_disable_current
#ifdef CONFIG_KASAN
-void kasan_unpoison_range(const void *address, size_t size);
-
-void kasan_alloc_pages(struct page *page, unsigned int order);
-void kasan_free_pages(struct page *page, unsigned int order);
-
-void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags);
-
-void kasan_poison_slab(struct page *page);
-void kasan_unpoison_object_data(struct kmem_cache *cache, void *object);
-void kasan_poison_object_data(struct kmem_cache *cache, void *object);
-void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
- const void *object);
-
-void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
- gfp_t flags);
-void kasan_kfree_large(void *ptr, unsigned long ip);
-void kasan_poison_kfree(void *ptr, unsigned long ip);
-void * __must_check kasan_kmalloc(struct kmem_cache *s, const void *object,
- size_t size, gfp_t flags);
-void * __must_check kasan_krealloc(const void *object, size_t new_size,
- gfp_t flags);
-
-void * __must_check kasan_slab_alloc(struct kmem_cache *s, void *object,
- gfp_t flags);
-bool kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip);
-
struct kasan_cache {
int alloc_meta_offset;
int free_meta_offset;
};
-size_t kasan_metadata_size(struct kmem_cache *cache);
+#ifdef CONFIG_KASAN_HW_TAGS
+DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
+static __always_inline bool kasan_enabled(void)
+{
+ return static_branch_likely(&kasan_flag_enabled);
+}
+#else
+static inline bool kasan_enabled(void)
+{
+ return true;
+}
+#endif
+
+void __kasan_unpoison_range(const void *addr, size_t size);
+static __always_inline void kasan_unpoison_range(const void *addr, size_t size)
+{
+ if (kasan_enabled())
+ __kasan_unpoison_range(addr, size);
+}
+
+void __kasan_alloc_pages(struct page *page, unsigned int order);
+static __always_inline void kasan_alloc_pages(struct page *page,
+ unsigned int order)
+{
+ if (kasan_enabled())
+ __kasan_alloc_pages(page, order);
+}
+
+void __kasan_free_pages(struct page *page, unsigned int order);
+static __always_inline void kasan_free_pages(struct page *page,
+ unsigned int order)
+{
+ if (kasan_enabled())
+ __kasan_free_pages(page, order);
+}
+
+void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags);
+static __always_inline void kasan_cache_create(struct kmem_cache *cache,
+ unsigned int *size, slab_flags_t *flags)
+{
+ if (kasan_enabled())
+ __kasan_cache_create(cache, size, flags);
+}
+
+size_t __kasan_metadata_size(struct kmem_cache *cache);
+static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache)
+{
+ if (kasan_enabled())
+ return __kasan_metadata_size(cache);
+ return 0;
+}
+
+void __kasan_poison_slab(struct page *page);
+static __always_inline void kasan_poison_slab(struct page *page)
+{
+ if (kasan_enabled())
+ __kasan_poison_slab(page);
+}
+
+void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object);
+static __always_inline void kasan_unpoison_object_data(struct kmem_cache *cache,
+ void *object)
+{
+ if (kasan_enabled())
+ __kasan_unpoison_object_data(cache, object);
+}
+
+void __kasan_poison_object_data(struct kmem_cache *cache, void *object);
+static __always_inline void kasan_poison_object_data(struct kmem_cache *cache,
+ void *object)
+{
+ if (kasan_enabled())
+ __kasan_poison_object_data(cache, object);
+}
+
+void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
+ const void *object);
+static __always_inline void * __must_check kasan_init_slab_obj(
+ struct kmem_cache *cache, const void *object)
+{
+ if (kasan_enabled())
+ return __kasan_init_slab_obj(cache, object);
+ return (void *)object;
+}
+
+bool __kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip);
+static __always_inline bool kasan_slab_free(struct kmem_cache *s, void *object,
+ unsigned long ip)
+{
+ if (kasan_enabled())
+ return __kasan_slab_free(s, object, ip);
+ return false;
+}
+
+void * __must_check __kasan_slab_alloc(struct kmem_cache *s,
+ void *object, gfp_t flags);
+static __always_inline void * __must_check kasan_slab_alloc(
+ struct kmem_cache *s, void *object, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_slab_alloc(s, object, flags);
+ return object;
+}
+
+void * __must_check __kasan_kmalloc(struct kmem_cache *s, const void *object,
+ size_t size, gfp_t flags);
+static __always_inline void * __must_check kasan_kmalloc(struct kmem_cache *s,
+ const void *object, size_t size, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_kmalloc(s, object, size, flags);
+ return (void *)object;
+}
+
+void * __must_check __kasan_kmalloc_large(const void *ptr,
+ size_t size, gfp_t flags);
+static __always_inline void * __must_check kasan_kmalloc_large(const void *ptr,
+ size_t size, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_kmalloc_large(ptr, size, flags);
+ return (void *)ptr;
+}
+
+void * __must_check __kasan_krealloc(const void *object,
+ size_t new_size, gfp_t flags);
+static __always_inline void * __must_check kasan_krealloc(const void *object,
+ size_t new_size, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_krealloc(object, new_size, flags);
+ return (void *)object;
+}
+
+void __kasan_poison_kfree(void *ptr, unsigned long ip);
+static __always_inline void kasan_poison_kfree(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())
+ __kasan_poison_kfree(ptr, ip);
+}
+
+void __kasan_kfree_large(void *ptr, unsigned long ip);
+static __always_inline void kasan_kfree_large(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())
+ __kasan_kfree_large(ptr, ip);
+}
bool kasan_save_enable_multi_shot(void);
void kasan_restore_multi_shot(bool enabled);
#else /* CONFIG_KASAN */
+static inline bool kasan_enabled(void)
+{
+ return false;
+}
static inline void kasan_unpoison_range(const void *address, size_t size) {}
-
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}
-
static inline void kasan_cache_create(struct kmem_cache *cache,
unsigned int *size,
slab_flags_t *flags) {}
-
+static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
static inline void kasan_poison_slab(struct page *page) {}
static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
void *object) {}
@@ -133,36 +256,32 @@ static inline void *kasan_init_slab_obj(
{
return (void *)object;
}
-
-static inline void *kasan_kmalloc_large(void *ptr, size_t size, gfp_t flags)
+static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
+ unsigned long ip)
{
- return ptr;
+ return false;
+}
+static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,
+ gfp_t flags)
+{
+ return object;
}
-static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}
-static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}
static inline void *kasan_kmalloc(struct kmem_cache *s, const void *object,
size_t size, gfp_t flags)
{
return (void *)object;
}
+static inline void *kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags)
+{
+ return (void *)ptr;
+}
static inline void *kasan_krealloc(const void *object, size_t new_size,
gfp_t flags)
{
return (void *)object;
}
-
-static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,
- gfp_t flags)
-{
- return object;
-}
-static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
- unsigned long ip)
-{
- return false;
-}
-
-static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
+static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}
+static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}
#endif /* CONFIG_KASAN */
--- a/include/linux/mm.h~kasan-mm-check-kasan_enabled-in-annotations
+++ a/include/linux/mm.h
@@ -31,6 +31,7 @@
#include <linux/sizes.h>
#include <linux/sched.h>
#include <linux/pgtable.h>
+#include <linux/kasan.h>
struct mempolicy;
struct anon_vma;
@@ -1422,22 +1423,30 @@ static inline bool cpupid_match_pid(stru
#endif /* CONFIG_NUMA_BALANCING */
#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+
static inline u8 page_kasan_tag(const struct page *page)
{
- return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
+ if (kasan_enabled())
+ return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
+ return 0xff;
}
static inline void page_kasan_tag_set(struct page *page, u8 tag)
{
- page->flags &= ~(KASAN_TAG_MASK << KASAN_TAG_PGSHIFT);
- page->flags |= (tag & KASAN_TAG_MASK) << KASAN_TAG_PGSHIFT;
+ if (kasan_enabled()) {
+ page->flags &= ~(KASAN_TAG_MASK << KASAN_TAG_PGSHIFT);
+ page->flags |= (tag & KASAN_TAG_MASK) << KASAN_TAG_PGSHIFT;
+ }
}
static inline void page_kasan_tag_reset(struct page *page)
{
- page_kasan_tag_set(page, 0xff);
+ if (kasan_enabled())
+ page_kasan_tag_set(page, 0xff);
}
-#else
+
+#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
+
static inline u8 page_kasan_tag(const struct page *page)
{
return 0xff;
@@ -1445,7 +1454,8 @@ static inline u8 page_kasan_tag(const st
static inline void page_kasan_tag_set(struct page *page, u8 tag) { }
static inline void page_kasan_tag_reset(struct page *page) { }
-#endif
+
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
static inline struct zone *page_zone(const struct page *page)
{
--- a/mm/kasan/common.c~kasan-mm-check-kasan_enabled-in-annotations
+++ a/mm/kasan/common.c
@@ -58,7 +58,7 @@ void kasan_disable_current(void)
}
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
-void kasan_unpoison_range(const void *address, size_t size)
+void __kasan_unpoison_range(const void *address, size_t size)
{
unpoison_range(address, size);
}
@@ -86,7 +86,7 @@ asmlinkage void kasan_unpoison_task_stac
}
#endif /* CONFIG_KASAN_STACK */
-void kasan_alloc_pages(struct page *page, unsigned int order)
+void __kasan_alloc_pages(struct page *page, unsigned int order)
{
u8 tag;
unsigned long i;
@@ -100,7 +100,7 @@ void kasan_alloc_pages(struct page *page
unpoison_range(page_address(page), PAGE_SIZE << order);
}
-void kasan_free_pages(struct page *page, unsigned int order)
+void __kasan_free_pages(struct page *page, unsigned int order)
{
if (likely(!PageHighMem(page)))
poison_range(page_address(page),
@@ -127,8 +127,8 @@ static inline unsigned int optimal_redzo
object_size <= (1 << 16) - 1024 ? 1024 : 2048;
}
-void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags)
+void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags)
{
unsigned int orig_size = *size;
unsigned int redzone_size;
@@ -173,7 +173,7 @@ void kasan_cache_create(struct kmem_cach
*flags |= SLAB_KASAN;
}
-size_t kasan_metadata_size(struct kmem_cache *cache)
+size_t __kasan_metadata_size(struct kmem_cache *cache)
{
if (!kasan_stack_collection_enabled())
return 0;
@@ -196,7 +196,7 @@ struct kasan_free_meta *kasan_get_free_m
return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}
-void kasan_poison_slab(struct page *page)
+void __kasan_poison_slab(struct page *page)
{
unsigned long i;
@@ -206,12 +206,12 @@ void kasan_poison_slab(struct page *page
KASAN_KMALLOC_REDZONE);
}
-void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
+void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
{
unpoison_range(object, cache->object_size);
}
-void kasan_poison_object_data(struct kmem_cache *cache, void *object)
+void __kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
poison_range(object,
round_up(cache->object_size, KASAN_GRANULE_SIZE),
@@ -264,7 +264,7 @@ static u8 assign_tag(struct kmem_cache *
#endif
}
-void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
+void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
const void *object)
{
struct kasan_alloc_meta *alloc_meta;
@@ -283,7 +283,7 @@ void * __must_check kasan_init_slab_obj(
return (void *)object;
}
-static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
+static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
unsigned long ip, bool quarantine)
{
u8 tag;
@@ -326,9 +326,9 @@ static bool __kasan_slab_free(struct kme
return IS_ENABLED(CONFIG_KASAN_GENERIC);
}
-bool kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
+bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
{
- return __kasan_slab_free(cache, object, ip, true);
+ return ____kasan_slab_free(cache, object, ip, true);
}
static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
@@ -336,7 +336,7 @@ static void set_alloc_info(struct kmem_c
kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
}
-static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
+static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,
size_t size, gfp_t flags, bool keep_tag)
{
unsigned long redzone_start;
@@ -368,20 +368,20 @@ static void *__kasan_kmalloc(struct kmem
return set_tag(object, tag);
}
-void * __must_check kasan_slab_alloc(struct kmem_cache *cache, void *object,
- gfp_t flags)
+void * __must_check __kasan_slab_alloc(struct kmem_cache *cache,
+ void *object, gfp_t flags)
{
- return __kasan_kmalloc(cache, object, cache->object_size, flags, false);
+ return ____kasan_kmalloc(cache, object, cache->object_size, flags, false);
}
-void * __must_check kasan_kmalloc(struct kmem_cache *cache, const void *object,
- size_t size, gfp_t flags)
+void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *object,
+ size_t size, gfp_t flags)
{
- return __kasan_kmalloc(cache, object, size, flags, true);
+ return ____kasan_kmalloc(cache, object, size, flags, true);
}
-EXPORT_SYMBOL(kasan_kmalloc);
+EXPORT_SYMBOL(__kasan_kmalloc);
-void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
+void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size,
gfp_t flags)
{
struct page *page;
@@ -406,7 +406,7 @@ void * __must_check kasan_kmalloc_large(
return (void *)ptr;
}
-void * __must_check kasan_krealloc(const void *object, size_t size, gfp_t flags)
+void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flags)
{
struct page *page;
@@ -416,13 +416,13 @@ void * __must_check kasan_krealloc(const
page = virt_to_head_page(object);
if (unlikely(!PageSlab(page)))
- return kasan_kmalloc_large(object, size, flags);
+ return __kasan_kmalloc_large(object, size, flags);
else
- return __kasan_kmalloc(page->slab_cache, object, size,
+ return ____kasan_kmalloc(page->slab_cache, object, size,
flags, true);
}
-void kasan_poison_kfree(void *ptr, unsigned long ip)
+void __kasan_poison_kfree(void *ptr, unsigned long ip)
{
struct page *page;
@@ -435,11 +435,11 @@ void kasan_poison_kfree(void *ptr, unsig
}
poison_range(ptr, page_size(page), KASAN_FREE_PAGE);
} else {
- __kasan_slab_free(page->slab_cache, ptr, ip, false);
+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);
}
}
-void kasan_kfree_large(void *ptr, unsigned long ip)
+void __kasan_kfree_large(void *ptr, unsigned long ip)
{
if (ptr != page_address(virt_to_head_page(ptr)))
kasan_report_invalid_free(ptr, ip);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 54/60] kasan, mm: rename kasan_poison_kfree
2020-12-22 19:58 incoming Andrew Morton
` (52 preceding siblings ...)
2020-12-22 20:03 ` [patch 53/60] kasan, mm: check kasan_enabled in annotations Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 55/60] kasan: don't round_up too much Andrew Morton
` (6 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, mm: rename kasan_poison_kfree
Rename kasan_poison_kfree() to kasan_slab_free_mempool() as it better
reflects what this annotation does. Also add a comment that explains the
PageSlab() check.
No functional changes.
Link: https://lkml.kernel.org/r/141675fb493555e984c5dca555e9d9f768c7bbaa.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/I5026f87364e556b506ef1baee725144bb04b8810
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 16 ++++++++--------
mm/kasan/common.c | 40 +++++++++++++++++++++++-----------------
mm/mempool.c | 2 +-
3 files changed, 32 insertions(+), 26 deletions(-)
--- a/include/linux/kasan.h~kasan-mm-rename-kasan_poison_kfree
+++ a/include/linux/kasan.h
@@ -176,6 +176,13 @@ static __always_inline bool kasan_slab_f
return false;
}
+void __kasan_slab_free_mempool(void *ptr, unsigned long ip);
+static __always_inline void kasan_slab_free_mempool(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())
+ __kasan_slab_free_mempool(ptr, ip);
+}
+
void * __must_check __kasan_slab_alloc(struct kmem_cache *s,
void *object, gfp_t flags);
static __always_inline void * __must_check kasan_slab_alloc(
@@ -216,13 +223,6 @@ static __always_inline void * __must_che
return (void *)object;
}
-void __kasan_poison_kfree(void *ptr, unsigned long ip);
-static __always_inline void kasan_poison_kfree(void *ptr, unsigned long ip)
-{
- if (kasan_enabled())
- __kasan_poison_kfree(ptr, ip);
-}
-
void __kasan_kfree_large(void *ptr, unsigned long ip);
static __always_inline void kasan_kfree_large(void *ptr, unsigned long ip)
{
@@ -261,6 +261,7 @@ static inline bool kasan_slab_free(struc
{
return false;
}
+static inline void kasan_slab_free_mempool(void *ptr, unsigned long ip) {}
static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,
gfp_t flags)
{
@@ -280,7 +281,6 @@ static inline void *kasan_krealloc(const
{
return (void *)object;
}
-static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}
static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}
#endif /* CONFIG_KASAN */
--- a/mm/kasan/common.c~kasan-mm-rename-kasan_poison_kfree
+++ a/mm/kasan/common.c
@@ -331,6 +331,29 @@ bool __kasan_slab_free(struct kmem_cache
return ____kasan_slab_free(cache, object, ip, true);
}
+void __kasan_slab_free_mempool(void *ptr, unsigned long ip)
+{
+ struct page *page;
+
+ page = virt_to_head_page(ptr);
+
+ /*
+ * Even though this function is only called for kmem_cache_alloc and
+ * kmalloc backed mempool allocations, those allocations can still be
+ * !PageSlab() when the size provided to kmalloc is larger than
+ * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc.
+ */
+ if (unlikely(!PageSlab(page))) {
+ if (ptr != page_address(page)) {
+ kasan_report_invalid_free(ptr, ip);
+ return;
+ }
+ poison_range(ptr, page_size(page), KASAN_FREE_PAGE);
+ } else {
+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);
+ }
+}
+
static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
{
kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
@@ -422,23 +445,6 @@ void * __must_check __kasan_krealloc(con
flags, true);
}
-void __kasan_poison_kfree(void *ptr, unsigned long ip)
-{
- struct page *page;
-
- page = virt_to_head_page(ptr);
-
- if (unlikely(!PageSlab(page))) {
- if (ptr != page_address(page)) {
- kasan_report_invalid_free(ptr, ip);
- return;
- }
- poison_range(ptr, page_size(page), KASAN_FREE_PAGE);
- } else {
- ____kasan_slab_free(page->slab_cache, ptr, ip, false);
- }
-}
-
void __kasan_kfree_large(void *ptr, unsigned long ip)
{
if (ptr != page_address(virt_to_head_page(ptr)))
--- a/mm/mempool.c~kasan-mm-rename-kasan_poison_kfree
+++ a/mm/mempool.c
@@ -104,7 +104,7 @@ static inline void poison_element(mempoo
static __always_inline void kasan_poison_element(mempool_t *pool, void *element)
{
if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)
- kasan_poison_kfree(element, _RET_IP_);
+ kasan_slab_free_mempool(element, _RET_IP_);
else if (pool->alloc == mempool_alloc_pages)
kasan_free_pages(element, (unsigned long)pool->pool_data);
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 55/60] kasan: don't round_up too much
2020-12-22 19:58 incoming Andrew Morton
` (53 preceding siblings ...)
2020-12-22 20:03 ` [patch 54/60] kasan, mm: rename kasan_poison_kfree Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 56/60] kasan: simplify assign_tag and set_tag calls Andrew Morton
` (5 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: don't round_up too much
For hardware tag-based mode kasan_poison_memory() already rounds up the
size. Do the same for software modes and remove round_up() from the common
code.
Link: https://lkml.kernel.org/r/47b232474f1f89dc072aeda0fa58daa6efade377.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Ib397128fac6eba874008662b4964d65352db4aa4
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 8 ++------
mm/kasan/shadow.c | 1 +
2 files changed, 3 insertions(+), 6 deletions(-)
--- a/mm/kasan/common.c~kasan-dont-round_up-too-much
+++ a/mm/kasan/common.c
@@ -213,9 +213,7 @@ void __kasan_unpoison_object_data(struct
void __kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
- poison_range(object,
- round_up(cache->object_size, KASAN_GRANULE_SIZE),
- KASAN_KMALLOC_REDZONE);
+ poison_range(object, cache->object_size, KASAN_KMALLOC_REDZONE);
}
/*
@@ -288,7 +286,6 @@ static bool ____kasan_slab_free(struct k
{
u8 tag;
void *tagged_object;
- unsigned long rounded_up_size;
tag = get_tag(object);
tagged_object = object;
@@ -309,8 +306,7 @@ static bool ____kasan_slab_free(struct k
return true;
}
- rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
- poison_range(object, rounded_up_size, KASAN_KMALLOC_FREE);
+ poison_range(object, cache->object_size, KASAN_KMALLOC_FREE);
if (!kasan_stack_collection_enabled())
return false;
--- a/mm/kasan/shadow.c~kasan-dont-round_up-too-much
+++ a/mm/kasan/shadow.c
@@ -82,6 +82,7 @@ void poison_range(const void *address, s
* addresses to this function.
*/
address = kasan_reset_tag(address);
+ size = round_up(size, KASAN_GRANULE_SIZE);
shadow_start = kasan_mem_to_shadow(address);
shadow_end = kasan_mem_to_shadow(address + size);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 56/60] kasan: simplify assign_tag and set_tag calls
2020-12-22 19:58 incoming Andrew Morton
` (54 preceding siblings ...)
2020-12-22 20:03 ` [patch 55/60] kasan: don't round_up too much Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 57/60] kasan: clarify comment in __kasan_kfree_large Andrew Morton
` (4 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: simplify assign_tag and set_tag calls
set_tag() already ignores the tag for the generic mode, so just call it
as is. Add a check for the generic mode to assign_tag(), and simplify its
call in ____kasan_kmalloc().
Link: https://lkml.kernel.org/r/121eeab245f98555862b289d2ba9269c868fbbcf.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/I18905ca78fb4a3d60e1a34a4ca00247272480438
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
--- a/mm/kasan/common.c~kasan-simplify-assign_tag-and-set_tag-calls
+++ a/mm/kasan/common.c
@@ -233,6 +233,9 @@ void __kasan_poison_object_data(struct k
static u8 assign_tag(struct kmem_cache *cache, const void *object,
bool init, bool keep_tag)
{
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC))
+ return 0xff;
+
/*
* 1. When an object is kmalloc()'ed, two hooks are called:
* kasan_slab_alloc() and kasan_kmalloc(). We assign the
@@ -275,8 +278,8 @@ void * __must_check __kasan_init_slab_ob
__memset(alloc_meta, 0, sizeof(*alloc_meta));
}
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
- object = set_tag(object, assign_tag(cache, object, true, false));
+ /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */
+ object = set_tag(object, assign_tag(cache, object, true, false));
return (void *)object;
}
@@ -360,7 +363,7 @@ static void *____kasan_kmalloc(struct km
{
unsigned long redzone_start;
unsigned long redzone_end;
- u8 tag = 0xff;
+ u8 tag;
if (gfpflags_allow_blocking(flags))
quarantine_reduce();
@@ -372,9 +375,7 @@ static void *____kasan_kmalloc(struct km
KASAN_GRANULE_SIZE);
redzone_end = round_up((unsigned long)object + cache->object_size,
KASAN_GRANULE_SIZE);
-
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
- tag = assign_tag(cache, object, false, keep_tag);
+ tag = assign_tag(cache, object, false, keep_tag);
/* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */
unpoison_range(set_tag(object, tag), size);
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 57/60] kasan: clarify comment in __kasan_kfree_large
2020-12-22 19:58 incoming Andrew Morton
` (55 preceding siblings ...)
2020-12-22 20:03 ` [patch 56/60] kasan: simplify assign_tag and set_tag calls Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 58/60] kasan: sanitize objects when metadata doesn't fit Andrew Morton
` (3 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: clarify comment in __kasan_kfree_large
Currently it says that the memory gets poisoned by page_alloc code.
Clarify this by mentioning the specific callback that poisons the memory.
Link: https://lkml.kernel.org/r/1c8380fe0332a3bcc720fe29f1e0bef2e2974416.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/I1334dffb69b87d7986fab88a1a039cc3ea764725
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/mm/kasan/common.c~kasan-clarify-comment-in-__kasan_kfree_large
+++ a/mm/kasan/common.c
@@ -446,5 +446,5 @@ void __kasan_kfree_large(void *ptr, unsi
{
if (ptr != page_address(virt_to_head_page(ptr)))
kasan_report_invalid_free(ptr, ip);
- /* The object will be poisoned by page_alloc. */
+ /* The object will be poisoned by kasan_free_pages(). */
}
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 58/60] kasan: sanitize objects when metadata doesn't fit
2020-12-22 19:58 incoming Andrew Morton
` (56 preceding siblings ...)
2020-12-22 20:03 ` [patch 57/60] kasan: clarify comment in __kasan_kfree_large Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 59/60] kasan, mm: allow cache merging with no metadata Andrew Morton
` (2 subsequent siblings)
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, Vincenzo.Frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: sanitize objects when metadata doesn't fit
KASAN marks caches that are sanitized with the SLAB_KASAN cache flag.
Currently if the metadata that is appended after the object (stores e.g.
stack trace ids) doesn't fit into KMALLOC_MAX_SIZE (can only happen with
SLAB, see the comment in the patch), KASAN turns off sanitization
completely.
With this change sanitization of the object data is always enabled.
However the metadata is only stored when it fits. Instead of checking for
SLAB_KASAN flag accross the code to find out whether the metadata is
there, use cache->kasan_info.alloc/free_meta_offset. As 0 can be a valid
value for free_meta_offset, introduce KASAN_NO_FREE_META as an indicator
that the free metadata is missing.
Without this change all sanitized KASAN objects would be put into
quarantine with generic KASAN. With this change, only the objects that
have metadata (i.e. when it fits) are put into quarantine, the rest is
freed right away.
Along the way rework __kasan_cache_create() and add claryfying comments.
Link: https://lkml.kernel.org/r/aee34b87a5e4afe586c2ac6a0b32db8dc4dcc2dc.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Icd947e2bea054cb5cfbdc6cf6652227d97032dcb
Co-developed-by: Vincenzo Frascino <Vincenzo.Frascino@arm.com>
Signed-off-by: Vincenzo Frascino <Vincenzo.Frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/kasan/common.c | 116 ++++++++++++++++++++++--------------
mm/kasan/generic.c | 9 +-
mm/kasan/hw_tags.c | 6 +
mm/kasan/kasan.h | 17 ++++-
mm/kasan/quarantine.c | 18 ++++-
mm/kasan/report.c | 43 +++++++------
mm/kasan/report_sw_tags.c | 9 +-
mm/kasan/sw_tags.c | 4 +
8 files changed, 147 insertions(+), 75 deletions(-)
--- a/mm/kasan/common.c~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/common.c
@@ -114,9 +114,6 @@ void __kasan_free_pages(struct page *pag
*/
static inline unsigned int optimal_redzone(unsigned int object_size)
{
- if (!IS_ENABLED(CONFIG_KASAN_GENERIC))
- return 0;
-
return
object_size <= 64 - 16 ? 16 :
object_size <= 128 - 32 ? 32 :
@@ -130,47 +127,77 @@ static inline unsigned int optimal_redzo
void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
slab_flags_t *flags)
{
- unsigned int orig_size = *size;
- unsigned int redzone_size;
- int redzone_adjust;
+ unsigned int ok_size;
+ unsigned int optimal_size;
+
+ /*
+ * SLAB_KASAN is used to mark caches as ones that are sanitized by
+ * KASAN. Currently this flag is used in two places:
+ * 1. In slab_ksize() when calculating the size of the accessible
+ * memory within the object.
+ * 2. In slab_common.c to prevent merging of sanitized caches.
+ */
+ *flags |= SLAB_KASAN;
- if (!kasan_stack_collection_enabled()) {
- *flags |= SLAB_KASAN;
+ if (!kasan_stack_collection_enabled())
return;
- }
- /* Add alloc meta. */
+ ok_size = *size;
+
+ /* Add alloc meta into redzone. */
cache->kasan_info.alloc_meta_offset = *size;
*size += sizeof(struct kasan_alloc_meta);
- /* Add free meta. */
- if (IS_ENABLED(CONFIG_KASAN_GENERIC) &&
- (cache->flags & SLAB_TYPESAFE_BY_RCU || cache->ctor ||
- cache->object_size < sizeof(struct kasan_free_meta))) {
- cache->kasan_info.free_meta_offset = *size;
- *size += sizeof(struct kasan_free_meta);
+ /*
+ * If alloc meta doesn't fit, don't add it.
+ * This can only happen with SLAB, as it has KMALLOC_MAX_SIZE equal
+ * to KMALLOC_MAX_CACHE_SIZE and doesn't fall back to page_alloc for
+ * larger sizes.
+ */
+ if (*size > KMALLOC_MAX_SIZE) {
+ cache->kasan_info.alloc_meta_offset = 0;
+ *size = ok_size;
+ /* Continue, since free meta might still fit. */
}
- redzone_size = optimal_redzone(cache->object_size);
- redzone_adjust = redzone_size - (*size - cache->object_size);
- if (redzone_adjust > 0)
- *size += redzone_adjust;
-
- *size = min_t(unsigned int, KMALLOC_MAX_SIZE,
- max(*size, cache->object_size + redzone_size));
+ /* Only the generic mode uses free meta or flexible redzones. */
+ if (!IS_ENABLED(CONFIG_KASAN_GENERIC)) {
+ cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
+ return;
+ }
/*
- * If the metadata doesn't fit, don't enable KASAN at all.
+ * Add free meta into redzone when it's not possible to store
+ * it in the object. This is the case when:
+ * 1. Object is SLAB_TYPESAFE_BY_RCU, which means that it can
+ * be touched after it was freed, or
+ * 2. Object has a constructor, which means it's expected to
+ * retain its content until the next allocation, or
+ * 3. Object is too small.
+ * Otherwise cache->kasan_info.free_meta_offset = 0 is implied.
*/
- if (*size <= cache->kasan_info.alloc_meta_offset ||
- *size <= cache->kasan_info.free_meta_offset) {
- cache->kasan_info.alloc_meta_offset = 0;
- cache->kasan_info.free_meta_offset = 0;
- *size = orig_size;
- return;
+ if ((cache->flags & SLAB_TYPESAFE_BY_RCU) || cache->ctor ||
+ cache->object_size < sizeof(struct kasan_free_meta)) {
+ ok_size = *size;
+
+ cache->kasan_info.free_meta_offset = *size;
+ *size += sizeof(struct kasan_free_meta);
+
+ /* If free meta doesn't fit, don't add it. */
+ if (*size > KMALLOC_MAX_SIZE) {
+ cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
+ *size = ok_size;
+ }
}
- *flags |= SLAB_KASAN;
+ /* Calculate size with optimal redzone. */
+ optimal_size = cache->object_size + optimal_redzone(cache->object_size);
+ /* Limit it with KMALLOC_MAX_SIZE (relevant for SLAB only). */
+ if (optimal_size > KMALLOC_MAX_SIZE)
+ optimal_size = KMALLOC_MAX_SIZE;
+ /* Use optimal size if the size with added metas is not large enough. */
+ if (*size < optimal_size)
+ *size = optimal_size;
}
size_t __kasan_metadata_size(struct kmem_cache *cache)
@@ -186,15 +213,21 @@ size_t __kasan_metadata_size(struct kmem
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{
+ if (!cache->kasan_info.alloc_meta_offset)
+ return NULL;
return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}
+#ifdef CONFIG_KASAN_GENERIC
struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
+ if (cache->kasan_info.free_meta_offset == KASAN_NO_FREE_META)
+ return NULL;
return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}
+#endif
void __kasan_poison_slab(struct page *page)
{
@@ -271,11 +304,9 @@ void * __must_check __kasan_init_slab_ob
struct kasan_alloc_meta *alloc_meta;
if (kasan_stack_collection_enabled()) {
- if (!(cache->flags & SLAB_KASAN))
- return (void *)object;
-
alloc_meta = kasan_get_alloc_meta(cache, object);
- __memset(alloc_meta, 0, sizeof(*alloc_meta));
+ if (alloc_meta)
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));
}
/* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */
@@ -314,15 +345,12 @@ static bool ____kasan_slab_free(struct k
if (!kasan_stack_collection_enabled())
return false;
- if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
- unlikely(!(cache->flags & SLAB_KASAN)))
+ if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine))
return false;
kasan_set_free_info(cache, object, tag);
- quarantine_put(cache, object);
-
- return IS_ENABLED(CONFIG_KASAN_GENERIC);
+ return quarantine_put(cache, object);
}
bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
@@ -355,7 +383,11 @@ void __kasan_slab_free_mempool(void *ptr
static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
{
- kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
+ struct kasan_alloc_meta *alloc_meta;
+
+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (alloc_meta)
+ kasan_set_track(&alloc_meta->alloc_track, flags);
}
static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,
@@ -382,7 +414,7 @@ static void *____kasan_kmalloc(struct km
poison_range((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);
- if (kasan_stack_collection_enabled() && (cache->flags & SLAB_KASAN))
+ if (kasan_stack_collection_enabled())
set_alloc_info(cache, (void *)object, flags);
return set_tag(object, tag);
--- a/mm/kasan/generic.c~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/generic.c
@@ -348,11 +348,11 @@ void kasan_set_free_info(struct kmem_cac
struct kasan_free_meta *free_meta;
free_meta = kasan_get_free_meta(cache, object);
- kasan_set_track(&free_meta->free_track, GFP_NOWAIT);
+ if (!free_meta)
+ return;
- /*
- * the object was freed and has free track set
- */
+ kasan_set_track(&free_meta->free_track, GFP_NOWAIT);
+ /* The object was freed and has free track set. */
*(u8 *)kasan_mem_to_shadow(object) = KASAN_KMALLOC_FREETRACK;
}
@@ -361,5 +361,6 @@ struct kasan_track *kasan_get_free_track
{
if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_KMALLOC_FREETRACK)
return NULL;
+ /* Free meta must be present with KASAN_KMALLOC_FREETRACK. */
return &kasan_get_free_meta(cache, object)->free_track;
}
--- a/mm/kasan/hw_tags.c~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/hw_tags.c
@@ -187,7 +187,8 @@ void kasan_set_free_info(struct kmem_cac
struct kasan_alloc_meta *alloc_meta;
alloc_meta = kasan_get_alloc_meta(cache, object);
- kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
+ if (alloc_meta)
+ kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
}
struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
@@ -196,5 +197,8 @@ struct kasan_track *kasan_get_free_track
struct kasan_alloc_meta *alloc_meta;
alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return NULL;
+
return &alloc_meta->free_track[0];
}
--- a/mm/kasan/kasan.h~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/kasan.h
@@ -155,20 +155,31 @@ struct kasan_alloc_meta {
struct qlist_node {
struct qlist_node *next;
};
+
+/*
+ * Generic mode either stores free meta in the object itself or in the redzone
+ * after the object. In the former case free meta offset is 0, in the latter
+ * case it has some sane value smaller than INT_MAX. Use INT_MAX as free meta
+ * offset when free meta isn't present.
+ */
+#define KASAN_NO_FREE_META INT_MAX
+
struct kasan_free_meta {
+#ifdef CONFIG_KASAN_GENERIC
/* This field is used while the object is in the quarantine.
* Otherwise it might be used for the allocator freelist.
*/
struct qlist_node quarantine_link;
-#ifdef CONFIG_KASAN_GENERIC
struct kasan_track free_track;
#endif
};
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object);
+#ifdef CONFIG_KASAN_GENERIC
struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object);
+#endif
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
@@ -233,11 +244,11 @@ struct kasan_track *kasan_get_free_track
#if defined(CONFIG_KASAN_GENERIC) && \
(defined(CONFIG_SLAB) || defined(CONFIG_SLUB))
-void quarantine_put(struct kmem_cache *cache, void *object);
+bool quarantine_put(struct kmem_cache *cache, void *object);
void quarantine_reduce(void);
void quarantine_remove_cache(struct kmem_cache *cache);
#else
-static inline void quarantine_put(struct kmem_cache *cache, void *object) { }
+static inline bool quarantine_put(struct kmem_cache *cache, void *object) { return false; }
static inline void quarantine_reduce(void) { }
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif
--- a/mm/kasan/quarantine.c~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/quarantine.c
@@ -137,7 +137,12 @@ static void qlink_free(struct qlist_node
if (IS_ENABLED(CONFIG_SLAB))
local_irq_save(flags);
+ /*
+ * As the object now gets freed from the quaratine, assume that its
+ * free track is no longer valid.
+ */
*(u8 *)kasan_mem_to_shadow(object) = KASAN_KMALLOC_FREE;
+
___cache_free(cache, object, _THIS_IP_);
if (IS_ENABLED(CONFIG_SLAB))
@@ -163,7 +168,7 @@ static void qlist_free_all(struct qlist_
qlist_init(q);
}
-void quarantine_put(struct kmem_cache *cache, void *object)
+bool quarantine_put(struct kmem_cache *cache, void *object)
{
unsigned long flags;
struct qlist_head *q;
@@ -171,6 +176,13 @@ void quarantine_put(struct kmem_cache *c
struct kasan_free_meta *meta = kasan_get_free_meta(cache, object);
/*
+ * If there's no metadata for this object, don't put it into
+ * quarantine.
+ */
+ if (!meta)
+ return false;
+
+ /*
* Note: irq must be disabled until after we move the batch to the
* global quarantine. Otherwise quarantine_remove_cache() can miss
* some objects belonging to the cache if they are in our local temp
@@ -183,7 +195,7 @@ void quarantine_put(struct kmem_cache *c
q = this_cpu_ptr(&cpu_quarantine);
if (q->offline) {
local_irq_restore(flags);
- return;
+ return false;
}
qlist_put(q, &meta->quarantine_link, cache->size);
if (unlikely(q->bytes > QUARANTINE_PERCPU_SIZE)) {
@@ -206,6 +218,8 @@ void quarantine_put(struct kmem_cache *c
}
local_irq_restore(flags);
+
+ return true;
}
void quarantine_reduce(void)
--- a/mm/kasan/report.c~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/report.c
@@ -168,32 +168,35 @@ static void describe_object_addr(struct
static void describe_object_stacks(struct kmem_cache *cache, void *object,
const void *addr, u8 tag)
{
- struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);
-
- if (cache->flags & SLAB_KASAN) {
- struct kasan_track *free_track;
+ struct kasan_alloc_meta *alloc_meta;
+ struct kasan_track *free_track;
+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (alloc_meta) {
print_track(&alloc_meta->alloc_track, "Allocated");
pr_err("\n");
- free_track = kasan_get_free_track(cache, object, tag);
- if (free_track) {
- print_track(free_track, "Freed");
- pr_err("\n");
- }
+ }
+
+ free_track = kasan_get_free_track(cache, object, tag);
+ if (free_track) {
+ print_track(free_track, "Freed");
+ pr_err("\n");
+ }
#ifdef CONFIG_KASAN_GENERIC
- if (alloc_meta->aux_stack[0]) {
- pr_err("Last potentially related work creation:\n");
- print_stack(alloc_meta->aux_stack[0]);
- pr_err("\n");
- }
- if (alloc_meta->aux_stack[1]) {
- pr_err("Second to last potentially related work creation:\n");
- print_stack(alloc_meta->aux_stack[1]);
- pr_err("\n");
- }
-#endif
+ if (!alloc_meta)
+ return;
+ if (alloc_meta->aux_stack[0]) {
+ pr_err("Last potentially related work creation:\n");
+ print_stack(alloc_meta->aux_stack[0]);
+ pr_err("\n");
}
+ if (alloc_meta->aux_stack[1]) {
+ pr_err("Second to last potentially related work creation:\n");
+ print_stack(alloc_meta->aux_stack[1]);
+ pr_err("\n");
+ }
+#endif
}
static void describe_object(struct kmem_cache *cache, void *object,
--- a/mm/kasan/report_sw_tags.c~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/report_sw_tags.c
@@ -48,9 +48,12 @@ const char *get_bug_type(struct kasan_ac
object = nearest_obj(cache, page, (void *)addr);
alloc_meta = kasan_get_alloc_meta(cache, object);
- for (i = 0; i < KASAN_NR_FREE_STACKS; i++)
- if (alloc_meta->free_pointer_tag[i] == tag)
- return "use-after-free";
+ if (alloc_meta) {
+ for (i = 0; i < KASAN_NR_FREE_STACKS; i++) {
+ if (alloc_meta->free_pointer_tag[i] == tag)
+ return "use-after-free";
+ }
+ }
return "out-of-bounds";
}
--- a/mm/kasan/sw_tags.c~kasan-sanitize-objects-when-metadata-doesnt-fit
+++ a/mm/kasan/sw_tags.c
@@ -170,6 +170,8 @@ void kasan_set_free_info(struct kmem_cac
u8 idx = 0;
alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return;
#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
idx = alloc_meta->free_track_idx;
@@ -187,6 +189,8 @@ struct kasan_track *kasan_get_free_track
int i = 0;
alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return NULL;
#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
for (i = 0; i < KASAN_NR_FREE_STACKS; i++) {
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 59/60] kasan, mm: allow cache merging with no metadata
2020-12-22 19:58 incoming Andrew Morton
` (57 preceding siblings ...)
2020-12-22 20:03 ` [patch 58/60] kasan: sanitize objects when metadata doesn't fit Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 20:03 ` [patch 60/60] kasan: update documentation Andrew Morton
2020-12-22 21:43 ` incoming Linus Torvalds
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, Vincenzo.Frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan, mm: allow cache merging with no metadata
The reason cache merging is disabled with KASAN is because KASAN puts its
metadata right after the allocated object. When the merged caches have
slightly different sizes, the metadata ends up in different places, which
KASAN doesn't support.
It might be possible to adjust the metadata allocation algorithm and make
it friendly to the cache merging code. Instead this change takes a simpler
approach and allows merging caches when no metadata is present. Which is
the case for hardware tag-based KASAN with kasan.mode=prod.
Link: https://lkml.kernel.org/r/37497e940bfd4b32c0a93a702a9ae4cf061d5392.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Ia114847dfb2244f297d2cb82d592bf6a07455dba
Co-developed-by: Vincenzo Frascino <Vincenzo.Frascino@arm.com>
Signed-off-by: Vincenzo Frascino <Vincenzo.Frascino@arm.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/kasan.h | 21 +++++++++++++++++++--
mm/kasan/common.c | 11 +++++++++++
mm/slab_common.c | 3 ++-
3 files changed, 32 insertions(+), 3 deletions(-)
--- a/include/linux/kasan.h~kasan-mm-allow-cache-merging-with-no-metadata
+++ a/include/linux/kasan.h
@@ -82,17 +82,30 @@ struct kasan_cache {
};
#ifdef CONFIG_KASAN_HW_TAGS
+
DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
+
static __always_inline bool kasan_enabled(void)
{
return static_branch_likely(&kasan_flag_enabled);
}
-#else
+
+#else /* CONFIG_KASAN_HW_TAGS */
+
static inline bool kasan_enabled(void)
{
return true;
}
-#endif
+
+#endif /* CONFIG_KASAN_HW_TAGS */
+
+slab_flags_t __kasan_never_merge(void);
+static __always_inline slab_flags_t kasan_never_merge(void)
+{
+ if (kasan_enabled())
+ return __kasan_never_merge();
+ return 0;
+}
void __kasan_unpoison_range(const void *addr, size_t size);
static __always_inline void kasan_unpoison_range(const void *addr, size_t size)
@@ -239,6 +252,10 @@ static inline bool kasan_enabled(void)
{
return false;
}
+static inline slab_flags_t kasan_never_merge(void)
+{
+ return 0;
+}
static inline void kasan_unpoison_range(const void *address, size_t size) {}
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}
--- a/mm/kasan/common.c~kasan-mm-allow-cache-merging-with-no-metadata
+++ a/mm/kasan/common.c
@@ -86,6 +86,17 @@ asmlinkage void kasan_unpoison_task_stac
}
#endif /* CONFIG_KASAN_STACK */
+/*
+ * Only allow cache merging when stack collection is disabled and no metadata
+ * is present.
+ */
+slab_flags_t __kasan_never_merge(void)
+{
+ if (kasan_stack_collection_enabled())
+ return SLAB_KASAN;
+ return 0;
+}
+
void __kasan_alloc_pages(struct page *page, unsigned int order)
{
u8 tag;
--- a/mm/slab_common.c~kasan-mm-allow-cache-merging-with-no-metadata
+++ a/mm/slab_common.c
@@ -18,6 +18,7 @@
#include <linux/seq_file.h>
#include <linux/proc_fs.h>
#include <linux/debugfs.h>
+#include <linux/kasan.h>
#include <asm/cacheflush.h>
#include <asm/tlbflush.h>
#include <asm/page.h>
@@ -53,7 +54,7 @@ static DECLARE_WORK(slab_caches_to_rcu_d
*/
#define SLAB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
SLAB_TRACE | SLAB_TYPESAFE_BY_RCU | SLAB_NOLEAKTRACE | \
- SLAB_FAILSLAB | SLAB_KASAN)
+ SLAB_FAILSLAB | kasan_never_merge())
#define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \
SLAB_CACHE_DMA32 | SLAB_ACCOUNT)
_
^ permalink raw reply [flat|nested] 62+ messages in thread* [patch 60/60] kasan: update documentation
2020-12-22 19:58 incoming Andrew Morton
` (58 preceding siblings ...)
2020-12-22 20:03 ` [patch 59/60] kasan, mm: allow cache merging with no metadata Andrew Morton
@ 2020-12-22 20:03 ` Andrew Morton
2020-12-22 21:43 ` incoming Linus Torvalds
60 siblings, 0 replies; 62+ messages in thread
From: Andrew Morton @ 2020-12-22 20:03 UTC (permalink / raw)
To: akpm, andreyknvl, aryabinin, Branislav.Rankov, catalin.marinas,
dvyukov, elver, eugenis, glider, gor, kevin.brodsky, linux-mm,
mm-commits, torvalds, vincenzo.frascino, will.deacon
From: Andrey Konovalov <andreyknvl@google.com>
Subject: kasan: update documentation
This change updates KASAN documentation to reflect the addition of boot
parameters and also reworks and clarifies some of the existing sections,
in particular: defines what a memory granule is, mentions quarantine,
makes Kunit section more readable.
Link: https://lkml.kernel.org/r/748daf013e17d925b0fe00c1c3b5dce726dd2430.1606162397.git.andreyknvl@google.com
Link: https://linux-review.googlesource.com/id/Ib1f83e91be273264b25f42b04448ac96b858849f
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
Documentation/dev-tools/kasan.rst | 188 +++++++++++++++++-----------
1 file changed, 117 insertions(+), 71 deletions(-)
--- a/Documentation/dev-tools/kasan.rst~kasan-update-documentation
+++ a/Documentation/dev-tools/kasan.rst
@@ -4,8 +4,9 @@ The Kernel Address Sanitizer (KASAN)
Overview
--------
-KernelAddressSANitizer (KASAN) is a dynamic memory error detector designed to
-find out-of-bound and use-after-free bugs. KASAN has three modes:
+KernelAddressSANitizer (KASAN) is a dynamic memory safety error detector
+designed to find out-of-bound and use-after-free bugs. KASAN has three modes:
+
1. generic KASAN (similar to userspace ASan),
2. software tag-based KASAN (similar to userspace HWASan),
3. hardware tag-based KASAN (based on hardware memory tagging).
@@ -39,23 +40,13 @@ CONFIG_KASAN_INLINE. Outline and inline
The former produces smaller binary while the latter is 1.1 - 2 times faster.
Both software KASAN modes work with both SLUB and SLAB memory allocators,
-hardware tag-based KASAN currently only support SLUB.
-For better bug detection and nicer reporting, enable CONFIG_STACKTRACE.
+while the hardware tag-based KASAN currently only support SLUB.
+
+For better error reports that include stack traces, enable CONFIG_STACKTRACE.
To augment reports with last allocation and freeing stack of the physical page,
it is recommended to enable also CONFIG_PAGE_OWNER and boot with page_owner=on.
-To disable instrumentation for specific files or directories, add a line
-similar to the following to the respective kernel Makefile:
-
-- For a single file (e.g. main.o)::
-
- KASAN_SANITIZE_main.o := n
-
-- For all files in one directory::
-
- KASAN_SANITIZE := n
-
Error reports
~~~~~~~~~~~~~
@@ -140,22 +131,75 @@ freed (in case of a use-after-free bug r
the accessed slab object and information about the accessed memory page.
In the last section the report shows memory state around the accessed address.
-Reading this part requires some understanding of how KASAN works.
-
-The state of each 8 aligned bytes of memory is encoded in one shadow byte.
-Those 8 bytes can be accessible, partially accessible, freed or be a redzone.
-We use the following encoding for each shadow byte: 0 means that all 8 bytes
-of the corresponding memory region are accessible; number N (1 <= N <= 7) means
-that the first N bytes are accessible, and other (8 - N) bytes are not;
-any negative value indicates that the entire 8-byte word is inaccessible.
-We use different negative values to distinguish between different kinds of
-inaccessible memory like redzones or freed memory (see mm/kasan/kasan.h).
+Internally KASAN tracks memory state separately for each memory granule, which
+is either 8 or 16 aligned bytes depending on KASAN mode. Each number in the
+memory state section of the report shows the state of one of the memory
+granules that surround the accessed address.
+
+For generic KASAN the size of each memory granule is 8. The state of each
+granule is encoded in one shadow byte. Those 8 bytes can be accessible,
+partially accessible, freed or be a part of a redzone. KASAN uses the following
+encoding for each shadow byte: 0 means that all 8 bytes of the corresponding
+memory region are accessible; number N (1 <= N <= 7) means that the first N
+bytes are accessible, and other (8 - N) bytes are not; any negative value
+indicates that the entire 8-byte word is inaccessible. KASAN uses different
+negative values to distinguish between different kinds of inaccessible memory
+like redzones or freed memory (see mm/kasan/kasan.h).
In the report above the arrows point to the shadow byte 03, which means that
the accessed address is partially accessible.
For tag-based KASAN this last report section shows the memory tags around the
-accessed address (see Implementation details section).
+accessed address (see `Implementation details`_ section).
+
+Boot parameters
+~~~~~~~~~~~~~~~
+
+Hardware tag-based KASAN mode (see the section about different mode below) is
+intended for use in production as a security mitigation. Therefore it supports
+boot parameters that allow to disable KASAN competely or otherwise control
+particular KASAN features.
+
+The things that can be controlled are:
+
+1. Whether KASAN is enabled at all.
+2. Whether KASAN collects and saves alloc/free stacks.
+3. Whether KASAN panics on a detected bug or not.
+
+The ``kasan.mode`` boot parameter allows to choose one of three main modes:
+
+- ``kasan.mode=off`` - KASAN is disabled, no tag checks are performed
+- ``kasan.mode=prod`` - only essential production features are enabled
+- ``kasan.mode=full`` - all KASAN features are enabled
+
+The chosen mode provides default control values for the features mentioned
+above. However it's also possible to override the default values by providing:
+
+- ``kasan.stacktrace=off`` or ``=on`` - enable alloc/free stack collection
+ (default: ``on`` for ``mode=full``,
+ otherwise ``off``)
+- ``kasan.fault=report`` or ``=panic`` - only print KASAN report or also panic
+ (default: ``report``)
+
+If ``kasan.mode`` parameter is not provided, it defaults to ``full`` when
+``CONFIG_DEBUG_KERNEL`` is enabled, and to ``prod`` otherwise.
+
+For developers
+~~~~~~~~~~~~~~
+
+Software KASAN modes use compiler instrumentation to insert validity checks.
+Such instrumentation might be incompatible with some part of the kernel, and
+therefore needs to be disabled. To disable instrumentation for specific files
+or directories, add a line similar to the following to the respective kernel
+Makefile:
+
+- For a single file (e.g. main.o)::
+
+ KASAN_SANITIZE_main.o := n
+
+- For all files in one directory::
+
+ KASAN_SANITIZE := n
Implementation details
@@ -164,10 +208,10 @@ Implementation details
Generic KASAN
~~~~~~~~~~~~~
-From a high level, our approach to memory error detection is similar to that
-of kmemcheck: use shadow memory to record whether each byte of memory is safe
-to access, and use compile-time instrumentation to insert checks of shadow
-memory on each memory access.
+From a high level perspective, KASAN's approach to memory error detection is
+similar to that of kmemcheck: use shadow memory to record whether each byte of
+memory is safe to access, and use compile-time instrumentation to insert checks
+of shadow memory on each memory access.
Generic KASAN dedicates 1/8th of kernel memory to its shadow memory (e.g. 16TB
to cover 128TB on x86_64) and uses direct mapping with a scale and offset to
@@ -198,6 +242,9 @@ Generic KASAN also reports the last 2 ca
potentially has access to an object. Call stacks for the following are shown:
call_rcu() and workqueue queuing.
+Generic KASAN is the only mode that delays the reuse of freed object via
+quarantine (see mm/kasan/quarantine.c for implementation).
+
Software tag-based KASAN
~~~~~~~~~~~~~~~~~~~~~~~~
@@ -305,15 +352,15 @@ therefore be wasteful. Furthermore, to e
use different shadow pages, mappings would have to be aligned to
``KASAN_GRANULE_SIZE * PAGE_SIZE``.
-Instead, we share backing space across multiple mappings. We allocate
+Instead, KASAN shares backing space across multiple mappings. It allocates
a backing page when a mapping in vmalloc space uses a particular page
of the shadow region. This page can be shared by other vmalloc
mappings later on.
-We hook in to the vmap infrastructure to lazily clean up unused shadow
+KASAN hooks into the vmap infrastructure to lazily clean up unused shadow
memory.
-To avoid the difficulties around swapping mappings around, we expect
+To avoid the difficulties around swapping mappings around, KASAN expects
that the part of the shadow region that covers the vmalloc space will
not be covered by the early shadow page, but will be left
unmapped. This will require changes in arch-specific code.
@@ -324,24 +371,31 @@ architectures that do not have a fixed m
CONFIG_KASAN_KUNIT_TEST & CONFIG_TEST_KASAN_MODULE
--------------------------------------------------
-``CONFIG_KASAN_KUNIT_TEST`` utilizes the KUnit Test Framework for testing.
-This means each test focuses on a small unit of functionality and
-there are a few ways these tests can be run.
+KASAN tests consist on two parts:
+
+1. Tests that are integrated with the KUnit Test Framework. Enabled with
+``CONFIG_KASAN_KUNIT_TEST``. These tests can be run and partially verified
+automatically in a few different ways, see the instructions below.
+
+2. Tests that are currently incompatible with KUnit. Enabled with
+``CONFIG_TEST_KASAN_MODULE`` and can only be run as a module. These tests can
+only be verified manually, by loading the kernel module and inspecting the
+kernel log for KASAN reports.
-Each test will print the KASAN report if an error is detected and then
-print the number of the test and the status of the test:
+Each KUnit-compatible KASAN test prints a KASAN report if an error is detected.
+Then the test prints its number and status.
-pass::
+When a test passes::
ok 28 - kmalloc_double_kzfree
-or, if kmalloc failed::
+When a test fails due to a failed ``kmalloc``::
# kmalloc_large_oob_right: ASSERTION FAILED at lib/test_kasan.c:163
Expected ptr is not null, but is
not ok 4 - kmalloc_large_oob_right
-or, if a KASAN report was expected, but not found::
+When a test fails due to a missing KASAN report::
# kmalloc_double_kzfree: EXPECTATION FAILED at lib/test_kasan.c:629
Expected kasan_data->report_expected == kasan_data->report_found, but
@@ -349,46 +403,38 @@ or, if a KASAN report was expected, but
kasan_data->report_found == 0
not ok 28 - kmalloc_double_kzfree
-All test statuses are tracked as they run and an overall status will
-be printed at the end::
+At the end the cumulative status of all KASAN tests is printed. On success::
ok 1 - kasan
-or::
+Or, if one of the tests failed::
not ok 1 - kasan
-(1) Loadable Module
-~~~~~~~~~~~~~~~~~~~~
+
+There are a few ways to run KUnit-compatible KASAN tests.
+
+1. Loadable module
+~~~~~~~~~~~~~~~~~~
With ``CONFIG_KUNIT`` enabled, ``CONFIG_KASAN_KUNIT_TEST`` can be built as
-a loadable module and run on any architecture that supports KASAN
-using something like insmod or modprobe. The module is called ``test_kasan``.
+a loadable module and run on any architecture that supports KASAN by loading
+the module with insmod or modprobe. The module is called ``test_kasan``.
-(2) Built-In
-~~~~~~~~~~~~~
+2. Built-In
+~~~~~~~~~~~
With ``CONFIG_KUNIT`` built-in, ``CONFIG_KASAN_KUNIT_TEST`` can be built-in
-on any architecture that supports KASAN. These and any other KUnit
-tests enabled will run and print the results at boot as a late-init
-call.
-
-(3) Using kunit_tool
-~~~~~~~~~~~~~~~~~~~~~
-
-With ``CONFIG_KUNIT`` and ``CONFIG_KASAN_KUNIT_TEST`` built-in, we can also
-use kunit_tool to see the results of these along with other KUnit
-tests in a more readable way. This will not print the KASAN reports
-of tests that passed. Use `KUnit documentation <https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html>`_ for more up-to-date
-information on kunit_tool.
+on any architecure that supports KASAN. These and any other KUnit tests enabled
+will run and print the results at boot as a late-init call.
-.. _KUnit: https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html
+3. Using kunit_tool
+~~~~~~~~~~~~~~~~~~~
-``CONFIG_TEST_KASAN_MODULE`` is a set of KASAN tests that could not be
-converted to KUnit. These tests can be run only as a module with
-``CONFIG_TEST_KASAN_MODULE`` built as a loadable module and
-``CONFIG_KASAN`` built-in. The type of error expected and the
-function being run is printed before the expression expected to give
-an error. Then the error is printed, if found, and that test
-should be interpreted to pass only if the error was the one expected
-by the test.
+With ``CONFIG_KUNIT`` and ``CONFIG_KASAN_KUNIT_TEST`` built-in, it's also
+possible use ``kunit_tool`` to see the results of these and other KUnit tests
+in a more readable way. This will not print the KASAN reports of the tests that
+passed. Use `KUnit documentation <https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html>`_
+for more up-to-date information on ``kunit_tool``.
+
+.. _KUnit: https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html
_
^ permalink raw reply [flat|nested] 62+ messages in thread* Re: incoming
2020-12-22 19:58 incoming Andrew Morton
` (59 preceding siblings ...)
2020-12-22 20:03 ` [patch 60/60] kasan: update documentation Andrew Morton
@ 2020-12-22 21:43 ` Linus Torvalds
60 siblings, 0 replies; 62+ messages in thread
From: Linus Torvalds @ 2020-12-22 21:43 UTC (permalink / raw)
To: Andrew Morton; +Cc: Linux-MM, mm-commits
On Tue, Dec 22, 2020 at 11:58 AM Andrew Morton
<akpm@linux-foundation.org> wrote:
>
> 60 patches, based on 8653b778e454a7708847aeafe689bce07aeeb94e.
I see that you enabled renaming in the patches. Lovely.
Can you also enable it in the diffstat?
> 74 files changed, 2869 insertions(+), 1553 deletions(-)
With -M in the diffstat, you should have seen
72 files changed, 2775 insertions(+), 1460 deletions(-)
and if you add "--summary", you'll also see the rename part ofthe file
create/delete summary:
rename mm/kasan/{tags_report.c => report_sw_tags.c} (78%)
which is often nice to see in addition to the line stats..
Linus
^ permalink raw reply [flat|nested] 62+ messages in thread