* [PATCH 00/35] kasan: add hardware tag-based mode for arm64
@ 2020-08-14 17:26 Andrey Konovalov
2020-08-14 17:26 ` [PATCH 01/35] kasan: KASAN_VMALLOC depends on KASAN_GENERIC Andrey Konovalov
` (34 more replies)
0 siblings, 35 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This patchset adds a new hardware tag-based mode to KASAN [1]. The new mode
is similar to the existing software tag-based KASAN, but relies on arm64
Memory Tagging Extension (MTE) [2] to perform memory and pointer tagging
(instead of shadow memory and compiler instrumentation).
This patchset is available here:
https://github.com/xairy/linux/tree/up-kasan-mte-v1
and has also been uploaded to the Linux kernel Gerrit instance:
https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/2700
This patchset is based on the v7 of the user MTE patches [3], along with
some KASAN patches cherry-picked from the current mainline.
This patchset consists of three parts:
1. Rework KASAN code to allow easier integration of the hardware tag-based
mode.
2. Introduce core in-kernel MTE routines.
3. Combine the two parts together and introduce the new mode.
For testing in QEMU hardware tag-based KASAN requires:
1. QEMU built from master [4] with these patches [5] on top.
2. GCC version 10.
[1] https://www.kernel.org/doc/html/latest/dev-tools/kasan.html
[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux for-next/mte
[4] https://github.com/qemu/qemu
[5] https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg02677.html
====== Overview
The underlying ideas of the approach used by hardware tag-based KASAN are:
1. By relying on the Top Byte Ignore (TBI) arm64 CPU feature, pointer tags
are stored in the top byte of each kernel pointer.
2. With the Memory Tagging Extension (MTE) arm64 CPU feature, memory tags
for kernel memory allocations are stored in a dedicated memory not
accessible via normal instuctions.
3. On each memory allocation, a random tag is generated, embedded it into
the returned pointer, and the corresponding memory is tagged with the
same tag value.
4. With MTE the CPU performs a check on each memory access to make sure
that the pointer tag matches the memory tag.
5. On a tag mismatch the CPU generates a tag fault, and a KASAN report is
printed.
Same as other KASAN modes, hardware tag-based KASAN is intended as a
debugging feature at this point.
====== Rationale
There are two main reasons for this new hardware tag-based mode:
1. Previously implemented software tag-based KASAN is being successfully
used on dogfood testing devices due to its low memory overhead (as
initially planned). The new hardware mode keeps the same low memory
overhead, and is expected to have significantly lower performance
impact, due to the tag checks being performed by the hardware.
Therefore the new mode can be used as a better alternative in dogfood
testing for hardware that supports MTE.
2. The new mode lays the groundwork for the planned in-kernel MTE-based
memory corruption mitigation to be used in production.
====== Technical details
From the implementation perspective, hardware tag-based KASAN is almost
identical to the software mode. The key difference is using MTE for
assigning and checking tags.
Compared to the software mode, the hardware mode uses 4 bits per tag, as
dictated by MTE. Pointer tags are stored in bits [56:60), the top 4 bits
have the normal value 0xF. Having less distict tags increases the
probablity of false negatives (from ~1/256 to ~1/16) in certain cases.
Only synchronous exceptions are set up and used by hardware tag-based KASAN.
====== Benchmarks
Note: all measurements have been performed with software emulation of Memory
Tagging Extension, performance numbers for hardware tag-based KASAN on the
actual hardware are expected to be better.
Boot time [1]:
* 2.8 sec for clean kernel
* 5.7 sec for hardware tag-based KASAN
* 11.8 sec for software tag-based KASAN
* 11.6 sec for generic KASAN
Slab memory usage after boot [2]:
* 7.0 kb for clean kernel
* 9.7 kb for hardware tag-based KASAN
* 9.7 kb for software tag-based KASAN
* 41.3 kb for generic KASAN
Measurements have been performed with:
* defconfig-based configs
* Manually built QEMU master with these patches on top [3]
* QEMU arguments: -machine virt,mte=on -cpu max
* CONFIG_KASAN_STACK_ENABLE disabled
* CONFIG_KASAN_INLINE enabled
* clang-10 as the compiler and gcc-10 as the assembler
[1] Time before the ext4 driver is initialized.
[2] Measured as `cat /proc/meminfo | grep Slab`.
[3] https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg02677.html
====== Notes
The cover letter for software tag-based KASAN patchset can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0116523cfffa62aeb5aa3b85ce7419f3dae0c1b8
Andrey Konovalov (29):
kasan: KASAN_VMALLOC depends on KASAN_GENERIC
kasan: group vmalloc code
kasan: shadow declarations only for software modes
kasan: rename (un)poison_shadow to (un)poison_memory
kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_*
kasan: only build init.c for software modes
kasan: split out shadow.c from common.c
kasan: rename generic/tags_report.c files
kasan: don't duplicate config dependencies
kasan: hide invalid free check implementation
kasan: decode stack frame only with KASAN_STACK_ENABLE
kasan, arm64: only init shadow for software modes
kasan, arm64: only use kasan_depth for software modes
kasan: rename addr_has_shadow to addr_has_metadata
kasan: rename print_shadow_for_address to print_memory_metadata
kasan: kasan_non_canonical_hook only for software modes
kasan: rename SHADOW layout macros to META
kasan: separate metadata_fetch_row for each mode
kasan: don't allow SW_TAGS with ARM64_MTE
kasan: introduce CONFIG_KASAN_HW_TAGS
kasan, arm64: align allocations for HW_TAGS
kasan: define KASAN_GRANULE_SIZE for HW_TAGS
kasan, x86, s390: update undef CONFIG_KASAN
kasan, arm64: expand CONFIG_KASAN checks
kasan, arm64: implement HW_TAGS runtime
kasan, arm64: print report from tag fault handler
kasan, slub: reset tags when accessing metadata
kasan, arm64: enable CONFIG_KASAN_HW_TAGS
kasan: add documentation for hardware tag-based mode
Vincenzo Frascino (6):
arm64: mte: Add in-kernel MTE helpers
arm64: mte: Add in-kernel tag fault handler
arm64: mte: Enable in-kernel MTE
arm64: mte: Convert gcr_user into an exclude mask
arm64: mte: Switch GCR_EL1 in kernel entry and exit
kasan, arm64: Enable TBI EL1
Documentation/dev-tools/kasan.rst | 75 ++-
arch/arm64/Kconfig | 3 +-
arch/arm64/Makefile | 2 +-
arch/arm64/include/asm/assembler.h | 2 +-
arch/arm64/include/asm/cache.h | 3 +
arch/arm64/include/asm/esr.h | 1 +
arch/arm64/include/asm/kasan.h | 8 +-
arch/arm64/include/asm/memory.h | 6 +-
arch/arm64/include/asm/mte.h | 54 ++-
arch/arm64/include/asm/mte_asm.h | 10 +
arch/arm64/include/asm/processor.h | 2 +-
arch/arm64/include/asm/string.h | 5 +-
arch/arm64/kernel/asm-offsets.c | 3 +
arch/arm64/kernel/cpufeature.c | 11 +-
arch/arm64/kernel/entry.S | 28 ++
arch/arm64/kernel/head.S | 2 +-
arch/arm64/kernel/image-vars.h | 2 +-
arch/arm64/kernel/mte.c | 87 +++-
arch/arm64/kernel/setup.c | 1 -
arch/arm64/lib/mte.S | 41 ++
arch/arm64/mm/dump.c | 6 +-
arch/arm64/mm/fault.c | 59 ++-
arch/arm64/mm/kasan_init.c | 22 +-
arch/arm64/mm/proc.S | 2 +-
arch/s390/boot/string.c | 1 +
arch/x86/boot/compressed/misc.h | 1 +
include/linux/kasan-checks.h | 2 +-
include/linux/kasan.h | 110 +++--
include/linux/mm.h | 2 +-
include/linux/moduleloader.h | 3 +-
include/linux/page-flags-layout.h | 2 +-
include/linux/sched.h | 2 +-
include/linux/string.h | 2 +-
init/init_task.c | 2 +-
kernel/fork.c | 4 +-
lib/Kconfig.kasan | 60 ++-
lib/test_kasan.c | 2 +-
mm/kasan/Makefile | 21 +-
mm/kasan/common.c | 554 ++--------------------
mm/kasan/generic.c | 33 +-
mm/kasan/generic_report.c | 165 -------
mm/kasan/init.c | 10 +-
mm/kasan/kasan.h | 45 +-
mm/kasan/mte.c | 76 +++
mm/kasan/report.c | 254 ++--------
mm/kasan/report_generic.c | 331 +++++++++++++
mm/kasan/report_mte.c | 47 ++
mm/kasan/{tags_report.c => report_tags.c} | 7 +-
mm/kasan/shadow.c | 509 ++++++++++++++++++++
mm/kasan/tags.c | 14 +-
mm/page_poison.c | 2 +-
mm/ptdump.c | 13 +-
mm/slab_common.c | 2 +-
mm/slub.c | 25 +-
scripts/Makefile.lib | 2 +
55 files changed, 1653 insertions(+), 1085 deletions(-)
create mode 100644 arch/arm64/include/asm/mte_asm.h
delete mode 100644 mm/kasan/generic_report.c
create mode 100644 mm/kasan/mte.c
create mode 100644 mm/kasan/report_generic.c
create mode 100644 mm/kasan/report_mte.c
rename mm/kasan/{tags_report.c => report_tags.c} (94%)
create mode 100644 mm/kasan/shadow.c
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH 01/35] kasan: KASAN_VMALLOC depends on KASAN_GENERIC
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 02/35] kasan: group vmalloc code Andrey Konovalov
` (33 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Currently only generic KASAN mode supports vmalloc, reflect that
in the config.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
lib/Kconfig.kasan | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index 047b53dbfd58..e1d55331b618 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -156,7 +156,7 @@ config KASAN_SW_TAGS_IDENTIFY
config KASAN_VMALLOC
bool "Back mappings in vmalloc space with real shadow memory"
- depends on HAVE_ARCH_KASAN_VMALLOC
+ depends on KASAN_GENERIC && HAVE_ARCH_KASAN_VMALLOC
help
By default, the shadow region for vmalloc space is the read-only
zero page. This means that KASAN cannot detect errors involving
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 02/35] kasan: group vmalloc code
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
2020-08-14 17:26 ` [PATCH 01/35] kasan: KASAN_VMALLOC depends on KASAN_GENERIC Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 03/35] kasan: shadow declarations only for software modes Andrey Konovalov
` (32 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Group all vmalloc-related function declarations in include/linux/kasan.h,
and their implementations in mm/kasan/common.c.
No functional changes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
include/linux/kasan.h | 41 +++++++++++++----------
mm/kasan/common.c | 78 ++++++++++++++++++++++---------------------
2 files changed, 63 insertions(+), 56 deletions(-)
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 087fba34b209..bd5b4965a269 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -69,19 +69,6 @@ struct kasan_cache {
int free_meta_offset;
};
-/*
- * These functions provide a special case to support backing module
- * allocations with real shadow memory. With KASAN vmalloc, the special
- * case is unnecessary, as the work is handled in the generic case.
- */
-#ifndef CONFIG_KASAN_VMALLOC
-int kasan_module_alloc(void *addr, size_t size);
-void kasan_free_shadow(const struct vm_struct *vm);
-#else
-static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
-static inline void kasan_free_shadow(const struct vm_struct *vm) {}
-#endif
-
int kasan_add_zero_shadow(void *start, unsigned long size);
void kasan_remove_zero_shadow(void *start, unsigned long size);
@@ -150,9 +137,6 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
return false;
}
-static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
-static inline void kasan_free_shadow(const struct vm_struct *vm) {}
-
static inline int kasan_add_zero_shadow(void *start, unsigned long size)
{
return 0;
@@ -205,13 +189,16 @@ static inline void *kasan_reset_tag(const void *addr)
#endif /* CONFIG_KASAN_SW_TAGS */
#ifdef CONFIG_KASAN_VMALLOC
+
int kasan_populate_vmalloc(unsigned long addr, unsigned long size);
void kasan_poison_vmalloc(const void *start, unsigned long size);
void kasan_unpoison_vmalloc(const void *start, unsigned long size);
void kasan_release_vmalloc(unsigned long start, unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end);
-#else
+
+#else /* CONFIG_KASAN_VMALLOC */
+
static inline int kasan_populate_vmalloc(unsigned long start,
unsigned long size)
{
@@ -226,7 +213,25 @@ static inline void kasan_release_vmalloc(unsigned long start,
unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end) {}
-#endif
+
+#endif /* CONFIG_KASAN_VMALLOC */
+
+#if defined(CONFIG_KASAN) && !defined(CONFIG_KASAN_VMALLOC)
+
+/*
+ * These functions provide a special case to support backing module
+ * allocations with real shadow memory. With KASAN vmalloc, the special
+ * case is unnecessary, as the work is handled in the generic case.
+ */
+int kasan_module_alloc(void *addr, size_t size);
+void kasan_free_shadow(const struct vm_struct *vm);
+
+#else /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
+
+static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
+static inline void kasan_free_shadow(const struct vm_struct *vm) {}
+
+#endif /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
#ifdef CONFIG_KASAN_INLINE
void kasan_non_canonical_hook(unsigned long addr);
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 950fd372a07e..d1c987f324cd 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -541,44 +541,6 @@ void kasan_kfree_large(void *ptr, unsigned long ip)
/* The object will be poisoned by page_alloc. */
}
-#ifndef CONFIG_KASAN_VMALLOC
-int kasan_module_alloc(void *addr, size_t size)
-{
- void *ret;
- size_t scaled_size;
- size_t shadow_size;
- unsigned long shadow_start;
-
- shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
- scaled_size = (size + KASAN_SHADOW_MASK) >> KASAN_SHADOW_SCALE_SHIFT;
- shadow_size = round_up(scaled_size, PAGE_SIZE);
-
- if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
- return -EINVAL;
-
- ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
- shadow_start + shadow_size,
- GFP_KERNEL,
- PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
- __builtin_return_address(0));
-
- if (ret) {
- __memset(ret, KASAN_SHADOW_INIT, shadow_size);
- find_vm_area(addr)->flags |= VM_KASAN;
- kmemleak_ignore(ret);
- return 0;
- }
-
- return -ENOMEM;
-}
-
-void kasan_free_shadow(const struct vm_struct *vm)
-{
- if (vm->flags & VM_KASAN)
- vfree(kasan_mem_to_shadow(vm->addr));
-}
-#endif
-
#ifdef CONFIG_MEMORY_HOTPLUG
static bool shadow_mapped(unsigned long addr)
{
@@ -690,6 +652,7 @@ core_initcall(kasan_memhotplug_init);
#endif
#ifdef CONFIG_KASAN_VMALLOC
+
static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
void *unused)
{
@@ -928,4 +891,43 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,
(unsigned long)shadow_end);
}
}
+
+#else /* CONFIG_KASAN_VMALLOC */
+
+int kasan_module_alloc(void *addr, size_t size)
+{
+ void *ret;
+ size_t scaled_size;
+ size_t shadow_size;
+ unsigned long shadow_start;
+
+ shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
+ scaled_size = (size + KASAN_SHADOW_MASK) >> KASAN_SHADOW_SCALE_SHIFT;
+ shadow_size = round_up(scaled_size, PAGE_SIZE);
+
+ if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
+ return -EINVAL;
+
+ ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
+ shadow_start + shadow_size,
+ GFP_KERNEL,
+ PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
+ __builtin_return_address(0));
+
+ if (ret) {
+ __memset(ret, KASAN_SHADOW_INIT, shadow_size);
+ find_vm_area(addr)->flags |= VM_KASAN;
+ kmemleak_ignore(ret);
+ return 0;
+ }
+
+ return -ENOMEM;
+}
+
+void kasan_free_shadow(const struct vm_struct *vm)
+{
+ if (vm->flags & VM_KASAN)
+ vfree(kasan_mem_to_shadow(vm->addr));
+}
+
#endif
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 03/35] kasan: shadow declarations only for software modes
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
2020-08-14 17:26 ` [PATCH 01/35] kasan: KASAN_VMALLOC depends on KASAN_GENERIC Andrey Konovalov
2020-08-14 17:26 ` [PATCH 02/35] kasan: group vmalloc code Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-09-18 14:55 ` Marco Elver
2020-08-14 17:26 ` [PATCH 04/35] kasan: rename (un)poison_shadow to (un)poison_memory Andrey Konovalov
` (31 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Group shadow-related KASAN function declarations and only define them
for the two existing software modes.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
include/linux/kasan.h | 44 ++++++++++++++++++++++++++-----------------
1 file changed, 27 insertions(+), 17 deletions(-)
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index bd5b4965a269..44a9aae44138 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -3,16 +3,24 @@
#define _LINUX_KASAN_H
#include <linux/types.h>
+#include <asm/kasan.h>
struct kmem_cache;
struct page;
struct vm_struct;
struct task_struct;
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#include <linux/pgtable.h>
-#include <asm/kasan.h>
+
+/* Software KASAN implementations use shadow memory. */
+
+#ifdef CONFIG_KASAN_SW_TAGS
+#define KASAN_SHADOW_INIT 0xFF
+#else
+#define KASAN_SHADOW_INIT 0
+#endif
extern unsigned char kasan_early_shadow_page[PAGE_SIZE];
extern pte_t kasan_early_shadow_pte[PTRS_PER_PTE];
@@ -29,6 +37,23 @@ static inline void *kasan_mem_to_shadow(const void *addr)
+ KASAN_SHADOW_OFFSET;
}
+int kasan_add_zero_shadow(void *start, unsigned long size);
+void kasan_remove_zero_shadow(void *start, unsigned long size);
+
+#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+static inline int kasan_add_zero_shadow(void *start, unsigned long size)
+{
+ return 0;
+}
+static inline void kasan_remove_zero_shadow(void *start,
+ unsigned long size)
+{}
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+#ifdef CONFIG_KASAN
+
/* Enable reporting bugs after kasan_disable_current() */
extern void kasan_enable_current(void);
@@ -69,9 +94,6 @@ struct kasan_cache {
int free_meta_offset;
};
-int kasan_add_zero_shadow(void *start, unsigned long size);
-void kasan_remove_zero_shadow(void *start, unsigned long size);
-
size_t __ksize(const void *);
static inline void kasan_unpoison_slab(const void *ptr)
{
@@ -137,14 +159,6 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
return false;
}
-static inline int kasan_add_zero_shadow(void *start, unsigned long size)
-{
- return 0;
-}
-static inline void kasan_remove_zero_shadow(void *start,
- unsigned long size)
-{}
-
static inline void kasan_unpoison_slab(const void *ptr) { }
static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
@@ -152,8 +166,6 @@ static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
#ifdef CONFIG_KASAN_GENERIC
-#define KASAN_SHADOW_INIT 0
-
void kasan_cache_shrink(struct kmem_cache *cache);
void kasan_cache_shutdown(struct kmem_cache *cache);
void kasan_record_aux_stack(void *ptr);
@@ -168,8 +180,6 @@ static inline void kasan_record_aux_stack(void *ptr) {}
#ifdef CONFIG_KASAN_SW_TAGS
-#define KASAN_SHADOW_INIT 0xFF
-
void kasan_init_tags(void);
void *kasan_reset_tag(const void *addr);
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 04/35] kasan: rename (un)poison_shadow to (un)poison_memory
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (2 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 03/35] kasan: shadow declarations only for software modes Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 05/35] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_* Andrey Konovalov
` (30 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory, but will reuse the same
functions. Rename kasan_unpoison_shadow to kasan_unpoison_memory,
and kasan_poison_shadow to kasan_poison_memory.
No functional changes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
include/linux/kasan.h | 6 +++---
kernel/fork.c | 4 ++--
mm/kasan/common.c | 38 +++++++++++++++++++-------------------
mm/kasan/generic.c | 12 ++++++------
mm/kasan/kasan.h | 2 +-
mm/kasan/tags.c | 2 +-
mm/slab_common.c | 2 +-
7 files changed, 33 insertions(+), 33 deletions(-)
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 44a9aae44138..18617d5c4cd7 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -60,7 +60,7 @@ extern void kasan_enable_current(void);
/* Disable reporting bugs for current task */
extern void kasan_disable_current(void);
-void kasan_unpoison_shadow(const void *address, size_t size);
+void kasan_unpoison_memory(const void *address, size_t size);
void kasan_unpoison_task_stack(struct task_struct *task);
@@ -97,7 +97,7 @@ struct kasan_cache {
size_t __ksize(const void *);
static inline void kasan_unpoison_slab(const void *ptr)
{
- kasan_unpoison_shadow(ptr, __ksize(ptr));
+ kasan_unpoison_memory(ptr, __ksize(ptr));
}
size_t kasan_metadata_size(struct kmem_cache *cache);
@@ -106,7 +106,7 @@ void kasan_restore_multi_shot(bool enabled);
#else /* CONFIG_KASAN */
-static inline void kasan_unpoison_shadow(const void *address, size_t size) {}
+static inline void kasan_unpoison_memory(const void *address, size_t size) {}
static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
diff --git a/kernel/fork.c b/kernel/fork.c
index c9c76a4d1180..c93e93cfbab8 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -224,8 +224,8 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
if (!s)
continue;
- /* Clear the KASAN shadow of the stack. */
- kasan_unpoison_shadow(s->addr, THREAD_SIZE);
+ /* Mark stack accessible for KASAN. */
+ kasan_unpoison_memory(s->addr, THREAD_SIZE);
/* Clear stale pointers from reused stack. */
memset(s->addr, 0, THREAD_SIZE);
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index d1c987f324cd..65933b27df81 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -113,7 +113,7 @@ void *memcpy(void *dest, const void *src, size_t len)
* Poisons the shadow memory for 'size' bytes starting from 'addr'.
* Memory addresses should be aligned to KASAN_SHADOW_SCALE_SIZE.
*/
-void kasan_poison_shadow(const void *address, size_t size, u8 value)
+void kasan_poison_memory(const void *address, size_t size, u8 value)
{
void *shadow_start, *shadow_end;
@@ -130,7 +130,7 @@ void kasan_poison_shadow(const void *address, size_t size, u8 value)
__memset(shadow_start, value, shadow_end - shadow_start);
}
-void kasan_unpoison_shadow(const void *address, size_t size)
+void kasan_unpoison_memory(const void *address, size_t size)
{
u8 tag = get_tag(address);
@@ -141,7 +141,7 @@ void kasan_unpoison_shadow(const void *address, size_t size)
*/
address = reset_tag(address);
- kasan_poison_shadow(address, size, tag);
+ kasan_poison_memory(address, size, tag);
if (size & KASAN_SHADOW_MASK) {
u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
@@ -158,7 +158,7 @@ static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
void *base = task_stack_page(task);
size_t size = sp - base;
- kasan_unpoison_shadow(base, size);
+ kasan_unpoison_memory(base, size);
}
/* Unpoison the entire stack for a task. */
@@ -177,7 +177,7 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)
*/
void *base = (void *)((unsigned long)watermark & ~(THREAD_SIZE - 1));
- kasan_unpoison_shadow(base, watermark - base);
+ kasan_unpoison_memory(base, watermark - base);
}
void kasan_alloc_pages(struct page *page, unsigned int order)
@@ -191,13 +191,13 @@ void kasan_alloc_pages(struct page *page, unsigned int order)
tag = random_tag();
for (i = 0; i < (1 << order); i++)
page_kasan_tag_set(page + i, tag);
- kasan_unpoison_shadow(page_address(page), PAGE_SIZE << order);
+ kasan_unpoison_memory(page_address(page), PAGE_SIZE << order);
}
void kasan_free_pages(struct page *page, unsigned int order)
{
if (likely(!PageHighMem(page)))
- kasan_poison_shadow(page_address(page),
+ kasan_poison_memory(page_address(page),
PAGE_SIZE << order,
KASAN_FREE_PAGE);
}
@@ -289,18 +289,18 @@ void kasan_poison_slab(struct page *page)
for (i = 0; i < compound_nr(page); i++)
page_kasan_tag_reset(page + i);
- kasan_poison_shadow(page_address(page), page_size(page),
+ kasan_poison_memory(page_address(page), page_size(page),
KASAN_KMALLOC_REDZONE);
}
void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
{
- kasan_unpoison_shadow(object, cache->object_size);
+ kasan_unpoison_memory(object, cache->object_size);
}
void kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
- kasan_poison_shadow(object,
+ kasan_poison_memory(object,
round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE),
KASAN_KMALLOC_REDZONE);
}
@@ -413,7 +413,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
}
rounded_up_size = round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE);
- kasan_poison_shadow(object, rounded_up_size, KASAN_KMALLOC_FREE);
+ kasan_poison_memory(object, rounded_up_size, KASAN_KMALLOC_FREE);
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
unlikely(!(cache->flags & SLAB_KASAN)))
@@ -453,8 +453,8 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
tag = assign_tag(cache, object, false, keep_tag);
/* Tag is ignored in set_tag without CONFIG_KASAN_SW_TAGS */
- kasan_unpoison_shadow(set_tag(object, tag), size);
- kasan_poison_shadow((void *)redzone_start, redzone_end - redzone_start,
+ kasan_unpoison_memory(set_tag(object, tag), size);
+ kasan_poison_memory((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);
if (cache->flags & SLAB_KASAN)
@@ -494,8 +494,8 @@ void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
KASAN_SHADOW_SCALE_SIZE);
redzone_end = (unsigned long)ptr + page_size(page);
- kasan_unpoison_shadow(ptr, size);
- kasan_poison_shadow((void *)redzone_start, redzone_end - redzone_start,
+ kasan_unpoison_memory(ptr, size);
+ kasan_poison_memory((void *)redzone_start, redzone_end - redzone_start,
KASAN_PAGE_REDZONE);
return (void *)ptr;
@@ -528,7 +528,7 @@ void kasan_poison_kfree(void *ptr, unsigned long ip)
kasan_report_invalid_free(ptr, ip);
return;
}
- kasan_poison_shadow(ptr, page_size(page), KASAN_FREE_PAGE);
+ kasan_poison_memory(ptr, page_size(page), KASAN_FREE_PAGE);
} else {
__kasan_slab_free(page->slab_cache, ptr, ip, false);
}
@@ -714,7 +714,7 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
* // vmalloc() allocates memory
* // let a = area->addr
* // we reach kasan_populate_vmalloc
- * // and call kasan_unpoison_shadow:
+ * // and call kasan_unpoison_memory:
* STORE shadow(a), unpoison_val
* ...
* STORE shadow(a+99), unpoison_val x = LOAD p
@@ -749,7 +749,7 @@ void kasan_poison_vmalloc(const void *start, unsigned long size)
return;
size = round_up(size, KASAN_SHADOW_SCALE_SIZE);
- kasan_poison_shadow(start, size, KASAN_VMALLOC_INVALID);
+ kasan_poison_memory(start, size, KASAN_VMALLOC_INVALID);
}
void kasan_unpoison_vmalloc(const void *start, unsigned long size)
@@ -757,7 +757,7 @@ void kasan_unpoison_vmalloc(const void *start, unsigned long size)
if (!is_vmalloc_or_module_addr(start))
return;
- kasan_unpoison_shadow(start, size);
+ kasan_unpoison_memory(start, size);
}
static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index 248264b9cb76..4b5f905198d8 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -207,9 +207,9 @@ static void register_global(struct kasan_global *global)
{
size_t aligned_size = round_up(global->size, KASAN_SHADOW_SCALE_SIZE);
- kasan_unpoison_shadow(global->beg, global->size);
+ kasan_unpoison_memory(global->beg, global->size);
- kasan_poison_shadow(global->beg + aligned_size,
+ kasan_poison_memory(global->beg + aligned_size,
global->size_with_redzone - aligned_size,
KASAN_GLOBAL_REDZONE);
}
@@ -290,11 +290,11 @@ void __asan_alloca_poison(unsigned long addr, size_t size)
WARN_ON(!IS_ALIGNED(addr, KASAN_ALLOCA_REDZONE_SIZE));
- kasan_unpoison_shadow((const void *)(addr + rounded_down_size),
+ kasan_unpoison_memory((const void *)(addr + rounded_down_size),
size - rounded_down_size);
- kasan_poison_shadow(left_redzone, KASAN_ALLOCA_REDZONE_SIZE,
+ kasan_poison_memory(left_redzone, KASAN_ALLOCA_REDZONE_SIZE,
KASAN_ALLOCA_LEFT);
- kasan_poison_shadow(right_redzone,
+ kasan_poison_memory(right_redzone,
padding_size + KASAN_ALLOCA_REDZONE_SIZE,
KASAN_ALLOCA_RIGHT);
}
@@ -306,7 +306,7 @@ void __asan_allocas_unpoison(const void *stack_top, const void *stack_bottom)
if (unlikely(!stack_top || stack_top > stack_bottom))
return;
- kasan_unpoison_shadow(stack_top, stack_bottom - stack_top);
+ kasan_unpoison_memory(stack_top, stack_bottom - stack_top);
}
EXPORT_SYMBOL(__asan_allocas_unpoison);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index ac499456740f..03450d3b31f7 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -150,7 +150,7 @@ static inline bool addr_has_shadow(const void *addr)
return (addr >= kasan_shadow_to_mem((void *)KASAN_SHADOW_START));
}
-void kasan_poison_shadow(const void *address, size_t size, u8 value);
+void kasan_poison_memory(const void *address, size_t size, u8 value);
/**
* check_memory_region - Check memory region, and report if invalid access.
diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c
index e02a36a51f42..4d5a1fe8251f 100644
--- a/mm/kasan/tags.c
+++ b/mm/kasan/tags.c
@@ -158,7 +158,7 @@ EXPORT_SYMBOL(__hwasan_storeN_noabort);
void __hwasan_tag_memory(unsigned long addr, u8 tag, unsigned long size)
{
- kasan_poison_shadow((void *)addr, size, tag);
+ kasan_poison_memory((void *)addr, size, tag);
}
EXPORT_SYMBOL(__hwasan_tag_memory);
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 37d48a56431d..24bbf3704a51 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1772,7 +1772,7 @@ size_t ksize(const void *objp)
* We assume that ksize callers could use whole allocated area,
* so we need to unpoison this area.
*/
- kasan_unpoison_shadow(objp, size);
+ kasan_unpoison_memory(objp, size);
return size;
}
EXPORT_SYMBOL(ksize);
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 05/35] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_*
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (3 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 04/35] kasan: rename (un)poison_shadow to (un)poison_memory Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 06/35] kasan: only build init.c for software modes Andrey Konovalov
` (29 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory, but will still use the concept
of memory granules. Rename KASAN_SHADOW_SCALE_SIZE to KASAN_GRANULE_SIZE,
and KASAN_SHADOW_MASK to KASAN_GRANULE_MASK.
Also use MASK when used as a mask, otherwise use SIZE.
No functional changes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
Documentation/dev-tools/kasan.rst | 2 +-
lib/test_kasan.c | 2 +-
mm/kasan/common.c | 39 ++++++++++++++++---------------
mm/kasan/generic.c | 14 +++++------
mm/kasan/generic_report.c | 8 +++----
mm/kasan/init.c | 8 +++----
mm/kasan/kasan.h | 4 ++--
mm/kasan/report.c | 10 ++++----
mm/kasan/tags_report.c | 2 +-
9 files changed, 45 insertions(+), 44 deletions(-)
diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
index 38fd5681fade..a3030fc6afe5 100644
--- a/Documentation/dev-tools/kasan.rst
+++ b/Documentation/dev-tools/kasan.rst
@@ -264,7 +264,7 @@ Most mappings in vmalloc space are small, requiring less than a full
page of shadow space. Allocating a full shadow page per mapping would
therefore be wasteful. Furthermore, to ensure that different mappings
use different shadow pages, mappings would have to be aligned to
-``KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE``.
+``KASAN_GRANULE_SIZE * PAGE_SIZE``.
Instead, we share backing space across multiple mappings. We allocate
a backing page when a mapping in vmalloc space uses a particular page
diff --git a/lib/test_kasan.c b/lib/test_kasan.c
index 5d3f496893ef..247a14f40016 100644
--- a/lib/test_kasan.c
+++ b/lib/test_kasan.c
@@ -25,7 +25,7 @@
#include "../mm/kasan/kasan.h"
-#define OOB_TAG_OFF (IS_ENABLED(CONFIG_KASAN_GENERIC) ? 0 : KASAN_SHADOW_SCALE_SIZE)
+#define OOB_TAG_OFF (IS_ENABLED(CONFIG_KASAN_GENERIC) ? 0 : KASAN_GRANULE_SIZE)
/*
* We assign some test results to these globals to make sure the tests
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 65933b27df81..c9daf2c33651 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -111,7 +111,7 @@ void *memcpy(void *dest, const void *src, size_t len)
/*
* Poisons the shadow memory for 'size' bytes starting from 'addr'.
- * Memory addresses should be aligned to KASAN_SHADOW_SCALE_SIZE.
+ * Memory addresses should be aligned to KASAN_GRANULE_SIZE.
*/
void kasan_poison_memory(const void *address, size_t size, u8 value)
{
@@ -143,13 +143,13 @@ void kasan_unpoison_memory(const void *address, size_t size)
kasan_poison_memory(address, size, tag);
- if (size & KASAN_SHADOW_MASK) {
+ if (size & KASAN_GRANULE_MASK) {
u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
*shadow = tag;
else
- *shadow = size & KASAN_SHADOW_MASK;
+ *shadow = size & KASAN_GRANULE_MASK;
}
}
@@ -301,7 +301,7 @@ void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
void kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
kasan_poison_memory(object,
- round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE),
+ round_up(cache->object_size, KASAN_GRANULE_SIZE),
KASAN_KMALLOC_REDZONE);
}
@@ -373,7 +373,7 @@ static inline bool shadow_invalid(u8 tag, s8 shadow_byte)
{
if (IS_ENABLED(CONFIG_KASAN_GENERIC))
return shadow_byte < 0 ||
- shadow_byte >= KASAN_SHADOW_SCALE_SIZE;
+ shadow_byte >= KASAN_GRANULE_SIZE;
/* else CONFIG_KASAN_SW_TAGS: */
if ((u8)shadow_byte == KASAN_TAG_INVALID)
@@ -412,7 +412,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
return true;
}
- rounded_up_size = round_up(cache->object_size, KASAN_SHADOW_SCALE_SIZE);
+ rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
kasan_poison_memory(object, rounded_up_size, KASAN_KMALLOC_FREE);
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
@@ -445,9 +445,9 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
return NULL;
redzone_start = round_up((unsigned long)(object + size),
- KASAN_SHADOW_SCALE_SIZE);
+ KASAN_GRANULE_SIZE);
redzone_end = round_up((unsigned long)object + cache->object_size,
- KASAN_SHADOW_SCALE_SIZE);
+ KASAN_GRANULE_SIZE);
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
tag = assign_tag(cache, object, false, keep_tag);
@@ -491,7 +491,7 @@ void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
page = virt_to_page(ptr);
redzone_start = round_up((unsigned long)(ptr + size),
- KASAN_SHADOW_SCALE_SIZE);
+ KASAN_GRANULE_SIZE);
redzone_end = (unsigned long)ptr + page_size(page);
kasan_unpoison_memory(ptr, size);
@@ -589,8 +589,8 @@ static int __meminit kasan_mem_notifier(struct notifier_block *nb,
shadow_size = nr_shadow_pages << PAGE_SHIFT;
shadow_end = shadow_start + shadow_size;
- if (WARN_ON(mem_data->nr_pages % KASAN_SHADOW_SCALE_SIZE) ||
- WARN_ON(start_kaddr % (KASAN_SHADOW_SCALE_SIZE << PAGE_SHIFT)))
+ if (WARN_ON(mem_data->nr_pages % KASAN_GRANULE_SIZE) ||
+ WARN_ON(start_kaddr % (KASAN_GRANULE_SIZE << PAGE_SHIFT)))
return NOTIFY_BAD;
switch (action) {
@@ -748,7 +748,7 @@ void kasan_poison_vmalloc(const void *start, unsigned long size)
if (!is_vmalloc_or_module_addr(start))
return;
- size = round_up(size, KASAN_SHADOW_SCALE_SIZE);
+ size = round_up(size, KASAN_GRANULE_SIZE);
kasan_poison_memory(start, size, KASAN_VMALLOC_INVALID);
}
@@ -861,22 +861,22 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,
unsigned long region_start, region_end;
unsigned long size;
- region_start = ALIGN(start, PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
- region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
+ region_start = ALIGN(start, PAGE_SIZE * KASAN_GRANULE_SIZE);
+ region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_GRANULE_SIZE);
free_region_start = ALIGN(free_region_start,
- PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
if (start != region_start &&
free_region_start < region_start)
- region_start -= PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE;
+ region_start -= PAGE_SIZE * KASAN_GRANULE_SIZE;
free_region_end = ALIGN_DOWN(free_region_end,
- PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE);
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
if (end != region_end &&
free_region_end > region_end)
- region_end += PAGE_SIZE * KASAN_SHADOW_SCALE_SIZE;
+ region_end += PAGE_SIZE * KASAN_GRANULE_SIZE;
shadow_start = kasan_mem_to_shadow((void *)region_start);
shadow_end = kasan_mem_to_shadow((void *)region_end);
@@ -902,7 +902,8 @@ int kasan_module_alloc(void *addr, size_t size)
unsigned long shadow_start;
shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
- scaled_size = (size + KASAN_SHADOW_MASK) >> KASAN_SHADOW_SCALE_SHIFT;
+ scaled_size = (size + KASAN_GRANULE_SIZE - 1) >>
+ KASAN_SHADOW_SCALE_SHIFT;
shadow_size = round_up(scaled_size, PAGE_SIZE);
if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index 4b5f905198d8..f6d68aa9872f 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -51,7 +51,7 @@ static __always_inline bool memory_is_poisoned_1(unsigned long addr)
s8 shadow_value = *(s8 *)kasan_mem_to_shadow((void *)addr);
if (unlikely(shadow_value)) {
- s8 last_accessible_byte = addr & KASAN_SHADOW_MASK;
+ s8 last_accessible_byte = addr & KASAN_GRANULE_MASK;
return unlikely(last_accessible_byte >= shadow_value);
}
@@ -67,7 +67,7 @@ static __always_inline bool memory_is_poisoned_2_4_8(unsigned long addr,
* Access crosses 8(shadow size)-byte boundary. Such access maps
* into 2 shadow bytes, so we need to check them both.
*/
- if (unlikely(((addr + size - 1) & KASAN_SHADOW_MASK) < size - 1))
+ if (unlikely(((addr + size - 1) & KASAN_GRANULE_MASK) < size - 1))
return *shadow_addr || memory_is_poisoned_1(addr + size - 1);
return memory_is_poisoned_1(addr + size - 1);
@@ -78,7 +78,7 @@ static __always_inline bool memory_is_poisoned_16(unsigned long addr)
u16 *shadow_addr = (u16 *)kasan_mem_to_shadow((void *)addr);
/* Unaligned 16-bytes access maps into 3 shadow bytes. */
- if (unlikely(!IS_ALIGNED(addr, KASAN_SHADOW_SCALE_SIZE)))
+ if (unlikely(!IS_ALIGNED(addr, KASAN_GRANULE_SIZE)))
return *shadow_addr || memory_is_poisoned_1(addr + 15);
return *shadow_addr;
@@ -139,7 +139,7 @@ static __always_inline bool memory_is_poisoned_n(unsigned long addr,
s8 *last_shadow = (s8 *)kasan_mem_to_shadow((void *)last_byte);
if (unlikely(ret != (unsigned long)last_shadow ||
- ((long)(last_byte & KASAN_SHADOW_MASK) >= *last_shadow)))
+ ((long)(last_byte & KASAN_GRANULE_MASK) >= *last_shadow)))
return true;
}
return false;
@@ -205,7 +205,7 @@ void kasan_cache_shutdown(struct kmem_cache *cache)
static void register_global(struct kasan_global *global)
{
- size_t aligned_size = round_up(global->size, KASAN_SHADOW_SCALE_SIZE);
+ size_t aligned_size = round_up(global->size, KASAN_GRANULE_SIZE);
kasan_unpoison_memory(global->beg, global->size);
@@ -279,10 +279,10 @@ EXPORT_SYMBOL(__asan_handle_no_return);
/* Emitted by compiler to poison alloca()ed objects. */
void __asan_alloca_poison(unsigned long addr, size_t size)
{
- size_t rounded_up_size = round_up(size, KASAN_SHADOW_SCALE_SIZE);
+ size_t rounded_up_size = round_up(size, KASAN_GRANULE_SIZE);
size_t padding_size = round_up(size, KASAN_ALLOCA_REDZONE_SIZE) -
rounded_up_size;
- size_t rounded_down_size = round_down(size, KASAN_SHADOW_SCALE_SIZE);
+ size_t rounded_down_size = round_down(size, KASAN_GRANULE_SIZE);
const void *left_redzone = (const void *)(addr -
KASAN_ALLOCA_REDZONE_SIZE);
diff --git a/mm/kasan/generic_report.c b/mm/kasan/generic_report.c
index a38c7a9e192a..4dce1633b082 100644
--- a/mm/kasan/generic_report.c
+++ b/mm/kasan/generic_report.c
@@ -39,7 +39,7 @@ void *find_first_bad_addr(void *addr, size_t size)
void *p = addr;
while (p < addr + size && !(*(u8 *)kasan_mem_to_shadow(p)))
- p += KASAN_SHADOW_SCALE_SIZE;
+ p += KASAN_GRANULE_SIZE;
return p;
}
@@ -51,14 +51,14 @@ static const char *get_shadow_bug_type(struct kasan_access_info *info)
shadow_addr = (u8 *)kasan_mem_to_shadow(info->first_bad_addr);
/*
- * If shadow byte value is in [0, KASAN_SHADOW_SCALE_SIZE) we can look
+ * If shadow byte value is in [0, KASAN_GRANULE_SIZE) we can look
* at the next shadow byte to determine the type of the bad access.
*/
- if (*shadow_addr > 0 && *shadow_addr <= KASAN_SHADOW_SCALE_SIZE - 1)
+ if (*shadow_addr > 0 && *shadow_addr <= KASAN_GRANULE_SIZE - 1)
shadow_addr++;
switch (*shadow_addr) {
- case 0 ... KASAN_SHADOW_SCALE_SIZE - 1:
+ case 0 ... KASAN_GRANULE_SIZE - 1:
/*
* In theory it's still possible to see these shadow values
* due to a data race in the kernel code.
diff --git a/mm/kasan/init.c b/mm/kasan/init.c
index fe6be0be1f76..754b641c83c7 100644
--- a/mm/kasan/init.c
+++ b/mm/kasan/init.c
@@ -447,8 +447,8 @@ void kasan_remove_zero_shadow(void *start, unsigned long size)
end = addr + (size >> KASAN_SHADOW_SCALE_SHIFT);
if (WARN_ON((unsigned long)start %
- (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)) ||
- WARN_ON(size % (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)))
+ (KASAN_GRANULE_SIZE * PAGE_SIZE)) ||
+ WARN_ON(size % (KASAN_GRANULE_SIZE * PAGE_SIZE)))
return;
for (; addr < end; addr = next) {
@@ -482,8 +482,8 @@ int kasan_add_zero_shadow(void *start, unsigned long size)
shadow_end = shadow_start + (size >> KASAN_SHADOW_SCALE_SHIFT);
if (WARN_ON((unsigned long)start %
- (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)) ||
- WARN_ON(size % (KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE)))
+ (KASAN_GRANULE_SIZE * PAGE_SIZE)) ||
+ WARN_ON(size % (KASAN_GRANULE_SIZE * PAGE_SIZE)))
return -EINVAL;
ret = kasan_populate_early_shadow(shadow_start, shadow_end);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 03450d3b31f7..c31e2c739301 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -5,8 +5,8 @@
#include <linux/kasan.h>
#include <linux/stackdepot.h>
-#define KASAN_SHADOW_SCALE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
-#define KASAN_SHADOW_MASK (KASAN_SHADOW_SCALE_SIZE - 1)
+#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
+#define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
#define KASAN_TAG_INVALID 0xFE /* inaccessible memory tag */
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 4f49fa6cd1aa..7c025d792e2f 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -317,24 +317,24 @@ static bool __must_check get_address_stack_frame_info(const void *addr,
return false;
aligned_addr = round_down((unsigned long)addr, sizeof(long));
- mem_ptr = round_down(aligned_addr, KASAN_SHADOW_SCALE_SIZE);
+ mem_ptr = round_down(aligned_addr, KASAN_GRANULE_SIZE);
shadow_ptr = kasan_mem_to_shadow((void *)aligned_addr);
shadow_bottom = kasan_mem_to_shadow(end_of_stack(current));
while (shadow_ptr >= shadow_bottom && *shadow_ptr != KASAN_STACK_LEFT) {
shadow_ptr--;
- mem_ptr -= KASAN_SHADOW_SCALE_SIZE;
+ mem_ptr -= KASAN_GRANULE_SIZE;
}
while (shadow_ptr >= shadow_bottom && *shadow_ptr == KASAN_STACK_LEFT) {
shadow_ptr--;
- mem_ptr -= KASAN_SHADOW_SCALE_SIZE;
+ mem_ptr -= KASAN_GRANULE_SIZE;
}
if (shadow_ptr < shadow_bottom)
return false;
- frame = (const unsigned long *)(mem_ptr + KASAN_SHADOW_SCALE_SIZE);
+ frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",
frame[0]);
@@ -572,6 +572,6 @@ void kasan_non_canonical_hook(unsigned long addr)
else
bug_type = "maybe wild-memory-access";
pr_alert("KASAN: %s in range [0x%016lx-0x%016lx]\n", bug_type,
- orig_addr, orig_addr + KASAN_SHADOW_MASK);
+ orig_addr, orig_addr + KASAN_GRANULE_SIZE - 1);
}
#endif
diff --git a/mm/kasan/tags_report.c b/mm/kasan/tags_report.c
index bee43717d6f0..6ddb55676a7c 100644
--- a/mm/kasan/tags_report.c
+++ b/mm/kasan/tags_report.c
@@ -81,7 +81,7 @@ void *find_first_bad_addr(void *addr, size_t size)
void *end = p + size;
while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p))
- p += KASAN_SHADOW_SCALE_SIZE;
+ p += KASAN_GRANULE_SIZE;
return p;
}
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 06/35] kasan: only build init.c for software modes
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (4 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 05/35] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_* Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 07/35] kasan: split out shadow.c from common.c Andrey Konovalov
` (28 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory, so only build init.c that
contains shadow initialization code for software modes.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/Makefile | 6 +++---
mm/kasan/init.c | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/mm/kasan/Makefile b/mm/kasan/Makefile
index d532c2587731..b5517de7fc87 100644
--- a/mm/kasan/Makefile
+++ b/mm/kasan/Makefile
@@ -29,6 +29,6 @@ CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags_report.o := $(CC_FLAGS_KASAN_RUNTIME)
-obj-$(CONFIG_KASAN) := common.o init.o report.o
-obj-$(CONFIG_KASAN_GENERIC) += generic.o generic_report.o quarantine.o
-obj-$(CONFIG_KASAN_SW_TAGS) += tags.o tags_report.o
+obj-$(CONFIG_KASAN) := common.o report.o
+obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o quarantine.o
+obj-$(CONFIG_KASAN_SW_TAGS) += init.o tags.o tags_report.o
diff --git a/mm/kasan/init.c b/mm/kasan/init.c
index 754b641c83c7..20f5e1ab8d95 100644
--- a/mm/kasan/init.c
+++ b/mm/kasan/init.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains some kasan initialization code.
+ * This file contains KASAN shadow initialization code.
*
* Copyright (c) 2015 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 07/35] kasan: split out shadow.c from common.c
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (5 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 06/35] kasan: only build init.c for software modes Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 08/35] kasan: rename generic/tags_report.c files Andrey Konovalov
` (27 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
The new mode won't be using shadow memory. Move all shadow-related code
to shadow.c, which is only enabled for software KASAN modes that use
shadow memory.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/Makefile | 6 +-
mm/kasan/common.c | 486 +------------------------------------------
mm/kasan/shadow.c | 509 ++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 514 insertions(+), 487 deletions(-)
create mode 100644 mm/kasan/shadow.c
diff --git a/mm/kasan/Makefile b/mm/kasan/Makefile
index b5517de7fc87..40366d706b7c 100644
--- a/mm/kasan/Makefile
+++ b/mm/kasan/Makefile
@@ -10,6 +10,7 @@ CFLAGS_REMOVE_generic_report.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_init.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_quarantine.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_shadow.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_tags_report.o = $(CC_FLAGS_FTRACE)
@@ -26,9 +27,10 @@ CFLAGS_generic_report.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_init.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_quarantine.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_shadow.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags_report.o := $(CC_FLAGS_KASAN_RUNTIME)
obj-$(CONFIG_KASAN) := common.o report.o
-obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o quarantine.o
-obj-$(CONFIG_KASAN_SW_TAGS) += init.o tags.o tags_report.o
+obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o shadow.o quarantine.o
+obj-$(CONFIG_KASAN_SW_TAGS) += init.o shadow.o tags.o tags_report.o
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index c9daf2c33651..43a927e70067 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains common generic and tag-based KASAN code.
+ * This file contains common KASAN code.
*
* Copyright (c) 2014 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
@@ -18,7 +18,6 @@
#include <linux/init.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
-#include <linux/kmemleak.h>
#include <linux/linkage.h>
#include <linux/memblock.h>
#include <linux/memory.h>
@@ -31,12 +30,8 @@
#include <linux/stacktrace.h>
#include <linux/string.h>
#include <linux/types.h>
-#include <linux/vmalloc.h>
#include <linux/bug.h>
-#include <asm/cacheflush.h>
-#include <asm/tlbflush.h>
-
#include "kasan.h"
#include "../slab.h"
@@ -66,93 +61,6 @@ void kasan_disable_current(void)
current->kasan_depth--;
}
-bool __kasan_check_read(const volatile void *p, unsigned int size)
-{
- return check_memory_region((unsigned long)p, size, false, _RET_IP_);
-}
-EXPORT_SYMBOL(__kasan_check_read);
-
-bool __kasan_check_write(const volatile void *p, unsigned int size)
-{
- return check_memory_region((unsigned long)p, size, true, _RET_IP_);
-}
-EXPORT_SYMBOL(__kasan_check_write);
-
-#undef memset
-void *memset(void *addr, int c, size_t len)
-{
- if (!check_memory_region((unsigned long)addr, len, true, _RET_IP_))
- return NULL;
-
- return __memset(addr, c, len);
-}
-
-#ifdef __HAVE_ARCH_MEMMOVE
-#undef memmove
-void *memmove(void *dest, const void *src, size_t len)
-{
- if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
- !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
- return NULL;
-
- return __memmove(dest, src, len);
-}
-#endif
-
-#undef memcpy
-void *memcpy(void *dest, const void *src, size_t len)
-{
- if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
- !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
- return NULL;
-
- return __memcpy(dest, src, len);
-}
-
-/*
- * Poisons the shadow memory for 'size' bytes starting from 'addr'.
- * Memory addresses should be aligned to KASAN_GRANULE_SIZE.
- */
-void kasan_poison_memory(const void *address, size_t size, u8 value)
-{
- void *shadow_start, *shadow_end;
-
- /*
- * Perform shadow offset calculation based on untagged address, as
- * some of the callers (e.g. kasan_poison_object_data) pass tagged
- * addresses to this function.
- */
- address = reset_tag(address);
-
- shadow_start = kasan_mem_to_shadow(address);
- shadow_end = kasan_mem_to_shadow(address + size);
-
- __memset(shadow_start, value, shadow_end - shadow_start);
-}
-
-void kasan_unpoison_memory(const void *address, size_t size)
-{
- u8 tag = get_tag(address);
-
- /*
- * Perform shadow offset calculation based on untagged address, as
- * some of the callers (e.g. kasan_unpoison_object_data) pass tagged
- * addresses to this function.
- */
- address = reset_tag(address);
-
- kasan_poison_memory(address, size, tag);
-
- if (size & KASAN_GRANULE_MASK) {
- u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
-
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
- *shadow = tag;
- else
- *shadow = size & KASAN_GRANULE_MASK;
- }
-}
-
static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
{
void *base = task_stack_page(task);
@@ -540,395 +448,3 @@ void kasan_kfree_large(void *ptr, unsigned long ip)
kasan_report_invalid_free(ptr, ip);
/* The object will be poisoned by page_alloc. */
}
-
-#ifdef CONFIG_MEMORY_HOTPLUG
-static bool shadow_mapped(unsigned long addr)
-{
- pgd_t *pgd = pgd_offset_k(addr);
- p4d_t *p4d;
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
-
- if (pgd_none(*pgd))
- return false;
- p4d = p4d_offset(pgd, addr);
- if (p4d_none(*p4d))
- return false;
- pud = pud_offset(p4d, addr);
- if (pud_none(*pud))
- return false;
-
- /*
- * We can't use pud_large() or pud_huge(), the first one is
- * arch-specific, the last one depends on HUGETLB_PAGE. So let's abuse
- * pud_bad(), if pud is bad then it's bad because it's huge.
- */
- if (pud_bad(*pud))
- return true;
- pmd = pmd_offset(pud, addr);
- if (pmd_none(*pmd))
- return false;
-
- if (pmd_bad(*pmd))
- return true;
- pte = pte_offset_kernel(pmd, addr);
- return !pte_none(*pte);
-}
-
-static int __meminit kasan_mem_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- struct memory_notify *mem_data = data;
- unsigned long nr_shadow_pages, start_kaddr, shadow_start;
- unsigned long shadow_end, shadow_size;
-
- nr_shadow_pages = mem_data->nr_pages >> KASAN_SHADOW_SCALE_SHIFT;
- start_kaddr = (unsigned long)pfn_to_kaddr(mem_data->start_pfn);
- shadow_start = (unsigned long)kasan_mem_to_shadow((void *)start_kaddr);
- shadow_size = nr_shadow_pages << PAGE_SHIFT;
- shadow_end = shadow_start + shadow_size;
-
- if (WARN_ON(mem_data->nr_pages % KASAN_GRANULE_SIZE) ||
- WARN_ON(start_kaddr % (KASAN_GRANULE_SIZE << PAGE_SHIFT)))
- return NOTIFY_BAD;
-
- switch (action) {
- case MEM_GOING_ONLINE: {
- void *ret;
-
- /*
- * If shadow is mapped already than it must have been mapped
- * during the boot. This could happen if we onlining previously
- * offlined memory.
- */
- if (shadow_mapped(shadow_start))
- return NOTIFY_OK;
-
- ret = __vmalloc_node_range(shadow_size, PAGE_SIZE, shadow_start,
- shadow_end, GFP_KERNEL,
- PAGE_KERNEL, VM_NO_GUARD,
- pfn_to_nid(mem_data->start_pfn),
- __builtin_return_address(0));
- if (!ret)
- return NOTIFY_BAD;
-
- kmemleak_ignore(ret);
- return NOTIFY_OK;
- }
- case MEM_CANCEL_ONLINE:
- case MEM_OFFLINE: {
- struct vm_struct *vm;
-
- /*
- * shadow_start was either mapped during boot by kasan_init()
- * or during memory online by __vmalloc_node_range().
- * In the latter case we can use vfree() to free shadow.
- * Non-NULL result of the find_vm_area() will tell us if
- * that was the second case.
- *
- * Currently it's not possible to free shadow mapped
- * during boot by kasan_init(). It's because the code
- * to do that hasn't been written yet. So we'll just
- * leak the memory.
- */
- vm = find_vm_area((void *)shadow_start);
- if (vm)
- vfree((void *)shadow_start);
- }
- }
-
- return NOTIFY_OK;
-}
-
-static int __init kasan_memhotplug_init(void)
-{
- hotplug_memory_notifier(kasan_mem_notifier, 0);
-
- return 0;
-}
-
-core_initcall(kasan_memhotplug_init);
-#endif
-
-#ifdef CONFIG_KASAN_VMALLOC
-
-static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
- void *unused)
-{
- unsigned long page;
- pte_t pte;
-
- if (likely(!pte_none(*ptep)))
- return 0;
-
- page = __get_free_page(GFP_KERNEL);
- if (!page)
- return -ENOMEM;
-
- memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
- pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
-
- spin_lock(&init_mm.page_table_lock);
- if (likely(pte_none(*ptep))) {
- set_pte_at(&init_mm, addr, ptep, pte);
- page = 0;
- }
- spin_unlock(&init_mm.page_table_lock);
- if (page)
- free_page(page);
- return 0;
-}
-
-int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
-{
- unsigned long shadow_start, shadow_end;
- int ret;
-
- if (!is_vmalloc_or_module_addr((void *)addr))
- return 0;
-
- shadow_start = (unsigned long)kasan_mem_to_shadow((void *)addr);
- shadow_start = ALIGN_DOWN(shadow_start, PAGE_SIZE);
- shadow_end = (unsigned long)kasan_mem_to_shadow((void *)addr + size);
- shadow_end = ALIGN(shadow_end, PAGE_SIZE);
-
- ret = apply_to_page_range(&init_mm, shadow_start,
- shadow_end - shadow_start,
- kasan_populate_vmalloc_pte, NULL);
- if (ret)
- return ret;
-
- flush_cache_vmap(shadow_start, shadow_end);
-
- /*
- * We need to be careful about inter-cpu effects here. Consider:
- *
- * CPU#0 CPU#1
- * WRITE_ONCE(p, vmalloc(100)); while (x = READ_ONCE(p)) ;
- * p[99] = 1;
- *
- * With compiler instrumentation, that ends up looking like this:
- *
- * CPU#0 CPU#1
- * // vmalloc() allocates memory
- * // let a = area->addr
- * // we reach kasan_populate_vmalloc
- * // and call kasan_unpoison_memory:
- * STORE shadow(a), unpoison_val
- * ...
- * STORE shadow(a+99), unpoison_val x = LOAD p
- * // rest of vmalloc process <data dependency>
- * STORE p, a LOAD shadow(x+99)
- *
- * If there is no barrier between the end of unpoisioning the shadow
- * and the store of the result to p, the stores could be committed
- * in a different order by CPU#0, and CPU#1 could erroneously observe
- * poison in the shadow.
- *
- * We need some sort of barrier between the stores.
- *
- * In the vmalloc() case, this is provided by a smp_wmb() in
- * clear_vm_uninitialized_flag(). In the per-cpu allocator and in
- * get_vm_area() and friends, the caller gets shadow allocated but
- * doesn't have any pages mapped into the virtual address space that
- * has been reserved. Mapping those pages in will involve taking and
- * releasing a page-table lock, which will provide the barrier.
- */
-
- return 0;
-}
-
-/*
- * Poison the shadow for a vmalloc region. Called as part of the
- * freeing process at the time the region is freed.
- */
-void kasan_poison_vmalloc(const void *start, unsigned long size)
-{
- if (!is_vmalloc_or_module_addr(start))
- return;
-
- size = round_up(size, KASAN_GRANULE_SIZE);
- kasan_poison_memory(start, size, KASAN_VMALLOC_INVALID);
-}
-
-void kasan_unpoison_vmalloc(const void *start, unsigned long size)
-{
- if (!is_vmalloc_or_module_addr(start))
- return;
-
- kasan_unpoison_memory(start, size);
-}
-
-static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,
- void *unused)
-{
- unsigned long page;
-
- page = (unsigned long)__va(pte_pfn(*ptep) << PAGE_SHIFT);
-
- spin_lock(&init_mm.page_table_lock);
-
- if (likely(!pte_none(*ptep))) {
- pte_clear(&init_mm, addr, ptep);
- free_page(page);
- }
- spin_unlock(&init_mm.page_table_lock);
-
- return 0;
-}
-
-/*
- * Release the backing for the vmalloc region [start, end), which
- * lies within the free region [free_region_start, free_region_end).
- *
- * This can be run lazily, long after the region was freed. It runs
- * under vmap_area_lock, so it's not safe to interact with the vmalloc/vmap
- * infrastructure.
- *
- * How does this work?
- * -------------------
- *
- * We have a region that is page aligned, labelled as A.
- * That might not map onto the shadow in a way that is page-aligned:
- *
- * start end
- * v v
- * |????????|????????|AAAAAAAA|AA....AA|AAAAAAAA|????????| < vmalloc
- * -------- -------- -------- -------- --------
- * | | | | |
- * | | | /-------/ |
- * \-------\|/------/ |/---------------/
- * ||| ||
- * |??AAAAAA|AAAAAAAA|AA??????| < shadow
- * (1) (2) (3)
- *
- * First we align the start upwards and the end downwards, so that the
- * shadow of the region aligns with shadow page boundaries. In the
- * example, this gives us the shadow page (2). This is the shadow entirely
- * covered by this allocation.
- *
- * Then we have the tricky bits. We want to know if we can free the
- * partially covered shadow pages - (1) and (3) in the example. For this,
- * we are given the start and end of the free region that contains this
- * allocation. Extending our previous example, we could have:
- *
- * free_region_start free_region_end
- * | start end |
- * v v v v
- * |FFFFFFFF|FFFFFFFF|AAAAAAAA|AA....AA|AAAAAAAA|FFFFFFFF| < vmalloc
- * -------- -------- -------- -------- --------
- * | | | | |
- * | | | /-------/ |
- * \-------\|/------/ |/---------------/
- * ||| ||
- * |FFAAAAAA|AAAAAAAA|AAF?????| < shadow
- * (1) (2) (3)
- *
- * Once again, we align the start of the free region up, and the end of
- * the free region down so that the shadow is page aligned. So we can free
- * page (1) - we know no allocation currently uses anything in that page,
- * because all of it is in the vmalloc free region. But we cannot free
- * page (3), because we can't be sure that the rest of it is unused.
- *
- * We only consider pages that contain part of the original region for
- * freeing: we don't try to free other pages from the free region or we'd
- * end up trying to free huge chunks of virtual address space.
- *
- * Concurrency
- * -----------
- *
- * How do we know that we're not freeing a page that is simultaneously
- * being used for a fresh allocation in kasan_populate_vmalloc(_pte)?
- *
- * We _can_ have kasan_release_vmalloc and kasan_populate_vmalloc running
- * at the same time. While we run under free_vmap_area_lock, the population
- * code does not.
- *
- * free_vmap_area_lock instead operates to ensure that the larger range
- * [free_region_start, free_region_end) is safe: because __alloc_vmap_area and
- * the per-cpu region-finding algorithm both run under free_vmap_area_lock,
- * no space identified as free will become used while we are running. This
- * means that so long as we are careful with alignment and only free shadow
- * pages entirely covered by the free region, we will not run in to any
- * trouble - any simultaneous allocations will be for disjoint regions.
- */
-void kasan_release_vmalloc(unsigned long start, unsigned long end,
- unsigned long free_region_start,
- unsigned long free_region_end)
-{
- void *shadow_start, *shadow_end;
- unsigned long region_start, region_end;
- unsigned long size;
-
- region_start = ALIGN(start, PAGE_SIZE * KASAN_GRANULE_SIZE);
- region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_GRANULE_SIZE);
-
- free_region_start = ALIGN(free_region_start,
- PAGE_SIZE * KASAN_GRANULE_SIZE);
-
- if (start != region_start &&
- free_region_start < region_start)
- region_start -= PAGE_SIZE * KASAN_GRANULE_SIZE;
-
- free_region_end = ALIGN_DOWN(free_region_end,
- PAGE_SIZE * KASAN_GRANULE_SIZE);
-
- if (end != region_end &&
- free_region_end > region_end)
- region_end += PAGE_SIZE * KASAN_GRANULE_SIZE;
-
- shadow_start = kasan_mem_to_shadow((void *)region_start);
- shadow_end = kasan_mem_to_shadow((void *)region_end);
-
- if (shadow_end > shadow_start) {
- size = shadow_end - shadow_start;
- apply_to_existing_page_range(&init_mm,
- (unsigned long)shadow_start,
- size, kasan_depopulate_vmalloc_pte,
- NULL);
- flush_tlb_kernel_range((unsigned long)shadow_start,
- (unsigned long)shadow_end);
- }
-}
-
-#else /* CONFIG_KASAN_VMALLOC */
-
-int kasan_module_alloc(void *addr, size_t size)
-{
- void *ret;
- size_t scaled_size;
- size_t shadow_size;
- unsigned long shadow_start;
-
- shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
- scaled_size = (size + KASAN_GRANULE_SIZE - 1) >>
- KASAN_SHADOW_SCALE_SHIFT;
- shadow_size = round_up(scaled_size, PAGE_SIZE);
-
- if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
- return -EINVAL;
-
- ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
- shadow_start + shadow_size,
- GFP_KERNEL,
- PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
- __builtin_return_address(0));
-
- if (ret) {
- __memset(ret, KASAN_SHADOW_INIT, shadow_size);
- find_vm_area(addr)->flags |= VM_KASAN;
- kmemleak_ignore(ret);
- return 0;
- }
-
- return -ENOMEM;
-}
-
-void kasan_free_shadow(const struct vm_struct *vm)
-{
- if (vm->flags & VM_KASAN)
- vfree(kasan_mem_to_shadow(vm->addr));
-}
-
-#endif
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
new file mode 100644
index 000000000000..4888084ecdfc
--- /dev/null
+++ b/mm/kasan/shadow.c
@@ -0,0 +1,509 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file contains KASAN shadow runtime code.
+ *
+ * Copyright (c) 2014 Samsung Electronics Co., Ltd.
+ * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
+ *
+ * Some code borrowed from https://github.com/xairy/kasan-prototype by
+ * Andrey Konovalov <andreyknvl@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/init.h>
+#include <linux/kasan.h>
+#include <linux/kernel.h>
+#include <linux/kmemleak.h>
+#include <linux/memory.h>
+#include <linux/mm.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/vmalloc.h>
+
+#include <asm/cacheflush.h>
+#include <asm/tlbflush.h>
+
+#include "kasan.h"
+
+bool __kasan_check_read(const volatile void *p, unsigned int size)
+{
+ return check_memory_region((unsigned long)p, size, false, _RET_IP_);
+}
+EXPORT_SYMBOL(__kasan_check_read);
+
+bool __kasan_check_write(const volatile void *p, unsigned int size)
+{
+ return check_memory_region((unsigned long)p, size, true, _RET_IP_);
+}
+EXPORT_SYMBOL(__kasan_check_write);
+
+#undef memset
+void *memset(void *addr, int c, size_t len)
+{
+ if (!check_memory_region((unsigned long)addr, len, true, _RET_IP_))
+ return NULL;
+
+ return __memset(addr, c, len);
+}
+
+#ifdef __HAVE_ARCH_MEMMOVE
+#undef memmove
+void *memmove(void *dest, const void *src, size_t len)
+{
+ if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
+ !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
+ return NULL;
+
+ return __memmove(dest, src, len);
+}
+#endif
+
+#undef memcpy
+void *memcpy(void *dest, const void *src, size_t len)
+{
+ if (!check_memory_region((unsigned long)src, len, false, _RET_IP_) ||
+ !check_memory_region((unsigned long)dest, len, true, _RET_IP_))
+ return NULL;
+
+ return __memcpy(dest, src, len);
+}
+
+/*
+ * Poisons the shadow memory for 'size' bytes starting from 'addr'.
+ * Memory addresses should be aligned to KASAN_GRANULE_SIZE.
+ */
+void kasan_poison_memory(const void *address, size_t size, u8 value)
+{
+ void *shadow_start, *shadow_end;
+
+ /*
+ * Perform shadow offset calculation based on untagged address, as
+ * some of the callers (e.g. kasan_poison_object_data) pass tagged
+ * addresses to this function.
+ */
+ address = reset_tag(address);
+
+ shadow_start = kasan_mem_to_shadow(address);
+ shadow_end = kasan_mem_to_shadow(address + size);
+
+ __memset(shadow_start, value, shadow_end - shadow_start);
+}
+
+void kasan_unpoison_memory(const void *address, size_t size)
+{
+ u8 tag = get_tag(address);
+
+ /*
+ * Perform shadow offset calculation based on untagged address, as
+ * some of the callers (e.g. kasan_unpoison_object_data) pass tagged
+ * addresses to this function.
+ */
+ address = reset_tag(address);
+
+ kasan_poison_memory(address, size, tag);
+
+ if (size & KASAN_GRANULE_MASK) {
+ u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size);
+
+ if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ *shadow = tag;
+ else
+ *shadow = size & KASAN_GRANULE_MASK;
+ }
+}
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static bool shadow_mapped(unsigned long addr)
+{
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+ pte_t *pte;
+
+ if (pgd_none(*pgd))
+ return false;
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(*p4d))
+ return false;
+ pud = pud_offset(p4d, addr);
+ if (pud_none(*pud))
+ return false;
+
+ /*
+ * We can't use pud_large() or pud_huge(), the first one is
+ * arch-specific, the last one depends on HUGETLB_PAGE. So let's abuse
+ * pud_bad(), if pud is bad then it's bad because it's huge.
+ */
+ if (pud_bad(*pud))
+ return true;
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return false;
+
+ if (pmd_bad(*pmd))
+ return true;
+ pte = pte_offset_kernel(pmd, addr);
+ return !pte_none(*pte);
+}
+
+static int __meminit kasan_mem_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ struct memory_notify *mem_data = data;
+ unsigned long nr_shadow_pages, start_kaddr, shadow_start;
+ unsigned long shadow_end, shadow_size;
+
+ nr_shadow_pages = mem_data->nr_pages >> KASAN_SHADOW_SCALE_SHIFT;
+ start_kaddr = (unsigned long)pfn_to_kaddr(mem_data->start_pfn);
+ shadow_start = (unsigned long)kasan_mem_to_shadow((void *)start_kaddr);
+ shadow_size = nr_shadow_pages << PAGE_SHIFT;
+ shadow_end = shadow_start + shadow_size;
+
+ if (WARN_ON(mem_data->nr_pages % KASAN_GRANULE_SIZE) ||
+ WARN_ON(start_kaddr % (KASAN_GRANULE_SIZE << PAGE_SHIFT)))
+ return NOTIFY_BAD;
+
+ switch (action) {
+ case MEM_GOING_ONLINE: {
+ void *ret;
+
+ /*
+ * If shadow is mapped already than it must have been mapped
+ * during the boot. This could happen if we onlining previously
+ * offlined memory.
+ */
+ if (shadow_mapped(shadow_start))
+ return NOTIFY_OK;
+
+ ret = __vmalloc_node_range(shadow_size, PAGE_SIZE, shadow_start,
+ shadow_end, GFP_KERNEL,
+ PAGE_KERNEL, VM_NO_GUARD,
+ pfn_to_nid(mem_data->start_pfn),
+ __builtin_return_address(0));
+ if (!ret)
+ return NOTIFY_BAD;
+
+ kmemleak_ignore(ret);
+ return NOTIFY_OK;
+ }
+ case MEM_CANCEL_ONLINE:
+ case MEM_OFFLINE: {
+ struct vm_struct *vm;
+
+ /*
+ * shadow_start was either mapped during boot by kasan_init()
+ * or during memory online by __vmalloc_node_range().
+ * In the latter case we can use vfree() to free shadow.
+ * Non-NULL result of the find_vm_area() will tell us if
+ * that was the second case.
+ *
+ * Currently it's not possible to free shadow mapped
+ * during boot by kasan_init(). It's because the code
+ * to do that hasn't been written yet. So we'll just
+ * leak the memory.
+ */
+ vm = find_vm_area((void *)shadow_start);
+ if (vm)
+ vfree((void *)shadow_start);
+ }
+ }
+
+ return NOTIFY_OK;
+}
+
+static int __init kasan_memhotplug_init(void)
+{
+ hotplug_memory_notifier(kasan_mem_notifier, 0);
+
+ return 0;
+}
+
+core_initcall(kasan_memhotplug_init);
+#endif
+
+#ifdef CONFIG_KASAN_VMALLOC
+
+static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
+ void *unused)
+{
+ unsigned long page;
+ pte_t pte;
+
+ if (likely(!pte_none(*ptep)))
+ return 0;
+
+ page = __get_free_page(GFP_KERNEL);
+ if (!page)
+ return -ENOMEM;
+
+ memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
+ pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
+
+ spin_lock(&init_mm.page_table_lock);
+ if (likely(pte_none(*ptep))) {
+ set_pte_at(&init_mm, addr, ptep, pte);
+ page = 0;
+ }
+ spin_unlock(&init_mm.page_table_lock);
+ if (page)
+ free_page(page);
+ return 0;
+}
+
+int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
+{
+ unsigned long shadow_start, shadow_end;
+ int ret;
+
+ if (!is_vmalloc_or_module_addr((void *)addr))
+ return 0;
+
+ shadow_start = (unsigned long)kasan_mem_to_shadow((void *)addr);
+ shadow_start = ALIGN_DOWN(shadow_start, PAGE_SIZE);
+ shadow_end = (unsigned long)kasan_mem_to_shadow((void *)addr + size);
+ shadow_end = ALIGN(shadow_end, PAGE_SIZE);
+
+ ret = apply_to_page_range(&init_mm, shadow_start,
+ shadow_end - shadow_start,
+ kasan_populate_vmalloc_pte, NULL);
+ if (ret)
+ return ret;
+
+ flush_cache_vmap(shadow_start, shadow_end);
+
+ /*
+ * We need to be careful about inter-cpu effects here. Consider:
+ *
+ * CPU#0 CPU#1
+ * WRITE_ONCE(p, vmalloc(100)); while (x = READ_ONCE(p)) ;
+ * p[99] = 1;
+ *
+ * With compiler instrumentation, that ends up looking like this:
+ *
+ * CPU#0 CPU#1
+ * // vmalloc() allocates memory
+ * // let a = area->addr
+ * // we reach kasan_populate_vmalloc
+ * // and call kasan_unpoison_memory:
+ * STORE shadow(a), unpoison_val
+ * ...
+ * STORE shadow(a+99), unpoison_val x = LOAD p
+ * // rest of vmalloc process <data dependency>
+ * STORE p, a LOAD shadow(x+99)
+ *
+ * If there is no barrier between the end of unpoisioning the shadow
+ * and the store of the result to p, the stores could be committed
+ * in a different order by CPU#0, and CPU#1 could erroneously observe
+ * poison in the shadow.
+ *
+ * We need some sort of barrier between the stores.
+ *
+ * In the vmalloc() case, this is provided by a smp_wmb() in
+ * clear_vm_uninitialized_flag(). In the per-cpu allocator and in
+ * get_vm_area() and friends, the caller gets shadow allocated but
+ * doesn't have any pages mapped into the virtual address space that
+ * has been reserved. Mapping those pages in will involve taking and
+ * releasing a page-table lock, which will provide the barrier.
+ */
+
+ return 0;
+}
+
+/*
+ * Poison the shadow for a vmalloc region. Called as part of the
+ * freeing process at the time the region is freed.
+ */
+void kasan_poison_vmalloc(const void *start, unsigned long size)
+{
+ if (!is_vmalloc_or_module_addr(start))
+ return;
+
+ size = round_up(size, KASAN_GRANULE_SIZE);
+ kasan_poison_memory(start, size, KASAN_VMALLOC_INVALID);
+}
+
+void kasan_unpoison_vmalloc(const void *start, unsigned long size)
+{
+ if (!is_vmalloc_or_module_addr(start))
+ return;
+
+ kasan_unpoison_memory(start, size);
+}
+
+static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,
+ void *unused)
+{
+ unsigned long page;
+
+ page = (unsigned long)__va(pte_pfn(*ptep) << PAGE_SHIFT);
+
+ spin_lock(&init_mm.page_table_lock);
+
+ if (likely(!pte_none(*ptep))) {
+ pte_clear(&init_mm, addr, ptep);
+ free_page(page);
+ }
+ spin_unlock(&init_mm.page_table_lock);
+
+ return 0;
+}
+
+/*
+ * Release the backing for the vmalloc region [start, end), which
+ * lies within the free region [free_region_start, free_region_end).
+ *
+ * This can be run lazily, long after the region was freed. It runs
+ * under vmap_area_lock, so it's not safe to interact with the vmalloc/vmap
+ * infrastructure.
+ *
+ * How does this work?
+ * -------------------
+ *
+ * We have a region that is page aligned, labelled as A.
+ * That might not map onto the shadow in a way that is page-aligned:
+ *
+ * start end
+ * v v
+ * |????????|????????|AAAAAAAA|AA....AA|AAAAAAAA|????????| < vmalloc
+ * -------- -------- -------- -------- --------
+ * | | | | |
+ * | | | /-------/ |
+ * \-------\|/------/ |/---------------/
+ * ||| ||
+ * |??AAAAAA|AAAAAAAA|AA??????| < shadow
+ * (1) (2) (3)
+ *
+ * First we align the start upwards and the end downwards, so that the
+ * shadow of the region aligns with shadow page boundaries. In the
+ * example, this gives us the shadow page (2). This is the shadow entirely
+ * covered by this allocation.
+ *
+ * Then we have the tricky bits. We want to know if we can free the
+ * partially covered shadow pages - (1) and (3) in the example. For this,
+ * we are given the start and end of the free region that contains this
+ * allocation. Extending our previous example, we could have:
+ *
+ * free_region_start free_region_end
+ * | start end |
+ * v v v v
+ * |FFFFFFFF|FFFFFFFF|AAAAAAAA|AA....AA|AAAAAAAA|FFFFFFFF| < vmalloc
+ * -------- -------- -------- -------- --------
+ * | | | | |
+ * | | | /-------/ |
+ * \-------\|/------/ |/---------------/
+ * ||| ||
+ * |FFAAAAAA|AAAAAAAA|AAF?????| < shadow
+ * (1) (2) (3)
+ *
+ * Once again, we align the start of the free region up, and the end of
+ * the free region down so that the shadow is page aligned. So we can free
+ * page (1) - we know no allocation currently uses anything in that page,
+ * because all of it is in the vmalloc free region. But we cannot free
+ * page (3), because we can't be sure that the rest of it is unused.
+ *
+ * We only consider pages that contain part of the original region for
+ * freeing: we don't try to free other pages from the free region or we'd
+ * end up trying to free huge chunks of virtual address space.
+ *
+ * Concurrency
+ * -----------
+ *
+ * How do we know that we're not freeing a page that is simultaneously
+ * being used for a fresh allocation in kasan_populate_vmalloc(_pte)?
+ *
+ * We _can_ have kasan_release_vmalloc and kasan_populate_vmalloc running
+ * at the same time. While we run under free_vmap_area_lock, the population
+ * code does not.
+ *
+ * free_vmap_area_lock instead operates to ensure that the larger range
+ * [free_region_start, free_region_end) is safe: because __alloc_vmap_area and
+ * the per-cpu region-finding algorithm both run under free_vmap_area_lock,
+ * no space identified as free will become used while we are running. This
+ * means that so long as we are careful with alignment and only free shadow
+ * pages entirely covered by the free region, we will not run in to any
+ * trouble - any simultaneous allocations will be for disjoint regions.
+ */
+void kasan_release_vmalloc(unsigned long start, unsigned long end,
+ unsigned long free_region_start,
+ unsigned long free_region_end)
+{
+ void *shadow_start, *shadow_end;
+ unsigned long region_start, region_end;
+ unsigned long size;
+
+ region_start = ALIGN(start, PAGE_SIZE * KASAN_GRANULE_SIZE);
+ region_end = ALIGN_DOWN(end, PAGE_SIZE * KASAN_GRANULE_SIZE);
+
+ free_region_start = ALIGN(free_region_start,
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
+
+ if (start != region_start &&
+ free_region_start < region_start)
+ region_start -= PAGE_SIZE * KASAN_GRANULE_SIZE;
+
+ free_region_end = ALIGN_DOWN(free_region_end,
+ PAGE_SIZE * KASAN_GRANULE_SIZE);
+
+ if (end != region_end &&
+ free_region_end > region_end)
+ region_end += PAGE_SIZE * KASAN_GRANULE_SIZE;
+
+ shadow_start = kasan_mem_to_shadow((void *)region_start);
+ shadow_end = kasan_mem_to_shadow((void *)region_end);
+
+ if (shadow_end > shadow_start) {
+ size = shadow_end - shadow_start;
+ apply_to_existing_page_range(&init_mm,
+ (unsigned long)shadow_start,
+ size, kasan_depopulate_vmalloc_pte,
+ NULL);
+ flush_tlb_kernel_range((unsigned long)shadow_start,
+ (unsigned long)shadow_end);
+ }
+}
+
+#else /* CONFIG_KASAN_VMALLOC */
+
+int kasan_module_alloc(void *addr, size_t size)
+{
+ void *ret;
+ size_t scaled_size;
+ size_t shadow_size;
+ unsigned long shadow_start;
+
+ shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
+ scaled_size = (size + KASAN_GRANULE_SIZE - 1) >>
+ KASAN_SHADOW_SCALE_SHIFT;
+ shadow_size = round_up(scaled_size, PAGE_SIZE);
+
+ if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
+ return -EINVAL;
+
+ ret = __vmalloc_node_range(shadow_size, 1, shadow_start,
+ shadow_start + shadow_size,
+ GFP_KERNEL,
+ PAGE_KERNEL, VM_NO_GUARD, NUMA_NO_NODE,
+ __builtin_return_address(0));
+
+ if (ret) {
+ __memset(ret, KASAN_SHADOW_INIT, shadow_size);
+ find_vm_area(addr)->flags |= VM_KASAN;
+ kmemleak_ignore(ret);
+ return 0;
+ }
+
+ return -ENOMEM;
+}
+
+void kasan_free_shadow(const struct vm_struct *vm)
+{
+ if (vm->flags & VM_KASAN)
+ vfree(kasan_mem_to_shadow(vm->addr));
+}
+
+#endif
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 08/35] kasan: rename generic/tags_report.c files
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (6 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 07/35] kasan: split out shadow.c from common.c Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 09/35] kasan: don't duplicate config dependencies Andrey Konovalov
` (26 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Rename generic_report.c to report_generic.c and tags_report.c to
report_tags.c, as their content is more relevant to report.c file,
then to generic.c or tags.c.
No functional changes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/Makefile | 12 ++++++------
mm/kasan/report.c | 2 +-
mm/kasan/{generic_report.c => report_generic.c} | 0
mm/kasan/{tags_report.c => report_tags.c} | 0
4 files changed, 7 insertions(+), 7 deletions(-)
rename mm/kasan/{generic_report.c => report_generic.c} (100%)
rename mm/kasan/{tags_report.c => report_tags.c} (100%)
diff --git a/mm/kasan/Makefile b/mm/kasan/Makefile
index 40366d706b7c..007c824f6f43 100644
--- a/mm/kasan/Makefile
+++ b/mm/kasan/Makefile
@@ -6,13 +6,13 @@ KCOV_INSTRUMENT := n
# Disable ftrace to avoid recursion.
CFLAGS_REMOVE_common.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_generic.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_generic_report.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_init.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_quarantine.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_report_generic.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_report_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_shadow.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_tags.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_tags_report.o = $(CC_FLAGS_FTRACE)
# Function splitter causes unnecessary splits in __asan_load1/__asan_store1
# see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63533
@@ -23,14 +23,14 @@ CC_FLAGS_KASAN_RUNTIME += -DDISABLE_BRANCH_PROFILING
CFLAGS_common.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_generic.o := $(CC_FLAGS_KASAN_RUNTIME)
-CFLAGS_generic_report.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_init.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_quarantine.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_report_generic.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_report_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_shadow.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
-CFLAGS_tags_report.o := $(CC_FLAGS_KASAN_RUNTIME)
obj-$(CONFIG_KASAN) := common.o report.o
-obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o generic_report.o shadow.o quarantine.o
-obj-$(CONFIG_KASAN_SW_TAGS) += init.o shadow.o tags.o tags_report.o
+obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o report_generic.o shadow.o quarantine.o
+obj-$(CONFIG_KASAN_SW_TAGS) += init.o report_tags.o shadow.o tags.o
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 7c025d792e2f..f16591ba9e2e 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * This file contains common generic and tag-based KASAN error reporting code.
+ * This file contains common KASAN error reporting code.
*
* Copyright (c) 2014 Samsung Electronics Co., Ltd.
* Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
diff --git a/mm/kasan/generic_report.c b/mm/kasan/report_generic.c
similarity index 100%
rename from mm/kasan/generic_report.c
rename to mm/kasan/report_generic.c
diff --git a/mm/kasan/tags_report.c b/mm/kasan/report_tags.c
similarity index 100%
rename from mm/kasan/tags_report.c
rename to mm/kasan/report_tags.c
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 09/35] kasan: don't duplicate config dependencies
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (7 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 08/35] kasan: rename generic/tags_report.c files Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 10/35] kasan: hide invalid free check implementation Andrey Konovalov
` (25 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Both KASAN_GENERIC and KASAN_SW_TAGS have common dependencies, move
those to KASAN.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
lib/Kconfig.kasan | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index e1d55331b618..b4cf6c519d71 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -24,6 +24,9 @@ menuconfig KASAN
(HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS)
depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
depends on CC_HAS_WORKING_NOSANITIZE_ADDRESS
+ select SLUB_DEBUG if SLUB
+ select CONSTRUCTORS
+ select STACKDEPOT
help
Enables KASAN (KernelAddressSANitizer) - runtime memory debugger,
designed to find out-of-bounds accesses and use-after-free bugs.
@@ -46,10 +49,6 @@ choice
config KASAN_GENERIC
bool "Generic mode"
depends on HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC
- depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
- select SLUB_DEBUG if SLUB
- select CONSTRUCTORS
- select STACKDEPOT
help
Enables generic KASAN mode.
@@ -70,10 +69,6 @@ config KASAN_GENERIC
config KASAN_SW_TAGS
bool "Software tag-based mode"
depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
- depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
- select SLUB_DEBUG if SLUB
- select CONSTRUCTORS
- select STACKDEPOT
help
Enables software tag-based KASAN mode.
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 10/35] kasan: hide invalid free check implementation
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (8 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 09/35] kasan: don't duplicate config dependencies Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 11/35] kasan: decode stack frame only with KASAN_STACK_ENABLE Andrey Konovalov
` (24 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
For software KASAN modes the check is based on the value in the shadow
memory. Hardware tag-based KASAN won't be using shadow, so hide the
implementation of the check in check_invalid_free().
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/common.c | 19 +------------------
mm/kasan/generic.c | 7 +++++++
mm/kasan/kasan.h | 2 ++
mm/kasan/tags.c | 12 ++++++++++++
4 files changed, 22 insertions(+), 18 deletions(-)
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 43a927e70067..a2321d35390e 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -277,25 +277,9 @@ void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
return (void *)object;
}
-static inline bool shadow_invalid(u8 tag, s8 shadow_byte)
-{
- if (IS_ENABLED(CONFIG_KASAN_GENERIC))
- return shadow_byte < 0 ||
- shadow_byte >= KASAN_GRANULE_SIZE;
-
- /* else CONFIG_KASAN_SW_TAGS: */
- if ((u8)shadow_byte == KASAN_TAG_INVALID)
- return true;
- if ((tag != KASAN_TAG_KERNEL) && (tag != (u8)shadow_byte))
- return true;
-
- return false;
-}
-
static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
unsigned long ip, bool quarantine)
{
- s8 shadow_byte;
u8 tag;
void *tagged_object;
unsigned long rounded_up_size;
@@ -314,8 +298,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
if (unlikely(cache->flags & SLAB_TYPESAFE_BY_RCU))
return false;
- shadow_byte = READ_ONCE(*(s8 *)kasan_mem_to_shadow(object));
- if (shadow_invalid(tag, shadow_byte)) {
+ if (check_invalid_free(tagged_object)) {
kasan_report_invalid_free(tagged_object, ip);
return true;
}
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index f6d68aa9872f..73f4d786ad5d 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -192,6 +192,13 @@ bool check_memory_region(unsigned long addr, size_t size, bool write,
return check_memory_region_inline(addr, size, write, ret_ip);
}
+bool check_invalid_free(void *addr)
+{
+ s8 shadow_byte = READ_ONCE(*(s8 *)kasan_mem_to_shadow(addr));
+
+ return shadow_byte < 0 || shadow_byte >= KASAN_GRANULE_SIZE;
+}
+
void kasan_cache_shrink(struct kmem_cache *cache)
{
quarantine_remove_cache(cache);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index c31e2c739301..cf6a135860f2 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -163,6 +163,8 @@ void kasan_poison_memory(const void *address, size_t size, u8 value);
bool check_memory_region(unsigned long addr, size_t size, bool write,
unsigned long ret_ip);
+bool check_invalid_free(void *addr);
+
void *find_first_bad_addr(void *addr, size_t size);
const char *get_bug_type(struct kasan_access_info *info);
diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c
index 4d5a1fe8251f..feb42c1763b8 100644
--- a/mm/kasan/tags.c
+++ b/mm/kasan/tags.c
@@ -126,6 +126,18 @@ bool check_memory_region(unsigned long addr, size_t size, bool write,
return true;
}
+bool check_invalid_free(void *addr)
+{
+ u8 tag = get_tag(addr);
+ u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(reset_tag(addr)));
+
+ if (shadow_byte == KASAN_TAG_INVALID)
+ return true;
+ if (tag != KASAN_TAG_KERNEL && tag != shadow_byte)
+ return true;
+ return false;
+}
+
#define DEFINE_HWASAN_LOAD_STORE(size) \
void __hwasan_load##size##_noabort(unsigned long addr) \
{ \
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 11/35] kasan: decode stack frame only with KASAN_STACK_ENABLE
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (9 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 10/35] kasan: hide invalid free check implementation Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 12/35] kasan, arm64: only init shadow for software modes Andrey Konovalov
` (23 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Decoding routines aren't needed when CONFIG_KASAN_STACK_ENABLE is not
enabled. Currently only generic KASAN mode implements stack error
reporting.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/kasan.h | 6 ++
mm/kasan/report.c | 162 --------------------------------------
mm/kasan/report_generic.c | 161 +++++++++++++++++++++++++++++++++++++
3 files changed, 167 insertions(+), 162 deletions(-)
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index cf6a135860f2..15cf3e0018ae 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -168,6 +168,12 @@ bool check_invalid_free(void *addr);
void *find_first_bad_addr(void *addr, size_t size);
const char *get_bug_type(struct kasan_access_info *info);
+#ifdef CONFIG_KASAN_STACK_ENABLE
+void print_address_stack_frame(const void *addr);
+#else
+static inline void print_address_stack_frame(const void *addr) { }
+#endif
+
bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
void kasan_report_invalid_free(void *object, unsigned long ip);
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index f16591ba9e2e..ddaf9d14ca81 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -214,168 +214,6 @@ static inline bool init_task_stack_addr(const void *addr)
sizeof(init_thread_union.stack));
}
-static bool __must_check tokenize_frame_descr(const char **frame_descr,
- char *token, size_t max_tok_len,
- unsigned long *value)
-{
- const char *sep = strchr(*frame_descr, ' ');
-
- if (sep == NULL)
- sep = *frame_descr + strlen(*frame_descr);
-
- if (token != NULL) {
- const size_t tok_len = sep - *frame_descr;
-
- if (tok_len + 1 > max_tok_len) {
- pr_err("KASAN internal error: frame description too long: %s\n",
- *frame_descr);
- return false;
- }
-
- /* Copy token (+ 1 byte for '\0'). */
- strlcpy(token, *frame_descr, tok_len + 1);
- }
-
- /* Advance frame_descr past separator. */
- *frame_descr = sep + 1;
-
- if (value != NULL && kstrtoul(token, 10, value)) {
- pr_err("KASAN internal error: not a valid number: %s\n", token);
- return false;
- }
-
- return true;
-}
-
-static void print_decoded_frame_descr(const char *frame_descr)
-{
- /*
- * We need to parse the following string:
- * "n alloc_1 alloc_2 ... alloc_n"
- * where alloc_i looks like
- * "offset size len name"
- * or "offset size len name:line".
- */
-
- char token[64];
- unsigned long num_objects;
-
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- &num_objects))
- return;
-
- pr_err("\n");
- pr_err("this frame has %lu %s:\n", num_objects,
- num_objects == 1 ? "object" : "objects");
-
- while (num_objects--) {
- unsigned long offset;
- unsigned long size;
-
- /* access offset */
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- &offset))
- return;
- /* access size */
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- &size))
- return;
- /* name length (unused) */
- if (!tokenize_frame_descr(&frame_descr, NULL, 0, NULL))
- return;
- /* object name */
- if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
- NULL))
- return;
-
- /* Strip line number; without filename it's not very helpful. */
- strreplace(token, ':', '\0');
-
- /* Finally, print object information. */
- pr_err(" [%lu, %lu) '%s'", offset, offset + size, token);
- }
-}
-
-static bool __must_check get_address_stack_frame_info(const void *addr,
- unsigned long *offset,
- const char **frame_descr,
- const void **frame_pc)
-{
- unsigned long aligned_addr;
- unsigned long mem_ptr;
- const u8 *shadow_bottom;
- const u8 *shadow_ptr;
- const unsigned long *frame;
-
- BUILD_BUG_ON(IS_ENABLED(CONFIG_STACK_GROWSUP));
-
- /*
- * NOTE: We currently only support printing frame information for
- * accesses to the task's own stack.
- */
- if (!object_is_on_stack(addr))
- return false;
-
- aligned_addr = round_down((unsigned long)addr, sizeof(long));
- mem_ptr = round_down(aligned_addr, KASAN_GRANULE_SIZE);
- shadow_ptr = kasan_mem_to_shadow((void *)aligned_addr);
- shadow_bottom = kasan_mem_to_shadow(end_of_stack(current));
-
- while (shadow_ptr >= shadow_bottom && *shadow_ptr != KASAN_STACK_LEFT) {
- shadow_ptr--;
- mem_ptr -= KASAN_GRANULE_SIZE;
- }
-
- while (shadow_ptr >= shadow_bottom && *shadow_ptr == KASAN_STACK_LEFT) {
- shadow_ptr--;
- mem_ptr -= KASAN_GRANULE_SIZE;
- }
-
- if (shadow_ptr < shadow_bottom)
- return false;
-
- frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
- if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
- pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",
- frame[0]);
- return false;
- }
-
- *offset = (unsigned long)addr - (unsigned long)frame;
- *frame_descr = (const char *)frame[1];
- *frame_pc = (void *)frame[2];
-
- return true;
-}
-
-static void print_address_stack_frame(const void *addr)
-{
- unsigned long offset;
- const char *frame_descr;
- const void *frame_pc;
-
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
- return;
-
- if (!get_address_stack_frame_info(addr, &offset, &frame_descr,
- &frame_pc))
- return;
-
- /*
- * get_address_stack_frame_info only returns true if the given addr is
- * on the current task's stack.
- */
- pr_err("\n");
- pr_err("addr %px is located in stack of task %s/%d at offset %lu in frame:\n",
- addr, current->comm, task_pid_nr(current), offset);
- pr_err(" %pS\n", frame_pc);
-
- if (!frame_descr)
- return;
-
- print_decoded_frame_descr(frame_descr);
-}
-
static void print_address_description(void *addr, u8 tag)
{
struct page *page = kasan_addr_to_page(addr);
diff --git a/mm/kasan/report_generic.c b/mm/kasan/report_generic.c
index 4dce1633b082..427f4ac80cca 100644
--- a/mm/kasan/report_generic.c
+++ b/mm/kasan/report_generic.c
@@ -127,6 +127,167 @@ const char *get_bug_type(struct kasan_access_info *info)
return get_wild_bug_type(info);
}
+#ifdef CONFIG_KASAN_STACK_ENABLE
+static bool __must_check tokenize_frame_descr(const char **frame_descr,
+ char *token, size_t max_tok_len,
+ unsigned long *value)
+{
+ const char *sep = strchr(*frame_descr, ' ');
+
+ if (sep == NULL)
+ sep = *frame_descr + strlen(*frame_descr);
+
+ if (token != NULL) {
+ const size_t tok_len = sep - *frame_descr;
+
+ if (tok_len + 1 > max_tok_len) {
+ pr_err("KASAN internal error: frame description too long: %s\n",
+ *frame_descr);
+ return false;
+ }
+
+ /* Copy token (+ 1 byte for '\0'). */
+ strlcpy(token, *frame_descr, tok_len + 1);
+ }
+
+ /* Advance frame_descr past separator. */
+ *frame_descr = sep + 1;
+
+ if (value != NULL && kstrtoul(token, 10, value)) {
+ pr_err("KASAN internal error: not a valid number: %s\n", token);
+ return false;
+ }
+
+ return true;
+}
+
+static void print_decoded_frame_descr(const char *frame_descr)
+{
+ /*
+ * We need to parse the following string:
+ * "n alloc_1 alloc_2 ... alloc_n"
+ * where alloc_i looks like
+ * "offset size len name"
+ * or "offset size len name:line".
+ */
+
+ char token[64];
+ unsigned long num_objects;
+
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ &num_objects))
+ return;
+
+ pr_err("\n");
+ pr_err("this frame has %lu %s:\n", num_objects,
+ num_objects == 1 ? "object" : "objects");
+
+ while (num_objects--) {
+ unsigned long offset;
+ unsigned long size;
+
+ /* access offset */
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ &offset))
+ return;
+ /* access size */
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ &size))
+ return;
+ /* name length (unused) */
+ if (!tokenize_frame_descr(&frame_descr, NULL, 0, NULL))
+ return;
+ /* object name */
+ if (!tokenize_frame_descr(&frame_descr, token, sizeof(token),
+ NULL))
+ return;
+
+ /* Strip line number; without filename it's not very helpful. */
+ strreplace(token, ':', '\0');
+
+ /* Finally, print object information. */
+ pr_err(" [%lu, %lu) '%s'", offset, offset + size, token);
+ }
+}
+
+static bool __must_check get_address_stack_frame_info(const void *addr,
+ unsigned long *offset,
+ const char **frame_descr,
+ const void **frame_pc)
+{
+ unsigned long aligned_addr;
+ unsigned long mem_ptr;
+ const u8 *shadow_bottom;
+ const u8 *shadow_ptr;
+ const unsigned long *frame;
+
+ BUILD_BUG_ON(IS_ENABLED(CONFIG_STACK_GROWSUP));
+
+ /*
+ * NOTE: We currently only support printing frame information for
+ * accesses to the task's own stack.
+ */
+ if (!object_is_on_stack(addr))
+ return false;
+
+ aligned_addr = round_down((unsigned long)addr, sizeof(long));
+ mem_ptr = round_down(aligned_addr, KASAN_GRANULE_SIZE);
+ shadow_ptr = kasan_mem_to_shadow((void *)aligned_addr);
+ shadow_bottom = kasan_mem_to_shadow(end_of_stack(current));
+
+ while (shadow_ptr >= shadow_bottom && *shadow_ptr != KASAN_STACK_LEFT) {
+ shadow_ptr--;
+ mem_ptr -= KASAN_GRANULE_SIZE;
+ }
+
+ while (shadow_ptr >= shadow_bottom && *shadow_ptr == KASAN_STACK_LEFT) {
+ shadow_ptr--;
+ mem_ptr -= KASAN_GRANULE_SIZE;
+ }
+
+ if (shadow_ptr < shadow_bottom)
+ return false;
+
+ frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
+ if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
+ pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",
+ frame[0]);
+ return false;
+ }
+
+ *offset = (unsigned long)addr - (unsigned long)frame;
+ *frame_descr = (const char *)frame[1];
+ *frame_pc = (void *)frame[2];
+
+ return true;
+}
+
+void print_address_stack_frame(const void *addr)
+{
+ unsigned long offset;
+ const char *frame_descr;
+ const void *frame_pc;
+
+ if (!get_address_stack_frame_info(addr, &offset, &frame_descr,
+ &frame_pc))
+ return;
+
+ /*
+ * get_address_stack_frame_info only returns true if the given addr is
+ * on the current task's stack.
+ */
+ pr_err("\n");
+ pr_err("addr %px is located in stack of task %s/%d at offset %lu in frame:\n",
+ addr, current->comm, task_pid_nr(current), offset);
+ pr_err(" %pS\n", frame_pc);
+
+ if (!frame_descr)
+ return;
+
+ print_decoded_frame_descr(frame_descr);
+}
+#endif /* CONFIG_KASAN_STACK_ENABLE */
+
#define DEFINE_ASAN_REPORT_LOAD(size) \
void __asan_report_load##size##_noabort(unsigned long addr) \
{ \
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 12/35] kasan, arm64: only init shadow for software modes
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (10 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 11/35] kasan: decode stack frame only with KASAN_STACK_ENABLE Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 13/35] kasan, arm64: only use kasan_depth " Andrey Konovalov
` (22 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory. Only initialize
it when one of the software KASAN modes are enabled.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/include/asm/kasan.h | 8 ++++++--
arch/arm64/mm/kasan_init.c | 15 ++++++++++++++-
2 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h
index b0dc4abc3589..f7ea70d02cab 100644
--- a/arch/arm64/include/asm/kasan.h
+++ b/arch/arm64/include/asm/kasan.h
@@ -13,6 +13,12 @@
#define arch_kasan_get_tag(addr) __tag_get(addr)
#ifdef CONFIG_KASAN
+void kasan_init(void);
+#else
+static inline void kasan_init(void) { }
+#endif
+
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
/*
* KASAN_SHADOW_START: beginning of the kernel virtual addresses.
@@ -33,12 +39,10 @@
#define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT)))
#define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual)
-void kasan_init(void);
void kasan_copy_shadow(pgd_t *pgdir);
asmlinkage void kasan_early_init(void);
#else
-static inline void kasan_init(void) { }
static inline void kasan_copy_shadow(pgd_t *pgdir) { }
#endif
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 7291b26ce788..4d35eaf3ec97 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -21,6 +21,8 @@
#include <asm/sections.h>
#include <asm/tlbflush.h>
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+
static pgd_t tmp_pg_dir[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE);
/*
@@ -208,7 +210,7 @@ static void __init clear_pgds(unsigned long start,
set_pgd(pgd_offset_k(start), __pgd(0));
}
-void __init kasan_init(void)
+static void __init kasan_init_shadow(void)
{
u64 kimg_shadow_start, kimg_shadow_end;
u64 mod_shadow_start, mod_shadow_end;
@@ -269,6 +271,17 @@ void __init kasan_init(void)
memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE);
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
+}
+
+#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) */
+
+static inline void __init kasan_init_shadow(void) { }
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+void __init kasan_init(void)
+{
+ kasan_init_shadow();
/* At this point kasan is fully initialized. Enable error messages */
init_task.kasan_depth = 0;
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 13/35] kasan, arm64: only use kasan_depth for software modes
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (11 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 12/35] kasan, arm64: only init shadow for software modes Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 14/35] kasan: rename addr_has_shadow to addr_has_metadata Andrey Konovalov
` (21 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't use kasan_depth. Only define and use it
when one of the software KASAN modes are enabled.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/mm/kasan_init.c | 11 ++++++++---
include/linux/kasan.h | 14 ++++++++++----
include/linux/sched.h | 2 +-
init/init_task.c | 2 +-
mm/kasan/common.c | 2 ++
mm/kasan/report.c | 2 ++
6 files changed, 24 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 4d35eaf3ec97..b6b9d55bb72e 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -273,17 +273,22 @@ static void __init kasan_init_shadow(void)
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
}
+void __init kasan_init_depth(void)
+{
+ init_task.kasan_depth = 0;
+}
+
#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) */
static inline void __init kasan_init_shadow(void) { }
+static inline void __init kasan_init_depth(void) { }
+
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
void __init kasan_init(void)
{
kasan_init_shadow();
-
- /* At this point kasan is fully initialized. Enable error messages */
- init_task.kasan_depth = 0;
+ kasan_init_depth();
pr_info("KernelAddressSanitizer initialized\n");
}
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 18617d5c4cd7..894f4d9163ee 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -52,7 +52,7 @@ static inline void kasan_remove_zero_shadow(void *start,
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
/* Enable reporting bugs after kasan_disable_current() */
extern void kasan_enable_current(void);
@@ -60,6 +60,15 @@ extern void kasan_enable_current(void);
/* Disable reporting bugs for current task */
extern void kasan_disable_current(void);
+#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+static inline void kasan_enable_current(void) {}
+static inline void kasan_disable_current(void) {}
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+#ifdef CONFIG_KASAN
+
void kasan_unpoison_memory(const void *address, size_t size);
void kasan_unpoison_task_stack(struct task_struct *task);
@@ -110,9 +119,6 @@ static inline void kasan_unpoison_memory(const void *address, size_t size) {}
static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
-static inline void kasan_enable_current(void) {}
-static inline void kasan_disable_current(void) {}
-
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 692e327d7455..6dca19f2516c 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1194,7 +1194,7 @@ struct task_struct {
u64 timer_slack_ns;
u64 default_timer_slack_ns;
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
unsigned int kasan_depth;
#endif
#ifdef CONFIG_KCSAN
diff --git a/init/init_task.c b/init/init_task.c
index 15089d15010a..13f1cf21412b 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -171,7 +171,7 @@ struct task_struct init_task
.numa_group = NULL,
.numa_faults = NULL,
#endif
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
.kasan_depth = 1,
#endif
#ifdef CONFIG_KCSAN
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index a2321d35390e..41c7f1105eaa 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -51,6 +51,7 @@ void kasan_set_track(struct kasan_track *track, gfp_t flags)
track->stack = kasan_save_stack(flags);
}
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
void kasan_enable_current(void)
{
current->kasan_depth++;
@@ -60,6 +61,7 @@ void kasan_disable_current(void)
{
current->kasan_depth--;
}
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
{
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index ddaf9d14ca81..8463e35b489f 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -295,8 +295,10 @@ static void print_shadow_for_address(const void *addr)
static bool report_enabled(void)
{
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
if (current->kasan_depth)
return false;
+#endif
if (test_bit(KASAN_BIT_MULTI_SHOT, &kasan_flags))
return true;
return !test_and_set_bit(KASAN_BIT_REPORTED, &kasan_flags);
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 14/35] kasan: rename addr_has_shadow to addr_has_metadata
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (12 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 13/35] kasan, arm64: only use kasan_depth " Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 15/35] kasan: rename print_shadow_for_address to print_memory_metadata Andrey Konovalov
` (20 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory, but will reuse
this function. Rename "shadow" to implementation-neutral "metadata".
No functional changes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/kasan.h | 2 +-
mm/kasan/report.c | 6 +++---
mm/kasan/report_generic.c | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 15cf3e0018ae..38fa4c202e9a 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -145,7 +145,7 @@ static inline const void *kasan_shadow_to_mem(const void *shadow_addr)
<< KASAN_SHADOW_SCALE_SHIFT);
}
-static inline bool addr_has_shadow(const void *addr)
+static inline bool addr_has_metadata(const void *addr)
{
return (addr >= kasan_shadow_to_mem((void *)KASAN_SHADOW_START));
}
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 8463e35b489f..ada3cfb43764 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -334,7 +334,7 @@ static void __kasan_report(unsigned long addr, size_t size, bool is_write,
untagged_addr = reset_tag(tagged_addr);
info.access_addr = tagged_addr;
- if (addr_has_shadow(untagged_addr))
+ if (addr_has_metadata(untagged_addr))
info.first_bad_addr = find_first_bad_addr(tagged_addr, size);
else
info.first_bad_addr = untagged_addr;
@@ -345,11 +345,11 @@ static void __kasan_report(unsigned long addr, size_t size, bool is_write,
start_report(&flags);
print_error_description(&info);
- if (addr_has_shadow(untagged_addr))
+ if (addr_has_metadata(untagged_addr))
print_tags(get_tag(tagged_addr), info.first_bad_addr);
pr_err("\n");
- if (addr_has_shadow(untagged_addr)) {
+ if (addr_has_metadata(untagged_addr)) {
print_address_description(untagged_addr, get_tag(tagged_addr));
pr_err("\n");
print_shadow_for_address(info.first_bad_addr);
diff --git a/mm/kasan/report_generic.c b/mm/kasan/report_generic.c
index 427f4ac80cca..29d30fae9421 100644
--- a/mm/kasan/report_generic.c
+++ b/mm/kasan/report_generic.c
@@ -122,7 +122,7 @@ const char *get_bug_type(struct kasan_access_info *info)
if (info->access_addr + info->access_size < info->access_addr)
return "out-of-bounds";
- if (addr_has_shadow(info->access_addr))
+ if (addr_has_metadata(info->access_addr))
return get_shadow_bug_type(info);
return get_wild_bug_type(info);
}
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 15/35] kasan: rename print_shadow_for_address to print_memory_metadata
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (13 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 14/35] kasan: rename addr_has_shadow to addr_has_metadata Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 16/35] kasan: kasan_non_canonical_hook only for software modes Andrey Konovalov
` (19 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory, but will reuse
this function. Rename "shadow" to implementation-neutral "metadata".
No functional changes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/report.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index ada3cfb43764..8ad1ced1607d 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -255,7 +255,7 @@ static int shadow_pointer_offset(const void *row, const void *shadow)
(shadow - row) / SHADOW_BYTES_PER_BLOCK + 1;
}
-static void print_shadow_for_address(const void *addr)
+static void print_memory_metadata(const void *addr)
{
int i;
const void *shadow = kasan_mem_to_shadow(addr);
@@ -316,7 +316,7 @@ void kasan_report_invalid_free(void *object, unsigned long ip)
pr_err("\n");
print_address_description(object, tag);
pr_err("\n");
- print_shadow_for_address(object);
+ print_memory_metadata(object);
end_report(&flags);
}
@@ -352,7 +352,7 @@ static void __kasan_report(unsigned long addr, size_t size, bool is_write,
if (addr_has_metadata(untagged_addr)) {
print_address_description(untagged_addr, get_tag(tagged_addr));
pr_err("\n");
- print_shadow_for_address(info.first_bad_addr);
+ print_memory_metadata(info.first_bad_addr);
} else {
dump_stack();
}
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 16/35] kasan: kasan_non_canonical_hook only for software modes
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (14 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 15/35] kasan: rename print_shadow_for_address to print_memory_metadata Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 17/35] kasan: rename SHADOW layout macros to META Andrey Konovalov
` (18 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
kasan_non_canonical_hook() is only applicable to KASAN modes that use
shadow memory, and won't be needed for hardware tag-based KASAN.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/report.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 8ad1ced1607d..2cce7c9beea3 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -376,7 +376,8 @@ bool kasan_report(unsigned long addr, size_t size, bool is_write,
return ret;
}
-#ifdef CONFIG_KASAN_INLINE
+#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
+ defined(CONFIG_KASAN_INLINE)
/*
* With CONFIG_KASAN_INLINE, accesses to bogus pointers (outside the high
* canonical half of the address space) cause out-of-bounds shadow memory reads
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 17/35] kasan: rename SHADOW layout macros to META
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (15 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 16/35] kasan: kasan_non_canonical_hook only for software modes Andrey Konovalov
@ 2020-08-14 17:26 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 18/35] kasan: separate metadata_fetch_row for each mode Andrey Konovalov
` (17 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:26 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Hardware tag-based KASAN won't be using shadow memory, but will reuse
these macros. Rename "SHADOW" to implementation-neutral "META".
No functional changes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/report.c | 30 +++++++++++++++---------------
1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 2cce7c9beea3..6306673e7062 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -36,11 +36,11 @@
#include "kasan.h"
#include "../slab.h"
-/* Shadow layout customization. */
-#define SHADOW_BYTES_PER_BLOCK 1
-#define SHADOW_BLOCKS_PER_ROW 16
-#define SHADOW_BYTES_PER_ROW (SHADOW_BLOCKS_PER_ROW * SHADOW_BYTES_PER_BLOCK)
-#define SHADOW_ROWS_AROUND_ADDR 2
+/* Metadata layout customization. */
+#define META_BYTES_PER_BLOCK 1
+#define META_BLOCKS_PER_ROW 16
+#define META_BYTES_PER_ROW (META_BLOCKS_PER_ROW * META_BYTES_PER_BLOCK)
+#define META_ROWS_AROUND_ADDR 2
static unsigned long kasan_flags;
@@ -243,7 +243,7 @@ static void print_address_description(void *addr, u8 tag)
static bool row_is_guilty(const void *row, const void *guilty)
{
- return (row <= guilty) && (guilty < row + SHADOW_BYTES_PER_ROW);
+ return (row <= guilty) && (guilty < row + META_BYTES_PER_ROW);
}
static int shadow_pointer_offset(const void *row, const void *shadow)
@@ -252,7 +252,7 @@ static int shadow_pointer_offset(const void *row, const void *shadow)
* 3 + (BITS_PER_LONG/8)*2 chars.
*/
return 3 + (BITS_PER_LONG/8)*2 + (shadow - row)*2 +
- (shadow - row) / SHADOW_BYTES_PER_BLOCK + 1;
+ (shadow - row) / META_BYTES_PER_BLOCK + 1;
}
static void print_memory_metadata(const void *addr)
@@ -262,15 +262,15 @@ static void print_memory_metadata(const void *addr)
const void *shadow_row;
shadow_row = (void *)round_down((unsigned long)shadow,
- SHADOW_BYTES_PER_ROW)
- - SHADOW_ROWS_AROUND_ADDR * SHADOW_BYTES_PER_ROW;
+ META_BYTES_PER_ROW)
+ - META_ROWS_AROUND_ADDR * META_BYTES_PER_ROW;
pr_err("Memory state around the buggy address:\n");
- for (i = -SHADOW_ROWS_AROUND_ADDR; i <= SHADOW_ROWS_AROUND_ADDR; i++) {
+ for (i = -META_ROWS_AROUND_ADDR; i <= META_ROWS_AROUND_ADDR; i++) {
const void *kaddr = kasan_shadow_to_mem(shadow_row);
char buffer[4 + (BITS_PER_LONG/8)*2];
- char shadow_buf[SHADOW_BYTES_PER_ROW];
+ char shadow_buf[META_BYTES_PER_ROW];
snprintf(buffer, sizeof(buffer),
(i == 0) ? ">%px: " : " %px: ", kaddr);
@@ -279,17 +279,17 @@ static void print_memory_metadata(const void *addr)
* function, because generic functions may try to
* access kasan mapping for the passed address.
*/
- memcpy(shadow_buf, shadow_row, SHADOW_BYTES_PER_ROW);
+ memcpy(shadow_buf, shadow_row, META_BYTES_PER_ROW);
print_hex_dump(KERN_ERR, buffer,
- DUMP_PREFIX_NONE, SHADOW_BYTES_PER_ROW, 1,
- shadow_buf, SHADOW_BYTES_PER_ROW, 0);
+ DUMP_PREFIX_NONE, META_BYTES_PER_ROW, 1,
+ shadow_buf, META_BYTES_PER_ROW, 0);
if (row_is_guilty(shadow_row, shadow))
pr_err("%*c\n",
shadow_pointer_offset(shadow_row, shadow),
'^');
- shadow_row += SHADOW_BYTES_PER_ROW;
+ shadow_row += META_BYTES_PER_ROW;
}
}
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 18/35] kasan: separate metadata_fetch_row for each mode
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (16 preceding siblings ...)
2020-08-14 17:26 ` [PATCH 17/35] kasan: rename SHADOW layout macros to META Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE Andrey Konovalov
` (16 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This is a preparatory commit for the upcoming addition of a new hardware
tag-based (MTE-based) KASAN mode.
Rework print_memory_metadata() to make it agnostic with regard to the
way metadata is stored. Allow providing a separate metadata_fetch_row()
implementation for each KASAN mode. Hardware tag-based KASAN will provide
its own implementation that doesn't use shadow memory.
No functional changes for software modes.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/kasan.h | 8 ++++++
mm/kasan/report.c | 56 +++++++++++++++++++--------------------
mm/kasan/report_generic.c | 5 ++++
mm/kasan/report_tags.c | 5 ++++
4 files changed, 45 insertions(+), 29 deletions(-)
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 38fa4c202e9a..1d3c7c6ce771 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -56,6 +56,13 @@
#define KASAN_ABI_VERSION 1
#endif
+/* Metadata layout customization. */
+#define META_BYTES_PER_BLOCK 1
+#define META_BLOCKS_PER_ROW 16
+#define META_BYTES_PER_ROW (META_BLOCKS_PER_ROW * META_BYTES_PER_BLOCK)
+#define META_MEM_BYTES_PER_ROW (META_BYTES_PER_ROW * KASAN_GRANULE_SIZE)
+#define META_ROWS_AROUND_ADDR 2
+
struct kasan_access_info {
const void *access_addr;
const void *first_bad_addr;
@@ -167,6 +174,7 @@ bool check_invalid_free(void *addr);
void *find_first_bad_addr(void *addr, size_t size);
const char *get_bug_type(struct kasan_access_info *info);
+void metadata_fetch_row(char *buffer, void *row);
#ifdef CONFIG_KASAN_STACK_ENABLE
void print_address_stack_frame(const void *addr);
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 6306673e7062..c904edab33b8 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -36,12 +36,6 @@
#include "kasan.h"
#include "../slab.h"
-/* Metadata layout customization. */
-#define META_BYTES_PER_BLOCK 1
-#define META_BLOCKS_PER_ROW 16
-#define META_BYTES_PER_ROW (META_BLOCKS_PER_ROW * META_BYTES_PER_BLOCK)
-#define META_ROWS_AROUND_ADDR 2
-
static unsigned long kasan_flags;
#define KASAN_BIT_REPORTED 0
@@ -241,55 +235,59 @@ static void print_address_description(void *addr, u8 tag)
print_address_stack_frame(addr);
}
-static bool row_is_guilty(const void *row, const void *guilty)
+static bool meta_row_is_guilty(const void *row, const void *addr)
{
- return (row <= guilty) && (guilty < row + META_BYTES_PER_ROW);
+ return (row <= addr) && (addr < row + META_MEM_BYTES_PER_ROW);
}
-static int shadow_pointer_offset(const void *row, const void *shadow)
+static int meta_pointer_offset(const void *row, const void *addr)
{
- /* The length of ">ff00ff00ff00ff00: " is
- * 3 + (BITS_PER_LONG/8)*2 chars.
+ /*
+ * Memory state around the buggy address:
+ * ff00ff00ff00ff00: 00 00 00 05 fe fe fe fe fe fe fe fe fe fe fe fe
+ * ...
+ *
+ * The length of ">ff00ff00ff00ff00: " is
+ * 3 + (BITS_PER_LONG / 8) * 2 chars.
+ * The length of each granule metadata is 2 bytes
+ * plus 1 byte for space.
*/
- return 3 + (BITS_PER_LONG/8)*2 + (shadow - row)*2 +
- (shadow - row) / META_BYTES_PER_BLOCK + 1;
+ return 3 + (BITS_PER_LONG / 8) * 2 +
+ (addr - row) / KASAN_GRANULE_SIZE * 3 + 1;
}
static void print_memory_metadata(const void *addr)
{
int i;
- const void *shadow = kasan_mem_to_shadow(addr);
- const void *shadow_row;
+ void *row;
- shadow_row = (void *)round_down((unsigned long)shadow,
- META_BYTES_PER_ROW)
- - META_ROWS_AROUND_ADDR * META_BYTES_PER_ROW;
+ row = (void *)round_down((unsigned long)addr, META_MEM_BYTES_PER_ROW)
+ - META_ROWS_AROUND_ADDR * META_MEM_BYTES_PER_ROW;
pr_err("Memory state around the buggy address:\n");
for (i = -META_ROWS_AROUND_ADDR; i <= META_ROWS_AROUND_ADDR; i++) {
- const void *kaddr = kasan_shadow_to_mem(shadow_row);
- char buffer[4 + (BITS_PER_LONG/8)*2];
- char shadow_buf[META_BYTES_PER_ROW];
+ char buffer[4 + (BITS_PER_LONG / 8) * 2];
+ char metadata[META_BYTES_PER_ROW];
snprintf(buffer, sizeof(buffer),
- (i == 0) ? ">%px: " : " %px: ", kaddr);
+ (i == 0) ? ">%px: " : " %px: ", row);
+
/*
* We should not pass a shadow pointer to generic
* function, because generic functions may try to
* access kasan mapping for the passed address.
*/
- memcpy(shadow_buf, shadow_row, META_BYTES_PER_ROW);
+ metadata_fetch_row(&metadata[0], row);
+
print_hex_dump(KERN_ERR, buffer,
DUMP_PREFIX_NONE, META_BYTES_PER_ROW, 1,
- shadow_buf, META_BYTES_PER_ROW, 0);
+ metadata, META_BYTES_PER_ROW, 0);
- if (row_is_guilty(shadow_row, shadow))
- pr_err("%*c\n",
- shadow_pointer_offset(shadow_row, shadow),
- '^');
+ if (meta_row_is_guilty(row, addr))
+ pr_err("%*c\n", meta_pointer_offset(row, addr), '^');
- shadow_row += META_BYTES_PER_ROW;
+ row += META_MEM_BYTES_PER_ROW;
}
}
diff --git a/mm/kasan/report_generic.c b/mm/kasan/report_generic.c
index 29d30fae9421..6524651b5d2e 100644
--- a/mm/kasan/report_generic.c
+++ b/mm/kasan/report_generic.c
@@ -127,6 +127,11 @@ const char *get_bug_type(struct kasan_access_info *info)
return get_wild_bug_type(info);
}
+void metadata_fetch_row(char *buffer, void *row)
+{
+ memcpy(buffer, kasan_mem_to_shadow(row), META_BYTES_PER_ROW);
+}
+
#ifdef CONFIG_KASAN_STACK_ENABLE
static bool __must_check tokenize_frame_descr(const char **frame_descr,
char *token, size_t max_tok_len,
diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c
index 6ddb55676a7c..4060d0503462 100644
--- a/mm/kasan/report_tags.c
+++ b/mm/kasan/report_tags.c
@@ -85,6 +85,11 @@ void *find_first_bad_addr(void *addr, size_t size)
return p;
}
+void metadata_fetch_row(char *buffer, void *row)
+{
+ memcpy(buffer, kasan_mem_to_shadow(row), META_BYTES_PER_ROW);
+}
+
void print_tags(u8 addr_tag, const void *addr)
{
u8 *shadow = (u8 *)kasan_mem_to_shadow(addr);
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (17 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 18/35] kasan: separate metadata_fetch_row for each mode Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 8:04 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers Andrey Konovalov
` (15 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Software tag-based KASAN provides its own tag checking machinery that
can conflict with MTE. Don't allow enabling software tag-based KASAN
when MTE is enabled.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
lib/Kconfig.kasan | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index b4cf6c519d71..e500c18cbe79 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -69,6 +69,7 @@ config KASAN_GENERIC
config KASAN_SW_TAGS
bool "Software tag-based mode"
depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
+ depends on !ARM64_MTE
help
Enables software tag-based KASAN mode.
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (18 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 9:38 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler Andrey Konovalov
` (14 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Provide helper functions to manipulate allocation and pointer tags for
kernel addresses.
Low-level helper functions (mte_assign_*, written in assembly) operate
tag values from the [0x0, 0xF] range. High-level helper functions
(mte_get/set_*) use the [0xF0, 0xFF] range to preserve compatibility
with normal kernel pointers that have 0xFF in their top byte.
MTE_GRANULE_SIZE definition is moved to mte_asm.h header that doesn't
have any dependencies and is safe to include into any low-level header.
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Co-developed-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/include/asm/esr.h | 1 +
arch/arm64/include/asm/mte.h | 46 +++++++++++++++++++++++++++++---
arch/arm64/include/asm/mte_asm.h | 10 +++++++
arch/arm64/kernel/mte.c | 43 +++++++++++++++++++++++++++++
arch/arm64/lib/mte.S | 41 ++++++++++++++++++++++++++++
5 files changed, 138 insertions(+), 3 deletions(-)
create mode 100644 arch/arm64/include/asm/mte_asm.h
diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
index 035003acfa87..bc0dc66a6a27 100644
--- a/arch/arm64/include/asm/esr.h
+++ b/arch/arm64/include/asm/esr.h
@@ -103,6 +103,7 @@
#define ESR_ELx_FSC (0x3F)
#define ESR_ELx_FSC_TYPE (0x3C)
#define ESR_ELx_FSC_EXTABT (0x10)
+#define ESR_ELx_FSC_MTE (0x11)
#define ESR_ELx_FSC_SERROR (0x11)
#define ESR_ELx_FSC_ACCESS (0x08)
#define ESR_ELx_FSC_FAULT (0x04)
diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 1c99fcadb58c..733be1cb5c95 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -5,14 +5,19 @@
#ifndef __ASM_MTE_H
#define __ASM_MTE_H
-#define MTE_GRANULE_SIZE UL(16)
+#include <asm/mte_asm.h>
+
#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
#define MTE_TAG_SHIFT 56
#define MTE_TAG_SIZE 4
+#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
+#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
#ifndef __ASSEMBLY__
+#include <linux/bitfield.h>
#include <linux/page-flags.h>
+#include <linux/types.h>
#include <asm/pgtable-types.h>
@@ -45,7 +50,16 @@ long get_mte_ctrl(struct task_struct *task);
int mte_ptrace_copy_tags(struct task_struct *child, long request,
unsigned long addr, unsigned long data);
-#else
+void *mte_assign_valid_ptr_tag(void *ptr);
+void *mte_assign_random_ptr_tag(void *ptr);
+void mte_assign_mem_tag_range(void *addr, size_t size);
+
+#define mte_get_ptr_tag(ptr) ((u8)(((u64)(ptr)) >> MTE_TAG_SHIFT))
+u8 mte_get_mem_tag(void *addr);
+u8 mte_get_random_tag(void);
+void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag);
+
+#else /* CONFIG_ARM64_MTE */
/* unused if !CONFIG_ARM64_MTE, silence the compiler */
#define PG_mte_tagged 0
@@ -80,7 +94,33 @@ static inline int mte_ptrace_copy_tags(struct task_struct *child,
return -EIO;
}
-#endif
+static inline void *mte_assign_valid_ptr_tag(void *ptr)
+{
+ return ptr;
+}
+static inline void *mte_assign_random_ptr_tag(void *ptr)
+{
+ return ptr;
+}
+static inline void mte_assign_mem_tag_range(void *addr, size_t size)
+{
+}
+
+#define mte_get_ptr_tag(ptr) 0xFF
+static inline u8 mte_get_mem_tag(void *addr)
+{
+ return 0xFF;
+}
+static inline u8 mte_get_random_tag(void)
+{
+ return 0xFF;
+}
+static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
+{
+ return addr;
+}
+
+#endif /* CONFIG_ARM64_MTE */
#endif /* __ASSEMBLY__ */
#endif /* __ASM_MTE_H */
diff --git a/arch/arm64/include/asm/mte_asm.h b/arch/arm64/include/asm/mte_asm.h
new file mode 100644
index 000000000000..aa532c1851e1
--- /dev/null
+++ b/arch/arm64/include/asm/mte_asm.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 ARM Ltd.
+ */
+#ifndef __ASM_MTE_ASM_H
+#define __ASM_MTE_ASM_H
+
+#define MTE_GRANULE_SIZE UL(16)
+
+#endif /* __ASM_MTE_ASM_H */
diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index eb39504e390a..e2d708b4583d 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -13,8 +13,10 @@
#include <linux/swap.h>
#include <linux/swapops.h>
#include <linux/thread_info.h>
+#include <linux/types.h>
#include <linux/uio.h>
+#include <asm/barrier.h>
#include <asm/cpufeature.h>
#include <asm/mte.h>
#include <asm/ptrace.h>
@@ -72,6 +74,47 @@ int memcmp_pages(struct page *page1, struct page *page2)
return ret;
}
+u8 mte_get_mem_tag(void *addr)
+{
+ if (system_supports_mte())
+ addr = mte_assign_valid_ptr_tag(addr);
+
+ return 0xF0 | mte_get_ptr_tag(addr);
+}
+
+u8 mte_get_random_tag(void)
+{
+ u8 tag = 0xF;
+
+ if (system_supports_mte())
+ tag = mte_get_ptr_tag(mte_assign_random_ptr_tag(NULL));
+
+ return 0xF0 | tag;
+}
+
+void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
+{
+ void *ptr = addr;
+
+ if ((!system_supports_mte()) || (size == 0))
+ return addr;
+
+ tag = 0xF0 | (tag & 0xF);
+ ptr = (void *)__tag_set(ptr, tag);
+ size = ALIGN(size, MTE_GRANULE_SIZE);
+
+ mte_assign_mem_tag_range(ptr, size);
+
+ /*
+ * mte_assign_mem_tag_range() can be invoked in a multi-threaded
+ * context, ensure that tags are written in memory before the
+ * reference is used.
+ */
+ smp_wmb();
+
+ return ptr;
+}
+
static void update_sctlr_el1_tcf0(u64 tcf0)
{
/* ISB required for the kernel uaccess routines */
diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
index 03ca6d8b8670..8c743540e32c 100644
--- a/arch/arm64/lib/mte.S
+++ b/arch/arm64/lib/mte.S
@@ -149,3 +149,44 @@ SYM_FUNC_START(mte_restore_page_tags)
ret
SYM_FUNC_END(mte_restore_page_tags)
+
+/*
+ * Assign pointer tag based on the allocation tag
+ * x0 - source pointer
+ * Returns:
+ * x0 - pointer with the correct tag to access memory
+ */
+SYM_FUNC_START(mte_assign_valid_ptr_tag)
+ ldg x0, [x0]
+ ret
+SYM_FUNC_END(mte_assign_valid_ptr_tag)
+
+/*
+ * Assign random pointer tag
+ * x0 - source pointer
+ * Returns:
+ * x0 - pointer with a random tag
+ */
+SYM_FUNC_START(mte_assign_random_ptr_tag)
+ irg x0, x0
+ ret
+SYM_FUNC_END(mte_assign_random_ptr_tag)
+
+/*
+ * Assign allocation tags for a region of memory based on the pointer tag
+ * x0 - source pointer
+ * x1 - size
+ *
+ * Note: size is expected to be MTE_GRANULE_SIZE aligned
+ */
+SYM_FUNC_START(mte_assign_mem_tag_range)
+ /* if (src == NULL) return; */
+ cbz x0, 2f
+ /* if (size == 0) return; */
+ cbz x1, 2f
+1: stg x0, [x0]
+ add x0, x0, #MTE_GRANULE_SIZE
+ sub x1, x1, #MTE_GRANULE_SIZE
+ cbnz x1, 1b
+2: ret
+SYM_FUNC_END(mte_assign_mem_tag_range)
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (19 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 9:54 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 22/35] arm64: mte: Enable in-kernel MTE Andrey Konovalov
` (13 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Add the implementation of the in-kernel fault handler.
When a tag fault happens on a kernel address:
* a warning is logged,
* the faulting instruction is skipped,
* the execution continues.
When a tag fault happens on a user address:
* the kernel executes do_bad_area() and panics.
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Co-developed-by: Andrey Konovalov <andreyknvl@google.com>
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/mm/fault.c | 50 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 49 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 5e832b3387f1..c62c8ba85c0e 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -33,6 +33,7 @@
#include <asm/debug-monitors.h>
#include <asm/esr.h>
#include <asm/kprobes.h>
+#include <asm/mte.h>
#include <asm/processor.h>
#include <asm/sysreg.h>
#include <asm/system_misc.h>
@@ -222,6 +223,20 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return 1;
}
+static bool is_el1_mte_sync_tag_check_fault(unsigned int esr)
+{
+ unsigned int ec = ESR_ELx_EC(esr);
+ unsigned int fsc = esr & ESR_ELx_FSC;
+
+ if (ec != ESR_ELx_EC_DABT_CUR)
+ return false;
+
+ if (fsc == ESR_ELx_FSC_MTE)
+ return true;
+
+ return false;
+}
+
static bool is_el1_instruction_abort(unsigned int esr)
{
return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_CUR;
@@ -294,6 +309,18 @@ static void die_kernel_fault(const char *msg, unsigned long addr,
do_exit(SIGKILL);
}
+static void report_tag_fault(unsigned long addr, unsigned int esr,
+ struct pt_regs *regs)
+{
+ bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
+
+ pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
+ pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
+ pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
+ mte_get_ptr_tag(addr),
+ mte_get_mem_tag((void *)addr));
+}
+
static void __do_kernel_fault(unsigned long addr, unsigned int esr,
struct pt_regs *regs)
{
@@ -317,12 +344,16 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
msg = "execute from non-executable memory";
else
msg = "read from unreadable memory";
+ } else if (is_el1_mte_sync_tag_check_fault(esr)) {
+ report_tag_fault(addr, esr, regs);
+ msg = "memory tagging extension fault";
} else if (addr < PAGE_SIZE) {
msg = "NULL pointer dereference";
} else {
msg = "paging request";
}
+
die_kernel_fault(msg, addr, esr, regs);
}
@@ -658,10 +689,27 @@ static int do_sea(unsigned long addr, unsigned int esr, struct pt_regs *regs)
return 0;
}
+static int do_tag_recovery(unsigned long addr, unsigned int esr,
+ struct pt_regs *regs)
+{
+ report_tag_fault(addr, esr, regs);
+
+ /* Skip over the faulting instruction and continue: */
+ arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
+
+ return 0;
+}
+
+
static int do_tag_check_fault(unsigned long addr, unsigned int esr,
struct pt_regs *regs)
{
- do_bad_area(addr, esr, regs);
+ /* The tag check fault (TCF) is per TTBR */
+ if (is_ttbr0_addr(addr))
+ do_bad_area(addr, esr, regs);
+ else
+ do_tag_recovery(addr, esr, regs);
+
return 0;
}
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 22/35] arm64: mte: Enable in-kernel MTE
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (20 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 10:01 ` Catalin Marinas
2020-09-08 14:39 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 23/35] arm64: mte: Convert gcr_user into an exclude mask Andrey Konovalov
` (12 subsequent siblings)
34 siblings, 2 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
The Tag Checking operation causes a synchronous data abort as
a consequence of a tag check fault when MTE is configured in
synchronous mode.
Enable MTE in Synchronous mode in EL1 to provide a more immediate
way of tag check failure detection in the kernel.
As part of this change enable match-all tag for EL1 to allow the
kernel to access user pages without faulting. This is required because
the kernel does not have knowledge of the tags set by the user in a
page.
Note: For MTE, the TCF bit field in SCTLR_EL1 affects only EL1 in a
similar way as TCF0 affects EL0.
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
---
arch/arm64/kernel/cpufeature.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 4d3abb51f7d4..4d94af19d8f6 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1670,6 +1670,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
write_sysreg_s(0, SYS_TFSR_EL1);
write_sysreg_s(0, SYS_TFSRE0_EL1);
+ /* Enable Match-All at EL1 */
+ sysreg_clear_set(tcr_el1, 0, SYS_TCR_EL1_TCMA1);
+
/*
* CnP must be enabled only after the MAIR_EL1 register has been set
* up. Inconsistent MAIR_EL1 between CPUs sharing the same TLB may
@@ -1687,6 +1690,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
mair &= ~MAIR_ATTRIDX(MAIR_ATTR_MASK, MT_NORMAL_TAGGED);
mair |= MAIR_ATTRIDX(MAIR_ATTR_NORMAL_TAGGED, MT_NORMAL_TAGGED);
write_sysreg_s(mair, SYS_MAIR_EL1);
+
+ /* Enable MTE Sync Mode for EL1 */
+ sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC);
isb();
local_flush_tlb_all();
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 23/35] arm64: mte: Convert gcr_user into an exclude mask
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (21 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 22/35] arm64: mte: Enable in-kernel MTE Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit Andrey Konovalov
` (11 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
The gcr_user mask is a per thread mask that represents the tags that are
excluded from random generation when the Memory Tagging Extension is
present and an 'irg' instruction is invoked.
gcr_user affects the behavior on EL0 only.
Currently that mask is an include mask and it is controlled by the user
via prctl() while GCR_EL1 accepts an exclude mask.
Convert the include mask into an exclude one to make it easier the
register setting.
Note: This change will affect gcr_kernel (for EL1) introduced with a
future patch.
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
---
arch/arm64/include/asm/processor.h | 2 +-
arch/arm64/kernel/mte.c | 29 +++++++++++++++--------------
2 files changed, 16 insertions(+), 15 deletions(-)
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index fec204d28fce..ed9efa5be8eb 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -153,7 +153,7 @@ struct thread_struct {
#endif
#ifdef CONFIG_ARM64_MTE
u64 sctlr_tcf0;
- u64 gcr_user_incl;
+ u64 gcr_user_excl;
#endif
};
diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index e2d708b4583d..7717ea9bc2a7 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -135,23 +135,22 @@ static void set_sctlr_el1_tcf0(u64 tcf0)
preempt_enable();
}
-static void update_gcr_el1_excl(u64 incl)
+static void update_gcr_el1_excl(u64 excl)
{
- u64 excl = ~incl & SYS_GCR_EL1_EXCL_MASK;
/*
- * Note that 'incl' is an include mask (controlled by the user via
- * prctl()) while GCR_EL1 accepts an exclude mask.
+ * Note that the mask controlled by the user via prctl() is an
+ * include while GCR_EL1 accepts an exclude mask.
* No need for ISB since this only affects EL0 currently, implicit
* with ERET.
*/
sysreg_clear_set_s(SYS_GCR_EL1, SYS_GCR_EL1_EXCL_MASK, excl);
}
-static void set_gcr_el1_excl(u64 incl)
+static void set_gcr_el1_excl(u64 excl)
{
- current->thread.gcr_user_incl = incl;
- update_gcr_el1_excl(incl);
+ current->thread.gcr_user_excl = excl;
+ update_gcr_el1_excl(excl);
}
void flush_mte_state(void)
@@ -166,7 +165,7 @@ void flush_mte_state(void)
/* disable tag checking */
set_sctlr_el1_tcf0(SCTLR_EL1_TCF0_NONE);
/* reset tag generation mask */
- set_gcr_el1_excl(0);
+ set_gcr_el1_excl(SYS_GCR_EL1_EXCL_MASK);
}
void mte_thread_switch(struct task_struct *next)
@@ -177,7 +176,7 @@ void mte_thread_switch(struct task_struct *next)
/* avoid expensive SCTLR_EL1 accesses if no change */
if (current->thread.sctlr_tcf0 != next->thread.sctlr_tcf0)
update_sctlr_el1_tcf0(next->thread.sctlr_tcf0);
- update_gcr_el1_excl(next->thread.gcr_user_incl);
+ update_gcr_el1_excl(next->thread.gcr_user_excl);
}
void mte_suspend_exit(void)
@@ -185,13 +184,14 @@ void mte_suspend_exit(void)
if (!system_supports_mte())
return;
- update_gcr_el1_excl(current->thread.gcr_user_incl);
+ update_gcr_el1_excl(current->thread.gcr_user_excl);
}
long set_mte_ctrl(struct task_struct *task, unsigned long arg)
{
u64 tcf0;
- u64 gcr_incl = (arg & PR_MTE_TAG_MASK) >> PR_MTE_TAG_SHIFT;
+ u64 gcr_excl = ~((arg & PR_MTE_TAG_MASK) >> PR_MTE_TAG_SHIFT) &
+ SYS_GCR_EL1_EXCL_MASK;
if (!system_supports_mte())
return 0;
@@ -212,10 +212,10 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg)
if (task != current) {
task->thread.sctlr_tcf0 = tcf0;
- task->thread.gcr_user_incl = gcr_incl;
+ task->thread.gcr_user_excl = gcr_excl;
} else {
set_sctlr_el1_tcf0(tcf0);
- set_gcr_el1_excl(gcr_incl);
+ set_gcr_el1_excl(gcr_excl);
}
return 0;
@@ -224,11 +224,12 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg)
long get_mte_ctrl(struct task_struct *task)
{
unsigned long ret;
+ u64 incl = ~task->thread.gcr_user_excl & SYS_GCR_EL1_EXCL_MASK;
if (!system_supports_mte())
return 0;
- ret = task->thread.gcr_user_incl << PR_MTE_TAG_SHIFT;
+ ret = incl << PR_MTE_TAG_SHIFT;
switch (task->thread.sctlr_tcf0) {
case SCTLR_EL1_TCF0_NONE:
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (22 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 23/35] arm64: mte: Convert gcr_user into an exclude mask Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 10:38 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 25/35] kasan: introduce CONFIG_KASAN_HW_TAGS Andrey Konovalov
` (10 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
When MTE is present, the GCR_EL1 register contains the tags mask that
allows to exclude tags from the random generation via the IRG instruction.
With the introduction of the new Tag-Based KASAN API that provides a
mechanism to reserve tags for special reasons, the MTE implementation
has to make sure that the GCR_EL1 setting for the kernel does not affect
the userspace processes and viceversa.
Save and restore the kernel/user mask in GCR_EL1 in kernel entry and exit.
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
---
arch/arm64/include/asm/mte.h | 8 ++++++++
arch/arm64/kernel/asm-offsets.c | 3 +++
arch/arm64/kernel/cpufeature.c | 5 +++--
arch/arm64/kernel/entry.S | 28 ++++++++++++++++++++++++++++
arch/arm64/kernel/mte.c | 19 +++++++++++++++++--
5 files changed, 59 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 733be1cb5c95..4929f744d103 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -21,6 +21,8 @@
#include <asm/pgtable-types.h>
+extern u64 gcr_kernel_excl;
+
void mte_clear_page_tags(void *addr);
unsigned long mte_copy_tags_from_user(void *to, const void __user *from,
unsigned long n);
@@ -59,6 +61,8 @@ u8 mte_get_mem_tag(void *addr);
u8 mte_get_random_tag(void);
void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag);
+void mte_init_tags(u64 max_tag);
+
#else /* CONFIG_ARM64_MTE */
/* unused if !CONFIG_ARM64_MTE, silence the compiler */
@@ -120,6 +124,10 @@ static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
return addr;
}
+static inline void mte_init_tags(u64 max_tag)
+{
+}
+
#endif /* CONFIG_ARM64_MTE */
#endif /* __ASSEMBLY__ */
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 0577e2142284..a1ef256cad4f 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -47,6 +47,9 @@ int main(void)
#ifdef CONFIG_ARM64_PTR_AUTH
DEFINE(THREAD_KEYS_USER, offsetof(struct task_struct, thread.keys_user));
DEFINE(THREAD_KEYS_KERNEL, offsetof(struct task_struct, thread.keys_kernel));
+#endif
+#ifdef CONFIG_ARM64_MTE
+ DEFINE(THREAD_GCR_EL1_USER, offsetof(struct task_struct, thread.gcr_user_excl));
#endif
BLANK();
DEFINE(S_X0, offsetof(struct pt_regs, regs[0]));
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 4d94af19d8f6..54bc3b315063 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1665,14 +1665,15 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
{
u64 mair;
- /* all non-zero tags excluded by default */
- write_sysreg_s(SYS_GCR_EL1_RRND | SYS_GCR_EL1_EXCL_MASK, SYS_GCR_EL1);
write_sysreg_s(0, SYS_TFSR_EL1);
write_sysreg_s(0, SYS_TFSRE0_EL1);
/* Enable Match-All at EL1 */
sysreg_clear_set(tcr_el1, 0, SYS_TCR_EL1_TCMA1);
+ /* Enable the kernel exclude mask for random tags generation */
+ write_sysreg_s((SYS_GCR_EL1_RRND | gcr_kernel_excl), SYS_GCR_EL1);
+
/*
* CnP must be enabled only after the MAIR_EL1 register has been set
* up. Inconsistent MAIR_EL1 between CPUs sharing the same TLB may
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index cde127508e38..a17fefb0571b 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -172,6 +172,29 @@ alternative_else_nop_endif
#endif
.endm
+ /* Note: tmp should always be a callee-saved register */
+ .macro mte_restore_gcr, el, tsk, tmp, tmp2
+#ifdef CONFIG_ARM64_MTE
+alternative_if_not ARM64_MTE
+ b 1f
+alternative_else_nop_endif
+ .if \el == 0
+ ldr \tmp, [\tsk, #THREAD_GCR_EL1_USER]
+ .else
+ ldr_l \tmp, gcr_kernel_excl
+ .endif
+ /*
+ * Calculate and set the exclude mask preserving
+ * the RRND (bit[16]) setting.
+ */
+ mrs_s \tmp2, SYS_GCR_EL1
+ bfi \tmp2, \tmp, #0, #16
+ msr_s SYS_GCR_EL1, \tmp2
+ isb
+1:
+#endif
+ .endm
+
.macro kernel_entry, el, regsize = 64
.if \regsize == 32
mov w0, w0 // zero upper 32 bits of x0
@@ -209,6 +232,8 @@ alternative_else_nop_endif
ptrauth_keys_install_kernel tsk, x20, x22, x23
+ mte_restore_gcr 1, tsk, x22, x23
+
scs_load tsk, x20
.else
add x21, sp, #S_FRAME_SIZE
@@ -386,6 +411,8 @@ alternative_else_nop_endif
/* No kernel C function calls after this as user keys are set. */
ptrauth_keys_install_user tsk, x0, x1, x2
+ mte_restore_gcr 0, tsk, x0, x1
+
apply_ssbd 0, x0, x1
.endif
@@ -957,6 +984,7 @@ SYM_FUNC_START(cpu_switch_to)
mov sp, x9
msr sp_el0, x1
ptrauth_keys_install_kernel x1, x8, x9, x10
+ mte_restore_gcr 1, x1, x8, x9
scs_save x0, x8
scs_load x1, x8
ret
diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index 7717ea9bc2a7..cfac7d02f032 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -18,10 +18,14 @@
#include <asm/barrier.h>
#include <asm/cpufeature.h>
+#include <asm/kasan.h>
+#include <asm/kprobes.h>
#include <asm/mte.h>
#include <asm/ptrace.h>
#include <asm/sysreg.h>
+u64 gcr_kernel_excl __read_mostly;
+
static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap)
{
pte_t old_pte = READ_ONCE(*ptep);
@@ -115,6 +119,13 @@ void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
return ptr;
}
+void mte_init_tags(u64 max_tag)
+{
+ u64 incl = ((1ULL << ((max_tag & MTE_TAG_MAX) + 1)) - 1);
+
+ gcr_kernel_excl = ~incl & SYS_GCR_EL1_EXCL_MASK;
+}
+
static void update_sctlr_el1_tcf0(u64 tcf0)
{
/* ISB required for the kernel uaccess routines */
@@ -150,7 +161,11 @@ static void update_gcr_el1_excl(u64 excl)
static void set_gcr_el1_excl(u64 excl)
{
current->thread.gcr_user_excl = excl;
- update_gcr_el1_excl(excl);
+
+ /*
+ * SYS_GCR_EL1 will be set to current->thread.gcr_user_incl value
+ * by mte_restore_gcr() in kernel_exit,
+ */
}
void flush_mte_state(void)
@@ -184,7 +199,7 @@ void mte_suspend_exit(void)
if (!system_supports_mte())
return;
- update_gcr_el1_excl(current->thread.gcr_user_excl);
+ update_gcr_el1_excl(gcr_kernel_excl);
}
long set_mte_ctrl(struct task_struct *task, unsigned long arg)
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 25/35] kasan: introduce CONFIG_KASAN_HW_TAGS
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (23 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 11:33 ` Vincenzo Frascino
2020-08-14 17:27 ` [PATCH 26/35] kasan, arm64: Enable TBI EL1 Andrey Konovalov
` (9 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
This patch adds a configuration option for a new KASAN mode called
hardware tag-based KASAN. This mode uses the memory tagging approach
like the software tag-based mode, but relies on arm64 Memory Tagging
Extension feature for tag management and access checking.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
lib/Kconfig.kasan | 46 ++++++++++++++++++++++++++++++++--------------
1 file changed, 32 insertions(+), 14 deletions(-)
diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index e500c18cbe79..0d4160ce5ee8 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -6,7 +6,10 @@ config HAVE_ARCH_KASAN
config HAVE_ARCH_KASAN_SW_TAGS
bool
-config HAVE_ARCH_KASAN_VMALLOC
+config HAVE_ARCH_KASAN_HW_TAGS
+ bool
+
+config HAVE_ARCH_KASAN_VMALLOC
bool
config CC_HAS_KASAN_GENERIC
@@ -20,10 +23,11 @@ config CC_HAS_WORKING_NOSANITIZE_ADDRESS
menuconfig KASAN
bool "KASAN: runtime memory debugger"
- depends on (HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC) || \
- (HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS)
+ depends on (((HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC) || \
+ (HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS)) && \
+ CC_HAS_WORKING_NOSANITIZE_ADDRESS) || \
+ HAVE_ARCH_KASAN_HW_TAGS
depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
- depends on CC_HAS_WORKING_NOSANITIZE_ADDRESS
select SLUB_DEBUG if SLUB
select CONSTRUCTORS
select STACKDEPOT
@@ -38,13 +42,18 @@ choice
prompt "KASAN mode"
default KASAN_GENERIC
help
- KASAN has two modes: generic KASAN (similar to userspace ASan,
- x86_64/arm64/xtensa, enabled with CONFIG_KASAN_GENERIC) and
- software tag-based KASAN (a version based on software memory
- tagging, arm64 only, similar to userspace HWASan, enabled with
- CONFIG_KASAN_SW_TAGS).
+ KASAN has three modes:
+ 1. generic KASAN (similar to userspace ASan,
+ x86_64/arm64/xtensa, enabled with CONFIG_KASAN_GENERIC),
+ 2. software tag-based KASAN (arm64 only, based on software
+ memory tagging (similar to userspace HWASan), enabled with
+ CONFIG_KASAN_SW_TAGS), and
+ 3. hardware tag-based KASAN (arm64-only, based hardware
+ memory tagging (MTE), enabled with CONFIG_KASAN_HW_TAGS).
+
+ All KASAN modes are strictly debugging features.
- Both generic and tag-based KASAN are strictly debugging features.
+ For better error detection enable CONFIG_STACKTRACE.
config KASAN_GENERIC
bool "Generic mode"
@@ -61,8 +70,6 @@ config KASAN_GENERIC
and introduces an overhead of ~x1.5 for the rest of the allocations.
The performance slowdown is ~x3.
- For better error detection enable CONFIG_STACKTRACE.
-
Currently CONFIG_KASAN_GENERIC doesn't work with CONFIG_DEBUG_SLAB
(the resulting kernel does not boot).
@@ -83,15 +90,25 @@ config KASAN_SW_TAGS
casting and comparison, as it embeds tags into the top byte of each
pointer.
- For better error detection enable CONFIG_STACKTRACE.
-
Currently CONFIG_KASAN_SW_TAGS doesn't work with CONFIG_DEBUG_SLAB
(the resulting kernel does not boot).
+config KASAN_HW_TAGS
+ bool "Hardware tag-based mode"
+ depends on HAVE_ARCH_KASAN_HW_TAGS
+ depends on SLUB
+ help
+ Enables hardware tag-based KASAN mode.
+
+ This mode requires both Memory Tagging Extension and Top Byte Ignore
+ support by the CPU and therefore is only supported for modern arm64
+ CPUs (MTE added in ARMv8.5 ISA).
+
endchoice
choice
prompt "Instrumentation type"
+ depends on KASAN_GENERIC || KASAN_SW_TAGS
default KASAN_OUTLINE
config KASAN_OUTLINE
@@ -115,6 +132,7 @@ endchoice
config KASAN_STACK_ENABLE
bool "Enable stack instrumentation (unsafe)" if CC_IS_CLANG && !COMPILE_TEST
+ depends on KASAN_GENERIC || KASAN_SW_TAGS
help
The LLVM stack address sanitizer has a know problem that
causes excessive stack usage in a lot of functions, see
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (24 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 25/35] kasan: introduce CONFIG_KASAN_HW_TAGS Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 10:40 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 27/35] kasan, arm64: align allocations for HW_TAGS Andrey Konovalov
` (8 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel
From: Vincenzo Frascino <vincenzo.frascino@arm.com>
Hardware tag-based KASAN relies on Memory Tagging Extension (MTE) that is
built on top of the Top Byte Ignore (TBI) feature.
Enable in-kernel TBI when CONFIG_KASAN_HW_TAGS is turned on by enabling
the TCR_TBI1 bit in proc.S.
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
---
arch/arm64/mm/proc.S | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 152d74f2cc9c..6880ddaa5144 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -38,7 +38,7 @@
/* PTWs cacheable, inner/outer WBWA */
#define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
#define TCR_KASAN_FLAGS TCR_TBI1
#else
#define TCR_KASAN_FLAGS 0
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 27/35] kasan, arm64: align allocations for HW_TAGS
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (25 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 26/35] kasan, arm64: Enable TBI EL1 Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE " Andrey Konovalov
` (7 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Hardware tag-based KASAN uses the memory tagging approach, which requires
all allocations to be aligned to the memory granule size. Align the
allocations to MTE_GRANULE_SIZE via ARCH_SLAB_MINALIGN when
CONFIG_KASAN_HW_TAGS is enabled.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/include/asm/cache.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
index a4d1b5f771f6..b8a0cae38470 100644
--- a/arch/arm64/include/asm/cache.h
+++ b/arch/arm64/include/asm/cache.h
@@ -6,6 +6,7 @@
#define __ASM_CACHE_H
#include <asm/cputype.h>
+#include <asm/mte_asm.h>
#define CTR_L1IP_SHIFT 14
#define CTR_L1IP_MASK 3
@@ -50,6 +51,8 @@
#ifdef CONFIG_KASAN_SW_TAGS
#define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT)
+#elif defined(CONFIG_KASAN_HW_TAGS)
+#define ARCH_SLAB_MINALIGN MTE_GRANULE_SIZE
#endif
#ifndef __ASSEMBLY__
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE for HW_TAGS
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (26 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 27/35] kasan, arm64: align allocations for HW_TAGS Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 10:41 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 29/35] kasan, x86, s390: update undef CONFIG_KASAN Andrey Konovalov
` (6 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Hardware tag-based KASAN has granules of MTE_GRANULE_SIZE. Define
KASAN_GRANULE_SIZE to MTE_GRANULE_SIZE for CONFIG_KASAN_HW_TAGS.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/kasan/kasan.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 1d3c7c6ce771..4d8e229f8e01 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -5,7 +5,13 @@
#include <linux/kasan.h>
#include <linux/stackdepot.h>
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
+#else
+#include <asm/mte.h>
+#define KASAN_GRANULE_SIZE (MTE_GRANULE_SIZE)
+#endif
+
#define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 29/35] kasan, x86, s390: update undef CONFIG_KASAN
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (27 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE " Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 30/35] kasan, arm64: expand CONFIG_KASAN checks Andrey Konovalov
` (5 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
With the intoduction of hardware tag-based KASAN some kernel checks of
this kind:
ifdef CONFIG_KASAN
will be updated to:
if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
x86 and s390 use a trick to #undef CONFIG_KASAN for some of the code
that isn't linked with KASAN runtime and shouldn't have any KASAN
annotations.
Also #undef CONFIG_KASAN_GENERIC with CONFIG_KASAN.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/s390/boot/string.c | 1 +
arch/x86/boot/compressed/misc.h | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/s390/boot/string.c b/arch/s390/boot/string.c
index b11e8108773a..faccb33b462c 100644
--- a/arch/s390/boot/string.c
+++ b/arch/s390/boot/string.c
@@ -3,6 +3,7 @@
#include <linux/kernel.h>
#include <linux/errno.h>
#undef CONFIG_KASAN
+#undef CONFIG_KASAN_GENERIC
#include "../lib/string.c"
int strncmp(const char *cs, const char *ct, size_t count)
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 726e264410ff..2ac973983a8e 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -12,6 +12,7 @@
#undef CONFIG_PARAVIRT_XXL
#undef CONFIG_PARAVIRT_SPINLOCKS
#undef CONFIG_KASAN
+#undef CONFIG_KASAN_GENERIC
/* cpu_feature_enabled() cannot be used this early */
#define USE_EARLY_PGTABLE_L5
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 30/35] kasan, arm64: expand CONFIG_KASAN checks
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (28 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 29/35] kasan, x86, s390: update undef CONFIG_KASAN Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime Andrey Konovalov
` (4 subsequent siblings)
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Some #ifdef CONFIG_KASAN checks are only relevant for software KASAN
modes (either related to shadow memory or compiler instrumentation).
Expand those into CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/Kconfig | 2 +-
arch/arm64/Makefile | 2 +-
arch/arm64/include/asm/assembler.h | 2 +-
arch/arm64/include/asm/memory.h | 2 +-
arch/arm64/include/asm/string.h | 5 +++--
arch/arm64/kernel/head.S | 2 +-
arch/arm64/kernel/image-vars.h | 2 +-
arch/arm64/mm/dump.c | 6 +++---
include/linux/kasan-checks.h | 2 +-
include/linux/kasan.h | 7 ++++---
include/linux/moduleloader.h | 3 ++-
include/linux/string.h | 2 +-
mm/ptdump.c | 13 ++++++++-----
scripts/Makefile.lib | 2 ++
14 files changed, 30 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 32ceff21acc1..70a7880d5145 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -320,7 +320,7 @@ config BROKEN_GAS_INST
config KASAN_SHADOW_OFFSET
hex
- depends on KASAN
+ depends on KASAN_GENERIC || KASAN_SW_TAGS
default 0xdfffa00000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && !KASAN_SW_TAGS
default 0xdfffd00000000000 if ARM64_VA_BITS_47 && !KASAN_SW_TAGS
default 0xdffffe8000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index a0d94d063fa8..89df029a3cff 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -125,7 +125,7 @@ TEXT_OFFSET := 0x0
ifeq ($(CONFIG_KASAN_SW_TAGS), y)
KASAN_SHADOW_SCALE_SHIFT := 4
-else
+else ifeq ($(CONFIG_KASAN_GENERIC), y)
KASAN_SHADOW_SCALE_SHIFT := 3
endif
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 54d181177656..bc9ace1e5f3a 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -464,7 +464,7 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU
#define NOKPROBE(x)
#endif
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define EXPORT_SYMBOL_NOKASAN(name)
#else
#define EXPORT_SYMBOL_NOKASAN(name) EXPORT_SYMBOL(name)
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 770535b7ca35..8881849929e3 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -75,7 +75,7 @@
* address space for the shadow region respectively. They can bloat the stack
* significantly, so double the (minimum) stack size when they are in use.
*/
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
#define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) \
+ KASAN_SHADOW_OFFSET)
diff --git a/arch/arm64/include/asm/string.h b/arch/arm64/include/asm/string.h
index b31e8e87a0db..3a3264ff47b9 100644
--- a/arch/arm64/include/asm/string.h
+++ b/arch/arm64/include/asm/string.h
@@ -5,7 +5,7 @@
#ifndef __ASM_STRING_H
#define __ASM_STRING_H
-#ifndef CONFIG_KASAN
+#if !(defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
#define __HAVE_ARCH_STRRCHR
extern char *strrchr(const char *, int c);
@@ -48,7 +48,8 @@ extern void *__memset(void *, int, __kernel_size_t);
void memcpy_flushcache(void *dst, const void *src, size_t cnt);
#endif
-#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__)
+#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
+ !defined(__SANITIZE_ADDRESS__)
/*
* For files that are not instrumented (e.g. mm/slub.c) we
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 037421c66b14..427ded9e68e8 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -452,7 +452,7 @@ SYM_FUNC_START_LOCAL(__primary_switched)
bl __pi_memset
dsb ishst // Make zero page visible to PTW
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
bl kasan_early_init
#endif
#ifdef CONFIG_RANDOMIZE_BASE
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index be0a63ffed23..b85372eba2d6 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -37,7 +37,7 @@ __efistub_strncmp = __pi_strncmp;
__efistub_strrchr = __pi_strrchr;
__efistub___clean_dcache_area_poc = __pi___clean_dcache_area_poc;
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
__efistub___memcpy = __pi_memcpy;
__efistub___memmove = __pi_memmove;
__efistub___memset = __pi_memset;
diff --git a/arch/arm64/mm/dump.c b/arch/arm64/mm/dump.c
index ba6d1d89f9b2..bf8ddeac5d8f 100644
--- a/arch/arm64/mm/dump.c
+++ b/arch/arm64/mm/dump.c
@@ -29,7 +29,7 @@
enum address_markers_idx {
PAGE_OFFSET_NR = 0,
PAGE_END_NR,
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
KASAN_START_NR,
#endif
};
@@ -37,7 +37,7 @@ enum address_markers_idx {
static struct addr_marker address_markers[] = {
{ PAGE_OFFSET, "Linear Mapping start" },
{ 0 /* PAGE_END */, "Linear Mapping end" },
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
{ 0 /* KASAN_SHADOW_START */, "Kasan shadow start" },
{ KASAN_SHADOW_END, "Kasan shadow end" },
#endif
@@ -381,7 +381,7 @@ void ptdump_check_wx(void)
static int ptdump_init(void)
{
address_markers[PAGE_END_NR].start_address = PAGE_END;
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
address_markers[KASAN_START_NR].start_address = KASAN_SHADOW_START;
#endif
ptdump_initialize();
diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h
index ac6aba632f2d..ca5e89fb10d3 100644
--- a/include/linux/kasan-checks.h
+++ b/include/linux/kasan-checks.h
@@ -9,7 +9,7 @@
* even in compilation units that selectively disable KASAN, but must use KASAN
* to validate access to an address. Never use these in header files!
*/
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
bool __kasan_check_read(const volatile void *p, unsigned int size);
bool __kasan_check_write(const volatile void *p, unsigned int size);
#else
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 894f4d9163ee..875bbcedd994 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -232,7 +232,8 @@ static inline void kasan_release_vmalloc(unsigned long start,
#endif /* CONFIG_KASAN_VMALLOC */
-#if defined(CONFIG_KASAN) && !defined(CONFIG_KASAN_VMALLOC)
+#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
+ !defined(CONFIG_KASAN_VMALLOC)
/*
* These functions provide a special case to support backing module
@@ -242,12 +243,12 @@ static inline void kasan_release_vmalloc(unsigned long start,
int kasan_module_alloc(void *addr, size_t size);
void kasan_free_shadow(const struct vm_struct *vm);
-#else /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
+#else /* (CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) && !CONFIG_KASAN_VMALLOC */
static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
static inline void kasan_free_shadow(const struct vm_struct *vm) {}
-#endif /* CONFIG_KASAN && !CONFIG_KASAN_VMALLOC */
+#endif /* (CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) && !CONFIG_KASAN_VMALLOC */
#ifdef CONFIG_KASAN_INLINE
void kasan_non_canonical_hook(unsigned long addr);
diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h
index 4fa67a8b2265..9e09d11ffe5b 100644
--- a/include/linux/moduleloader.h
+++ b/include/linux/moduleloader.h
@@ -96,7 +96,8 @@ void module_arch_cleanup(struct module *mod);
/* Any cleanup before freeing mod->module_init */
void module_arch_freeing_init(struct module *mod);
-#if defined(CONFIG_KASAN) && !defined(CONFIG_KASAN_VMALLOC)
+#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
+ !defined(CONFIG_KASAN_VMALLOC)
#include <linux/kasan.h>
#define MODULE_ALIGN (PAGE_SIZE << KASAN_SHADOW_SCALE_SHIFT)
#else
diff --git a/include/linux/string.h b/include/linux/string.h
index 9b7a0632e87a..607322616363 100644
--- a/include/linux/string.h
+++ b/include/linux/string.h
@@ -273,7 +273,7 @@ void __write_overflow(void) __compiletime_error("detected write beyond size of o
#if !defined(__NO_FORTIFY) && defined(__OPTIMIZE__) && defined(CONFIG_FORTIFY_SOURCE)
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
extern void *__underlying_memchr(const void *p, int c, __kernel_size_t size) __RENAME(memchr);
extern int __underlying_memcmp(const void *p, const void *q, __kernel_size_t size) __RENAME(memcmp);
extern void *__underlying_memcpy(void *p, const void *q, __kernel_size_t size) __RENAME(memcpy);
diff --git a/mm/ptdump.c b/mm/ptdump.c
index ba88ec43ff21..4354c1422d57 100644
--- a/mm/ptdump.c
+++ b/mm/ptdump.c
@@ -4,7 +4,7 @@
#include <linux/ptdump.h>
#include <linux/kasan.h>
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
/*
* This is an optimization for KASAN=y case. Since all kasan page tables
* eventually point to the kasan_early_shadow_page we could call note_page()
@@ -31,7 +31,8 @@ static int ptdump_pgd_entry(pgd_t *pgd, unsigned long addr,
struct ptdump_state *st = walk->private;
pgd_t val = READ_ONCE(*pgd);
-#if CONFIG_PGTABLE_LEVELS > 4 && defined(CONFIG_KASAN)
+#if CONFIG_PGTABLE_LEVELS > 4 && \
+ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
if (pgd_page(val) == virt_to_page(lm_alias(kasan_early_shadow_p4d)))
return note_kasan_page_table(walk, addr);
#endif
@@ -51,7 +52,8 @@ static int ptdump_p4d_entry(p4d_t *p4d, unsigned long addr,
struct ptdump_state *st = walk->private;
p4d_t val = READ_ONCE(*p4d);
-#if CONFIG_PGTABLE_LEVELS > 3 && defined(CONFIG_KASAN)
+#if CONFIG_PGTABLE_LEVELS > 3 && \
+ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
if (p4d_page(val) == virt_to_page(lm_alias(kasan_early_shadow_pud)))
return note_kasan_page_table(walk, addr);
#endif
@@ -71,7 +73,8 @@ static int ptdump_pud_entry(pud_t *pud, unsigned long addr,
struct ptdump_state *st = walk->private;
pud_t val = READ_ONCE(*pud);
-#if CONFIG_PGTABLE_LEVELS > 2 && defined(CONFIG_KASAN)
+#if CONFIG_PGTABLE_LEVELS > 2 && \
+ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS))
if (pud_page(val) == virt_to_page(lm_alias(kasan_early_shadow_pmd)))
return note_kasan_page_table(walk, addr);
#endif
@@ -91,7 +94,7 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long addr,
struct ptdump_state *st = walk->private;
pmd_t val = READ_ONCE(*pmd);
-#if defined(CONFIG_KASAN)
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
if (pmd_page(val) == virt_to_page(lm_alias(kasan_early_shadow_pte)))
return note_kasan_page_table(walk, addr);
#endif
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 99ac59c59826..e527d06b9da8 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -135,10 +135,12 @@ endif
# we don't want to check (depends on variables KASAN_SANITIZE_obj.o, KASAN_SANITIZE)
#
ifeq ($(CONFIG_KASAN),y)
+ifneq ($(CONFIG_KASAN_HW_TAGS),y)
_c_flags += $(if $(patsubst n%,, \
$(KASAN_SANITIZE_$(basetarget).o)$(KASAN_SANITIZE)y), \
$(CFLAGS_KASAN), $(CFLAGS_KASAN_NOSANITIZE))
endif
+endif
ifeq ($(CONFIG_UBSAN),y)
_c_flags += $(if $(patsubst n%,, \
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (29 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 30/35] kasan, arm64: expand CONFIG_KASAN checks Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 10:45 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 32/35] kasan, arm64: print report from tag fault handler Andrey Konovalov
` (3 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Provide implementation of KASAN functions required for the hardware
tag-based mode. Those include core functions for memory and pointer
tagging (mte.c) and bug reporting (report_mte.c). Also adapt common
KASAN code to support the new mode.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/include/asm/memory.h | 4 +-
arch/arm64/kernel/setup.c | 1 -
include/linux/kasan.h | 6 +--
include/linux/mm.h | 2 +-
include/linux/page-flags-layout.h | 2 +-
mm/kasan/Makefile | 5 ++
mm/kasan/common.c | 14 +++---
mm/kasan/kasan.h | 17 +++++--
mm/kasan/mte.c | 76 +++++++++++++++++++++++++++++++
mm/kasan/report_mte.c | 47 +++++++++++++++++++
mm/kasan/shadow.c | 2 +-
11 files changed, 158 insertions(+), 18 deletions(-)
create mode 100644 mm/kasan/mte.c
create mode 100644 mm/kasan/report_mte.c
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 8881849929e3..433341acf3f3 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -214,7 +214,7 @@ static inline unsigned long kaslr_offset(void)
(__force __typeof__(addr))__addr; \
})
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
#define __tag_shifted(tag) ((u64)(tag) << 56)
#define __tag_reset(addr) __untagged_addr(addr)
#define __tag_get(addr) (__u8)((u64)(addr) >> 56)
@@ -222,7 +222,7 @@ static inline unsigned long kaslr_offset(void)
#define __tag_shifted(tag) 0UL
#define __tag_reset(addr) (addr)
#define __tag_get(addr) 0
-#endif /* CONFIG_KASAN_SW_TAGS */
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
static inline const void *__tag_set(const void *addr, u8 tag)
{
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 575da075a2b9..4bee6e70eef4 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -352,7 +352,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
smp_init_cpus();
smp_build_mpidr_hash();
- /* Init percpu seeds for random tags after cpus are set up. */
kasan_init_tags();
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 875bbcedd994..613c9d38eee5 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -184,7 +184,7 @@ static inline void kasan_record_aux_stack(void *ptr) {}
#endif /* CONFIG_KASAN_GENERIC */
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
void kasan_init_tags(void);
@@ -193,7 +193,7 @@ void *kasan_reset_tag(const void *addr);
bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
-#else /* CONFIG_KASAN_SW_TAGS */
+#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
static inline void kasan_init_tags(void) { }
@@ -202,7 +202,7 @@ static inline void *kasan_reset_tag(const void *addr)
return (void *)addr;
}
-#endif /* CONFIG_KASAN_SW_TAGS */
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS*/
#ifdef CONFIG_KASAN_VMALLOC
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 65cbbfaa739b..94581f82c1b3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1395,7 +1395,7 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
}
#endif /* CONFIG_NUMA_BALANCING */
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
static inline u8 page_kasan_tag(const struct page *page)
{
return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-layout.h
index 71283739ffd2..75945732a58b 100644
--- a/include/linux/page-flags-layout.h
+++ b/include/linux/page-flags-layout.h
@@ -77,7 +77,7 @@
#define LAST_CPUPID_SHIFT 0
#endif
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
#define KASAN_TAG_WIDTH 8
#else
#define KASAN_TAG_WIDTH 0
diff --git a/mm/kasan/Makefile b/mm/kasan/Makefile
index 007c824f6f43..182095c6af28 100644
--- a/mm/kasan/Makefile
+++ b/mm/kasan/Makefile
@@ -10,9 +10,11 @@ CFLAGS_REMOVE_init.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_quarantine.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report_generic.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_report_mte.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_report_tags.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_shadow.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_tags.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_mte.o = $(CC_FLAGS_FTRACE)
# Function splitter causes unnecessary splits in __asan_load1/__asan_store1
# see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63533
@@ -27,10 +29,13 @@ CFLAGS_init.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_quarantine.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report_generic.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_report_mte.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_report_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_shadow.o := $(CC_FLAGS_KASAN_RUNTIME)
CFLAGS_tags.o := $(CC_FLAGS_KASAN_RUNTIME)
+CFLAGS_mte.o := $(CC_FLAGS_KASAN_RUNTIME)
obj-$(CONFIG_KASAN) := common.o report.o
obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o report_generic.o shadow.o quarantine.o
obj-$(CONFIG_KASAN_SW_TAGS) += init.o report_tags.o shadow.o tags.o
+obj-$(CONFIG_KASAN_HW_TAGS) += mte.o report_mte.o
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 41c7f1105eaa..412a23d1546b 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -118,7 +118,7 @@ void kasan_free_pages(struct page *page, unsigned int order)
*/
static inline unsigned int optimal_redzone(unsigned int object_size)
{
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ if (!IS_ENABLED(CONFIG_KASAN_GENERIC))
return 0;
return
@@ -183,14 +183,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache)
struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
const void *object)
{
- return (void *)object + cache->kasan_info.alloc_meta_offset;
+ return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}
struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
- return (void *)object + cache->kasan_info.free_meta_offset;
+ return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
}
void kasan_poison_slab(struct page *page)
@@ -272,7 +272,8 @@ void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
alloc_info = get_alloc_info(cache, object);
__memset(alloc_info, 0, sizeof(*alloc_info));
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) ||
+ IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object,
assign_tag(cache, object, true, false));
@@ -342,10 +343,11 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
redzone_end = round_up((unsigned long)object + cache->object_size,
KASAN_GRANULE_SIZE);
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) ||
+ IS_ENABLED(CONFIG_KASAN_HW_TAGS))
tag = assign_tag(cache, object, false, keep_tag);
- /* Tag is ignored in set_tag without CONFIG_KASAN_SW_TAGS */
+ /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */
kasan_unpoison_memory(set_tag(object, tag), size);
kasan_poison_memory((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 4d8e229f8e01..bc56cf8b9c48 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -152,6 +152,10 @@ struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
const void *object);
+void kasan_poison_memory(const void *address, size_t size, u8 value);
+
+#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+
static inline const void *kasan_shadow_to_mem(const void *shadow_addr)
{
return (void *)(((unsigned long)shadow_addr - KASAN_SHADOW_OFFSET)
@@ -163,8 +167,6 @@ static inline bool addr_has_metadata(const void *addr)
return (addr >= kasan_shadow_to_mem((void *)KASAN_SHADOW_START));
}
-void kasan_poison_memory(const void *address, size_t size, u8 value);
-
/**
* check_memory_region - Check memory region, and report if invalid access.
* @addr: the accessed address
@@ -176,6 +178,15 @@ void kasan_poison_memory(const void *address, size_t size, u8 value);
bool check_memory_region(unsigned long addr, size_t size, bool write,
unsigned long ret_ip);
+#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
+static inline bool addr_has_metadata(const void *addr)
+{
+ return true;
+}
+
+#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
+
bool check_invalid_free(void *addr);
void *find_first_bad_addr(void *addr, size_t size);
@@ -212,7 +223,7 @@ static inline void quarantine_reduce(void) { }
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif
-#ifdef CONFIG_KASAN_SW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
void print_tags(u8 addr_tag, const void *addr);
diff --git a/mm/kasan/mte.c b/mm/kasan/mte.c
new file mode 100644
index 000000000000..43b7d74161e5
--- /dev/null
+++ b/mm/kasan/mte.c
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file contains hardware tag-based (MTE-based) KASAN code.
+ *
+ * Copyright (c) 2020 Google, Inc.
+ * Author: Andrey Konovalov <andreyknvl@google.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/kasan.h>
+#include <linux/kernel.h>
+#include <linux/memory.h>
+#include <linux/mm.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+#include "kasan.h"
+
+void kasan_init_tags(void)
+{
+ mte_init_tags(KASAN_TAG_MAX);
+}
+
+void *kasan_reset_tag(const void *addr)
+{
+ return reset_tag(addr);
+}
+
+void kasan_poison_memory(const void *address, size_t size, u8 value)
+{
+ mte_set_mem_tag_range(reset_tag(address), size, value);
+}
+
+void kasan_unpoison_memory(const void *address, size_t size)
+{
+ mte_set_mem_tag_range(reset_tag(address), size, get_tag(address));
+}
+
+u8 random_tag(void)
+{
+ return mte_get_random_tag();
+}
+
+bool check_invalid_free(void *addr)
+{
+ u8 ptr_tag = get_tag(addr);
+ u8 mem_tag = mte_get_mem_tag(addr);
+
+ if (mem_tag == KASAN_TAG_INVALID)
+ return true;
+ if (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag)
+ return true;
+ return false;
+}
+
+void kasan_set_free_info(struct kmem_cache *cache,
+ void *object, u8 tag)
+{
+ struct kasan_alloc_meta *alloc_meta;
+
+ alloc_meta = get_alloc_info(cache, object);
+ kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
+}
+
+struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
+ void *object, u8 tag)
+{
+ struct kasan_alloc_meta *alloc_meta;
+
+ alloc_meta = get_alloc_info(cache, object);
+ return &alloc_meta->free_track[0];
+}
diff --git a/mm/kasan/report_mte.c b/mm/kasan/report_mte.c
new file mode 100644
index 000000000000..dbbf3aaa8798
--- /dev/null
+++ b/mm/kasan/report_mte.c
@@ -0,0 +1,47 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file contains Hardware Tag-Based (MTE-based) KASAN code.
+ *
+ * Copyright (c) 2020 Google, Inc.
+ * Author: Andrey Konovalov <andreyknvl@google.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/kasan.h>
+#include <linux/kernel.h>
+#include <linux/memory.h>
+#include <linux/mm.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+#include "kasan.h"
+
+const char *get_bug_type(struct kasan_access_info *info)
+{
+ return "invalid-access";
+}
+
+void *find_first_bad_addr(void *addr, size_t size)
+{
+ return reset_tag(addr);
+}
+
+void metadata_fetch_row(char *buffer, void *row)
+{
+ int i;
+
+ for (i = 0; i < META_BYTES_PER_ROW; i++)
+ buffer[i] = mte_get_mem_tag(row + i * KASAN_GRANULE_SIZE);
+}
+
+void print_tags(u8 addr_tag, const void *addr)
+{
+ u8 memory_tag = mte_get_mem_tag((void *)addr);
+
+ pr_err("Pointer tag: [%02x], memory tag: [%02x]\n",
+ addr_tag, memory_tag);
+}
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index 4888084ecdfc..ca69726adf8f 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -111,7 +111,7 @@ void kasan_unpoison_memory(const void *address, size_t size)
if (IS_ENABLED(CONFIG_KASAN_SW_TAGS))
*shadow = tag;
- else
+ else /* CONFIG_KASAN_GENERIC */
*shadow = size & KASAN_GRANULE_MASK;
}
}
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 32/35] kasan, arm64: print report from tag fault handler
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (30 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-27 10:48 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 33/35] kasan, slub: reset tags when accessing metadata Andrey Konovalov
` (2 subsequent siblings)
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Add error reporting for hardware tag-based KASAN. When CONFIG_KASAN_HW_TAGS
is enabled, print KASAN report from the arm64 tag fault handler.
SAS bits aren't set in ESR for all faults reported in EL1, so it's
impossible to find out the size of the access the caused the fault.
Adapt KASAN reporting code to handle this case.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/mm/fault.c | 9 +++++++++
mm/kasan/report.c | 11 ++++++++---
2 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index c62c8ba85c0e..cf00b3942564 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -14,6 +14,7 @@
#include <linux/mm.h>
#include <linux/hardirq.h>
#include <linux/init.h>
+#include <linux/kasan.h>
#include <linux/kprobes.h>
#include <linux/uaccess.h>
#include <linux/page-flags.h>
@@ -314,11 +315,19 @@ static void report_tag_fault(unsigned long addr, unsigned int esr,
{
bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
+#ifdef CONFIG_KASAN_HW_TAGS
+ /*
+ * SAS bits aren't set for all faults reported in EL1, so we can't
+ * find out access size.
+ */
+ kasan_report(addr, 0, is_write, regs->pc);
+#else
pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
mte_get_ptr_tag(addr),
mte_get_mem_tag((void *)addr));
+#endif
}
static void __do_kernel_fault(unsigned long addr, unsigned int esr,
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index c904edab33b8..34ef81736d73 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -65,9 +65,14 @@ static void print_error_description(struct kasan_access_info *info)
{
pr_err("BUG: KASAN: %s in %pS\n",
get_bug_type(info), (void *)info->ip);
- pr_err("%s of size %zu at addr %px by task %s/%d\n",
- info->is_write ? "Write" : "Read", info->access_size,
- info->access_addr, current->comm, task_pid_nr(current));
+ if (info->access_size)
+ pr_err("%s of size %zu at addr %px by task %s/%d\n",
+ info->is_write ? "Write" : "Read", info->access_size,
+ info->access_addr, current->comm, task_pid_nr(current));
+ else
+ pr_err("%s at addr %px by task %s/%d\n",
+ info->is_write ? "Write" : "Read",
+ info->access_addr, current->comm, task_pid_nr(current));
}
static DEFINE_SPINLOCK(report_lock);
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 33/35] kasan, slub: reset tags when accessing metadata
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (31 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 32/35] kasan, arm64: print report from tag fault handler Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 34/35] kasan, arm64: enable CONFIG_KASAN_HW_TAGS Andrey Konovalov
2020-08-14 17:27 ` [PATCH 35/35] kasan: add documentation for hardware tag-based mode Andrey Konovalov
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
SLUB allocator accesses metadata for slab objects, that may lie
out-of-bounds of the object itself, or be accessed when an object is freed.
Handle this for Hardware tag-based KASAN by resetting tags when accessing
metadata.
Hardware tag-based KASAN doesn't rely on metadata_access_disable/enable(),
and therefore requires resetting tags in the sections of code guarded
by those annotations.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
mm/page_poison.c | 2 +-
mm/slub.c | 25 ++++++++++++++-----------
2 files changed, 15 insertions(+), 12 deletions(-)
diff --git a/mm/page_poison.c b/mm/page_poison.c
index 34b9181ee5d1..d90d342a391f 100644
--- a/mm/page_poison.c
+++ b/mm/page_poison.c
@@ -43,7 +43,7 @@ static void poison_page(struct page *page)
/* KASAN still think the page is in-use, so skip it. */
kasan_disable_current();
- memset(addr, PAGE_POISON, PAGE_SIZE);
+ memset(kasan_reset_tag(addr), PAGE_POISON, PAGE_SIZE);
kasan_enable_current();
kunmap_atomic(addr);
}
diff --git a/mm/slub.c b/mm/slub.c
index ef303070d175..a786e1cee095 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -249,7 +249,7 @@ static inline void *freelist_ptr(const struct kmem_cache *s, void *ptr,
{
#ifdef CONFIG_SLAB_FREELIST_HARDENED
/*
- * When CONFIG_KASAN_SW_TAGS is enabled, ptr_addr might be tagged.
+ * When CONFIG_KASAN_SW/HW_TAGS is enabled, ptr_addr might be tagged.
* Normally, this doesn't cause any issues, as both set_freepointer()
* and get_freepointer() are called with a pointer with the same tag.
* However, there are some issues with CONFIG_SLUB_DEBUG code. For
@@ -275,6 +275,7 @@ static inline void *freelist_dereference(const struct kmem_cache *s,
static inline void *get_freepointer(struct kmem_cache *s, void *object)
{
+ object = kasan_reset_tag(object);
return freelist_dereference(s, object + s->offset);
}
@@ -304,6 +305,7 @@ static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp)
BUG_ON(object == fp); /* naive detection of double free or corruption */
#endif
+ freeptr_addr = (unsigned long)kasan_reset_tag((void *)freeptr_addr);
*(void **)freeptr_addr = freelist_ptr(s, fp, freeptr_addr);
}
@@ -546,8 +548,8 @@ static void print_section(char *level, char *text, u8 *addr,
unsigned int length)
{
metadata_access_enable();
- print_hex_dump(level, text, DUMP_PREFIX_ADDRESS, 16, 1, addr,
- length, 1);
+ print_hex_dump(level, kasan_reset_tag(text), DUMP_PREFIX_ADDRESS,
+ 16, 1, addr, length, 1);
metadata_access_disable();
}
@@ -578,7 +580,7 @@ static struct track *get_track(struct kmem_cache *s, void *object,
p = object + get_info_end(s);
- return p + alloc;
+ return kasan_reset_tag(p + alloc);
}
static void set_track(struct kmem_cache *s, void *object,
@@ -591,7 +593,8 @@ static void set_track(struct kmem_cache *s, void *object,
unsigned int nr_entries;
metadata_access_enable();
- nr_entries = stack_trace_save(p->addrs, TRACK_ADDRS_COUNT, 3);
+ nr_entries = stack_trace_save(kasan_reset_tag(p->addrs),
+ TRACK_ADDRS_COUNT, 3);
metadata_access_disable();
if (nr_entries < TRACK_ADDRS_COUNT)
@@ -755,7 +758,7 @@ static __printf(3, 4) void slab_err(struct kmem_cache *s, struct page *page,
static void init_object(struct kmem_cache *s, void *object, u8 val)
{
- u8 *p = object;
+ u8 *p = kasan_reset_tag(object);
if (s->flags & SLAB_RED_ZONE)
memset(p - s->red_left_pad, val, s->red_left_pad);
@@ -785,7 +788,7 @@ static int check_bytes_and_report(struct kmem_cache *s, struct page *page,
u8 *addr = page_address(page);
metadata_access_enable();
- fault = memchr_inv(start, value, bytes);
+ fault = memchr_inv(kasan_reset_tag(start), value, bytes);
metadata_access_disable();
if (!fault)
return 1;
@@ -881,7 +884,7 @@ static int slab_pad_check(struct kmem_cache *s, struct page *page)
pad = end - remainder;
metadata_access_enable();
- fault = memchr_inv(pad, POISON_INUSE, remainder);
+ fault = memchr_inv(kasan_reset_tag(pad), POISON_INUSE, remainder);
metadata_access_disable();
if (!fault)
return 1;
@@ -1126,7 +1129,7 @@ void setup_page_debug(struct kmem_cache *s, struct page *page, void *addr)
return;
metadata_access_enable();
- memset(addr, POISON_INUSE, page_size(page));
+ memset(kasan_reset_tag(addr), POISON_INUSE, page_size(page));
metadata_access_disable();
}
@@ -2816,10 +2819,10 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
stat(s, ALLOC_FASTPATH);
}
- maybe_wipe_obj_freeptr(s, object);
+ maybe_wipe_obj_freeptr(s, kasan_reset_tag(object));
if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object)
- memset(object, 0, s->object_size);
+ memset(kasan_reset_tag(object), 0, s->object_size);
slab_post_alloc_hook(s, gfpflags, 1, &object);
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 34/35] kasan, arm64: enable CONFIG_KASAN_HW_TAGS
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (32 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 33/35] kasan, slub: reset tags when accessing metadata Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 35/35] kasan: add documentation for hardware tag-based mode Andrey Konovalov
34 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Hardware tag-based KASAN is now ready, enable the configuration option.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
arch/arm64/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 70a7880d5145..0d95d8391d0f 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -131,6 +131,7 @@ config ARM64
select HAVE_ARCH_JUMP_LABEL_RELATIVE
select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
+ select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE)
select HAVE_ARCH_KGDB
select HAVE_ARCH_MMAP_RND_BITS
select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH 35/35] kasan: add documentation for hardware tag-based mode
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
` (33 preceding siblings ...)
2020-08-14 17:27 ` [PATCH 34/35] kasan, arm64: enable CONFIG_KASAN_HW_TAGS Andrey Konovalov
@ 2020-08-14 17:27 ` Andrey Konovalov
2020-08-28 11:12 ` Marco Elver
34 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-14 17:27 UTC (permalink / raw)
To: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel, Andrey Konovalov
Add documentation for hardware tag-based KASAN mode and also add some
clarifications for software tag-based mode.
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
---
Documentation/dev-tools/kasan.rst | 73 +++++++++++++++++++++----------
1 file changed, 51 insertions(+), 22 deletions(-)
diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
index a3030fc6afe5..aeed89d6eaf5 100644
--- a/Documentation/dev-tools/kasan.rst
+++ b/Documentation/dev-tools/kasan.rst
@@ -5,12 +5,14 @@ Overview
--------
KernelAddressSANitizer (KASAN) is a dynamic memory error detector designed to
-find out-of-bound and use-after-free bugs. KASAN has two modes: generic KASAN
-(similar to userspace ASan) and software tag-based KASAN (similar to userspace
-HWASan).
+find out-of-bound and use-after-free bugs. KASAN has three modes:
+1. generic KASAN (similar to userspace ASan),
+2. software tag-based KASAN (similar to userspace HWASan),
+3. hardware tag-based KASAN (based on hardware memory tagging).
-KASAN uses compile-time instrumentation to insert validity checks before every
-memory access, and therefore requires a compiler version that supports that.
+Software KASAN modes (1 and 2) use compile-time instrumentation to insert
+validity checks before every memory access, and therefore require a compiler
+version that supports that.
Generic KASAN is supported in both GCC and Clang. With GCC it requires version
8.3.0 or later. With Clang it requires version 7.0.0 or later, but detection of
@@ -19,7 +21,7 @@ out-of-bounds accesses for global variables is only supported since Clang 11.
Tag-based KASAN is only supported in Clang and requires version 7.0.0 or later.
Currently generic KASAN is supported for the x86_64, arm64, xtensa, s390 and
-riscv architectures, and tag-based KASAN is supported only for arm64.
+riscv architectures, and tag-based KASAN modes are supported only for arm64.
Usage
-----
@@ -28,14 +30,16 @@ To enable KASAN configure kernel with::
CONFIG_KASAN = y
-and choose between CONFIG_KASAN_GENERIC (to enable generic KASAN) and
-CONFIG_KASAN_SW_TAGS (to enable software tag-based KASAN).
+and choose between CONFIG_KASAN_GENERIC (to enable generic KASAN),
+CONFIG_KASAN_SW_TAGS (to enable software tag-based KASAN), and
+CONFIG_KASAN_HW_TAGS (to enable hardware tag-based KASAN).
-You also need to choose between CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE.
-Outline and inline are compiler instrumentation types. The former produces
-smaller binary while the latter is 1.1 - 2 times faster.
+For software modes, you also need to choose between CONFIG_KASAN_OUTLINE and
+CONFIG_KASAN_INLINE. Outline and inline are compiler instrumentation types.
+The former produces smaller binary while the latter is 1.1 - 2 times faster.
-Both KASAN modes work with both SLUB and SLAB memory allocators.
+Both software KASAN modes work with both SLUB and SLAB memory allocators,
+hardware tag-based KASAN currently only support SLUB.
For better bug detection and nicer reporting, enable CONFIG_STACKTRACE.
To augment reports with last allocation and freeing stack of the physical page,
@@ -196,17 +200,20 @@ and the second to last.
Software tag-based KASAN
~~~~~~~~~~~~~~~~~~~~~~~~
-Tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64 CPUs to
-store a pointer tag in the top byte of kernel pointers. Like generic KASAN it
-uses shadow memory to store memory tags associated with each 16-byte memory
-cell (therefore it dedicates 1/16th of the kernel memory for shadow memory).
+Software tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64
+CPUs to store a pointer tag in the top byte of kernel pointers. Like generic
+KASAN it uses shadow memory to store memory tags associated with each 16-byte
+memory cell (therefore it dedicates 1/16th of the kernel memory for shadow
+memory).
+
+On each memory allocation software tag-based KASAN generates a random tag, tags
+the allocated memory with this tag, and embeds this tag into the returned
+pointer.
-On each memory allocation tag-based KASAN generates a random tag, tags the
-allocated memory with this tag, and embeds this tag into the returned pointer.
Software tag-based KASAN uses compile-time instrumentation to insert checks
before each memory access. These checks make sure that tag of the memory that
is being accessed is equal to tag of the pointer that is used to access this
-memory. In case of a tag mismatch tag-based KASAN prints a bug report.
+memory. In case of a tag mismatch software tag-based KASAN prints a bug report.
Software tag-based KASAN also has two instrumentation modes (outline, that
emits callbacks to check memory accesses; and inline, that performs the shadow
@@ -215,9 +222,31 @@ simply printed from the function that performs the access check. With inline
instrumentation a brk instruction is emitted by the compiler, and a dedicated
brk handler is used to print bug reports.
-A potential expansion of this mode is a hardware tag-based mode, which would
-use hardware memory tagging support instead of compiler instrumentation and
-manual shadow memory manipulation.
+Software tag-based KASAN uses 0xFF as a match-all pointer tag (accesses aren't
+checked).
+
+Software tag-based KASAN currently only supports tagging of slab memory.
+
+Hardware tag-based KASAN
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Hardware tag-based KASAN is similar to the software mode in concept, but uses
+hardware memory tagging support instead of compiler instrumentation and
+shadow memory.
+
+Hardware tag-based KASAN is based on both arm64 Memory Tagging Extension (MTE)
+introduced in ARMv8.5 Instruction Set Architecture, and Top Byte Ignore (TBI).
+
+Special arm64 instructions are used to assign memory tags for each allocation.
+Same tags are assigned to pointers to those allocations. On every memory
+access, hardware makes sure that tag of the memory that is being accessed is
+equal to tag of the pointer that is used to access this memory. In case of a
+tag mismatch a fault is generated and a report is printed.
+
+Hardware tag-based KASAN uses 0xFF as a match-all pointer tag (accesses aren't
+checked).
+
+Hardware tag-based KASAN currently only supports tagging of slab memory.
What memory accesses are sanitised by KASAN?
--------------------------------------------
--
2.28.0.220.ged08abb693-goog
^ permalink raw reply related [flat|nested] 94+ messages in thread
* Re: [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE
2020-08-14 17:27 ` [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE Andrey Konovalov
@ 2020-08-27 8:04 ` Catalin Marinas
2020-08-27 9:54 ` Vincenzo Frascino
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 8:04 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:01PM +0200, Andrey Konovalov wrote:
> Software tag-based KASAN provides its own tag checking machinery that
> can conflict with MTE. Don't allow enabling software tag-based KASAN
> when MTE is enabled.
>
> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> ---
> lib/Kconfig.kasan | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
> index b4cf6c519d71..e500c18cbe79 100644
> --- a/lib/Kconfig.kasan
> +++ b/lib/Kconfig.kasan
> @@ -69,6 +69,7 @@ config KASAN_GENERIC
> config KASAN_SW_TAGS
> bool "Software tag-based mode"
> depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
> + depends on !ARM64_MTE
I think that's better as:
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 10cf81d70657..736c32bd8905 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -131,7 +131,7 @@ config ARM64
select HAVE_ARCH_JUMP_LABEL
select HAVE_ARCH_JUMP_LABEL_RELATIVE
select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
- select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
+ select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN && !ARM64_MTE
select HAVE_ARCH_KGDB
select HAVE_ARCH_MMAP_RND_BITS
select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
--
Catalin
^ permalink raw reply related [flat|nested] 94+ messages in thread
* Re: [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-08-14 17:27 ` [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers Andrey Konovalov
@ 2020-08-27 9:38 ` Catalin Marinas
2020-08-27 10:31 ` Vincenzo Frascino
` (2 more replies)
0 siblings, 3 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 9:38 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:02PM +0200, Andrey Konovalov wrote:
> diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
> index 1c99fcadb58c..733be1cb5c95 100644
> --- a/arch/arm64/include/asm/mte.h
> +++ b/arch/arm64/include/asm/mte.h
> @@ -5,14 +5,19 @@
> #ifndef __ASM_MTE_H
> #define __ASM_MTE_H
>
> -#define MTE_GRANULE_SIZE UL(16)
> +#include <asm/mte_asm.h>
So the reason for this move is to include it in asm/cache.h. Fine by
me but...
> #define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
> #define MTE_TAG_SHIFT 56
> #define MTE_TAG_SIZE 4
> +#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
> +#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
... I'd rather move all these definitions in a file with a more
meaningful name like mte-def.h. The _asm implies being meant for .S
files inclusion which isn't the case.
> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> index eb39504e390a..e2d708b4583d 100644
> --- a/arch/arm64/kernel/mte.c
> +++ b/arch/arm64/kernel/mte.c
> @@ -72,6 +74,47 @@ int memcmp_pages(struct page *page1, struct page *page2)
> return ret;
> }
>
> +u8 mte_get_mem_tag(void *addr)
> +{
> + if (system_supports_mte())
> + addr = mte_assign_valid_ptr_tag(addr);
The mte_assign_valid_ptr_tag() is slightly misleading. All it does is
read the allocation tag from memory.
I also think this should be inline asm, possibly using alternatives.
It's just an LDG instruction (and it saves us from having to invent a
better function name).
> +
> + return 0xF0 | mte_get_ptr_tag(addr);
> +}
> +
> +u8 mte_get_random_tag(void)
> +{
> + u8 tag = 0xF;
> +
> + if (system_supports_mte())
> + tag = mte_get_ptr_tag(mte_assign_random_ptr_tag(NULL));
Another alternative inline asm with an IRG instruction.
> +
> + return 0xF0 | tag;
> +}
> +
> +void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
> +{
> + void *ptr = addr;
> +
> + if ((!system_supports_mte()) || (size == 0))
> + return addr;
> +
> + tag = 0xF0 | (tag & 0xF);
> + ptr = (void *)__tag_set(ptr, tag);
> + size = ALIGN(size, MTE_GRANULE_SIZE);
I think aligning the size is dangerous. Can we instead turn it into a
WARN_ON if not already aligned? At a quick look, the callers of
kasan_{un,}poison_memory() already align the size.
> +
> + mte_assign_mem_tag_range(ptr, size);
> +
> + /*
> + * mte_assign_mem_tag_range() can be invoked in a multi-threaded
> + * context, ensure that tags are written in memory before the
> + * reference is used.
> + */
> + smp_wmb();
> +
> + return ptr;
I'm not sure I understand the barrier here. It ensures the relative
ordering of memory (or tag) accesses on a CPU as observed by other CPUs.
While the first access here is setting the tag, I can't see what other
access on _this_ CPU it is ordered with.
> +}
> +
> static void update_sctlr_el1_tcf0(u64 tcf0)
> {
> /* ISB required for the kernel uaccess routines */
> diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
> index 03ca6d8b8670..8c743540e32c 100644
> --- a/arch/arm64/lib/mte.S
> +++ b/arch/arm64/lib/mte.S
> @@ -149,3 +149,44 @@ SYM_FUNC_START(mte_restore_page_tags)
>
> ret
> SYM_FUNC_END(mte_restore_page_tags)
> +
> +/*
> + * Assign pointer tag based on the allocation tag
> + * x0 - source pointer
> + * Returns:
> + * x0 - pointer with the correct tag to access memory
> + */
> +SYM_FUNC_START(mte_assign_valid_ptr_tag)
> + ldg x0, [x0]
> + ret
> +SYM_FUNC_END(mte_assign_valid_ptr_tag)
> +
> +/*
> + * Assign random pointer tag
> + * x0 - source pointer
> + * Returns:
> + * x0 - pointer with a random tag
> + */
> +SYM_FUNC_START(mte_assign_random_ptr_tag)
> + irg x0, x0
> + ret
> +SYM_FUNC_END(mte_assign_random_ptr_tag)
As I said above, these two can be inline asm.
> +
> +/*
> + * Assign allocation tags for a region of memory based on the pointer tag
> + * x0 - source pointer
> + * x1 - size
> + *
> + * Note: size is expected to be MTE_GRANULE_SIZE aligned
> + */
> +SYM_FUNC_START(mte_assign_mem_tag_range)
> + /* if (src == NULL) return; */
> + cbz x0, 2f
> + /* if (size == 0) return; */
You could skip the cbz here and just document that the size should be
non-zero and aligned. The caller already takes care of this check.
> + cbz x1, 2f
> +1: stg x0, [x0]
> + add x0, x0, #MTE_GRANULE_SIZE
> + sub x1, x1, #MTE_GRANULE_SIZE
> + cbnz x1, 1b
> +2: ret
> +SYM_FUNC_END(mte_assign_mem_tag_range)
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE
2020-08-27 8:04 ` Catalin Marinas
@ 2020-08-27 9:54 ` Vincenzo Frascino
2020-08-27 12:02 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 9:54 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
Hi Andrey,
On 8/27/20 9:04 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:01PM +0200, Andrey Konovalov wrote:
>> Software tag-based KASAN provides its own tag checking machinery that
>> can conflict with MTE. Don't allow enabling software tag-based KASAN
>> when MTE is enabled.
>>
>> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
>> ---
>> lib/Kconfig.kasan | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
>> index b4cf6c519d71..e500c18cbe79 100644
>> --- a/lib/Kconfig.kasan
>> +++ b/lib/Kconfig.kasan
>> @@ -69,6 +69,7 @@ config KASAN_GENERIC
>> config KASAN_SW_TAGS
>> bool "Software tag-based mode"
>> depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
>> + depends on !ARM64_MTE
>
> I think that's better as:
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 10cf81d70657..736c32bd8905 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -131,7 +131,7 @@ config ARM64
> select HAVE_ARCH_JUMP_LABEL
> select HAVE_ARCH_JUMP_LABEL_RELATIVE
> select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
> - select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
> + select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN && !ARM64_MTE
> select HAVE_ARCH_KGDB
> select HAVE_ARCH_MMAP_RND_BITS
> select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
>
I agree with Catalin here, "select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN &&
!ARM64_MTE" should be sufficient.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-14 17:27 ` [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler Andrey Konovalov
@ 2020-08-27 9:54 ` Catalin Marinas
2020-08-27 10:44 ` Vincenzo Frascino
2020-08-27 12:31 ` Andrey Konovalov
0 siblings, 2 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 9:54 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 5e832b3387f1..c62c8ba85c0e 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -33,6 +33,7 @@
> #include <asm/debug-monitors.h>
> #include <asm/esr.h>
> #include <asm/kprobes.h>
> +#include <asm/mte.h>
> #include <asm/processor.h>
> #include <asm/sysreg.h>
> #include <asm/system_misc.h>
> @@ -222,6 +223,20 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
> return 1;
> }
>
> +static bool is_el1_mte_sync_tag_check_fault(unsigned int esr)
> +{
> + unsigned int ec = ESR_ELx_EC(esr);
> + unsigned int fsc = esr & ESR_ELx_FSC;
> +
> + if (ec != ESR_ELx_EC_DABT_CUR)
> + return false;
> +
> + if (fsc == ESR_ELx_FSC_MTE)
> + return true;
> +
> + return false;
> +}
> +
> static bool is_el1_instruction_abort(unsigned int esr)
> {
> return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_CUR;
> @@ -294,6 +309,18 @@ static void die_kernel_fault(const char *msg, unsigned long addr,
> do_exit(SIGKILL);
> }
>
> +static void report_tag_fault(unsigned long addr, unsigned int esr,
> + struct pt_regs *regs)
> +{
> + bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
> +
> + pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
> + pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
> + pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
> + mte_get_ptr_tag(addr),
> + mte_get_mem_tag((void *)addr));
> +}
> +
> static void __do_kernel_fault(unsigned long addr, unsigned int esr,
> struct pt_regs *regs)
> {
> @@ -317,12 +344,16 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
> msg = "execute from non-executable memory";
> else
> msg = "read from unreadable memory";
> + } else if (is_el1_mte_sync_tag_check_fault(esr)) {
> + report_tag_fault(addr, esr, regs);
> + msg = "memory tagging extension fault";
IIUC, that's dead code. See my comment below on do_tag_check_fault().
> } else if (addr < PAGE_SIZE) {
> msg = "NULL pointer dereference";
> } else {
> msg = "paging request";
> }
>
> +
Unnecessary empty line.
> die_kernel_fault(msg, addr, esr, regs);
> }
>
> @@ -658,10 +689,27 @@ static int do_sea(unsigned long addr, unsigned int esr, struct pt_regs *regs)
> return 0;
> }
>
> +static int do_tag_recovery(unsigned long addr, unsigned int esr,
> + struct pt_regs *regs)
> +{
> + report_tag_fault(addr, esr, regs);
> +
> + /* Skip over the faulting instruction and continue: */
> + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
Ooooh, do we expect the kernel to still behave correctly after this? I
thought the recovery means disabling tag checking altogether and
restarting the instruction rather than skipping over it. We only skip if
we emulated it.
> +
> + return 0;
> +}
> +
> +
> static int do_tag_check_fault(unsigned long addr, unsigned int esr,
> struct pt_regs *regs)
> {
> - do_bad_area(addr, esr, regs);
> + /* The tag check fault (TCF) is per TTBR */
> + if (is_ttbr0_addr(addr))
> + do_bad_area(addr, esr, regs);
> + else
> + do_tag_recovery(addr, esr, regs);
So we never invoke __do_kernel_fault() for a synchronous tag check in
the kernel. What's with all the is_el1_mte_sync_tag_check_fault() check
above?
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 22/35] arm64: mte: Enable in-kernel MTE
2020-08-14 17:27 ` [PATCH 22/35] arm64: mte: Enable in-kernel MTE Andrey Konovalov
@ 2020-08-27 10:01 ` Catalin Marinas
2020-08-27 10:46 ` Vincenzo Frascino
2020-09-08 14:39 ` Andrey Konovalov
1 sibling, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 10:01 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:04PM +0200, Andrey Konovalov wrote:
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 4d3abb51f7d4..4d94af19d8f6 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -1670,6 +1670,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
> write_sysreg_s(0, SYS_TFSR_EL1);
> write_sysreg_s(0, SYS_TFSRE0_EL1);
>
> + /* Enable Match-All at EL1 */
> + sysreg_clear_set(tcr_el1, 0, SYS_TCR_EL1_TCMA1);
> +
> /*
> * CnP must be enabled only after the MAIR_EL1 register has been set
> * up. Inconsistent MAIR_EL1 between CPUs sharing the same TLB may
> @@ -1687,6 +1690,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
> mair &= ~MAIR_ATTRIDX(MAIR_ATTR_MASK, MT_NORMAL_TAGGED);
> mair |= MAIR_ATTRIDX(MAIR_ATTR_NORMAL_TAGGED, MT_NORMAL_TAGGED);
> write_sysreg_s(mair, SYS_MAIR_EL1);
> +
> + /* Enable MTE Sync Mode for EL1 */
> + sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC);
In the 8th incarnation of the user MTE patches, this initialisation
moved to proc.S before the MMU is initialised. When rebasing, please
take this into account.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-08-27 9:38 ` Catalin Marinas
@ 2020-08-27 10:31 ` Vincenzo Frascino
2020-08-27 11:10 ` Catalin Marinas
2020-08-27 12:46 ` Andrey Konovalov
2020-09-08 13:23 ` Andrey Konovalov
2 siblings, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 10:31 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
Hi Catalin,
On 8/27/20 10:38 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:02PM +0200, Andrey Konovalov wrote:
>> diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
>> index 1c99fcadb58c..733be1cb5c95 100644
>> --- a/arch/arm64/include/asm/mte.h
>> +++ b/arch/arm64/include/asm/mte.h
>> @@ -5,14 +5,19 @@
>> #ifndef __ASM_MTE_H
>> #define __ASM_MTE_H
>>
>> -#define MTE_GRANULE_SIZE UL(16)
>> +#include <asm/mte_asm.h>
>
> So the reason for this move is to include it in asm/cache.h. Fine by
> me but...
>
>> #define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
>> #define MTE_TAG_SHIFT 56
>> #define MTE_TAG_SIZE 4
>> +#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
>> +#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
>
> ... I'd rather move all these definitions in a file with a more
> meaningful name like mte-def.h. The _asm implies being meant for .S
> files inclusion which isn't the case.
>
mte-asm.h was originally called mte_helper.h hence it made sense to have these
defines here. But I agree with your proposal it makes things more readable and
it is in line with the rest of the arm64 code (e.g. page-def.h).
We should as well update the commit message accordingly.
>> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
>> index eb39504e390a..e2d708b4583d 100644
>> --- a/arch/arm64/kernel/mte.c
>> +++ b/arch/arm64/kernel/mte.c
>> @@ -72,6 +74,47 @@ int memcmp_pages(struct page *page1, struct page *page2)
>> return ret;
>> }
>>
>> +u8 mte_get_mem_tag(void *addr)
>> +{
>> + if (system_supports_mte())
>> + addr = mte_assign_valid_ptr_tag(addr);
>
> The mte_assign_valid_ptr_tag() is slightly misleading. All it does is
> read the allocation tag from memory.
>
> I also think this should be inline asm, possibly using alternatives.
> It's just an LDG instruction (and it saves us from having to invent a
> better function name).
>
Yes, I agree, I implemented this code in the early days and never got around to
refactor it.
>> +
>> + return 0xF0 | mte_get_ptr_tag(addr);
>> +}
>> +
>> +u8 mte_get_random_tag(void)
>> +{
>> + u8 tag = 0xF;
>> +
>> + if (system_supports_mte())
>> + tag = mte_get_ptr_tag(mte_assign_random_ptr_tag(NULL));
>
> Another alternative inline asm with an IRG instruction.
>
As per above.
>> +
>> + return 0xF0 | tag;
>> +}
>> +
>> +void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
>> +{
>> + void *ptr = addr;
>> +
>> + if ((!system_supports_mte()) || (size == 0))
>> + return addr;
>> +
>> + tag = 0xF0 | (tag & 0xF);
>> + ptr = (void *)__tag_set(ptr, tag);
>> + size = ALIGN(size, MTE_GRANULE_SIZE);
>
> I think aligning the size is dangerous. Can we instead turn it into a
> WARN_ON if not already aligned? At a quick look, the callers of
> kasan_{un,}poison_memory() already align the size.
>
The size here is used only for tagging purposes and if we want to tag a
subgranule amount of memory we end up tagging the granule anyway. Why do you
think it can be dangerous?
Anyway I agree on the fact that is seems redundant, a WARN_ON here should be
sufficient.
>> +
>> + mte_assign_mem_tag_range(ptr, size);
>> +
>> + /*
>> + * mte_assign_mem_tag_range() can be invoked in a multi-threaded
>> + * context, ensure that tags are written in memory before the
>> + * reference is used.
>> + */
>> + smp_wmb();
>> +
>> + return ptr;
>
> I'm not sure I understand the barrier here. It ensures the relative
> ordering of memory (or tag) accesses on a CPU as observed by other CPUs.
> While the first access here is setting the tag, I can't see what other
> access on _this_ CPU it is ordered with.
>
You are right it can be removed. I was just overthinking here.
>> +}
>> +
>> static void update_sctlr_el1_tcf0(u64 tcf0)
>> {
>> /* ISB required for the kernel uaccess routines */
>> diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
>> index 03ca6d8b8670..8c743540e32c 100644
>> --- a/arch/arm64/lib/mte.S
>> +++ b/arch/arm64/lib/mte.S
>> @@ -149,3 +149,44 @@ SYM_FUNC_START(mte_restore_page_tags)
>>
>> ret
>> SYM_FUNC_END(mte_restore_page_tags)
>> +
>> +/*
>> + * Assign pointer tag based on the allocation tag
>> + * x0 - source pointer
>> + * Returns:
>> + * x0 - pointer with the correct tag to access memory
>> + */
>> +SYM_FUNC_START(mte_assign_valid_ptr_tag)
>> + ldg x0, [x0]
>> + ret
>> +SYM_FUNC_END(mte_assign_valid_ptr_tag)
>> +
>> +/*
>> + * Assign random pointer tag
>> + * x0 - source pointer
>> + * Returns:
>> + * x0 - pointer with a random tag
>> + */
>> +SYM_FUNC_START(mte_assign_random_ptr_tag)
>> + irg x0, x0
>> + ret
>> +SYM_FUNC_END(mte_assign_random_ptr_tag)
>
> As I said above, these two can be inline asm.
>
Agreed.
>> +
>> +/*
>> + * Assign allocation tags for a region of memory based on the pointer tag
>> + * x0 - source pointer
>> + * x1 - size
>> + *
>> + * Note: size is expected to be MTE_GRANULE_SIZE aligned
>> + */
>> +SYM_FUNC_START(mte_assign_mem_tag_range)
>> + /* if (src == NULL) return; */
>> + cbz x0, 2f
>> + /* if (size == 0) return; */
>
> You could skip the cbz here and just document that the size should be
> non-zero and aligned. The caller already takes care of this check.
>
I would prefer to keep the check here, unless there is a valid reason, since
allocate(0) is a viable option hence tag(x, 0) should be as well. The caller
takes care of it in one place, today, but I do not know where the API will be
used in future.
>> + cbz x1, 2f
>> +1: stg x0, [x0]
>> + add x0, x0, #MTE_GRANULE_SIZE
>> + sub x1, x1, #MTE_GRANULE_SIZE
>> + cbnz x1, 1b
>> +2: ret
>> +SYM_FUNC_END(mte_assign_mem_tag_range)
>
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-08-14 17:27 ` [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit Andrey Konovalov
@ 2020-08-27 10:38 ` Catalin Marinas
2020-08-27 10:56 ` Vincenzo Frascino
2020-09-08 13:58 ` Andrey Konovalov
0 siblings, 2 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 10:38 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index cde127508e38..a17fefb0571b 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -172,6 +172,29 @@ alternative_else_nop_endif
> #endif
> .endm
>
> + /* Note: tmp should always be a callee-saved register */
Why callee-saved? Do you preserve it anywhere here?
> + .macro mte_restore_gcr, el, tsk, tmp, tmp2
> +#ifdef CONFIG_ARM64_MTE
> +alternative_if_not ARM64_MTE
> + b 1f
> +alternative_else_nop_endif
> + .if \el == 0
> + ldr \tmp, [\tsk, #THREAD_GCR_EL1_USER]
> + .else
> + ldr_l \tmp, gcr_kernel_excl
> + .endif
> + /*
> + * Calculate and set the exclude mask preserving
> + * the RRND (bit[16]) setting.
> + */
> + mrs_s \tmp2, SYS_GCR_EL1
> + bfi \tmp2, \tmp, #0, #16
> + msr_s SYS_GCR_EL1, \tmp2
> + isb
> +1:
> +#endif
> + .endm
> +
> .macro kernel_entry, el, regsize = 64
> .if \regsize == 32
> mov w0, w0 // zero upper 32 bits of x0
> @@ -209,6 +232,8 @@ alternative_else_nop_endif
>
> ptrauth_keys_install_kernel tsk, x20, x22, x23
>
> + mte_restore_gcr 1, tsk, x22, x23
> +
> scs_load tsk, x20
> .else
> add x21, sp, #S_FRAME_SIZE
> @@ -386,6 +411,8 @@ alternative_else_nop_endif
> /* No kernel C function calls after this as user keys are set. */
> ptrauth_keys_install_user tsk, x0, x1, x2
>
> + mte_restore_gcr 0, tsk, x0, x1
> +
> apply_ssbd 0, x0, x1
> .endif
>
> @@ -957,6 +984,7 @@ SYM_FUNC_START(cpu_switch_to)
> mov sp, x9
> msr sp_el0, x1
> ptrauth_keys_install_kernel x1, x8, x9, x10
> + mte_restore_gcr 1, x1, x8, x9
> scs_save x0, x8
> scs_load x1, x8
> ret
Since we set GCR_EL1 on exception entry and return, why is this needed?
We don't have a per-kernel thread GCR_EL1, it's global to all threads,
so I think cpu_switch_to() should not be touched.
> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> index 7717ea9bc2a7..cfac7d02f032 100644
> --- a/arch/arm64/kernel/mte.c
> +++ b/arch/arm64/kernel/mte.c
> @@ -18,10 +18,14 @@
>
> #include <asm/barrier.h>
> #include <asm/cpufeature.h>
> +#include <asm/kasan.h>
> +#include <asm/kprobes.h>
> #include <asm/mte.h>
> #include <asm/ptrace.h>
> #include <asm/sysreg.h>
>
> +u64 gcr_kernel_excl __read_mostly;
Could we make this __ro_after_init?
> +
> static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap)
> {
> pte_t old_pte = READ_ONCE(*ptep);
> @@ -115,6 +119,13 @@ void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
> return ptr;
> }
>
> +void mte_init_tags(u64 max_tag)
> +{
> + u64 incl = ((1ULL << ((max_tag & MTE_TAG_MAX) + 1)) - 1);
I'd rather use GENMASK here, it is more readable.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-14 17:27 ` [PATCH 26/35] kasan, arm64: Enable TBI EL1 Andrey Konovalov
@ 2020-08-27 10:40 ` Catalin Marinas
2020-08-27 11:05 ` Vincenzo Frascino
2020-09-08 13:18 ` Andrey Konovalov
0 siblings, 2 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 10:40 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 152d74f2cc9c..6880ddaa5144 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -38,7 +38,7 @@
> /* PTWs cacheable, inner/outer WBWA */
> #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
>
> -#ifdef CONFIG_KASAN_SW_TAGS
> +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> #define TCR_KASAN_FLAGS TCR_TBI1
> #else
> #define TCR_KASAN_FLAGS 0
I prefer to turn TBI1 on only if MTE is present. So on top of the v8
user series, just do this in __cpu_setup.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE for HW_TAGS
2020-08-14 17:27 ` [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE " Andrey Konovalov
@ 2020-08-27 10:41 ` Catalin Marinas
2020-08-27 11:07 ` Vincenzo Frascino
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 10:41 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:10PM +0200, Andrey Konovalov wrote:
> Hardware tag-based KASAN has granules of MTE_GRANULE_SIZE. Define
> KASAN_GRANULE_SIZE to MTE_GRANULE_SIZE for CONFIG_KASAN_HW_TAGS.
>
> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> ---
> mm/kasan/kasan.h | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
> index 1d3c7c6ce771..4d8e229f8e01 100644
> --- a/mm/kasan/kasan.h
> +++ b/mm/kasan/kasan.h
> @@ -5,7 +5,13 @@
> #include <linux/kasan.h>
> #include <linux/stackdepot.h>
>
> +#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
> #define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
> +#else
> +#include <asm/mte.h>
You could only include the new asm/mte-def.h file (currently mte_asm.h).
> +#define KASAN_GRANULE_SIZE (MTE_GRANULE_SIZE)
> +#endif
> +
> #define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
>
> #define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
> --
> 2.28.0.220.ged08abb693-goog
>
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-27 9:54 ` Catalin Marinas
@ 2020-08-27 10:44 ` Vincenzo Frascino
2020-08-27 12:31 ` Andrey Konovalov
1 sibling, 0 replies; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 10:44 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
On 8/27/20 10:54 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
>> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
>> index 5e832b3387f1..c62c8ba85c0e 100644
>> --- a/arch/arm64/mm/fault.c
>> +++ b/arch/arm64/mm/fault.c
>> @@ -33,6 +33,7 @@
>> #include <asm/debug-monitors.h>
>> #include <asm/esr.h>
>> #include <asm/kprobes.h>
>> +#include <asm/mte.h>
>> #include <asm/processor.h>
>> #include <asm/sysreg.h>
>> #include <asm/system_misc.h>
>> @@ -222,6 +223,20 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
>> return 1;
>> }
>>
>> +static bool is_el1_mte_sync_tag_check_fault(unsigned int esr)
>> +{
>> + unsigned int ec = ESR_ELx_EC(esr);
>> + unsigned int fsc = esr & ESR_ELx_FSC;
>> +
>> + if (ec != ESR_ELx_EC_DABT_CUR)
>> + return false;
>> +
>> + if (fsc == ESR_ELx_FSC_MTE)
>> + return true;
>> +
>> + return false;
>> +}
>> +
>> static bool is_el1_instruction_abort(unsigned int esr)
>> {
>> return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_CUR;
>> @@ -294,6 +309,18 @@ static void die_kernel_fault(const char *msg, unsigned long addr,
>> do_exit(SIGKILL);
>> }
>>
>> +static void report_tag_fault(unsigned long addr, unsigned int esr,
>> + struct pt_regs *regs)
>> +{
>> + bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
>> +
>> + pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
>> + pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
>> + pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
>> + mte_get_ptr_tag(addr),
>> + mte_get_mem_tag((void *)addr));
>> +}
>> +
>> static void __do_kernel_fault(unsigned long addr, unsigned int esr,
>> struct pt_regs *regs)
>> {
>> @@ -317,12 +344,16 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
>> msg = "execute from non-executable memory";
>> else
>> msg = "read from unreadable memory";
>> + } else if (is_el1_mte_sync_tag_check_fault(esr)) {
>> + report_tag_fault(addr, esr, regs);
>> + msg = "memory tagging extension fault";
>
> IIUC, that's dead code. See my comment below on do_tag_check_fault().
>
That's correct. This was useful with "panic_on_mte_fault" kernel command line
parameter. Since it has now been replaced by a similar kasan feature, this code
can be safely removed.
>> } else if (addr < PAGE_SIZE) {
>> msg = "NULL pointer dereference";
>> } else {
>> msg = "paging request";
>> }
>>
>> +
>
> Unnecessary empty line.
>
Agree.
>> die_kernel_fault(msg, addr, esr, regs);
>> }
>>
>> @@ -658,10 +689,27 @@ static int do_sea(unsigned long addr, unsigned int esr, struct pt_regs *regs)
>> return 0;
>> }
>>
>> +static int do_tag_recovery(unsigned long addr, unsigned int esr,
>> + struct pt_regs *regs)
>> +{
>> + report_tag_fault(addr, esr, regs);
>> +
>> + /* Skip over the faulting instruction and continue: */
>> + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
>
> Ooooh, do we expect the kernel to still behave correctly after this? I
> thought the recovery means disabling tag checking altogether and
> restarting the instruction rather than skipping over it. We only skip if
> we emulated it.
>
I tried to dig it out but I am not sure why we need this as well.
>> +
>> + return 0;
>> +}
>> +
>> +
>> static int do_tag_check_fault(unsigned long addr, unsigned int esr,
>> struct pt_regs *regs)
>> {
>> - do_bad_area(addr, esr, regs);
>> + /* The tag check fault (TCF) is per TTBR */
>> + if (is_ttbr0_addr(addr))
>> + do_bad_area(addr, esr, regs);
>> + else
>> + do_tag_recovery(addr, esr, regs);
>
> So we never invoke __do_kernel_fault() for a synchronous tag check in
> the kernel. What's with all the is_el1_mte_sync_tag_check_fault() check
> above?
>
That's correct. This had a meaning with "panic_on_mte_fault" but since the
feature has been replaced is_el1_mte_sync_tag_check_fault() is not useful anymore.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime
2020-08-14 17:27 ` [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime Andrey Konovalov
@ 2020-08-27 10:45 ` Catalin Marinas
2020-08-27 11:35 ` Vincenzo Frascino
2020-08-27 12:37 ` Andrey Konovalov
0 siblings, 2 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 10:45 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:13PM +0200, Andrey Konovalov wrote:
> diff --git a/mm/kasan/mte.c b/mm/kasan/mte.c
> new file mode 100644
> index 000000000000..43b7d74161e5
> --- /dev/null
> +++ b/mm/kasan/mte.c
Since this is an arm64-specific kasan backend, I wonder whether it makes
more sense to keep it under arch/arm64 (mte-kasan.c).
> diff --git a/mm/kasan/report_mte.c b/mm/kasan/report_mte.c
> new file mode 100644
> index 000000000000..dbbf3aaa8798
> --- /dev/null
> +++ b/mm/kasan/report_mte.c
Same for this one.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 22/35] arm64: mte: Enable in-kernel MTE
2020-08-27 10:01 ` Catalin Marinas
@ 2020-08-27 10:46 ` Vincenzo Frascino
0 siblings, 0 replies; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 10:46 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
On 8/27/20 11:01 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:04PM +0200, Andrey Konovalov wrote:
>> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
>> index 4d3abb51f7d4..4d94af19d8f6 100644
>> --- a/arch/arm64/kernel/cpufeature.c
>> +++ b/arch/arm64/kernel/cpufeature.c
>> @@ -1670,6 +1670,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
>> write_sysreg_s(0, SYS_TFSR_EL1);
>> write_sysreg_s(0, SYS_TFSRE0_EL1);
>>
>> + /* Enable Match-All at EL1 */
>> + sysreg_clear_set(tcr_el1, 0, SYS_TCR_EL1_TCMA1);
>> +
>> /*
>> * CnP must be enabled only after the MAIR_EL1 register has been set
>> * up. Inconsistent MAIR_EL1 between CPUs sharing the same TLB may
>> @@ -1687,6 +1690,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
>> mair &= ~MAIR_ATTRIDX(MAIR_ATTR_MASK, MT_NORMAL_TAGGED);
>> mair |= MAIR_ATTRIDX(MAIR_ATTR_NORMAL_TAGGED, MT_NORMAL_TAGGED);
>> write_sysreg_s(mair, SYS_MAIR_EL1);
>> +
>> + /* Enable MTE Sync Mode for EL1 */
>> + sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC);
>
> In the 8th incarnation of the user MTE patches, this initialisation
> moved to proc.S before the MMU is initialised. When rebasing, please
> take this into account.
>
Thank you for the heads up.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 32/35] kasan, arm64: print report from tag fault handler
2020-08-14 17:27 ` [PATCH 32/35] kasan, arm64: print report from tag fault handler Andrey Konovalov
@ 2020-08-27 10:48 ` Catalin Marinas
2020-08-27 12:11 ` Vincenzo Frascino
2020-08-27 12:34 ` Andrey Konovalov
0 siblings, 2 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 10:48 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27:14PM +0200, Andrey Konovalov wrote:
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index c62c8ba85c0e..cf00b3942564 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -14,6 +14,7 @@
> #include <linux/mm.h>
> #include <linux/hardirq.h>
> #include <linux/init.h>
> +#include <linux/kasan.h>
> #include <linux/kprobes.h>
> #include <linux/uaccess.h>
> #include <linux/page-flags.h>
> @@ -314,11 +315,19 @@ static void report_tag_fault(unsigned long addr, unsigned int esr,
> {
> bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
>
> +#ifdef CONFIG_KASAN_HW_TAGS
> + /*
> + * SAS bits aren't set for all faults reported in EL1, so we can't
> + * find out access size.
> + */
> + kasan_report(addr, 0, is_write, regs->pc);
> +#else
> pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
> pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
> pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
> mte_get_ptr_tag(addr),
> mte_get_mem_tag((void *)addr));
> +#endif
> }
More dead code. So what's the point of keeping the pr_alert() introduced
earlier? CONFIG_KASAN_HW_TAGS is always on for in-kernel MTE. If MTE is
disabled, this function isn't called anyway.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-08-27 10:38 ` Catalin Marinas
@ 2020-08-27 10:56 ` Vincenzo Frascino
2020-08-27 12:16 ` Catalin Marinas
2020-09-08 13:58 ` Andrey Konovalov
1 sibling, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 10:56 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
On 8/27/20 11:38 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
>> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
>> index cde127508e38..a17fefb0571b 100644
>> --- a/arch/arm64/kernel/entry.S
>> +++ b/arch/arm64/kernel/entry.S
>> @@ -172,6 +172,29 @@ alternative_else_nop_endif
>> #endif
>> .endm
>>
>> + /* Note: tmp should always be a callee-saved register */
>
> Why callee-saved? Do you preserve it anywhere here?
>
Aargh, this is an old comment, I forgot to remove it after the last refactor.
Thank you for pointing this out.
>> + .macro mte_restore_gcr, el, tsk, tmp, tmp2
>> +#ifdef CONFIG_ARM64_MTE
>> +alternative_if_not ARM64_MTE
>> + b 1f
>> +alternative_else_nop_endif
>> + .if \el == 0
>> + ldr \tmp, [\tsk, #THREAD_GCR_EL1_USER]
>> + .else
>> + ldr_l \tmp, gcr_kernel_excl
>> + .endif
>> + /*
>> + * Calculate and set the exclude mask preserving
>> + * the RRND (bit[16]) setting.
>> + */
>> + mrs_s \tmp2, SYS_GCR_EL1
>> + bfi \tmp2, \tmp, #0, #16
>> + msr_s SYS_GCR_EL1, \tmp2
>> + isb
>> +1:
>> +#endif
>> + .endm
>> +
>> .macro kernel_entry, el, regsize = 64
>> .if \regsize == 32
>> mov w0, w0 // zero upper 32 bits of x0
>> @@ -209,6 +232,8 @@ alternative_else_nop_endif
>>
>> ptrauth_keys_install_kernel tsk, x20, x22, x23
>>
>> + mte_restore_gcr 1, tsk, x22, x23
>> +
>> scs_load tsk, x20
>> .else
>> add x21, sp, #S_FRAME_SIZE
>> @@ -386,6 +411,8 @@ alternative_else_nop_endif
>> /* No kernel C function calls after this as user keys are set. */
>> ptrauth_keys_install_user tsk, x0, x1, x2
>>
>> + mte_restore_gcr 0, tsk, x0, x1
>> +
>> apply_ssbd 0, x0, x1
>> .endif
>>
>> @@ -957,6 +984,7 @@ SYM_FUNC_START(cpu_switch_to)
>> mov sp, x9
>> msr sp_el0, x1
>> ptrauth_keys_install_kernel x1, x8, x9, x10
>> + mte_restore_gcr 1, x1, x8, x9
>> scs_save x0, x8
>> scs_load x1, x8
>> ret
>
> Since we set GCR_EL1 on exception entry and return, why is this needed?
> We don't have a per-kernel thread GCR_EL1, it's global to all threads,
> so I think cpu_switch_to() should not be touched.
>
I agree, we can remove it. We only require the kernel entry and the kernel exit
ones.
>> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
>> index 7717ea9bc2a7..cfac7d02f032 100644
>> --- a/arch/arm64/kernel/mte.c
>> +++ b/arch/arm64/kernel/mte.c
>> @@ -18,10 +18,14 @@
>>
>> #include <asm/barrier.h>
>> #include <asm/cpufeature.h>
>> +#include <asm/kasan.h>
>> +#include <asm/kprobes.h>
>> #include <asm/mte.h>
>> #include <asm/ptrace.h>
>> #include <asm/sysreg.h>
>>
>> +u64 gcr_kernel_excl __read_mostly;
>
> Could we make this __ro_after_init?
>
Yes, it makes sense, it should be updated only once through mte_init_tags().
Something to consider though here is that this might not be the right approach
if in future we want to add stack tagging. In such a case we need to know the
kernel exclude mask before any C code is executed. Initializing the mask via
mte_init_tags() it is too late.
I was thinking to add a compilation define instead of having gcr_kernel_excl in
place. This might not work if the kernel excl mask is meant to change during the
execution.
Thoughts?
>> +
>> static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap)
>> {
>> pte_t old_pte = READ_ONCE(*ptep);
>> @@ -115,6 +119,13 @@ void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
>> return ptr;
>> }
>>
>> +void mte_init_tags(u64 max_tag)
>> +{
>> + u64 incl = ((1ULL << ((max_tag & MTE_TAG_MAX) + 1)) - 1);
>
> I'd rather use GENMASK here, it is more readable.
>
Agree, we can change it.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-27 10:40 ` Catalin Marinas
@ 2020-08-27 11:05 ` Vincenzo Frascino
2020-08-27 11:13 ` Catalin Marinas
2020-09-08 13:18 ` Andrey Konovalov
1 sibling, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 11:05 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
On 8/27/20 11:40 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
>> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
>> index 152d74f2cc9c..6880ddaa5144 100644
>> --- a/arch/arm64/mm/proc.S
>> +++ b/arch/arm64/mm/proc.S
>> @@ -38,7 +38,7 @@
>> /* PTWs cacheable, inner/outer WBWA */
>> #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
>>
>> -#ifdef CONFIG_KASAN_SW_TAGS
>> +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
>> #define TCR_KASAN_FLAGS TCR_TBI1
>> #else
>> #define TCR_KASAN_FLAGS 0
>
> I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> user series, just do this in __cpu_setup.
>
Not sure I understand... Enabling TBI1 only if MTE is present would break
KASAN_SW_TAGS which is based on TBI1 but not on MTE.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE for HW_TAGS
2020-08-27 10:41 ` Catalin Marinas
@ 2020-08-27 11:07 ` Vincenzo Frascino
2020-08-27 12:05 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 11:07 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
On 8/27/20 11:41 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:10PM +0200, Andrey Konovalov wrote:
>> Hardware tag-based KASAN has granules of MTE_GRANULE_SIZE. Define
>> KASAN_GRANULE_SIZE to MTE_GRANULE_SIZE for CONFIG_KASAN_HW_TAGS.
>>
>> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
>> ---
>> mm/kasan/kasan.h | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
>> index 1d3c7c6ce771..4d8e229f8e01 100644
>> --- a/mm/kasan/kasan.h
>> +++ b/mm/kasan/kasan.h
>> @@ -5,7 +5,13 @@
>> #include <linux/kasan.h>
>> #include <linux/stackdepot.h>
>>
>> +#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
>> #define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
>> +#else
>> +#include <asm/mte.h>
>
> You could only include the new asm/mte-def.h file (currently mte_asm.h).
>
Agreed, we should only include asm/mte-def.h here since after the suggested
modification will be sufficient for the purpose.
>> +#define KASAN_GRANULE_SIZE (MTE_GRANULE_SIZE)
>> +#endif
>> +
>> #define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
>>
>> #define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
>> --
>> 2.28.0.220.ged08abb693-goog
>>
>
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-08-27 10:31 ` Vincenzo Frascino
@ 2020-08-27 11:10 ` Catalin Marinas
2020-08-27 11:24 ` Vincenzo Frascino
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 11:10 UTC (permalink / raw)
To: Vincenzo Frascino
Cc: Andrey Konovalov, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Thu, Aug 27, 2020 at 11:31:56AM +0100, Vincenzo Frascino wrote:
> On 8/27/20 10:38 AM, Catalin Marinas wrote:
> > On Fri, Aug 14, 2020 at 07:27:02PM +0200, Andrey Konovalov wrote:
> >> +void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
> >> +{
> >> + void *ptr = addr;
> >> +
> >> + if ((!system_supports_mte()) || (size == 0))
> >> + return addr;
> >> +
> >> + tag = 0xF0 | (tag & 0xF);
> >> + ptr = (void *)__tag_set(ptr, tag);
> >> + size = ALIGN(size, MTE_GRANULE_SIZE);
> >
> > I think aligning the size is dangerous. Can we instead turn it into a
> > WARN_ON if not already aligned? At a quick look, the callers of
> > kasan_{un,}poison_memory() already align the size.
>
> The size here is used only for tagging purposes and if we want to tag a
> subgranule amount of memory we end up tagging the granule anyway. Why do you
> think it can be dangerous?
In principle, I don't like expanding the size unless you are an
allocator. Since this code doesn't control the placement of the object
it was given, a warn seems more appropriate.
> >> +/*
> >> + * Assign allocation tags for a region of memory based on the pointer tag
> >> + * x0 - source pointer
> >> + * x1 - size
> >> + *
> >> + * Note: size is expected to be MTE_GRANULE_SIZE aligned
> >> + */
> >> +SYM_FUNC_START(mte_assign_mem_tag_range)
> >> + /* if (src == NULL) return; */
> >> + cbz x0, 2f
> >> + /* if (size == 0) return; */
> >
> > You could skip the cbz here and just document that the size should be
> > non-zero and aligned. The caller already takes care of this check.
>
> I would prefer to keep the check here, unless there is a valid reason, since
> allocate(0) is a viable option hence tag(x, 0) should be as well. The caller
> takes care of it in one place, today, but I do not know where the API will be
> used in future.
That's why I said just document it in the comment above the function.
The check is also insufficient if the size is not aligned to an MTE
granule, so it's not really consistent. This function should end with a
subs followed by b.gt as cbnz will get stuck in a loop for unaligned
size.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-27 11:05 ` Vincenzo Frascino
@ 2020-08-27 11:13 ` Catalin Marinas
2020-08-27 11:17 ` Vincenzo Frascino
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 11:13 UTC (permalink / raw)
To: Vincenzo Frascino
Cc: Andrey Konovalov, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Thu, Aug 27, 2020 at 12:05:55PM +0100, Vincenzo Frascino wrote:
> On 8/27/20 11:40 AM, Catalin Marinas wrote:
> > On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> >> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> >> index 152d74f2cc9c..6880ddaa5144 100644
> >> --- a/arch/arm64/mm/proc.S
> >> +++ b/arch/arm64/mm/proc.S
> >> @@ -38,7 +38,7 @@
> >> /* PTWs cacheable, inner/outer WBWA */
> >> #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
> >>
> >> -#ifdef CONFIG_KASAN_SW_TAGS
> >> +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> >> #define TCR_KASAN_FLAGS TCR_TBI1
> >> #else
> >> #define TCR_KASAN_FLAGS 0
> >
> > I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> > user series, just do this in __cpu_setup.
>
> Not sure I understand... Enabling TBI1 only if MTE is present would break
> KASAN_SW_TAGS which is based on TBI1 but not on MTE.
You keep the KASAN_SW_TAGS as above but for HW_TAGS, only set TBI1 later
in __cpu_setup().
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-27 11:13 ` Catalin Marinas
@ 2020-08-27 11:17 ` Vincenzo Frascino
2020-08-27 12:43 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 11:17 UTC (permalink / raw)
To: Catalin Marinas
Cc: Andrey Konovalov, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On 8/27/20 12:13 PM, Catalin Marinas wrote:
> On Thu, Aug 27, 2020 at 12:05:55PM +0100, Vincenzo Frascino wrote:
>> On 8/27/20 11:40 AM, Catalin Marinas wrote:
>>> On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
>>>> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
>>>> index 152d74f2cc9c..6880ddaa5144 100644
>>>> --- a/arch/arm64/mm/proc.S
>>>> +++ b/arch/arm64/mm/proc.S
>>>> @@ -38,7 +38,7 @@
>>>> /* PTWs cacheable, inner/outer WBWA */
>>>> #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
>>>>
>>>> -#ifdef CONFIG_KASAN_SW_TAGS
>>>> +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
>>>> #define TCR_KASAN_FLAGS TCR_TBI1
>>>> #else
>>>> #define TCR_KASAN_FLAGS 0
>>>
>>> I prefer to turn TBI1 on only if MTE is present. So on top of the v8
>>> user series, just do this in __cpu_setup.
>>
>> Not sure I understand... Enabling TBI1 only if MTE is present would break
>> KASAN_SW_TAGS which is based on TBI1 but not on MTE.
>
> You keep the KASAN_SW_TAGS as above but for HW_TAGS, only set TBI1 later
> in __cpu_setup().
>
Ok, sounds good.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-08-27 11:10 ` Catalin Marinas
@ 2020-08-27 11:24 ` Vincenzo Frascino
0 siblings, 0 replies; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 11:24 UTC (permalink / raw)
To: Catalin Marinas
Cc: Andrey Konovalov, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On 8/27/20 12:10 PM, Catalin Marinas wrote:
> On Thu, Aug 27, 2020 at 11:31:56AM +0100, Vincenzo Frascino wrote:
>> On 8/27/20 10:38 AM, Catalin Marinas wrote:
>>> On Fri, Aug 14, 2020 at 07:27:02PM +0200, Andrey Konovalov wrote:
>>>> +void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
>>>> +{
>>>> + void *ptr = addr;
>>>> +
>>>> + if ((!system_supports_mte()) || (size == 0))
>>>> + return addr;
>>>> +
>>>> + tag = 0xF0 | (tag & 0xF);
>>>> + ptr = (void *)__tag_set(ptr, tag);
>>>> + size = ALIGN(size, MTE_GRANULE_SIZE);
>>>
>>> I think aligning the size is dangerous. Can we instead turn it into a
>>> WARN_ON if not already aligned? At a quick look, the callers of
>>> kasan_{un,}poison_memory() already align the size.
>>
>> The size here is used only for tagging purposes and if we want to tag a
>> subgranule amount of memory we end up tagging the granule anyway. Why do you
>> think it can be dangerous?
>
> In principle, I don't like expanding the size unless you are an
> allocator. Since this code doesn't control the placement of the object
> it was given, a warn seems more appropriate.
>
That's a good point. Ok, we can change this in a warning.
>>>> +/*
>>>> + * Assign allocation tags for a region of memory based on the pointer tag
>>>> + * x0 - source pointer
>>>> + * x1 - size
>>>> + *
>>>> + * Note: size is expected to be MTE_GRANULE_SIZE aligned
>>>> + */
>>>> +SYM_FUNC_START(mte_assign_mem_tag_range)
>>>> + /* if (src == NULL) return; */
>>>> + cbz x0, 2f
>>>> + /* if (size == 0) return; */
>>>
>>> You could skip the cbz here and just document that the size should be
>>> non-zero and aligned. The caller already takes care of this check.
>>
>> I would prefer to keep the check here, unless there is a valid reason, since
>> allocate(0) is a viable option hence tag(x, 0) should be as well. The caller
>> takes care of it in one place, today, but I do not know where the API will be
>> used in future.
>
> That's why I said just document it in the comment above the function.
>
> The check is also insufficient if the size is not aligned to an MTE
> granule, so it's not really consistent. This function should end with a
> subs followed by b.gt as cbnz will get stuck in a loop for unaligned
> size.
>
That's correct. Thanks for pointing this out. I currently used it only in places
where the caller took care to align the size. But in future we cannot know hence
we should harden the function with what you are suggesting.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 25/35] kasan: introduce CONFIG_KASAN_HW_TAGS
2020-08-14 17:27 ` [PATCH 25/35] kasan: introduce CONFIG_KASAN_HW_TAGS Andrey Konovalov
@ 2020-08-27 11:33 ` Vincenzo Frascino
2020-08-27 12:22 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 11:33 UTC (permalink / raw)
To: Andrey Konovalov, Dmitry Vyukov, Catalin Marinas, kasan-dev
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, linux-arm-kernel, linux-mm,
linux-kernel
Hi Andrey,
On 8/14/20 6:27 PM, Andrey Konovalov wrote:
> +config·KASAN_HW_TAGS
> +» bool·"Hardware·tag-based·mode"
> +» depends·on·HAVE_ARCH_KASAN_HW_TAGS
> +» depends·on·SLUB
> +» help
> +» ··Enables·hardware·tag-based·KASAN·mode.
> +
> +» ··This·mode·requires·both·Memory·Tagging·Extension·and·Top·Byte·Ignore
> +» ··support·by·the·CPU·and·therefore·is·only·supported·for·modern·arm64
> +» ··CPUs·(MTE·added·in·ARMv8.5·ISA).
> +
I do not thing we should make KASAN_HW_TAGS MTE specific especially because it
is in the common code (e.g. SPARC ADI might want to implement it in future).
Probably would be better to provide some indirection in the generic code an
implement the MTE backend entirely in arch code.
Thoughts?
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime
2020-08-27 10:45 ` Catalin Marinas
@ 2020-08-27 11:35 ` Vincenzo Frascino
2020-08-27 12:37 ` Andrey Konovalov
1 sibling, 0 replies; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 11:35 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
On 8/27/20 11:45 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:13PM +0200, Andrey Konovalov wrote:
>> diff --git a/mm/kasan/mte.c b/mm/kasan/mte.c
>> new file mode 100644
>> index 000000000000..43b7d74161e5
>> --- /dev/null
>> +++ b/mm/kasan/mte.c
>
> Since this is an arm64-specific kasan backend, I wonder whether it makes
> more sense to keep it under arch/arm64 (mte-kasan.c).
>
Yes I agree, I had a similar comment in patch 25. I think we should implement
the mte backend entirely in arch code because other architectures might want to
enable the feature (e.g. Sparc ADI).
>> diff --git a/mm/kasan/report_mte.c b/mm/kasan/report_mte.c
>> new file mode 100644
>> index 000000000000..dbbf3aaa8798
>> --- /dev/null
>> +++ b/mm/kasan/report_mte.c
>
> Same for this one.
>
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE
2020-08-27 9:54 ` Vincenzo Frascino
@ 2020-08-27 12:02 ` Andrey Konovalov
0 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:02 UTC (permalink / raw)
To: Vincenzo Frascino
Cc: Catalin Marinas, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 11:52 AM Vincenzo Frascino
<vincenzo.frascino@arm.com> wrote:
>
> Hi Andrey,
>
> On 8/27/20 9:04 AM, Catalin Marinas wrote:
> > On Fri, Aug 14, 2020 at 07:27:01PM +0200, Andrey Konovalov wrote:
> >> Software tag-based KASAN provides its own tag checking machinery that
> >> can conflict with MTE. Don't allow enabling software tag-based KASAN
> >> when MTE is enabled.
> >>
> >> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> >> ---
> >> lib/Kconfig.kasan | 1 +
> >> 1 file changed, 1 insertion(+)
> >>
> >> diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
> >> index b4cf6c519d71..e500c18cbe79 100644
> >> --- a/lib/Kconfig.kasan
> >> +++ b/lib/Kconfig.kasan
> >> @@ -69,6 +69,7 @@ config KASAN_GENERIC
> >> config KASAN_SW_TAGS
> >> bool "Software tag-based mode"
> >> depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
> >> + depends on !ARM64_MTE
> >
> > I think that's better as:
> >
> > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> > index 10cf81d70657..736c32bd8905 100644
> > --- a/arch/arm64/Kconfig
> > +++ b/arch/arm64/Kconfig
> > @@ -131,7 +131,7 @@ config ARM64
> > select HAVE_ARCH_JUMP_LABEL
> > select HAVE_ARCH_JUMP_LABEL_RELATIVE
> > select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
> > - select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
> > + select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN && !ARM64_MTE
> > select HAVE_ARCH_KGDB
> > select HAVE_ARCH_MMAP_RND_BITS
> > select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
> >
>
> I agree with Catalin here, "select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN &&
> !ARM64_MTE" should be sufficient.
Sounds good, will do in v2, thanks!
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE for HW_TAGS
2020-08-27 11:07 ` Vincenzo Frascino
@ 2020-08-27 12:05 ` Andrey Konovalov
0 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:05 UTC (permalink / raw)
To: Vincenzo Frascino, Catalin Marinas
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, Linux ARM,
Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 1:05 PM Vincenzo Frascino
<vincenzo.frascino@arm.com> wrote:
>
>
>
> On 8/27/20 11:41 AM, Catalin Marinas wrote:
> > On Fri, Aug 14, 2020 at 07:27:10PM +0200, Andrey Konovalov wrote:
> >> Hardware tag-based KASAN has granules of MTE_GRANULE_SIZE. Define
> >> KASAN_GRANULE_SIZE to MTE_GRANULE_SIZE for CONFIG_KASAN_HW_TAGS.
> >>
> >> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> >> ---
> >> mm/kasan/kasan.h | 6 ++++++
> >> 1 file changed, 6 insertions(+)
> >>
> >> diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
> >> index 1d3c7c6ce771..4d8e229f8e01 100644
> >> --- a/mm/kasan/kasan.h
> >> +++ b/mm/kasan/kasan.h
> >> @@ -5,7 +5,13 @@
> >> #include <linux/kasan.h>
> >> #include <linux/stackdepot.h>
> >>
> >> +#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
> >> #define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
> >> +#else
> >> +#include <asm/mte.h>
> >
> > You could only include the new asm/mte-def.h file (currently mte_asm.h).
> >
>
> Agreed, we should only include asm/mte-def.h here since after the suggested
> modification will be sufficient for the purpose.
Will do in v2.
>
> >> +#define KASAN_GRANULE_SIZE (MTE_GRANULE_SIZE)
> >> +#endif
> >> +
> >> #define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1)
> >>
> >> #define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
> >> --
> >> 2.28.0.220.ged08abb693-goog
> >>
> >
>
> --
> Regards,
> Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 32/35] kasan, arm64: print report from tag fault handler
2020-08-27 10:48 ` Catalin Marinas
@ 2020-08-27 12:11 ` Vincenzo Frascino
2020-08-27 12:34 ` Andrey Konovalov
1 sibling, 0 replies; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 12:11 UTC (permalink / raw)
To: Catalin Marinas, Andrey Konovalov
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, linux-arm-kernel,
linux-mm, linux-kernel
On 8/27/20 11:48 AM, Catalin Marinas wrote:
> On Fri, Aug 14, 2020 at 07:27:14PM +0200, Andrey Konovalov wrote:
>> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
>> index c62c8ba85c0e..cf00b3942564 100644
>> --- a/arch/arm64/mm/fault.c
>> +++ b/arch/arm64/mm/fault.c
>> @@ -14,6 +14,7 @@
>> #include <linux/mm.h>
>> #include <linux/hardirq.h>
>> #include <linux/init.h>
>> +#include <linux/kasan.h>
>> #include <linux/kprobes.h>
>> #include <linux/uaccess.h>
>> #include <linux/page-flags.h>
>> @@ -314,11 +315,19 @@ static void report_tag_fault(unsigned long addr, unsigned int esr,
>> {
>> bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
>>
>> +#ifdef CONFIG_KASAN_HW_TAGS
>> + /*
>> + * SAS bits aren't set for all faults reported in EL1, so we can't
>> + * find out access size.
>> + */
>> + kasan_report(addr, 0, is_write, regs->pc);
>> +#else
>> pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
>> pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
>> pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
>> mte_get_ptr_tag(addr),
>> mte_get_mem_tag((void *)addr));
>> +#endif
>> }
>
> More dead code. So what's the point of keeping the pr_alert() introduced
> earlier? CONFIG_KASAN_HW_TAGS is always on for in-kernel MTE. If MTE is
> disabled, this function isn't called anyway.
>
I agree we should remove them (togheter with '#ifdef CONFIG_KASAN_HW_TAGS') or
integrate them with the kasan code if still meaningful.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-08-27 10:56 ` Vincenzo Frascino
@ 2020-08-27 12:16 ` Catalin Marinas
2020-09-08 14:02 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 12:16 UTC (permalink / raw)
To: Vincenzo Frascino
Cc: Andrey Konovalov, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Thu, Aug 27, 2020 at 11:56:49AM +0100, Vincenzo Frascino wrote:
> On 8/27/20 11:38 AM, Catalin Marinas wrote:
> > On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> >> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> >> index 7717ea9bc2a7..cfac7d02f032 100644
> >> --- a/arch/arm64/kernel/mte.c
> >> +++ b/arch/arm64/kernel/mte.c
> >> @@ -18,10 +18,14 @@
> >>
> >> #include <asm/barrier.h>
> >> #include <asm/cpufeature.h>
> >> +#include <asm/kasan.h>
> >> +#include <asm/kprobes.h>
> >> #include <asm/mte.h>
> >> #include <asm/ptrace.h>
> >> #include <asm/sysreg.h>
> >>
> >> +u64 gcr_kernel_excl __read_mostly;
> >
> > Could we make this __ro_after_init?
>
> Yes, it makes sense, it should be updated only once through mte_init_tags().
>
> Something to consider though here is that this might not be the right approach
> if in future we want to add stack tagging. In such a case we need to know the
> kernel exclude mask before any C code is executed. Initializing the mask via
> mte_init_tags() it is too late.
It depends on how stack tagging ends up in the kernel, whether it uses
ADDG/SUBG or not. If it's only IRG, I think it can cope with changing
the GCR_EL1.Excl in the middle of a function.
> I was thinking to add a compilation define instead of having gcr_kernel_excl in
> place. This might not work if the kernel excl mask is meant to change during the
> execution.
A macro with the default value works for me. That's what it basically is
currently, only that it ends up in a variable.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 25/35] kasan: introduce CONFIG_KASAN_HW_TAGS
2020-08-27 11:33 ` Vincenzo Frascino
@ 2020-08-27 12:22 ` Andrey Konovalov
0 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:22 UTC (permalink / raw)
To: Vincenzo Frascino, Catalin Marinas
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, Linux ARM,
Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 1:31 PM Vincenzo Frascino
<vincenzo.frascino@arm.com> wrote:
>
> Hi Andrey,
>
> On 8/14/20 6:27 PM, Andrey Konovalov wrote:
> > +config·KASAN_HW_TAGS
> > +» bool·"Hardware·tag-based·mode"
> > +» depends·on·HAVE_ARCH_KASAN_HW_TAGS
> > +» depends·on·SLUB
> > +» help
> > +» ··Enables·hardware·tag-based·KASAN·mode.
> > +
> > +» ··This·mode·requires·both·Memory·Tagging·Extension·and·Top·Byte·Ignore
> > +» ··support·by·the·CPU·and·therefore·is·only·supported·for·modern·arm64
> > +» ··CPUs·(MTE·added·in·ARMv8.5·ISA).
> > +
>
> I do not thing we should make KASAN_HW_TAGS MTE specific especially because it
> is in the common code (e.g. SPARC ADI might want to implement it in future).
>
> Probably would be better to provide some indirection in the generic code an
> implement the MTE backend entirely in arch code.
>
> Thoughts?
I think we can reword the help text to say that it enables tag-based
KASAN mode that is backed by the hardware in general, and mention that
this is currently only implemented for arm64 through MTE. I don't
think it makes sense to provide a common arch interface at this point
to keep the code simpler. We can do that when (and if) another
hardware backend is added.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-27 9:54 ` Catalin Marinas
2020-08-27 10:44 ` Vincenzo Frascino
@ 2020-08-27 12:31 ` Andrey Konovalov
2020-08-27 13:10 ` Catalin Marinas
1 sibling, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:31 UTC (permalink / raw)
To: Catalin Marinas, Vincenzo Frascino
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, Linux ARM,
Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 11:54 AM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
> > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> > index 5e832b3387f1..c62c8ba85c0e 100644
> > --- a/arch/arm64/mm/fault.c
> > +++ b/arch/arm64/mm/fault.c
> > @@ -33,6 +33,7 @@
> > #include <asm/debug-monitors.h>
> > #include <asm/esr.h>
> > #include <asm/kprobes.h>
> > +#include <asm/mte.h>
> > #include <asm/processor.h>
> > #include <asm/sysreg.h>
> > #include <asm/system_misc.h>
> > @@ -222,6 +223,20 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
> > return 1;
> > }
> >
> > +static bool is_el1_mte_sync_tag_check_fault(unsigned int esr)
> > +{
> > + unsigned int ec = ESR_ELx_EC(esr);
> > + unsigned int fsc = esr & ESR_ELx_FSC;
> > +
> > + if (ec != ESR_ELx_EC_DABT_CUR)
> > + return false;
> > +
> > + if (fsc == ESR_ELx_FSC_MTE)
> > + return true;
> > +
> > + return false;
> > +}
> > +
> > static bool is_el1_instruction_abort(unsigned int esr)
> > {
> > return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_CUR;
> > @@ -294,6 +309,18 @@ static void die_kernel_fault(const char *msg, unsigned long addr,
> > do_exit(SIGKILL);
> > }
> >
> > +static void report_tag_fault(unsigned long addr, unsigned int esr,
> > + struct pt_regs *regs)
> > +{
> > + bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
> > +
> > + pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
> > + pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
> > + pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
> > + mte_get_ptr_tag(addr),
> > + mte_get_mem_tag((void *)addr));
> > +}
> > +
> > static void __do_kernel_fault(unsigned long addr, unsigned int esr,
> > struct pt_regs *regs)
> > {
> > @@ -317,12 +344,16 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
> > msg = "execute from non-executable memory";
> > else
> > msg = "read from unreadable memory";
> > + } else if (is_el1_mte_sync_tag_check_fault(esr)) {
> > + report_tag_fault(addr, esr, regs);
> > + msg = "memory tagging extension fault";
>
> IIUC, that's dead code. See my comment below on do_tag_check_fault().
>
> > } else if (addr < PAGE_SIZE) {
> > msg = "NULL pointer dereference";
> > } else {
> > msg = "paging request";
> > }
> >
> > +
>
> Unnecessary empty line.
>
> > die_kernel_fault(msg, addr, esr, regs);
> > }
> >
> > @@ -658,10 +689,27 @@ static int do_sea(unsigned long addr, unsigned int esr, struct pt_regs *regs)
> > return 0;
> > }
> >
> > +static int do_tag_recovery(unsigned long addr, unsigned int esr,
> > + struct pt_regs *regs)
> > +{
> > + report_tag_fault(addr, esr, regs);
> > +
> > + /* Skip over the faulting instruction and continue: */
> > + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
>
> Ooooh, do we expect the kernel to still behave correctly after this? I
> thought the recovery means disabling tag checking altogether and
> restarting the instruction rather than skipping over it.
The intention is to be able to catch multiple MTE faults without
panicking or disabling MTE when executing KASAN tests (those do
multiple bad accesses one after another). We do
arm64_skip_faulting_instruction() for software tag-based KASAN too,
it's not ideal, but works for testing purposes.
Can we disable MTE, reexecute the instruction, and then reenable MTE,
or something like that?
When running in-kernel MTE in production, we'll either panic or
disable MTE after the first fault. This was controlled by the
panic_on_mte_fault option Vincenzo initially had.
> We only skip if we emulated it.
I'm not sure I understand this part, what do you mean by emulating?
>
> > +
> > + return 0;
> > +}
> > +
> > +
> > static int do_tag_check_fault(unsigned long addr, unsigned int esr,
> > struct pt_regs *regs)
> > {
> > - do_bad_area(addr, esr, regs);
> > + /* The tag check fault (TCF) is per TTBR */
> > + if (is_ttbr0_addr(addr))
> > + do_bad_area(addr, esr, regs);
> > + else
> > + do_tag_recovery(addr, esr, regs);
>
> So we never invoke __do_kernel_fault() for a synchronous tag check in
> the kernel. What's with all the is_el1_mte_sync_tag_check_fault() check
> above?
>
> --
> Catalin
>
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20200827095429.GC29264%40gaia.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 32/35] kasan, arm64: print report from tag fault handler
2020-08-27 10:48 ` Catalin Marinas
2020-08-27 12:11 ` Vincenzo Frascino
@ 2020-08-27 12:34 ` Andrey Konovalov
2020-08-27 14:21 ` Catalin Marinas
1 sibling, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:34 UTC (permalink / raw)
To: Catalin Marinas, Vincenzo Frascino
Cc: Dmitry Vyukov, kasan-dev, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, Linux ARM,
Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 12:48 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27:14PM +0200, Andrey Konovalov wrote:
> > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> > index c62c8ba85c0e..cf00b3942564 100644
> > --- a/arch/arm64/mm/fault.c
> > +++ b/arch/arm64/mm/fault.c
> > @@ -14,6 +14,7 @@
> > #include <linux/mm.h>
> > #include <linux/hardirq.h>
> > #include <linux/init.h>
> > +#include <linux/kasan.h>
> > #include <linux/kprobes.h>
> > #include <linux/uaccess.h>
> > #include <linux/page-flags.h>
> > @@ -314,11 +315,19 @@ static void report_tag_fault(unsigned long addr, unsigned int esr,
> > {
> > bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
> >
> > +#ifdef CONFIG_KASAN_HW_TAGS
> > + /*
> > + * SAS bits aren't set for all faults reported in EL1, so we can't
> > + * find out access size.
> > + */
> > + kasan_report(addr, 0, is_write, regs->pc);
> > +#else
> > pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
> > pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
> > pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
> > mte_get_ptr_tag(addr),
> > mte_get_mem_tag((void *)addr));
> > +#endif
> > }
>
> More dead code. So what's the point of keeping the pr_alert() introduced
> earlier? CONFIG_KASAN_HW_TAGS is always on for in-kernel MTE. If MTE is
> disabled, this function isn't called anyway.
I was considering that we can enable in-kernel MTE without enabling
CONFIG_KASAN_HW_TAGS, but perhaps this isn't what we want. I'll drop
this part in v2, but then we also need to make sure that in-kernel MTE
is only enabled when CONFIG_KASAN_HW_TAGS is enabled. Do we need more
ifdefs in arm64 patches when we write to MTE-related registers, or
does this work as is?
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime
2020-08-27 10:45 ` Catalin Marinas
2020-08-27 11:35 ` Vincenzo Frascino
@ 2020-08-27 12:37 ` Andrey Konovalov
1 sibling, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:37 UTC (permalink / raw)
To: Catalin Marinas
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 12:45 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27:13PM +0200, Andrey Konovalov wrote:
> > diff --git a/mm/kasan/mte.c b/mm/kasan/mte.c
> > new file mode 100644
> > index 000000000000..43b7d74161e5
> > --- /dev/null
> > +++ b/mm/kasan/mte.c
>
> Since this is an arm64-specific kasan backend, I wonder whether it makes
> more sense to keep it under arch/arm64 (mte-kasan.c).
I'm not sure if we do. I'd rather keep everything together, spreading
the implementation around the kernel is inconvenient. We already have
software tag-based KASAN implementation (which is also arm64-specific)
in the common code. We could, perhaps, rename mte.c into something
more generic, with other potential future hardware modes in mind.
> > diff --git a/mm/kasan/report_mte.c b/mm/kasan/report_mte.c
> > new file mode 100644
> > index 000000000000..dbbf3aaa8798
> > --- /dev/null
> > +++ b/mm/kasan/report_mte.c
>
> Same for this one.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-27 11:17 ` Vincenzo Frascino
@ 2020-08-27 12:43 ` Andrey Konovalov
2020-08-27 13:45 ` Vincenzo Frascino
0 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:43 UTC (permalink / raw)
To: Vincenzo Frascino
Cc: Catalin Marinas, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 1:15 PM Vincenzo Frascino
<vincenzo.frascino@arm.com> wrote:
>
>
>
> On 8/27/20 12:13 PM, Catalin Marinas wrote:
> > On Thu, Aug 27, 2020 at 12:05:55PM +0100, Vincenzo Frascino wrote:
> >> On 8/27/20 11:40 AM, Catalin Marinas wrote:
> >>> On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> >>>> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> >>>> index 152d74f2cc9c..6880ddaa5144 100644
> >>>> --- a/arch/arm64/mm/proc.S
> >>>> +++ b/arch/arm64/mm/proc.S
> >>>> @@ -38,7 +38,7 @@
> >>>> /* PTWs cacheable, inner/outer WBWA */
> >>>> #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
> >>>>
> >>>> -#ifdef CONFIG_KASAN_SW_TAGS
> >>>> +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> >>>> #define TCR_KASAN_FLAGS TCR_TBI1
> >>>> #else
> >>>> #define TCR_KASAN_FLAGS 0
> >>>
> >>> I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> >>> user series, just do this in __cpu_setup.
> >>
> >> Not sure I understand... Enabling TBI1 only if MTE is present would break
> >> KASAN_SW_TAGS which is based on TBI1 but not on MTE.
> >
> > You keep the KASAN_SW_TAGS as above but for HW_TAGS, only set TBI1 later
> > in __cpu_setup().
> >
>
> Ok, sounds good.
Sounds good to me too.
Vincenzo, could you take care of Catalin's comments on your (arm64)
patches, do the rebase onto user mte v8, and share it with me? I'll
work on KASAN changes in the meantime, and then integrate everything
together for v2.
Perhaps the best way to test only the arm64 part is writing a simple
module that causes an MTE fault. (At least that's what I did when I
was testing core in-kernel MTE patches separately.) Or reuse this
series, all KASAN patches should rebase cleanly on top of the latest
mainline.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-08-27 9:38 ` Catalin Marinas
2020-08-27 10:31 ` Vincenzo Frascino
@ 2020-08-27 12:46 ` Andrey Konovalov
2020-09-08 13:23 ` Andrey Konovalov
2 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 12:46 UTC (permalink / raw)
To: Catalin Marinas
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 11:38 AM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27:02PM +0200, Andrey Konovalov wrote:
> > diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
> > index 1c99fcadb58c..733be1cb5c95 100644
> > --- a/arch/arm64/include/asm/mte.h
> > +++ b/arch/arm64/include/asm/mte.h
> > @@ -5,14 +5,19 @@
> > #ifndef __ASM_MTE_H
> > #define __ASM_MTE_H
> >
> > -#define MTE_GRANULE_SIZE UL(16)
> > +#include <asm/mte_asm.h>
>
> So the reason for this move is to include it in asm/cache.h. Fine by
> me but...
>
> > #define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
> > #define MTE_TAG_SHIFT 56
> > #define MTE_TAG_SIZE 4
> > +#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
> > +#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
>
> ... I'd rather move all these definitions in a file with a more
> meaningful name like mte-def.h. The _asm implies being meant for .S
> files inclusion which isn't the case.
Sounds good, I'll leave fixing this and other arm64-specific comments
to Vincenzo. I'll change KASAN code to use mte-def.h once I have
patches where this file is renamed.
>
> > diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> > index eb39504e390a..e2d708b4583d 100644
> > --- a/arch/arm64/kernel/mte.c
> > +++ b/arch/arm64/kernel/mte.c
> > @@ -72,6 +74,47 @@ int memcmp_pages(struct page *page1, struct page *page2)
> > return ret;
> > }
> >
> > +u8 mte_get_mem_tag(void *addr)
> > +{
> > + if (system_supports_mte())
> > + addr = mte_assign_valid_ptr_tag(addr);
>
> The mte_assign_valid_ptr_tag() is slightly misleading. All it does is
> read the allocation tag from memory.
>
> I also think this should be inline asm, possibly using alternatives.
> It's just an LDG instruction (and it saves us from having to invent a
> better function name).
>
> > +
> > + return 0xF0 | mte_get_ptr_tag(addr);
> > +}
> > +
> > +u8 mte_get_random_tag(void)
> > +{
> > + u8 tag = 0xF;
> > +
> > + if (system_supports_mte())
> > + tag = mte_get_ptr_tag(mte_assign_random_ptr_tag(NULL));
>
> Another alternative inline asm with an IRG instruction.
>
> > +
> > + return 0xF0 | tag;
> > +}
> > +
> > +void * __must_check mte_set_mem_tag_range(void *addr, size_t size, u8 tag)
> > +{
> > + void *ptr = addr;
> > +
> > + if ((!system_supports_mte()) || (size == 0))
> > + return addr;
> > +
> > + tag = 0xF0 | (tag & 0xF);
> > + ptr = (void *)__tag_set(ptr, tag);
> > + size = ALIGN(size, MTE_GRANULE_SIZE);
>
> I think aligning the size is dangerous. Can we instead turn it into a
> WARN_ON if not already aligned? At a quick look, the callers of
> kasan_{un,}poison_memory() already align the size.
>
> > +
> > + mte_assign_mem_tag_range(ptr, size);
> > +
> > + /*
> > + * mte_assign_mem_tag_range() can be invoked in a multi-threaded
> > + * context, ensure that tags are written in memory before the
> > + * reference is used.
> > + */
> > + smp_wmb();
> > +
> > + return ptr;
>
> I'm not sure I understand the barrier here. It ensures the relative
> ordering of memory (or tag) accesses on a CPU as observed by other CPUs.
> While the first access here is setting the tag, I can't see what other
> access on _this_ CPU it is ordered with.
>
> > +}
> > +
> > static void update_sctlr_el1_tcf0(u64 tcf0)
> > {
> > /* ISB required for the kernel uaccess routines */
> > diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
> > index 03ca6d8b8670..8c743540e32c 100644
> > --- a/arch/arm64/lib/mte.S
> > +++ b/arch/arm64/lib/mte.S
> > @@ -149,3 +149,44 @@ SYM_FUNC_START(mte_restore_page_tags)
> >
> > ret
> > SYM_FUNC_END(mte_restore_page_tags)
> > +
> > +/*
> > + * Assign pointer tag based on the allocation tag
> > + * x0 - source pointer
> > + * Returns:
> > + * x0 - pointer with the correct tag to access memory
> > + */
> > +SYM_FUNC_START(mte_assign_valid_ptr_tag)
> > + ldg x0, [x0]
> > + ret
> > +SYM_FUNC_END(mte_assign_valid_ptr_tag)
> > +
> > +/*
> > + * Assign random pointer tag
> > + * x0 - source pointer
> > + * Returns:
> > + * x0 - pointer with a random tag
> > + */
> > +SYM_FUNC_START(mte_assign_random_ptr_tag)
> > + irg x0, x0
> > + ret
> > +SYM_FUNC_END(mte_assign_random_ptr_tag)
>
> As I said above, these two can be inline asm.
>
> > +
> > +/*
> > + * Assign allocation tags for a region of memory based on the pointer tag
> > + * x0 - source pointer
> > + * x1 - size
> > + *
> > + * Note: size is expected to be MTE_GRANULE_SIZE aligned
> > + */
> > +SYM_FUNC_START(mte_assign_mem_tag_range)
> > + /* if (src == NULL) return; */
> > + cbz x0, 2f
> > + /* if (size == 0) return; */
>
> You could skip the cbz here and just document that the size should be
> non-zero and aligned. The caller already takes care of this check.
>
> > + cbz x1, 2f
> > +1: stg x0, [x0]
> > + add x0, x0, #MTE_GRANULE_SIZE
> > + sub x1, x1, #MTE_GRANULE_SIZE
> > + cbnz x1, 1b
> > +2: ret
> > +SYM_FUNC_END(mte_assign_mem_tag_range)
>
> --
> Catalin
>
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20200827093808.GB29264%40gaia.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-27 12:31 ` Andrey Konovalov
@ 2020-08-27 13:10 ` Catalin Marinas
2020-08-27 13:34 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 13:10 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Vincenzo Frascino, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 02:31:23PM +0200, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 11:54 AM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
> > > +static int do_tag_recovery(unsigned long addr, unsigned int esr,
> > > + struct pt_regs *regs)
> > > +{
> > > + report_tag_fault(addr, esr, regs);
> > > +
> > > + /* Skip over the faulting instruction and continue: */
> > > + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
> >
> > Ooooh, do we expect the kernel to still behave correctly after this? I
> > thought the recovery means disabling tag checking altogether and
> > restarting the instruction rather than skipping over it.
>
> The intention is to be able to catch multiple MTE faults without
> panicking or disabling MTE when executing KASAN tests (those do
> multiple bad accesses one after another).
The problem is that for MTE synchronous tag check faults, the access has
not happened, so you basically introduce memory corruption by skipping
the access.
> We do arm64_skip_faulting_instruction() for software tag-based KASAN
> too, it's not ideal, but works for testing purposes.
IIUC, KASAN only skips over the brk instruction which doesn't have any
other side-effects. Has the actual memory access taken place when it
hits the brk?
> Can we disable MTE, reexecute the instruction, and then reenable MTE,
> or something like that?
If you want to preserve the MTE enabled, you could single-step the
instruction or execute it out of line, though it's a bit more convoluted
(we have a similar mechanism for kprobes/uprobes).
Another option would be to attempt to set the matching tag in memory,
under the assumption that it is writable (if it's not, maybe it's fine
to panic). Not sure how this interacts with the slub allocator since,
presumably, the logical tag in the pointer is wrong rather than the
allocation one.
Yet another option would be to change the tag in the register and
re-execute but this may confuse the compiler.
> When running in-kernel MTE in production, we'll either panic or
> disable MTE after the first fault. This was controlled by the
> panic_on_mte_fault option Vincenzo initially had.
I prefer to disable MTE, print something and continue, but no panic.
> > We only skip if we emulated it.
>
> I'm not sure I understand this part, what do you mean by emulating?
Executing it out of line or other form of instruction emulation (see
arch/arm64/kernel/probes/simulate-insn.c) so that the access actually
takes place. But you can single-step or experiment with some of the
other tricks above.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-27 13:10 ` Catalin Marinas
@ 2020-08-27 13:34 ` Andrey Konovalov
2020-08-27 14:56 ` Catalin Marinas
0 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 13:34 UTC (permalink / raw)
To: Catalin Marinas
Cc: Vincenzo Frascino, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 3:10 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Aug 27, 2020 at 02:31:23PM +0200, Andrey Konovalov wrote:
> > On Thu, Aug 27, 2020 at 11:54 AM Catalin Marinas
> > <catalin.marinas@arm.com> wrote:
> > > On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
> > > > +static int do_tag_recovery(unsigned long addr, unsigned int esr,
> > > > + struct pt_regs *regs)
> > > > +{
> > > > + report_tag_fault(addr, esr, regs);
> > > > +
> > > > + /* Skip over the faulting instruction and continue: */
> > > > + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
> > >
> > > Ooooh, do we expect the kernel to still behave correctly after this? I
> > > thought the recovery means disabling tag checking altogether and
> > > restarting the instruction rather than skipping over it.
> >
> > The intention is to be able to catch multiple MTE faults without
> > panicking or disabling MTE when executing KASAN tests (those do
> > multiple bad accesses one after another).
>
> The problem is that for MTE synchronous tag check faults, the access has
> not happened, so you basically introduce memory corruption by skipping
> the access.
Yes, you're right.
> > We do arm64_skip_faulting_instruction() for software tag-based KASAN
> > too, it's not ideal, but works for testing purposes.
>
> IIUC, KASAN only skips over the brk instruction which doesn't have any
> other side-effects.
Oh, yes, indeed. For some reason I confused myself thinking that we
also skip the access for software KASAN.
> Has the actual memory access taken place when it
> hits the brk?
IIRC, no, but it will be executed right after we skip the brk.
> > Can we disable MTE, reexecute the instruction, and then reenable MTE,
> > or something like that?
>
> If you want to preserve the MTE enabled, you could single-step the
> instruction or execute it out of line, though it's a bit more convoluted
> (we have a similar mechanism for kprobes/uprobes).
>
> Another option would be to attempt to set the matching tag in memory,
> under the assumption that it is writable (if it's not, maybe it's fine
> to panic). Not sure how this interacts with the slub allocator since,
> presumably, the logical tag in the pointer is wrong rather than the
> allocation one.
>
> Yet another option would be to change the tag in the register and
> re-execute but this may confuse the compiler.
Which one of these would be simpler to implement?
Perhaps we could somehow only skip faulting instructions that happen
in the KASAN test module?.. Decoding stack trace would be an option,
but that's a bit weird.
Overall, this feature is not essential, but will make testing simpler.
> > When running in-kernel MTE in production, we'll either panic or
> > disable MTE after the first fault. This was controlled by the
> > panic_on_mte_fault option Vincenzo initially had.
>
> I prefer to disable MTE, print something and continue, but no panic.
OK, we can do this.
> > > We only skip if we emulated it.
> >
> > I'm not sure I understand this part, what do you mean by emulating?
>
> Executing it out of line or other form of instruction emulation (see
> arch/arm64/kernel/probes/simulate-insn.c) so that the access actually
> takes place. But you can single-step or experiment with some of the
> other tricks above.
>
> --
> Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-27 12:43 ` Andrey Konovalov
@ 2020-08-27 13:45 ` Vincenzo Frascino
2020-08-27 14:36 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Vincenzo Frascino @ 2020-08-27 13:45 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Catalin Marinas, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
Hi Andrey,
On 8/27/20 1:43 PM, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 1:15 PM Vincenzo Frascino
> <vincenzo.frascino@arm.com> wrote:
>>
>>
>>
>> On 8/27/20 12:13 PM, Catalin Marinas wrote:
>>> On Thu, Aug 27, 2020 at 12:05:55PM +0100, Vincenzo Frascino wrote:
>>>> On 8/27/20 11:40 AM, Catalin Marinas wrote:
>>>>> On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
>>>>>> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
>>>>>> index 152d74f2cc9c..6880ddaa5144 100644
>>>>>> --- a/arch/arm64/mm/proc.S
>>>>>> +++ b/arch/arm64/mm/proc.S
>>>>>> @@ -38,7 +38,7 @@
>>>>>> /* PTWs cacheable, inner/outer WBWA */
>>>>>> #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
>>>>>>
>>>>>> -#ifdef CONFIG_KASAN_SW_TAGS
>>>>>> +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
>>>>>> #define TCR_KASAN_FLAGS TCR_TBI1
>>>>>> #else
>>>>>> #define TCR_KASAN_FLAGS 0
>>>>>
>>>>> I prefer to turn TBI1 on only if MTE is present. So on top of the v8
>>>>> user series, just do this in __cpu_setup.
>>>>
>>>> Not sure I understand... Enabling TBI1 only if MTE is present would break
>>>> KASAN_SW_TAGS which is based on TBI1 but not on MTE.
>>>
>>> You keep the KASAN_SW_TAGS as above but for HW_TAGS, only set TBI1 later
>>> in __cpu_setup().
>>>
>>
>> Ok, sounds good.
>
> Sounds good to me too.
>
> Vincenzo, could you take care of Catalin's comments on your (arm64)
> patches, do the rebase onto user mte v8, and share it with me? I'll
> work on KASAN changes in the meantime, and then integrate everything
> together for v2.
>
I am happy to do that. I will be on holiday though from this Saturday till the
September, 9. After that I will start the rebasing.
> Perhaps the best way to test only the arm64 part is writing a simple
> module that causes an MTE fault. (At least that's what I did when I
> was testing core in-kernel MTE patches separately.) Or reuse this
> series, all KASAN patches should rebase cleanly on top of the latest
> mainline.
>
I can reuse the patches as they are, unless they require changes when I start
rebasing. In such a case to not duplicate the work I will scale back to use a
simple module.
--
Regards,
Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 32/35] kasan, arm64: print report from tag fault handler
2020-08-27 12:34 ` Andrey Konovalov
@ 2020-08-27 14:21 ` Catalin Marinas
0 siblings, 0 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 14:21 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Vincenzo Frascino, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 02:34:31PM +0200, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 12:48 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Fri, Aug 14, 2020 at 07:27:14PM +0200, Andrey Konovalov wrote:
> > > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> > > index c62c8ba85c0e..cf00b3942564 100644
> > > --- a/arch/arm64/mm/fault.c
> > > +++ b/arch/arm64/mm/fault.c
> > > @@ -14,6 +14,7 @@
> > > #include <linux/mm.h>
> > > #include <linux/hardirq.h>
> > > #include <linux/init.h>
> > > +#include <linux/kasan.h>
> > > #include <linux/kprobes.h>
> > > #include <linux/uaccess.h>
> > > #include <linux/page-flags.h>
> > > @@ -314,11 +315,19 @@ static void report_tag_fault(unsigned long addr, unsigned int esr,
> > > {
> > > bool is_write = ((esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT) != 0;
> > >
> > > +#ifdef CONFIG_KASAN_HW_TAGS
> > > + /*
> > > + * SAS bits aren't set for all faults reported in EL1, so we can't
> > > + * find out access size.
> > > + */
> > > + kasan_report(addr, 0, is_write, regs->pc);
> > > +#else
> > > pr_alert("Memory Tagging Extension Fault in %pS\n", (void *)regs->pc);
> > > pr_alert(" %s at address %lx\n", is_write ? "Write" : "Read", addr);
> > > pr_alert(" Pointer tag: [%02x], memory tag: [%02x]\n",
> > > mte_get_ptr_tag(addr),
> > > mte_get_mem_tag((void *)addr));
> > > +#endif
> > > }
> >
> > More dead code. So what's the point of keeping the pr_alert() introduced
> > earlier? CONFIG_KASAN_HW_TAGS is always on for in-kernel MTE. If MTE is
> > disabled, this function isn't called anyway.
>
> I was considering that we can enable in-kernel MTE without enabling
> CONFIG_KASAN_HW_TAGS, but perhaps this isn't what we want. I'll drop
> this part in v2, but then we also need to make sure that in-kernel MTE
> is only enabled when CONFIG_KASAN_HW_TAGS is enabled. Do we need more
> ifdefs in arm64 patches when we write to MTE-related registers, or
> does this work as is?
I think the in-kernel MTE for the time being should only mean
CONFIG_KASAN_HW_TAGS, with a dependency on CONFIG_MTE. KASAN carries
some additional debugging features but if we can trim it down, we may
not need a separate in-kernel MTE option for production systems (maybe a
CONFIG_KASAN_HW_TAGS_LITE).
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-27 13:45 ` Vincenzo Frascino
@ 2020-08-27 14:36 ` Andrey Konovalov
0 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-27 14:36 UTC (permalink / raw)
To: Vincenzo Frascino
Cc: Catalin Marinas, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 3:42 PM Vincenzo Frascino
<vincenzo.frascino@arm.com> wrote:
>
> Hi Andrey,
>
> On 8/27/20 1:43 PM, Andrey Konovalov wrote:
> > On Thu, Aug 27, 2020 at 1:15 PM Vincenzo Frascino
> > <vincenzo.frascino@arm.com> wrote:
> >>
> >>
> >>
> >> On 8/27/20 12:13 PM, Catalin Marinas wrote:
> >>> On Thu, Aug 27, 2020 at 12:05:55PM +0100, Vincenzo Frascino wrote:
> >>>> On 8/27/20 11:40 AM, Catalin Marinas wrote:
> >>>>> On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> >>>>>> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> >>>>>> index 152d74f2cc9c..6880ddaa5144 100644
> >>>>>> --- a/arch/arm64/mm/proc.S
> >>>>>> +++ b/arch/arm64/mm/proc.S
> >>>>>> @@ -38,7 +38,7 @@
> >>>>>> /* PTWs cacheable, inner/outer WBWA */
> >>>>>> #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
> >>>>>>
> >>>>>> -#ifdef CONFIG_KASAN_SW_TAGS
> >>>>>> +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> >>>>>> #define TCR_KASAN_FLAGS TCR_TBI1
> >>>>>> #else
> >>>>>> #define TCR_KASAN_FLAGS 0
> >>>>>
> >>>>> I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> >>>>> user series, just do this in __cpu_setup.
> >>>>
> >>>> Not sure I understand... Enabling TBI1 only if MTE is present would break
> >>>> KASAN_SW_TAGS which is based on TBI1 but not on MTE.
> >>>
> >>> You keep the KASAN_SW_TAGS as above but for HW_TAGS, only set TBI1 later
> >>> in __cpu_setup().
> >>>
> >>
> >> Ok, sounds good.
> >
> > Sounds good to me too.
> >
> > Vincenzo, could you take care of Catalin's comments on your (arm64)
> > patches, do the rebase onto user mte v8, and share it with me? I'll
> > work on KASAN changes in the meantime, and then integrate everything
> > together for v2.
> >
>
> I am happy to do that. I will be on holiday though from this Saturday till the
> September, 9. After that I will start the rebasing.
Ah, OK. I'll see if I can do the rebase and fix some of Catalin's
comments myself then. I'll let you know the current status once you're
back.
>
> > Perhaps the best way to test only the arm64 part is writing a simple
> > module that causes an MTE fault. (At least that's what I did when I
> > was testing core in-kernel MTE patches separately.) Or reuse this
> > series, all KASAN patches should rebase cleanly on top of the latest
> > mainline.
> >
>
> I can reuse the patches as they are, unless they require changes when I start
> rebasing. In such a case to not duplicate the work I will scale back to use a
> simple module.
>
> --
> Regards,
> Vincenzo
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-27 13:34 ` Andrey Konovalov
@ 2020-08-27 14:56 ` Catalin Marinas
2020-08-27 19:14 ` Evgenii Stepanov
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-08-27 14:56 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Vincenzo Frascino, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 03:34:42PM +0200, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 3:10 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Aug 27, 2020 at 02:31:23PM +0200, Andrey Konovalov wrote:
> > > On Thu, Aug 27, 2020 at 11:54 AM Catalin Marinas
> > > <catalin.marinas@arm.com> wrote:
> > > > On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
> > > > > +static int do_tag_recovery(unsigned long addr, unsigned int esr,
> > > > > + struct pt_regs *regs)
> > > > > +{
> > > > > + report_tag_fault(addr, esr, regs);
> > > > > +
> > > > > + /* Skip over the faulting instruction and continue: */
> > > > > + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
> > > >
> > > > Ooooh, do we expect the kernel to still behave correctly after this? I
> > > > thought the recovery means disabling tag checking altogether and
> > > > restarting the instruction rather than skipping over it.
[...]
> > > Can we disable MTE, reexecute the instruction, and then reenable MTE,
> > > or something like that?
> >
> > If you want to preserve the MTE enabled, you could single-step the
> > instruction or execute it out of line, though it's a bit more convoluted
> > (we have a similar mechanism for kprobes/uprobes).
> >
> > Another option would be to attempt to set the matching tag in memory,
> > under the assumption that it is writable (if it's not, maybe it's fine
> > to panic). Not sure how this interacts with the slub allocator since,
> > presumably, the logical tag in the pointer is wrong rather than the
> > allocation one.
> >
> > Yet another option would be to change the tag in the register and
> > re-execute but this may confuse the compiler.
>
> Which one of these would be simpler to implement?
Either 2 or 3 would be simpler (re-tag the memory location or the
pointer) with the caveats I mentioned. Also, does the slab allocator
need to touch the memory on free with a tagged pointer? Otherwise slab
may hit an MTE fault itself.
> Perhaps we could somehow only skip faulting instructions that happen
> in the KASAN test module?.. Decoding stack trace would be an option,
> but that's a bit weird.
If you want to restrict this to the KASAN tests, just add some
MTE-specific accessors with a fixup entry similar to get_user/put_user.
__do_kernel_fault() (if actually called) will invoke the fixup code
which skips the access and returns an error. This way KASAN tests can
actually verify that tag checking works, I'd find this a lot more
useful.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-27 14:56 ` Catalin Marinas
@ 2020-08-27 19:14 ` Evgenii Stepanov
2020-08-28 9:56 ` Catalin Marinas
0 siblings, 1 reply; 94+ messages in thread
From: Evgenii Stepanov @ 2020-08-27 19:14 UTC (permalink / raw)
To: Catalin Marinas
Cc: Andrey Konovalov, Vincenzo Frascino, Dmitry Vyukov, kasan-dev,
Andrey Ryabinin, Alexander Potapenko, Marco Elver, Elena Petrova,
Branislav Rankov, Kevin Brodsky, Will Deacon, Andrew Morton,
Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 7:56 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Aug 27, 2020 at 03:34:42PM +0200, Andrey Konovalov wrote:
> > On Thu, Aug 27, 2020 at 3:10 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > On Thu, Aug 27, 2020 at 02:31:23PM +0200, Andrey Konovalov wrote:
> > > > On Thu, Aug 27, 2020 at 11:54 AM Catalin Marinas
> > > > <catalin.marinas@arm.com> wrote:
> > > > > On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
> > > > > > +static int do_tag_recovery(unsigned long addr, unsigned int esr,
> > > > > > + struct pt_regs *regs)
> > > > > > +{
> > > > > > + report_tag_fault(addr, esr, regs);
> > > > > > +
> > > > > > + /* Skip over the faulting instruction and continue: */
> > > > > > + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
> > > > >
> > > > > Ooooh, do we expect the kernel to still behave correctly after this? I
> > > > > thought the recovery means disabling tag checking altogether and
> > > > > restarting the instruction rather than skipping over it.
> [...]
> > > > Can we disable MTE, reexecute the instruction, and then reenable MTE,
> > > > or something like that?
> > >
> > > If you want to preserve the MTE enabled, you could single-step the
> > > instruction or execute it out of line, though it's a bit more convoluted
> > > (we have a similar mechanism for kprobes/uprobes).
> > >
> > > Another option would be to attempt to set the matching tag in memory,
> > > under the assumption that it is writable (if it's not, maybe it's fine
> > > to panic). Not sure how this interacts with the slub allocator since,
> > > presumably, the logical tag in the pointer is wrong rather than the
> > > allocation one.
> > >
> > > Yet another option would be to change the tag in the register and
> > > re-execute but this may confuse the compiler.
> >
> > Which one of these would be simpler to implement?
>
> Either 2 or 3 would be simpler (re-tag the memory location or the
> pointer) with the caveats I mentioned. Also, does the slab allocator
> need to touch the memory on free with a tagged pointer? Otherwise slab
> may hit an MTE fault itself.
Changing the memory tag can cause faults in other threads, and that
could be very confusing.
Probably the safest thing is to retag the register, single step and
then retag it back, but be careful with the instructions that change
the address register (like ldr x0, [x0]).
>
> > Perhaps we could somehow only skip faulting instructions that happen
> > in the KASAN test module?.. Decoding stack trace would be an option,
> > but that's a bit weird.
>
> If you want to restrict this to the KASAN tests, just add some
> MTE-specific accessors with a fixup entry similar to get_user/put_user.
> __do_kernel_fault() (if actually called) will invoke the fixup code
> which skips the access and returns an error. This way KASAN tests can
> actually verify that tag checking works, I'd find this a lot more
> useful.
>
> --
> Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler
2020-08-27 19:14 ` Evgenii Stepanov
@ 2020-08-28 9:56 ` Catalin Marinas
0 siblings, 0 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-08-28 9:56 UTC (permalink / raw)
To: Evgenii Stepanov
Cc: Andrey Konovalov, Vincenzo Frascino, Dmitry Vyukov, kasan-dev,
Andrey Ryabinin, Alexander Potapenko, Marco Elver, Elena Petrova,
Branislav Rankov, Kevin Brodsky, Will Deacon, Andrew Morton,
Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 12:14:26PM -0700, Evgenii Stepanov wrote:
> On Thu, Aug 27, 2020 at 7:56 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Aug 27, 2020 at 03:34:42PM +0200, Andrey Konovalov wrote:
> > > On Thu, Aug 27, 2020 at 3:10 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > > On Thu, Aug 27, 2020 at 02:31:23PM +0200, Andrey Konovalov wrote:
> > > > > On Thu, Aug 27, 2020 at 11:54 AM Catalin Marinas
> > > > > <catalin.marinas@arm.com> wrote:
> > > > > > On Fri, Aug 14, 2020 at 07:27:03PM +0200, Andrey Konovalov wrote:
> > > > > > > +static int do_tag_recovery(unsigned long addr, unsigned int esr,
> > > > > > > + struct pt_regs *regs)
> > > > > > > +{
> > > > > > > + report_tag_fault(addr, esr, regs);
> > > > > > > +
> > > > > > > + /* Skip over the faulting instruction and continue: */
> > > > > > > + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE);
> > > > > >
> > > > > > Ooooh, do we expect the kernel to still behave correctly after this? I
> > > > > > thought the recovery means disabling tag checking altogether and
> > > > > > restarting the instruction rather than skipping over it.
> > [...]
> > > > > Can we disable MTE, reexecute the instruction, and then reenable MTE,
> > > > > or something like that?
> > > >
> > > > If you want to preserve the MTE enabled, you could single-step the
> > > > instruction or execute it out of line, though it's a bit more convoluted
> > > > (we have a similar mechanism for kprobes/uprobes).
> > > >
> > > > Another option would be to attempt to set the matching tag in memory,
> > > > under the assumption that it is writable (if it's not, maybe it's fine
> > > > to panic). Not sure how this interacts with the slub allocator since,
> > > > presumably, the logical tag in the pointer is wrong rather than the
> > > > allocation one.
> > > >
> > > > Yet another option would be to change the tag in the register and
> > > > re-execute but this may confuse the compiler.
> > >
> > > Which one of these would be simpler to implement?
> >
> > Either 2 or 3 would be simpler (re-tag the memory location or the
> > pointer) with the caveats I mentioned. Also, does the slab allocator
> > need to touch the memory on free with a tagged pointer? Otherwise slab
> > may hit an MTE fault itself.
>
> Changing the memory tag can cause faults in other threads, and that
> could be very confusing.
It could indeed trigger a chain of faults. It's not even other threads,
it could be the same thread in a different function.
> Probably the safest thing is to retag the register, single step and
> then retag it back, but be careful with the instructions that change
> the address register (like ldr x0, [x0]).
This gets complicated if you have to parse the opcode. If you can
single-step, just set PSTATE.TCO for the instruction. But the
single-step machinery gets more complicated, probably interacts badly
with kprobes.
I think the best option is to disable the MTE checks in TCF on an
_unhandled_ kernel fault, report and continue. For the KASAN tests, add
accessors similar to get_user/put_user which are able to handle the
fault and return an error. Such accessors, since they have a fixup
handler, would not lead to the MTE checks being disabled.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 35/35] kasan: add documentation for hardware tag-based mode
2020-08-14 17:27 ` [PATCH 35/35] kasan: add documentation for hardware tag-based mode Andrey Konovalov
@ 2020-08-28 11:12 ` Marco Elver
2020-08-28 12:28 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Marco Elver @ 2020-08-28 11:12 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev,
Andrey Ryabinin, Alexander Potapenko, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:27PM +0200, Andrey Konovalov wrote:
> Add documentation for hardware tag-based KASAN mode and also add some
> clarifications for software tag-based mode.
>
> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> ---
> Documentation/dev-tools/kasan.rst | 73 +++++++++++++++++++++----------
> 1 file changed, 51 insertions(+), 22 deletions(-)
>
> diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
> index a3030fc6afe5..aeed89d6eaf5 100644
[...]
> -Tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64 CPUs to
> -store a pointer tag in the top byte of kernel pointers. Like generic KASAN it
> -uses shadow memory to store memory tags associated with each 16-byte memory
> -cell (therefore it dedicates 1/16th of the kernel memory for shadow memory).
> +Software tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64
> +CPUs to store a pointer tag in the top byte of kernel pointers. Like generic
> +KASAN it uses shadow memory to store memory tags associated with each 16-byte
> +memory cell (therefore it dedicates 1/16th of the kernel memory for shadow
> +memory).
It might be helpful to be more specific vs. saying "modern arm64 CPUs".
Does the "modern" qualifier suggest not all arm64 CPUs support the
feature? (HW tag-based KASAN below is specific, and mentions ARMv8.5.)
> +On each memory allocation software tag-based KASAN generates a random tag, tags
> +the allocated memory with this tag, and embeds this tag into the returned
> +pointer.
>
> -On each memory allocation tag-based KASAN generates a random tag, tags the
> -allocated memory with this tag, and embeds this tag into the returned pointer.
> Software tag-based KASAN uses compile-time instrumentation to insert checks
> before each memory access. These checks make sure that tag of the memory that
> is being accessed is equal to tag of the pointer that is used to access this
> -memory. In case of a tag mismatch tag-based KASAN prints a bug report.
> +memory. In case of a tag mismatch software tag-based KASAN prints a bug report.
>
> Software tag-based KASAN also has two instrumentation modes (outline, that
> emits callbacks to check memory accesses; and inline, that performs the shadow
> @@ -215,9 +222,31 @@ simply printed from the function that performs the access check. With inline
> instrumentation a brk instruction is emitted by the compiler, and a dedicated
> brk handler is used to print bug reports.
>
> -A potential expansion of this mode is a hardware tag-based mode, which would
> -use hardware memory tagging support instead of compiler instrumentation and
> -manual shadow memory manipulation.
> +Software tag-based KASAN uses 0xFF as a match-all pointer tag (accesses aren't
> +checked).
> +
> +Software tag-based KASAN currently only supports tagging of slab memory.
> +
> +Hardware tag-based KASAN
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Hardware tag-based KASAN is similar to the software mode in concept, but uses
> +hardware memory tagging support instead of compiler instrumentation and
> +shadow memory.
> +
> +Hardware tag-based KASAN is based on both arm64 Memory Tagging Extension (MTE)
> +introduced in ARMv8.5 Instruction Set Architecture, and Top Byte Ignore (TBI).
Is there anything inherently tying tag-based KASAN to arm64? I guess if
some other architecture supports MTE, they just have to touch arch/,
right?
You could reword to say that "Hardware tag-based KASAN is currently only
supported on the ARM64 architecture.
On the ARM64 architecture, tag-based KASAN is based on both ..."
Thanks,
-- Marco
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 35/35] kasan: add documentation for hardware tag-based mode
2020-08-28 11:12 ` Marco Elver
@ 2020-08-28 12:28 ` Andrey Konovalov
0 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-08-28 12:28 UTC (permalink / raw)
To: Marco Elver
Cc: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev,
Andrey Ryabinin, Alexander Potapenko, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Fri, Aug 28, 2020 at 1:12 PM Marco Elver <elver@google.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27PM +0200, Andrey Konovalov wrote:
> > Add documentation for hardware tag-based KASAN mode and also add some
> > clarifications for software tag-based mode.
> >
> > Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> > ---
> > Documentation/dev-tools/kasan.rst | 73 +++++++++++++++++++++----------
> > 1 file changed, 51 insertions(+), 22 deletions(-)
> >
> > diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
> > index a3030fc6afe5..aeed89d6eaf5 100644
> [...]
> > -Tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64 CPUs to
> > -store a pointer tag in the top byte of kernel pointers. Like generic KASAN it
> > -uses shadow memory to store memory tags associated with each 16-byte memory
> > -cell (therefore it dedicates 1/16th of the kernel memory for shadow memory).
> > +Software tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64
> > +CPUs to store a pointer tag in the top byte of kernel pointers. Like generic
> > +KASAN it uses shadow memory to store memory tags associated with each 16-byte
> > +memory cell (therefore it dedicates 1/16th of the kernel memory for shadow
> > +memory).
>
> It might be helpful to be more specific vs. saying "modern arm64 CPUs".
> Does the "modern" qualifier suggest not all arm64 CPUs support the
> feature? (HW tag-based KASAN below is specific, and mentions ARMv8.5.)
Will clarify this in v2.
> > +On each memory allocation software tag-based KASAN generates a random tag, tags
> > +the allocated memory with this tag, and embeds this tag into the returned
> > +pointer.
> >
> > -On each memory allocation tag-based KASAN generates a random tag, tags the
> > -allocated memory with this tag, and embeds this tag into the returned pointer.
> > Software tag-based KASAN uses compile-time instrumentation to insert checks
> > before each memory access. These checks make sure that tag of the memory that
> > is being accessed is equal to tag of the pointer that is used to access this
> > -memory. In case of a tag mismatch tag-based KASAN prints a bug report.
> > +memory. In case of a tag mismatch software tag-based KASAN prints a bug report.
> >
> > Software tag-based KASAN also has two instrumentation modes (outline, that
> > emits callbacks to check memory accesses; and inline, that performs the shadow
> > @@ -215,9 +222,31 @@ simply printed from the function that performs the access check. With inline
> > instrumentation a brk instruction is emitted by the compiler, and a dedicated
> > brk handler is used to print bug reports.
> >
> > -A potential expansion of this mode is a hardware tag-based mode, which would
> > -use hardware memory tagging support instead of compiler instrumentation and
> > -manual shadow memory manipulation.
> > +Software tag-based KASAN uses 0xFF as a match-all pointer tag (accesses aren't
> > +checked).
> > +
> > +Software tag-based KASAN currently only supports tagging of slab memory.
> > +
> > +Hardware tag-based KASAN
> > +~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +Hardware tag-based KASAN is similar to the software mode in concept, but uses
> > +hardware memory tagging support instead of compiler instrumentation and
> > +shadow memory.
> > +
> > +Hardware tag-based KASAN is based on both arm64 Memory Tagging Extension (MTE)
> > +introduced in ARMv8.5 Instruction Set Architecture, and Top Byte Ignore (TBI).
>
> Is there anything inherently tying tag-based KASAN to arm64?
Not really, the approach is generic and can be used by any arch that
supports memory tagging.
> I guess if
> some other architecture supports MTE, they just have to touch arch/,
> right?
For the most part - yes, but maybe adjustments to the generic code
will be required. No way to know before one tries to integrate another
arch.
> You could reword to say that "Hardware tag-based KASAN is currently only
> supported on the ARM64 architecture.
>
> On the ARM64 architecture, tag-based KASAN is based on both ..."
Will do in v2, thanks!
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-08-27 10:40 ` Catalin Marinas
2020-08-27 11:05 ` Vincenzo Frascino
@ 2020-09-08 13:18 ` Andrey Konovalov
2020-09-08 14:06 ` Catalin Marinas
1 sibling, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-08 13:18 UTC (permalink / raw)
To: Catalin Marinas
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 12:40 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> > diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> > index 152d74f2cc9c..6880ddaa5144 100644
> > --- a/arch/arm64/mm/proc.S
> > +++ b/arch/arm64/mm/proc.S
> > @@ -38,7 +38,7 @@
> > /* PTWs cacheable, inner/outer WBWA */
> > #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
> >
> > -#ifdef CONFIG_KASAN_SW_TAGS
> > +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> > #define TCR_KASAN_FLAGS TCR_TBI1
> > #else
> > #define TCR_KASAN_FLAGS 0
>
> I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> user series, just do this in __cpu_setup.
Started working on this, but realized that I don't understand what
exactly is suggested here. TCR_KASAN_FLAGS are used in __cpu_setup(),
so this already happens in __cpu_setup().
Do you mean that TBI1 should be enabled when CONFIG_ARM64_MTE is
enabled, but CONFIG_KASAN_HW_TAGS is disabled?
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-08-27 9:38 ` Catalin Marinas
2020-08-27 10:31 ` Vincenzo Frascino
2020-08-27 12:46 ` Andrey Konovalov
@ 2020-09-08 13:23 ` Andrey Konovalov
2020-09-08 14:50 ` Catalin Marinas
2 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-08 13:23 UTC (permalink / raw)
To: Catalin Marinas
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 11:38 AM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27:02PM +0200, Andrey Konovalov wrote:
> > diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
> > index 1c99fcadb58c..733be1cb5c95 100644
> > --- a/arch/arm64/include/asm/mte.h
> > +++ b/arch/arm64/include/asm/mte.h
> > @@ -5,14 +5,19 @@
> > #ifndef __ASM_MTE_H
> > #define __ASM_MTE_H
> >
> > -#define MTE_GRANULE_SIZE UL(16)
> > +#include <asm/mte_asm.h>
>
> So the reason for this move is to include it in asm/cache.h. Fine by
> me but...
>
> > #define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
> > #define MTE_TAG_SHIFT 56
> > #define MTE_TAG_SIZE 4
> > +#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
> > +#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
>
> ... I'd rather move all these definitions in a file with a more
> meaningful name like mte-def.h. The _asm implies being meant for .S
> files inclusion which isn't the case.
>
> > diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> > index eb39504e390a..e2d708b4583d 100644
> > --- a/arch/arm64/kernel/mte.c
> > +++ b/arch/arm64/kernel/mte.c
> > @@ -72,6 +74,47 @@ int memcmp_pages(struct page *page1, struct page *page2)
> > return ret;
> > }
> >
> > +u8 mte_get_mem_tag(void *addr)
> > +{
> > + if (system_supports_mte())
> > + addr = mte_assign_valid_ptr_tag(addr);
>
> The mte_assign_valid_ptr_tag() is slightly misleading. All it does is
> read the allocation tag from memory.
>
> I also think this should be inline asm, possibly using alternatives.
> It's just an LDG instruction (and it saves us from having to invent a
> better function name).
Could you point me to an example of inline asm with alternatives if
there's any? I see alternative_if and other similar macros used in
arch/arm64/ code, is that what you mean? Those seem to always use
static conditions, like config values, but here we have a dynamic
system_supports_mte(). Could you elaborate on how I should implement
this?
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-08-27 10:38 ` Catalin Marinas
2020-08-27 10:56 ` Vincenzo Frascino
@ 2020-09-08 13:58 ` Andrey Konovalov
2020-09-08 15:16 ` Catalin Marinas
1 sibling, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-08 13:58 UTC (permalink / raw)
To: Catalin Marinas
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 12:38 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> > @@ -957,6 +984,7 @@ SYM_FUNC_START(cpu_switch_to)
> > mov sp, x9
> > msr sp_el0, x1
> > ptrauth_keys_install_kernel x1, x8, x9, x10
> > + mte_restore_gcr 1, x1, x8, x9
> > scs_save x0, x8
> > scs_load x1, x8
> > ret
>
> Since we set GCR_EL1 on exception entry and return, why is this needed?
> We don't have a per-kernel thread GCR_EL1, it's global to all threads,
> so I think cpu_switch_to() should not be touched.
Dropping this line from the diff leads to many false-positives... I'll
leave this to Vincenzo.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-08-27 12:16 ` Catalin Marinas
@ 2020-09-08 14:02 ` Andrey Konovalov
2020-09-08 14:53 ` Andrey Konovalov
2020-09-08 15:39 ` Catalin Marinas
0 siblings, 2 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-08 14:02 UTC (permalink / raw)
To: Catalin Marinas
Cc: Vincenzo Frascino, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Thu, Aug 27, 2020 at 2:16 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Aug 27, 2020 at 11:56:49AM +0100, Vincenzo Frascino wrote:
> > On 8/27/20 11:38 AM, Catalin Marinas wrote:
> > > On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> > >> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> > >> index 7717ea9bc2a7..cfac7d02f032 100644
> > >> --- a/arch/arm64/kernel/mte.c
> > >> +++ b/arch/arm64/kernel/mte.c
> > >> @@ -18,10 +18,14 @@
> > >>
> > >> #include <asm/barrier.h>
> > >> #include <asm/cpufeature.h>
> > >> +#include <asm/kasan.h>
> > >> +#include <asm/kprobes.h>
> > >> #include <asm/mte.h>
> > >> #include <asm/ptrace.h>
> > >> #include <asm/sysreg.h>
> > >>
> > >> +u64 gcr_kernel_excl __read_mostly;
> > >
> > > Could we make this __ro_after_init?
> >
> > Yes, it makes sense, it should be updated only once through mte_init_tags().
> >
> > Something to consider though here is that this might not be the right approach
> > if in future we want to add stack tagging. In such a case we need to know the
> > kernel exclude mask before any C code is executed. Initializing the mask via
> > mte_init_tags() it is too late.
>
> It depends on how stack tagging ends up in the kernel, whether it uses
> ADDG/SUBG or not. If it's only IRG, I think it can cope with changing
> the GCR_EL1.Excl in the middle of a function.
>
> > I was thinking to add a compilation define instead of having gcr_kernel_excl in
> > place. This might not work if the kernel excl mask is meant to change during the
> > execution.
>
> A macro with the default value works for me. That's what it basically is
> currently, only that it ends up in a variable.
Some thoughts on the topic: gcr_kernel_excl is currently initialized
in mte_init_tags() and depends on the max_tag value dynamically
provided to it, so it's not something that can be expressed with a
define. In the case of KASAN the max_tag value is static, but if we
rely on that we make core MTE code depend on KASAN, which doesn't seem
right from the design perspective.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-09-08 13:18 ` Andrey Konovalov
@ 2020-09-08 14:06 ` Catalin Marinas
2020-09-08 14:12 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-09-08 14:06 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Tue, Sep 08, 2020 at 03:18:04PM +0200, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 12:40 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> >
> > On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> > > diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> > > index 152d74f2cc9c..6880ddaa5144 100644
> > > --- a/arch/arm64/mm/proc.S
> > > +++ b/arch/arm64/mm/proc.S
> > > @@ -38,7 +38,7 @@
> > > /* PTWs cacheable, inner/outer WBWA */
> > > #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
> > >
> > > -#ifdef CONFIG_KASAN_SW_TAGS
> > > +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> > > #define TCR_KASAN_FLAGS TCR_TBI1
> > > #else
> > > #define TCR_KASAN_FLAGS 0
> >
> > I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> > user series, just do this in __cpu_setup.
>
> Started working on this, but realized that I don't understand what
> exactly is suggested here. TCR_KASAN_FLAGS are used in __cpu_setup(),
> so this already happens in __cpu_setup().
>
> Do you mean that TBI1 should be enabled when CONFIG_ARM64_MTE is
> enabled, but CONFIG_KASAN_HW_TAGS is disabled?
What I meant is that we should turn TBI1 only when the MTE is present in
hardware (and the ARM64_MTE option is on). But I probably missed the way
MTE is used with KASAN.
So what happens if CONFIG_KASAN_HW_TAGS and CONFIG_ARM64_MTE are both on
but the hardware does not support MTE? Does KASAN still generate tagged
pointers? If yes, then the current patch is fine, we should always set
TBI1.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-09-08 14:06 ` Catalin Marinas
@ 2020-09-08 14:12 ` Andrey Konovalov
2020-09-08 14:41 ` Catalin Marinas
0 siblings, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-08 14:12 UTC (permalink / raw)
To: Catalin Marinas
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Tue, Sep 8, 2020 at 4:06 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Tue, Sep 08, 2020 at 03:18:04PM +0200, Andrey Konovalov wrote:
> > On Thu, Aug 27, 2020 at 12:40 PM Catalin Marinas
> > <catalin.marinas@arm.com> wrote:
> > >
> > > On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> > > > diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> > > > index 152d74f2cc9c..6880ddaa5144 100644
> > > > --- a/arch/arm64/mm/proc.S
> > > > +++ b/arch/arm64/mm/proc.S
> > > > @@ -38,7 +38,7 @@
> > > > /* PTWs cacheable, inner/outer WBWA */
> > > > #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
> > > >
> > > > -#ifdef CONFIG_KASAN_SW_TAGS
> > > > +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> > > > #define TCR_KASAN_FLAGS TCR_TBI1
> > > > #else
> > > > #define TCR_KASAN_FLAGS 0
> > >
> > > I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> > > user series, just do this in __cpu_setup.
> >
> > Started working on this, but realized that I don't understand what
> > exactly is suggested here. TCR_KASAN_FLAGS are used in __cpu_setup(),
> > so this already happens in __cpu_setup().
> >
> > Do you mean that TBI1 should be enabled when CONFIG_ARM64_MTE is
> > enabled, but CONFIG_KASAN_HW_TAGS is disabled?
>
> What I meant is that we should turn TBI1 only when the MTE is present in
> hardware (and the ARM64_MTE option is on). But I probably missed the way
> MTE is used with KASAN.
>
> So what happens if CONFIG_KASAN_HW_TAGS and CONFIG_ARM64_MTE are both on
> but the hardware does not support MTE? Does KASAN still generate tagged
> pointers? If yes, then the current patch is fine, we should always set
> TBI1.
No, the tag is always 0xFF when MTE is not supported.
Should we then only enable TBI1 if system_supports_mte() or something like that?
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 22/35] arm64: mte: Enable in-kernel MTE
2020-08-14 17:27 ` [PATCH 22/35] arm64: mte: Enable in-kernel MTE Andrey Konovalov
2020-08-27 10:01 ` Catalin Marinas
@ 2020-09-08 14:39 ` Andrey Konovalov
2020-09-08 14:52 ` Catalin Marinas
1 sibling, 1 reply; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-08 14:39 UTC (permalink / raw)
To: Vincenzo Frascino, Catalin Marinas
Cc: Andrey Ryabinin, Alexander Potapenko, Marco Elver,
Evgenii Stepanov, Elena Petrova, Branislav Rankov, Kevin Brodsky,
Will Deacon, Andrew Morton, Linux ARM,
Linux Memory Management List, LKML, Dmitry Vyukov, kasan-dev
On Fri, Aug 14, 2020 at 7:28 PM Andrey Konovalov <andreyknvl@google.com> wrote:
>
> From: Vincenzo Frascino <vincenzo.frascino@arm.com>
>
> The Tag Checking operation causes a synchronous data abort as
> a consequence of a tag check fault when MTE is configured in
> synchronous mode.
>
> Enable MTE in Synchronous mode in EL1 to provide a more immediate
> way of tag check failure detection in the kernel.
>
> As part of this change enable match-all tag for EL1 to allow the
> kernel to access user pages without faulting. This is required because
> the kernel does not have knowledge of the tags set by the user in a
> page.
>
> Note: For MTE, the TCF bit field in SCTLR_EL1 affects only EL1 in a
> similar way as TCF0 affects EL0.
>
> Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
> ---
> arch/arm64/kernel/cpufeature.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 4d3abb51f7d4..4d94af19d8f6 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -1670,6 +1670,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
> write_sysreg_s(0, SYS_TFSR_EL1);
> write_sysreg_s(0, SYS_TFSRE0_EL1);
>
> + /* Enable Match-All at EL1 */
> + sysreg_clear_set(tcr_el1, 0, SYS_TCR_EL1_TCMA1);
> +
> /*
> * CnP must be enabled only after the MAIR_EL1 register has been set
> * up. Inconsistent MAIR_EL1 between CPUs sharing the same TLB may
> @@ -1687,6 +1690,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
> mair &= ~MAIR_ATTRIDX(MAIR_ATTR_MASK, MT_NORMAL_TAGGED);
> mair |= MAIR_ATTRIDX(MAIR_ATTR_NORMAL_TAGGED, MT_NORMAL_TAGGED);
> write_sysreg_s(mair, SYS_MAIR_EL1);
> +
> + /* Enable MTE Sync Mode for EL1 */
> + sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC);
> isb();
>
> local_flush_tlb_all();
> --
> 2.28.0.220.ged08abb693-goog
>
Should we change this commit to enable in-kernel MTE only if
KASAN_HW_TAGS is enabled?
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 26/35] kasan, arm64: Enable TBI EL1
2020-09-08 14:12 ` Andrey Konovalov
@ 2020-09-08 14:41 ` Catalin Marinas
0 siblings, 0 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-09-08 14:41 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Tue, Sep 08, 2020 at 04:12:49PM +0200, Andrey Konovalov wrote:
> On Tue, Sep 8, 2020 at 4:06 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Tue, Sep 08, 2020 at 03:18:04PM +0200, Andrey Konovalov wrote:
> > > On Thu, Aug 27, 2020 at 12:40 PM Catalin Marinas
> > > <catalin.marinas@arm.com> wrote:
> > > > On Fri, Aug 14, 2020 at 07:27:08PM +0200, Andrey Konovalov wrote:
> > > > > diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> > > > > index 152d74f2cc9c..6880ddaa5144 100644
> > > > > --- a/arch/arm64/mm/proc.S
> > > > > +++ b/arch/arm64/mm/proc.S
> > > > > @@ -38,7 +38,7 @@
> > > > > /* PTWs cacheable, inner/outer WBWA */
> > > > > #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA
> > > > >
> > > > > -#ifdef CONFIG_KASAN_SW_TAGS
> > > > > +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
> > > > > #define TCR_KASAN_FLAGS TCR_TBI1
> > > > > #else
> > > > > #define TCR_KASAN_FLAGS 0
> > > >
> > > > I prefer to turn TBI1 on only if MTE is present. So on top of the v8
> > > > user series, just do this in __cpu_setup.
> > >
> > > Started working on this, but realized that I don't understand what
> > > exactly is suggested here. TCR_KASAN_FLAGS are used in __cpu_setup(),
> > > so this already happens in __cpu_setup().
> > >
> > > Do you mean that TBI1 should be enabled when CONFIG_ARM64_MTE is
> > > enabled, but CONFIG_KASAN_HW_TAGS is disabled?
> >
> > What I meant is that we should turn TBI1 only when the MTE is present in
> > hardware (and the ARM64_MTE option is on). But I probably missed the way
> > MTE is used with KASAN.
> >
> > So what happens if CONFIG_KASAN_HW_TAGS and CONFIG_ARM64_MTE are both on
> > but the hardware does not support MTE? Does KASAN still generate tagged
> > pointers? If yes, then the current patch is fine, we should always set
> > TBI1.
>
> No, the tag is always 0xFF when MTE is not supported.
>
> Should we then only enable TBI1 if system_supports_mte() or something
> like that?
You could add it do this block in __cpu_setup:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/tree/arch/arm64/mm/proc.S?h=for-next/mte#n429
It needs a few changes to have "mov_q x10, TCR_..." before the MTE
check so that you can add the TBI1 bit in there.
system_supports_mte() would be called too late, you want this set before
the MMU is turned on.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers
2020-09-08 13:23 ` Andrey Konovalov
@ 2020-09-08 14:50 ` Catalin Marinas
0 siblings, 0 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-09-08 14:50 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Tue, Sep 08, 2020 at 03:23:20PM +0200, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 11:38 AM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Fri, Aug 14, 2020 at 07:27:02PM +0200, Andrey Konovalov wrote:
> > > diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
> > > index 1c99fcadb58c..733be1cb5c95 100644
> > > --- a/arch/arm64/include/asm/mte.h
> > > +++ b/arch/arm64/include/asm/mte.h
> > > @@ -5,14 +5,19 @@
> > > #ifndef __ASM_MTE_H
> > > #define __ASM_MTE_H
> > >
> > > -#define MTE_GRANULE_SIZE UL(16)
> > > +#include <asm/mte_asm.h>
> >
> > So the reason for this move is to include it in asm/cache.h. Fine by
> > me but...
> >
> > > #define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1))
> > > #define MTE_TAG_SHIFT 56
> > > #define MTE_TAG_SIZE 4
> > > +#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
> > > +#define MTE_TAG_MAX (MTE_TAG_MASK >> MTE_TAG_SHIFT)
> >
> > ... I'd rather move all these definitions in a file with a more
> > meaningful name like mte-def.h. The _asm implies being meant for .S
> > files inclusion which isn't the case.
> >
> > > diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> > > index eb39504e390a..e2d708b4583d 100644
> > > --- a/arch/arm64/kernel/mte.c
> > > +++ b/arch/arm64/kernel/mte.c
> > > @@ -72,6 +74,47 @@ int memcmp_pages(struct page *page1, struct page *page2)
> > > return ret;
> > > }
> > >
> > > +u8 mte_get_mem_tag(void *addr)
> > > +{
> > > + if (system_supports_mte())
> > > + addr = mte_assign_valid_ptr_tag(addr);
> >
> > The mte_assign_valid_ptr_tag() is slightly misleading. All it does is
> > read the allocation tag from memory.
> >
> > I also think this should be inline asm, possibly using alternatives.
> > It's just an LDG instruction (and it saves us from having to invent a
> > better function name).
>
> Could you point me to an example of inline asm with alternatives if
> there's any? I see alternative_if and other similar macros used in
> arch/arm64/ code, is that what you mean? Those seem to always use
> static conditions, like config values, but here we have a dynamic
> system_supports_mte(). Could you elaborate on how I should implement
> this?
There are plenty of ALTERNATIVE macro uses under arch/arm64, see
arch/arm64/include/asm/alternative.h for the definition and some simple
documentation.
In this case, something like (untested, haven't even checked whether it
matches the mte_assign_valid_ptr_tag() code):
asm(ALTERNATIVE("orr %0, %1, #0xff << 56", "ldg %0, [%1]", ARM64_HAS_MTE));
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 22/35] arm64: mte: Enable in-kernel MTE
2020-09-08 14:39 ` Andrey Konovalov
@ 2020-09-08 14:52 ` Catalin Marinas
0 siblings, 0 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-09-08 14:52 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Vincenzo Frascino, Andrey Ryabinin, Alexander Potapenko,
Marco Elver, Evgenii Stepanov, Elena Petrova, Branislav Rankov,
Kevin Brodsky, Will Deacon, Andrew Morton, Linux ARM,
Linux Memory Management List, LKML, Dmitry Vyukov, kasan-dev
On Tue, Sep 08, 2020 at 04:39:35PM +0200, Andrey Konovalov wrote:
> On Fri, Aug 14, 2020 at 7:28 PM Andrey Konovalov <andreyknvl@google.com> wrote:
> > From: Vincenzo Frascino <vincenzo.frascino@arm.com>
> >
> > The Tag Checking operation causes a synchronous data abort as
> > a consequence of a tag check fault when MTE is configured in
> > synchronous mode.
> >
> > Enable MTE in Synchronous mode in EL1 to provide a more immediate
> > way of tag check failure detection in the kernel.
> >
> > As part of this change enable match-all tag for EL1 to allow the
> > kernel to access user pages without faulting. This is required because
> > the kernel does not have knowledge of the tags set by the user in a
> > page.
> >
> > Note: For MTE, the TCF bit field in SCTLR_EL1 affects only EL1 in a
> > similar way as TCF0 affects EL0.
> >
> > Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
> > ---
> > arch/arm64/kernel/cpufeature.c | 6 ++++++
> > 1 file changed, 6 insertions(+)
> >
> > diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> > index 4d3abb51f7d4..4d94af19d8f6 100644
> > --- a/arch/arm64/kernel/cpufeature.c
> > +++ b/arch/arm64/kernel/cpufeature.c
> > @@ -1670,6 +1670,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
> > write_sysreg_s(0, SYS_TFSR_EL1);
> > write_sysreg_s(0, SYS_TFSRE0_EL1);
> >
> > + /* Enable Match-All at EL1 */
> > + sysreg_clear_set(tcr_el1, 0, SYS_TCR_EL1_TCMA1);
> > +
> > /*
> > * CnP must be enabled only after the MAIR_EL1 register has been set
> > * up. Inconsistent MAIR_EL1 between CPUs sharing the same TLB may
> > @@ -1687,6 +1690,9 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
> > mair &= ~MAIR_ATTRIDX(MAIR_ATTR_MASK, MT_NORMAL_TAGGED);
> > mair |= MAIR_ATTRIDX(MAIR_ATTR_NORMAL_TAGGED, MT_NORMAL_TAGGED);
> > write_sysreg_s(mair, SYS_MAIR_EL1);
> > +
> > + /* Enable MTE Sync Mode for EL1 */
> > + sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC);
> > isb();
> >
> > local_flush_tlb_all();
> > --
> > 2.28.0.220.ged08abb693-goog
> >
>
> Should we change this commit to enable in-kernel MTE only if
> KASAN_HW_TAGS is enabled?
I think so. We don't currently have any patchset decoupling MTE from
KASAN.
See my other comment on TCR_EL1.TBI1, you'd need to set TCMA1 as well in
the same proc.S file.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-09-08 14:02 ` Andrey Konovalov
@ 2020-09-08 14:53 ` Andrey Konovalov
2020-09-08 15:39 ` Catalin Marinas
1 sibling, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-08 14:53 UTC (permalink / raw)
To: Catalin Marinas
Cc: Vincenzo Frascino, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Tue, Sep 8, 2020 at 4:02 PM Andrey Konovalov <andreyknvl@google.com> wrote:
>
> On Thu, Aug 27, 2020 at 2:16 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> >
> > On Thu, Aug 27, 2020 at 11:56:49AM +0100, Vincenzo Frascino wrote:
> > > On 8/27/20 11:38 AM, Catalin Marinas wrote:
> > > > On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> > > >> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> > > >> index 7717ea9bc2a7..cfac7d02f032 100644
> > > >> --- a/arch/arm64/kernel/mte.c
> > > >> +++ b/arch/arm64/kernel/mte.c
> > > >> @@ -18,10 +18,14 @@
> > > >>
> > > >> #include <asm/barrier.h>
> > > >> #include <asm/cpufeature.h>
> > > >> +#include <asm/kasan.h>
> > > >> +#include <asm/kprobes.h>
> > > >> #include <asm/mte.h>
> > > >> #include <asm/ptrace.h>
> > > >> #include <asm/sysreg.h>
> > > >>
> > > >> +u64 gcr_kernel_excl __read_mostly;
> > > >
> > > > Could we make this __ro_after_init?
> > >
> > > Yes, it makes sense, it should be updated only once through mte_init_tags().
> > >
> > > Something to consider though here is that this might not be the right approach
> > > if in future we want to add stack tagging. In such a case we need to know the
> > > kernel exclude mask before any C code is executed. Initializing the mask via
> > > mte_init_tags() it is too late.
> >
> > It depends on how stack tagging ends up in the kernel, whether it uses
> > ADDG/SUBG or not. If it's only IRG, I think it can cope with changing
> > the GCR_EL1.Excl in the middle of a function.
> >
> > > I was thinking to add a compilation define instead of having gcr_kernel_excl in
> > > place. This might not work if the kernel excl mask is meant to change during the
> > > execution.
> >
> > A macro with the default value works for me. That's what it basically is
> > currently, only that it ends up in a variable.
>
> Some thoughts on the topic: gcr_kernel_excl is currently initialized
> in mte_init_tags() and depends on the max_tag value dynamically
> provided to it, so it's not something that can be expressed with a
> define. In the case of KASAN the max_tag value is static, but if we
> rely on that we make core MTE code depend on KASAN, which doesn't seem
> right from the design perspective.
Thinking more about this, I think we've actually discussed moving
KASAN_MAX_TAG to somewhere in low-level headers, so I guess we can
reuse that and make gcr_kernel_excl a define. I'll look into this.
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-09-08 13:58 ` Andrey Konovalov
@ 2020-09-08 15:16 ` Catalin Marinas
0 siblings, 0 replies; 94+ messages in thread
From: Catalin Marinas @ 2020-09-08 15:16 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Tue, Sep 08, 2020 at 03:58:07PM +0200, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 12:38 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> >
> > On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> > > @@ -957,6 +984,7 @@ SYM_FUNC_START(cpu_switch_to)
> > > mov sp, x9
> > > msr sp_el0, x1
> > > ptrauth_keys_install_kernel x1, x8, x9, x10
> > > + mte_restore_gcr 1, x1, x8, x9
> > > scs_save x0, x8
> > > scs_load x1, x8
> > > ret
> >
> > Since we set GCR_EL1 on exception entry and return, why is this needed?
> > We don't have a per-kernel thread GCR_EL1, it's global to all threads,
> > so I think cpu_switch_to() should not be touched.
>
> Dropping this line from the diff leads to many false-positives... I'll
> leave this to Vincenzo.
I wouldn't expect this to have any effect but maybe the
mte_thread_switch() code still touches GCR_EL1 (it does this in the
user-space support, Vincenzo's patches should move that to exception
entry/return).
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-09-08 14:02 ` Andrey Konovalov
2020-09-08 14:53 ` Andrey Konovalov
@ 2020-09-08 15:39 ` Catalin Marinas
2020-09-08 19:41 ` Derrick McKee
1 sibling, 1 reply; 94+ messages in thread
From: Catalin Marinas @ 2020-09-08 15:39 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Vincenzo Frascino, Dmitry Vyukov, kasan-dev, Andrey Ryabinin,
Alexander Potapenko, Marco Elver, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Tue, Sep 08, 2020 at 04:02:06PM +0200, Andrey Konovalov wrote:
> On Thu, Aug 27, 2020 at 2:16 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Aug 27, 2020 at 11:56:49AM +0100, Vincenzo Frascino wrote:
> > > On 8/27/20 11:38 AM, Catalin Marinas wrote:
> > > > On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> > > >> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> > > >> index 7717ea9bc2a7..cfac7d02f032 100644
> > > >> --- a/arch/arm64/kernel/mte.c
> > > >> +++ b/arch/arm64/kernel/mte.c
> > > >> @@ -18,10 +18,14 @@
> > > >>
> > > >> #include <asm/barrier.h>
> > > >> #include <asm/cpufeature.h>
> > > >> +#include <asm/kasan.h>
> > > >> +#include <asm/kprobes.h>
> > > >> #include <asm/mte.h>
> > > >> #include <asm/ptrace.h>
> > > >> #include <asm/sysreg.h>
> > > >>
> > > >> +u64 gcr_kernel_excl __read_mostly;
> > > >
> > > > Could we make this __ro_after_init?
> > >
> > > Yes, it makes sense, it should be updated only once through mte_init_tags().
> > >
> > > Something to consider though here is that this might not be the right approach
> > > if in future we want to add stack tagging. In such a case we need to know the
> > > kernel exclude mask before any C code is executed. Initializing the mask via
> > > mte_init_tags() it is too late.
> >
> > It depends on how stack tagging ends up in the kernel, whether it uses
> > ADDG/SUBG or not. If it's only IRG, I think it can cope with changing
> > the GCR_EL1.Excl in the middle of a function.
> >
> > > I was thinking to add a compilation define instead of having gcr_kernel_excl in
> > > place. This might not work if the kernel excl mask is meant to change during the
> > > execution.
> >
> > A macro with the default value works for me. That's what it basically is
> > currently, only that it ends up in a variable.
>
> Some thoughts on the topic: gcr_kernel_excl is currently initialized
> in mte_init_tags() and depends on the max_tag value dynamically
> provided to it, so it's not something that can be expressed with a
> define. In the case of KASAN the max_tag value is static, but if we
> rely on that we make core MTE code depend on KASAN, which doesn't seem
> right from the design perspective.
The design is debatable. If we want MTE to run on production devices, we
either (1) optimise out some bits of KASAN (configurable) or (2) we
decouple MTE and KASAN completely and add new callbacks in the core code
(slab allocator etc.) specific to MTE.
My first choice is (1), unless there is a strong technical argument why
it is not possible.
--
Catalin
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit
2020-09-08 15:39 ` Catalin Marinas
@ 2020-09-08 19:41 ` Derrick McKee
0 siblings, 0 replies; 94+ messages in thread
From: Derrick McKee @ 2020-09-08 19:41 UTC (permalink / raw)
To: Catalin Marinas
Cc: Andrey Konovalov, Linux ARM, Marco Elver, Elena Petrova,
Kevin Brodsky, Will Deacon, Branislav Rankov, kasan-dev, LKML,
Linux Memory Management List, Alexander Potapenko,
Evgenii Stepanov, Andrey Ryabinin, Andrew Morton,
Vincenzo Frascino, Dmitry Vyukov
Hello,
Is the branch where the MTE patches currently are being applied
for-net/mte? It looks like that's the place, but I want to confirm.
On Tue, Sep 8, 2020 at 11:42 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Tue, Sep 08, 2020 at 04:02:06PM +0200, Andrey Konovalov wrote:
> > On Thu, Aug 27, 2020 at 2:16 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > On Thu, Aug 27, 2020 at 11:56:49AM +0100, Vincenzo Frascino wrote:
> > > > On 8/27/20 11:38 AM, Catalin Marinas wrote:
> > > > > On Fri, Aug 14, 2020 at 07:27:06PM +0200, Andrey Konovalov wrote:
> > > > >> diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
> > > > >> index 7717ea9bc2a7..cfac7d02f032 100644
> > > > >> --- a/arch/arm64/kernel/mte.c
> > > > >> +++ b/arch/arm64/kernel/mte.c
> > > > >> @@ -18,10 +18,14 @@
> > > > >>
> > > > >> #include <asm/barrier.h>
> > > > >> #include <asm/cpufeature.h>
> > > > >> +#include <asm/kasan.h>
> > > > >> +#include <asm/kprobes.h>
> > > > >> #include <asm/mte.h>
> > > > >> #include <asm/ptrace.h>
> > > > >> #include <asm/sysreg.h>
> > > > >>
> > > > >> +u64 gcr_kernel_excl __read_mostly;
> > > > >
> > > > > Could we make this __ro_after_init?
> > > >
> > > > Yes, it makes sense, it should be updated only once through mte_init_tags().
> > > >
> > > > Something to consider though here is that this might not be the right approach
> > > > if in future we want to add stack tagging. In such a case we need to know the
> > > > kernel exclude mask before any C code is executed. Initializing the mask via
> > > > mte_init_tags() it is too late.
> > >
> > > It depends on how stack tagging ends up in the kernel, whether it uses
> > > ADDG/SUBG or not. If it's only IRG, I think it can cope with changing
> > > the GCR_EL1.Excl in the middle of a function.
> > >
> > > > I was thinking to add a compilation define instead of having gcr_kernel_excl in
> > > > place. This might not work if the kernel excl mask is meant to change during the
> > > > execution.
> > >
> > > A macro with the default value works for me. That's what it basically is
> > > currently, only that it ends up in a variable.
> >
> > Some thoughts on the topic: gcr_kernel_excl is currently initialized
> > in mte_init_tags() and depends on the max_tag value dynamically
> > provided to it, so it's not something that can be expressed with a
> > define. In the case of KASAN the max_tag value is static, but if we
> > rely on that we make core MTE code depend on KASAN, which doesn't seem
> > right from the design perspective.
>
> The design is debatable. If we want MTE to run on production devices, we
> either (1) optimise out some bits of KASAN (configurable) or (2) we
> decouple MTE and KASAN completely and add new callbacks in the core code
> (slab allocator etc.) specific to MTE.
>
> My first choice is (1), unless there is a strong technical argument why
> it is not possible.
>
> --
> Catalin
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
--
Derrick McKee
Phone: (703) 957-9362
Email: derrick.mckee@gmail.com
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 03/35] kasan: shadow declarations only for software modes
2020-08-14 17:26 ` [PATCH 03/35] kasan: shadow declarations only for software modes Andrey Konovalov
@ 2020-09-18 14:55 ` Marco Elver
2020-09-18 14:56 ` Andrey Konovalov
0 siblings, 1 reply; 94+ messages in thread
From: Marco Elver @ 2020-09-18 14:55 UTC (permalink / raw)
To: Andrey Konovalov
Cc: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev,
Andrey Ryabinin, Alexander Potapenko, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, linux-arm-kernel, linux-mm, linux-kernel
On Fri, Aug 14, 2020 at 07:26PM +0200, Andrey Konovalov wrote:
> This is a preparatory commit for the upcoming addition of a new hardware
> tag-based (MTE-based) KASAN mode.
>
> Group shadow-related KASAN function declarations and only define them
> for the two existing software modes.
>
> No functional changes for software modes.
>
> Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> ---
> include/linux/kasan.h | 44 ++++++++++++++++++++++++++-----------------
> 1 file changed, 27 insertions(+), 17 deletions(-)
>
> diff --git a/include/linux/kasan.h b/include/linux/kasan.h
> index bd5b4965a269..44a9aae44138 100644
> --- a/include/linux/kasan.h
> +++ b/include/linux/kasan.h
[...]
> +static inline int kasan_add_zero_shadow(void *start, unsigned long size)
> +{
> + return 0;
> +}
> +static inline void kasan_remove_zero_shadow(void *start,
> + unsigned long size)
> +{}
Readability suggestion (latest checkpatch.pl allows up to 100 cols):
-static inline void kasan_remove_zero_shadow(void *start,
- unsigned long size)
-{}
+static inline void kasan_remove_zero_shadow(void *start, unsigned long size) {}
Thanks,
-- Marco
^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH 03/35] kasan: shadow declarations only for software modes
2020-09-18 14:55 ` Marco Elver
@ 2020-09-18 14:56 ` Andrey Konovalov
0 siblings, 0 replies; 94+ messages in thread
From: Andrey Konovalov @ 2020-09-18 14:56 UTC (permalink / raw)
To: Marco Elver
Cc: Dmitry Vyukov, Vincenzo Frascino, Catalin Marinas, kasan-dev,
Andrey Ryabinin, Alexander Potapenko, Evgenii Stepanov,
Elena Petrova, Branislav Rankov, Kevin Brodsky, Will Deacon,
Andrew Morton, Linux ARM, Linux Memory Management List, LKML
On Fri, Sep 18, 2020 at 4:55 PM Marco Elver <elver@google.com> wrote:
>
> On Fri, Aug 14, 2020 at 07:26PM +0200, Andrey Konovalov wrote:
> > This is a preparatory commit for the upcoming addition of a new hardware
> > tag-based (MTE-based) KASAN mode.
> >
> > Group shadow-related KASAN function declarations and only define them
> > for the two existing software modes.
> >
> > No functional changes for software modes.
> >
> > Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
> > ---
> > include/linux/kasan.h | 44 ++++++++++++++++++++++++++-----------------
> > 1 file changed, 27 insertions(+), 17 deletions(-)
> >
> > diff --git a/include/linux/kasan.h b/include/linux/kasan.h
> > index bd5b4965a269..44a9aae44138 100644
> > --- a/include/linux/kasan.h
> > +++ b/include/linux/kasan.h
> [...]
> > +static inline int kasan_add_zero_shadow(void *start, unsigned long size)
> > +{
> > + return 0;
> > +}
> > +static inline void kasan_remove_zero_shadow(void *start,
> > + unsigned long size)
> > +{}
>
> Readability suggestion (latest checkpatch.pl allows up to 100 cols):
>
> -static inline void kasan_remove_zero_shadow(void *start,
> - unsigned long size)
> -{}
> +static inline void kasan_remove_zero_shadow(void *start, unsigned long size) {}
Will do in v3, thanks!
^ permalink raw reply [flat|nested] 94+ messages in thread
end of thread, other threads:[~2020-09-18 14:56 UTC | newest]
Thread overview: 94+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-14 17:26 [PATCH 00/35] kasan: add hardware tag-based mode for arm64 Andrey Konovalov
2020-08-14 17:26 ` [PATCH 01/35] kasan: KASAN_VMALLOC depends on KASAN_GENERIC Andrey Konovalov
2020-08-14 17:26 ` [PATCH 02/35] kasan: group vmalloc code Andrey Konovalov
2020-08-14 17:26 ` [PATCH 03/35] kasan: shadow declarations only for software modes Andrey Konovalov
2020-09-18 14:55 ` Marco Elver
2020-09-18 14:56 ` Andrey Konovalov
2020-08-14 17:26 ` [PATCH 04/35] kasan: rename (un)poison_shadow to (un)poison_memory Andrey Konovalov
2020-08-14 17:26 ` [PATCH 05/35] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_* Andrey Konovalov
2020-08-14 17:26 ` [PATCH 06/35] kasan: only build init.c for software modes Andrey Konovalov
2020-08-14 17:26 ` [PATCH 07/35] kasan: split out shadow.c from common.c Andrey Konovalov
2020-08-14 17:26 ` [PATCH 08/35] kasan: rename generic/tags_report.c files Andrey Konovalov
2020-08-14 17:26 ` [PATCH 09/35] kasan: don't duplicate config dependencies Andrey Konovalov
2020-08-14 17:26 ` [PATCH 10/35] kasan: hide invalid free check implementation Andrey Konovalov
2020-08-14 17:26 ` [PATCH 11/35] kasan: decode stack frame only with KASAN_STACK_ENABLE Andrey Konovalov
2020-08-14 17:26 ` [PATCH 12/35] kasan, arm64: only init shadow for software modes Andrey Konovalov
2020-08-14 17:26 ` [PATCH 13/35] kasan, arm64: only use kasan_depth " Andrey Konovalov
2020-08-14 17:26 ` [PATCH 14/35] kasan: rename addr_has_shadow to addr_has_metadata Andrey Konovalov
2020-08-14 17:26 ` [PATCH 15/35] kasan: rename print_shadow_for_address to print_memory_metadata Andrey Konovalov
2020-08-14 17:26 ` [PATCH 16/35] kasan: kasan_non_canonical_hook only for software modes Andrey Konovalov
2020-08-14 17:26 ` [PATCH 17/35] kasan: rename SHADOW layout macros to META Andrey Konovalov
2020-08-14 17:27 ` [PATCH 18/35] kasan: separate metadata_fetch_row for each mode Andrey Konovalov
2020-08-14 17:27 ` [PATCH 19/35] kasan: don't allow SW_TAGS with ARM64_MTE Andrey Konovalov
2020-08-27 8:04 ` Catalin Marinas
2020-08-27 9:54 ` Vincenzo Frascino
2020-08-27 12:02 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 20/35] arm64: mte: Add in-kernel MTE helpers Andrey Konovalov
2020-08-27 9:38 ` Catalin Marinas
2020-08-27 10:31 ` Vincenzo Frascino
2020-08-27 11:10 ` Catalin Marinas
2020-08-27 11:24 ` Vincenzo Frascino
2020-08-27 12:46 ` Andrey Konovalov
2020-09-08 13:23 ` Andrey Konovalov
2020-09-08 14:50 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 21/35] arm64: mte: Add in-kernel tag fault handler Andrey Konovalov
2020-08-27 9:54 ` Catalin Marinas
2020-08-27 10:44 ` Vincenzo Frascino
2020-08-27 12:31 ` Andrey Konovalov
2020-08-27 13:10 ` Catalin Marinas
2020-08-27 13:34 ` Andrey Konovalov
2020-08-27 14:56 ` Catalin Marinas
2020-08-27 19:14 ` Evgenii Stepanov
2020-08-28 9:56 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 22/35] arm64: mte: Enable in-kernel MTE Andrey Konovalov
2020-08-27 10:01 ` Catalin Marinas
2020-08-27 10:46 ` Vincenzo Frascino
2020-09-08 14:39 ` Andrey Konovalov
2020-09-08 14:52 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 23/35] arm64: mte: Convert gcr_user into an exclude mask Andrey Konovalov
2020-08-14 17:27 ` [PATCH 24/35] arm64: mte: Switch GCR_EL1 in kernel entry and exit Andrey Konovalov
2020-08-27 10:38 ` Catalin Marinas
2020-08-27 10:56 ` Vincenzo Frascino
2020-08-27 12:16 ` Catalin Marinas
2020-09-08 14:02 ` Andrey Konovalov
2020-09-08 14:53 ` Andrey Konovalov
2020-09-08 15:39 ` Catalin Marinas
2020-09-08 19:41 ` Derrick McKee
2020-09-08 13:58 ` Andrey Konovalov
2020-09-08 15:16 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 25/35] kasan: introduce CONFIG_KASAN_HW_TAGS Andrey Konovalov
2020-08-27 11:33 ` Vincenzo Frascino
2020-08-27 12:22 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 26/35] kasan, arm64: Enable TBI EL1 Andrey Konovalov
2020-08-27 10:40 ` Catalin Marinas
2020-08-27 11:05 ` Vincenzo Frascino
2020-08-27 11:13 ` Catalin Marinas
2020-08-27 11:17 ` Vincenzo Frascino
2020-08-27 12:43 ` Andrey Konovalov
2020-08-27 13:45 ` Vincenzo Frascino
2020-08-27 14:36 ` Andrey Konovalov
2020-09-08 13:18 ` Andrey Konovalov
2020-09-08 14:06 ` Catalin Marinas
2020-09-08 14:12 ` Andrey Konovalov
2020-09-08 14:41 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 27/35] kasan, arm64: align allocations for HW_TAGS Andrey Konovalov
2020-08-14 17:27 ` [PATCH 28/35] kasan: define KASAN_GRANULE_SIZE " Andrey Konovalov
2020-08-27 10:41 ` Catalin Marinas
2020-08-27 11:07 ` Vincenzo Frascino
2020-08-27 12:05 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 29/35] kasan, x86, s390: update undef CONFIG_KASAN Andrey Konovalov
2020-08-14 17:27 ` [PATCH 30/35] kasan, arm64: expand CONFIG_KASAN checks Andrey Konovalov
2020-08-14 17:27 ` [PATCH 31/35] kasan, arm64: implement HW_TAGS runtime Andrey Konovalov
2020-08-27 10:45 ` Catalin Marinas
2020-08-27 11:35 ` Vincenzo Frascino
2020-08-27 12:37 ` Andrey Konovalov
2020-08-14 17:27 ` [PATCH 32/35] kasan, arm64: print report from tag fault handler Andrey Konovalov
2020-08-27 10:48 ` Catalin Marinas
2020-08-27 12:11 ` Vincenzo Frascino
2020-08-27 12:34 ` Andrey Konovalov
2020-08-27 14:21 ` Catalin Marinas
2020-08-14 17:27 ` [PATCH 33/35] kasan, slub: reset tags when accessing metadata Andrey Konovalov
2020-08-14 17:27 ` [PATCH 34/35] kasan, arm64: enable CONFIG_KASAN_HW_TAGS Andrey Konovalov
2020-08-14 17:27 ` [PATCH 35/35] kasan: add documentation for hardware tag-based mode Andrey Konovalov
2020-08-28 11:12 ` Marco Elver
2020-08-28 12:28 ` Andrey Konovalov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).