From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15D2AC33CB6 for ; Fri, 17 Jan 2020 22:52:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DF07421D56 for ; Fri, 17 Jan 2020 22:52:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gVYEl7P1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730172AbgAQWwT (ORCPT ); Fri, 17 Jan 2020 17:52:19 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:54422 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729015AbgAQWwS (ORCPT ); Fri, 17 Jan 2020 17:52:18 -0500 Received: by mail-wm1-f67.google.com with SMTP id b19so8878370wmj.4; Fri, 17 Jan 2020 14:52:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LFs5m8wbYLJe7WnSk64bj60RyQWBR7/vDgMqYkBMoLM=; b=gVYEl7P1+QZZg4+jpdIOZvPowqFSmM1yGloq+VCyoENndNYETconbDhxaqbjLzzgwM SgMObFb70Z/EAY1IKRMVwkkopYHl/mE66txY4Atyoc3Hme2Z6gH6yHfge6pD7Qa/e0ou ObZ0mislha4cLNwgrXb80sgMyBppO5kNckrOAl/VnYxw5h6Nl6oLSeZtVBdPGtPqSKge JurXZAZYtjnp1p+9coDSinlwRQTcqxUFEQNog7NlPs0io+geVXvQhs2KBHl1LaaDmTcy 5D1ZA/ijU5QvyAXPltZS5zjAqcl9hWJATv9xVUNS5I0sLUoVo9LxBfiTsCnOTfZL5XoR TutA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LFs5m8wbYLJe7WnSk64bj60RyQWBR7/vDgMqYkBMoLM=; b=qAWRTp2h/1HztH8x5YMVIdvZG+yaFX9zBZoknXuRtSjlFMPKspbKxw/6FokSf8E1nd F1NgW8G5EpK9L6PvIxmG6XvUgceEJydVlUCXjiEEg/1/eky88QgZEie88qZVxzNs9rsJ Nx+EXfnaQ1lrvD0zPZgTxmKlWdixhnwMzZJtFrcFgzlOftFlTQD4MpRhBg8YNN9p57Df 02gbP/YjPOEnGI5UaIgx9G8HYaiJ2JUvY1CFLQu1+ELMSWuk/F4xrm0ijz5VMi4t3c7m FtZVtkVcuEM+pUnMOWLSpSux2bNVxHOihEdOSDgBTMhIHuyOhkRnryu4gaaVYFN+aBKY Tu1w== X-Gm-Message-State: APjAAAXC5a9m3UzTtcqqOy4ws7wDi0NwHwFMvbfHSG8zibX/KpW2Q/rc 7SpE7DFbUTTZXVxFpus2geE= X-Google-Smtp-Source: APXvYqxNuXck2Z6eF96n39rU/M9fvOkymwJ2uIEW5UGYE8OF849gIM7NBTPCQOjJjYfalkWxUuqY+Q== X-Received: by 2002:a1c:9c4c:: with SMTP id f73mr6535941wme.125.1579301534925; Fri, 17 Jan 2020 14:52:14 -0800 (PST) Received: from fainelli-desktop.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id l3sm32829387wrt.29.2020.01.17.14.52.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Jan 2020 14:52:14 -0800 (PST) From: Florian Fainelli To: linux-arm-kernel@lists.infradead.org Cc: Andrey Ryabinin , Abbott Liu , Florian Fainelli , bcm-kernel-feedback-list@broadcom.com, glider@google.com, dvyukov@google.com, corbet@lwn.net, linux@armlinux.org.uk, christoffer.dall@arm.com, marc.zyngier@arm.com, arnd@arndb.de, nico@fluxnic.net, vladimir.murzin@arm.com, keescook@chromium.org, jinb.park7@gmail.com, alexandre.belloni@bootlin.com, ard.biesheuvel@linaro.org, daniel.lezcano@linaro.org, pombredanne@nexb.com, rob@landley.net, gregkh@linuxfoundation.org, akpm@linux-foundation.org, mark.rutland@arm.com, catalin.marinas@arm.com, yamada.masahiro@socionext.com, tglx@linutronix.de, thgarnie@google.com, dhowells@redhat.com, geert@linux-m68k.org, andre.przywara@arm.com, julien.thierry@arm.com, drjones@redhat.com, philip@cog.systems, mhocko@suse.com, kirill.shutemov@linux.intel.com, kasan-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@lists.cs.columbia.edu, ryabinin.a.a@gmail.com Subject: [PATCH v7 6/7] ARM: Initialize the mapping of KASan shadow memory Date: Fri, 17 Jan 2020 14:48:38 -0800 Message-Id: <20200117224839.23531-7-f.fainelli@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200117224839.23531-1-f.fainelli@gmail.com> References: <20200117224839.23531-1-f.fainelli@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Andrey Ryabinin This patch initializes KASan shadow region's page table and memory. There are two stage for KASan initializing: 1. At early boot stage the whole shadow region is mapped to just one physical page (kasan_zero_page). It is finished by the function kasan_early_init which is called by __mmap_switched(arch/arm/kernel/ head-common.S) ---Andrey Ryabinin 2. After the calling of paging_init, we use kasan_zero_page as zero shadow for some memory that KASan does not need to track, and we allocate a new shadow space for the other memory that KASan need to track. These issues are finished by the function kasan_init which is call by setup_arch. ---Andrey Ryabinin 3. Add support ARM LPAE If LPAE is enabled, KASan shadow region's mapping table need be copied in the pgd_alloc() function. ---Abbott Liu 4. Change kasan_pte_populate,kasan_pmd_populate,kasan_pud_populate, kasan_pgd_populate from .meminit.text section to .init.text section. ---Reported by: Florian Fainelli ---Signed off by: Abbott Liu Cc: Andrey Ryabinin Co-Developed-by: Abbott Liu Tested-by: Linus Walleij Reported-by: Russell King - ARM Linux Reported-by: Florian Fainelli Signed-off-by: Abbott Liu Signed-off-by: Florian Fainelli --- arch/arm/include/asm/kasan.h | 35 ++++ arch/arm/include/asm/pgalloc.h | 9 +- arch/arm/include/asm/thread_info.h | 4 + arch/arm/kernel/head-common.S | 3 + arch/arm/kernel/setup.c | 2 + arch/arm/mm/Makefile | 3 + arch/arm/mm/kasan_init.c | 302 +++++++++++++++++++++++++++++ arch/arm/mm/pgd.c | 14 ++ 8 files changed, 370 insertions(+), 2 deletions(-) create mode 100644 arch/arm/include/asm/kasan.h create mode 100644 arch/arm/mm/kasan_init.c diff --git a/arch/arm/include/asm/kasan.h b/arch/arm/include/asm/kasan.h new file mode 100644 index 000000000000..1801f4d30993 --- /dev/null +++ b/arch/arm/include/asm/kasan.h @@ -0,0 +1,35 @@ +/* + * arch/arm/include/asm/kasan.h + * + * Copyright (c) 2015 Samsung Electronics Co., Ltd. + * Author: Andrey Ryabinin + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + */ + +#ifndef __ASM_KASAN_H +#define __ASM_KASAN_H + +#ifdef CONFIG_KASAN + +#include + +#define KASAN_SHADOW_SCALE_SHIFT 3 + +/* + * Compiler uses shadow offset assuming that addresses start + * from 0. Kernel addresses don't start from 0, so shadow + * for kernel really starts from 'compiler's shadow offset' + + * ('kernel address space start' >> KASAN_SHADOW_SCALE_SHIFT) + */ + +extern void kasan_init(void); + +#else +static inline void kasan_init(void) { } +#endif + +#endif diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h index 069da393110c..d969f8058b26 100644 --- a/arch/arm/include/asm/pgalloc.h +++ b/arch/arm/include/asm/pgalloc.h @@ -21,6 +21,7 @@ #define _PAGE_KERNEL_TABLE (PMD_TYPE_TABLE | PMD_BIT4 | PMD_DOMAIN(DOMAIN_KERNEL)) #ifdef CONFIG_ARM_LPAE +#define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { @@ -39,14 +40,18 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) } #else /* !CONFIG_ARM_LPAE */ +#define PGD_SIZE (PAGE_SIZE << 2) /* * Since we have only two-level page tables, these are trivial */ #define pmd_alloc_one(mm,addr) ({ BUG(); ((pmd_t *)2); }) #define pmd_free(mm, pmd) do { } while (0) -#define pud_populate(mm,pmd,pte) BUG() - +#ifndef CONFIG_KASAN +#define pud_populate(mm, pmd, pte) BUG() +#else +#define pud_populate(mm, pmd, pte) do { } while (0) +#endif #endif /* CONFIG_ARM_LPAE */ extern pgd_t *pgd_alloc(struct mm_struct *mm); diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 0d0d5178e2c3..2c940dcc953b 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -13,7 +13,11 @@ #include #include +#ifdef CONFIG_KASAN +#define THREAD_SIZE_ORDER 2 +#else #define THREAD_SIZE_ORDER 1 +#endif #define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER) #define THREAD_START_SP (THREAD_SIZE - 8) diff --git a/arch/arm/kernel/head-common.S b/arch/arm/kernel/head-common.S index 6840c7c60a85..89c80154b9ef 100644 --- a/arch/arm/kernel/head-common.S +++ b/arch/arm/kernel/head-common.S @@ -111,6 +111,9 @@ __mmap_switched: str r8, [r2] @ Save atags pointer cmp r3, #0 strne r10, [r3] @ Save control register values +#ifdef CONFIG_KASAN + bl kasan_early_init +#endif mov lr, #0 b start_kernel ENDPROC(__mmap_switched) diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index d0a464e317ea..b120df6325dc 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -58,6 +58,7 @@ #include #include #include +#include #include "atags.h" @@ -1130,6 +1131,7 @@ void __init setup_arch(char **cmdline_p) early_ioremap_reset(); paging_init(mdesc); + kasan_init(); request_standard_resources(mdesc); if (mdesc->restart) diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile index 432302911d6e..1c937135c9c4 100644 --- a/arch/arm/mm/Makefile +++ b/arch/arm/mm/Makefile @@ -112,3 +112,6 @@ obj-$(CONFIG_CACHE_L2X0_PMU) += cache-l2x0-pmu.o obj-$(CONFIG_CACHE_XSC3L2) += cache-xsc3l2.o obj-$(CONFIG_CACHE_TAUROS2) += cache-tauros2.o obj-$(CONFIG_CACHE_UNIPHIER) += cache-uniphier.o + +KASAN_SANITIZE_kasan_init.o := n +obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm/mm/kasan_init.c b/arch/arm/mm/kasan_init.c new file mode 100644 index 000000000000..7597efb36cb0 --- /dev/null +++ b/arch/arm/mm/kasan_init.c @@ -0,0 +1,302 @@ +/* + * This file contains kasan initialization code for ARM. + * + * Copyright (c) 2018 Samsung Electronics Co., Ltd. + * Author: Andrey Ryabinin + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + */ + +#define pr_fmt(fmt) "kasan: " fmt +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "mm.h" + +static pgd_t tmp_pgd_table[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE); + +pmd_t tmp_pmd_table[PTRS_PER_PMD] __page_aligned_bss; + +static __init void *kasan_alloc_block(size_t size, int node) +{ + return memblock_alloc_try_nid(size, size, __pa(MAX_DMA_ADDRESS), + MEMBLOCK_ALLOC_KASAN, node); +} + +static void __init kasan_early_pmd_populate(unsigned long start, + unsigned long end, pud_t *pud) +{ + unsigned long addr; + unsigned long next; + pmd_t *pmd; + + pmd = pmd_offset(pud, start); + for (addr = start; addr < end;) { + pmd_populate_kernel(&init_mm, pmd, kasan_early_shadow_pte); + next = pmd_addr_end(addr, end); + addr = next; + flush_pmd_entry(pmd); + pmd++; + } +} + +static void __init kasan_early_pud_populate(unsigned long start, + unsigned long end, pgd_t *pgd) +{ + unsigned long addr; + unsigned long next; + pud_t *pud; + + pud = pud_offset(pgd, start); + for (addr = start; addr < end;) { + next = pud_addr_end(addr, end); + kasan_early_pmd_populate(addr, next, pud); + addr = next; + pud++; + } +} + +void __init kasan_map_early_shadow(pgd_t *pgdp) +{ + int i; + unsigned long start = KASAN_SHADOW_START; + unsigned long end = KASAN_SHADOW_END; + unsigned long addr; + unsigned long next; + pgd_t *pgd; + + for (i = 0; i < PTRS_PER_PTE; i++) + set_pte_at(&init_mm, KASAN_SHADOW_START + i*PAGE_SIZE, + &kasan_early_shadow_pte[i], pfn_pte( + virt_to_pfn(kasan_early_shadow_page), + __pgprot(_L_PTE_DEFAULT | L_PTE_DIRTY + | L_PTE_XN))); + + pgd = pgd_offset_k(start); + for (addr = start; addr < end;) { + next = pgd_addr_end(addr, end); + kasan_early_pud_populate(addr, next, pgd); + addr = next; + pgd++; + } +} + +extern struct proc_info_list *lookup_processor_type(unsigned int); + +void __init kasan_early_init(void) +{ + struct proc_info_list *list; + + /* + * locate processor in the list of supported processor + * types. The linker builds this table for us from the + * entries in arch/arm/mm/proc-*.S + */ + list = lookup_processor_type(read_cpuid_id()); + if (list) { +#ifdef MULTI_CPU + processor = *list->proc; +#endif + } + + BUILD_BUG_ON((KASAN_SHADOW_END - (1UL << 29)) != KASAN_SHADOW_OFFSET); + kasan_map_early_shadow(swapper_pg_dir); +} + +static void __init clear_pgds(unsigned long start, + unsigned long end) +{ + for (; start && start < end; start += PMD_SIZE) + pmd_clear(pmd_off_k(start)); +} + +pte_t * __init kasan_pte_populate(pmd_t *pmd, unsigned long addr, int node) +{ + pte_t *pte = pte_offset_kernel(pmd, addr); + + if (pte_none(*pte)) { + pte_t entry; + void *p = kasan_alloc_block(PAGE_SIZE, node); + + if (!p) + return NULL; + entry = pfn_pte(virt_to_pfn(p), + __pgprot(pgprot_val(PAGE_KERNEL))); + set_pte_at(&init_mm, addr, pte, entry); + } + return pte; +} + +pmd_t * __init kasan_pmd_populate(pud_t *pud, unsigned long addr, int node) +{ + pmd_t *pmd = pmd_offset(pud, addr); + + if (pmd_none(*pmd)) { + void *p = kasan_alloc_block(PAGE_SIZE, node); + + if (!p) + return NULL; + pmd_populate_kernel(&init_mm, pmd, p); + } + return pmd; +} + +pud_t * __init kasan_pud_populate(pgd_t *pgd, unsigned long addr, int node) +{ + pud_t *pud = pud_offset(pgd, addr); + + if (pud_none(*pud)) { + void *p = kasan_alloc_block(PAGE_SIZE, node); + + if (!p) + return NULL; + pr_err("populating pud addr %lx\n", addr); + pud_populate(&init_mm, pud, p); + } + return pud; +} + +pgd_t * __init kasan_pgd_populate(unsigned long addr, int node) +{ + pgd_t *pgd = pgd_offset_k(addr); + + if (pgd_none(*pgd)) { + void *p = kasan_alloc_block(PAGE_SIZE, node); + + if (!p) + return NULL; + pgd_populate(&init_mm, pgd, p); + } + return pgd; +} + +static int __init create_mapping(unsigned long start, unsigned long end, + int node) +{ + unsigned long addr = start; + pgd_t *pgd; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pr_info("populating shadow for %lx, %lx\n", start, end); + + for (; addr < end; addr += PAGE_SIZE) { + pgd = kasan_pgd_populate(addr, node); + if (!pgd) + return -ENOMEM; + + pud = kasan_pud_populate(pgd, addr, node); + if (!pud) + return -ENOMEM; + + pmd = kasan_pmd_populate(pud, addr, node); + if (!pmd) + return -ENOMEM; + + pte = kasan_pte_populate(pmd, addr, node); + if (!pte) + return -ENOMEM; + } + return 0; +} + + +void __init kasan_init(void) +{ + struct memblock_region *reg; + u64 orig_ttbr0; + int i; + + /* + * We are going to perform proper setup of shadow memory. + * At first we should unmap early shadow (clear_pgds() call bellow). + * However, instrumented code couldn't execute without shadow memory. + * tmp_pgd_table and tmp_pmd_table used to keep early shadow mapped + * until full shadow setup will be finished. + */ + orig_ttbr0 = get_ttbr0(); + +#ifdef CONFIG_ARM_LPAE + memcpy(tmp_pmd_table, + pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_START)), + sizeof(tmp_pmd_table)); + memcpy(tmp_pgd_table, swapper_pg_dir, sizeof(tmp_pgd_table)); + set_pgd(&tmp_pgd_table[pgd_index(KASAN_SHADOW_START)], + __pgd(__pa(tmp_pmd_table) | PMD_TYPE_TABLE | L_PGD_SWAPPER)); + set_ttbr0(__pa(tmp_pgd_table)); +#else + memcpy(tmp_pgd_table, swapper_pg_dir, sizeof(tmp_pgd_table)); + set_ttbr0((u64)__pa(tmp_pgd_table)); +#endif + flush_cache_all(); + local_flush_bp_all(); + local_flush_tlb_all(); + + clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + + kasan_populate_early_shadow(kasan_mem_to_shadow((void *)VMALLOC_START), + kasan_mem_to_shadow((void *)-1UL) + 1); + + for_each_memblock(memory, reg) { + void *start = __va(reg->base); + void *end = __va(reg->base + reg->size); + + if (reg->base + reg->size > arm_lowmem_limit) + end = __va(arm_lowmem_limit); + if (start >= end) + break; + + create_mapping((unsigned long)kasan_mem_to_shadow(start), + (unsigned long)kasan_mem_to_shadow(end), + NUMA_NO_NODE); + } + + /*1.the module's global variable is in MODULES_VADDR ~ MODULES_END, + * so we need mapping. + *2.PKMAP_BASE ~ PKMAP_BASE+PMD_SIZE's shadow and MODULES_VADDR + * ~ MODULES_END's shadow is in the same PMD_SIZE, so we cant + * use kasan_populate_zero_shadow. + */ + create_mapping( + (unsigned long)kasan_mem_to_shadow((void *)MODULES_VADDR), + + (unsigned long)kasan_mem_to_shadow((void *)(PKMAP_BASE + + PMD_SIZE)), + NUMA_NO_NODE); + + /* + * KAsan may reuse the contents of kasan_early_shadow_pte directly, so + * we should make sure that it maps the zero page read-only. + */ + for (i = 0; i < PTRS_PER_PTE; i++) + set_pte_at(&init_mm, KASAN_SHADOW_START + i*PAGE_SIZE, + &kasan_early_shadow_pte[i], + pfn_pte(virt_to_pfn(kasan_early_shadow_page), + __pgprot(pgprot_val(PAGE_KERNEL) + | L_PTE_RDONLY))); + memset(kasan_early_shadow_page, 0, PAGE_SIZE); + set_ttbr0(orig_ttbr0); + flush_cache_all(); + local_flush_bp_all(); + local_flush_tlb_all(); + pr_info("Kernel address sanitizer initialized\n"); + init_task.kasan_depth = 0; +} diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index 478bd2c6aa50..92a408262df2 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -61,6 +61,20 @@ pgd_t *pgd_alloc(struct mm_struct *mm) new_pmd = pmd_alloc(mm, new_pud, 0); if (!new_pmd) goto no_pmd; +#ifdef CONFIG_KASAN + /* + *Copy PMD table for KASAN shadow mappings. + */ + init_pgd = pgd_offset_k(TASK_SIZE); + init_pud = pud_offset(init_pgd, TASK_SIZE); + init_pmd = pmd_offset(init_pud, TASK_SIZE); + new_pmd = pmd_offset(new_pud, TASK_SIZE); + memcpy(new_pmd, init_pmd, + (pmd_index(MODULES_VADDR)-pmd_index(TASK_SIZE)) + * sizeof(pmd_t)); + clean_dcache_area(new_pmd, PTRS_PER_PMD*sizeof(pmd_t)); +#endif + #endif if (!vectors_high()) { -- 2.17.1