From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D999C43461 for ; Mon, 7 Sep 2020 17:25:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 225F720658 for ; Mon, 7 Sep 2020 17:25:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="qrzGpgAI" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729768AbgIGRY6 (ORCPT ); Mon, 7 Sep 2020 13:24:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729620AbgIGNlR (ORCPT ); Mon, 7 Sep 2020 09:41:17 -0400 Received: from mail-wr1-x44a.google.com (mail-wr1-x44a.google.com [IPv6:2a00:1450:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D885C061757 for ; Mon, 7 Sep 2020 06:41:16 -0700 (PDT) Received: by mail-wr1-x44a.google.com with SMTP id f18so5708770wrv.19 for ; Mon, 07 Sep 2020 06:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5z/J5FmpUih5AunXUZcBxXMftgMmrnQQb5zqsYk89+U=; b=qrzGpgAIyNh1w1WAX6xz+D6eDKW3iGZQ1odGklPBY1aEhgeSVmHjibiYQHRsi5BhX7 pWGz513RdHCeDJcsAUIfZx/ErEkDGQP7Qh4xkOrC+juyvDNQJJuJEk3ym5W4ZN/odjpb 5eALUP8QxT/DyRhXS9wsBeQEG//vhpbeu9gkeqghAjUcgWpka7slfkhqVxqnO76Muijy eK/nm9eX2JWYZb1c5Ms682TiF/naUxEjIKpKgJElOceA1rM2RyBOGCviCKDiaxu5Hcgj KDBgU+km4pVvAncOD/zNAe7BpyvroPLAnErSVIbLQrPeBJ8PR0PyR53TNz6OZgIHFWjF oafw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5z/J5FmpUih5AunXUZcBxXMftgMmrnQQb5zqsYk89+U=; b=H0KGXYegp+hki+w1GQdFZQtMawQJqt7VILh9e3aua61+noRO21bi8AlzNId+6jrjGG UTHGSHNHrZHvn1ZSi6owmKsae97K1ZsUhZubL45dAUmV00ldYs7LD81fZnptR5kM7UJz HT1fDXVV98PtCWi5pZhllySTarOgDzkZgH15yQd+LFHi9Hsa3Dr2WlqpX7UmvWJrq8nP 7ft0hOS3bZ3cIVHzqsrt+zMtLYe9hUwyzaloZ9YFCpgg6RCcqkJS00XQmWpQjf/AJOCA OdfYDYPcDAwC73FaMi5l7NogOgIdORoMIWMRvaklfCJlhq/7i8WvPjAOnbDRmhf1azOA mvJw== X-Gm-Message-State: AOAM532RSqz0tIAPumFdme88eW4o74thEx2IS1hFywfItdCT2jLCydzE N1L3x4MdxkugBryc04lLMSd4vkbcMA== X-Google-Smtp-Source: ABdhPJxhActcecvve95wvtZfEwTJHuDIR+XsqED/73NlG4rCv/rWU9g+NLwhYgxlkuyB59M81PWVbgm4Nw== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a7b:c14f:: with SMTP id z15mr9137wmi.1.1599486074403; Mon, 07 Sep 2020 06:41:14 -0700 (PDT) Date: Mon, 7 Sep 2020 15:40:47 +0200 In-Reply-To: <20200907134055.2878499-1-elver@google.com> Message-Id: <20200907134055.2878499-3-elver@google.com> Mime-Version: 1.0 References: <20200907134055.2878499-1-elver@google.com> X-Mailer: git-send-email 2.28.0.526.ge36021eeef-goog Subject: [PATCH RFC 02/10] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, glider@google.com, akpm@linux-foundation.org, catalin.marinas@arm.com, cl@linux.com, rientjes@google.com, iamjoonsoo.kim@lge.com, mark.rutland@arm.com, penberg@kernel.org Cc: hpa@zytor.com, paulmck@kernel.org, andreyknvl@google.com, aryabinin@virtuozzo.com, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, dvyukov@google.com, edumazet@google.com, gregkh@linuxfoundation.org, mingo@redhat.com, jannh@google.com, corbet@lwn.net, keescook@chromium.org, peterz@infradead.org, cai@lca.pw, tglx@linutronix.de, will@kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..e22dc722698c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..cf09e377faf9 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* The alignment should be at least a 4K page. */ +#define KFENCE_POOL_ALIGNMENT PAGE_SIZE + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_initialize_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLBs. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (!pte || level != PG_LEVEL_4K) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..423e15ad5eb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, } #endif + if (kfence_handle_page_fault(address)) + return; + /* * 32-bit: * -- 2.28.0.526.ge36021eeef-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47571C433E2 for ; Mon, 7 Sep 2020 13:41:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D4DC1216C4 for ; Mon, 7 Sep 2020 13:41:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="qrzGpgAI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D4DC1216C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6E2366B005C; Mon, 7 Sep 2020 09:41:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 695166B005D; Mon, 7 Sep 2020 09:41:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 536046B0062; Mon, 7 Sep 2020 09:41:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id 3B2056B005C for ; Mon, 7 Sep 2020 09:41:17 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id ED9D7181AC9CC for ; Mon, 7 Sep 2020 13:41:16 +0000 (UTC) X-FDA: 77236376952.05.snail93_43148d5270cc Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id D12291826B6B4 for ; Mon, 7 Sep 2020 13:41:16 +0000 (UTC) X-HE-Tag: snail93_43148d5270cc X-Filterd-Recvd-Size: 7111 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Mon, 7 Sep 2020 13:41:16 +0000 (UTC) Received: by mail-wr1-f73.google.com with SMTP id v12so5725659wrm.9 for ; Mon, 07 Sep 2020 06:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5z/J5FmpUih5AunXUZcBxXMftgMmrnQQb5zqsYk89+U=; b=qrzGpgAIyNh1w1WAX6xz+D6eDKW3iGZQ1odGklPBY1aEhgeSVmHjibiYQHRsi5BhX7 pWGz513RdHCeDJcsAUIfZx/ErEkDGQP7Qh4xkOrC+juyvDNQJJuJEk3ym5W4ZN/odjpb 5eALUP8QxT/DyRhXS9wsBeQEG//vhpbeu9gkeqghAjUcgWpka7slfkhqVxqnO76Muijy eK/nm9eX2JWYZb1c5Ms682TiF/naUxEjIKpKgJElOceA1rM2RyBOGCviCKDiaxu5Hcgj KDBgU+km4pVvAncOD/zNAe7BpyvroPLAnErSVIbLQrPeBJ8PR0PyR53TNz6OZgIHFWjF oafw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5z/J5FmpUih5AunXUZcBxXMftgMmrnQQb5zqsYk89+U=; b=IJV+CfL+HFEmA6ijATtP/Mb6fYpPrtu1blCa846JYKVIOd9+pCioumYNaqfrU5mVv0 LkUsXT/UC/WTvOxpi1YzQAPNU5mM+RjOzNnt3SmvjVXv5qGHLXv2dyGesPGLdmGtBJma pSwvx1cV9kD7DyMtcJw6sxi0BYwVxlXZm/IMdyPaQlHLj1sJ9+fXbdm+23BeTNePIE7p Te06RO++DT5vj7wCQBNlkPhTmXY6+KXYYmr+1gzOokU/rgALJdcOrGGWXapTp+f0swZq DkZRaMOVRyfF2QCbdygvSdH3SPTSERyS26nUvJl3fEt9uV3VAiifFRslr5EN4F9zzrpI FFvg== X-Gm-Message-State: AOAM533QvPEnYuh1Z1YQafByHWwGisZm8r1mN1ObINwHs/Z3ns7Xc/zm dt+UR5CTMN21l3UUSmx5pq78qwWwlQ== X-Google-Smtp-Source: ABdhPJxhActcecvve95wvtZfEwTJHuDIR+XsqED/73NlG4rCv/rWU9g+NLwhYgxlkuyB59M81PWVbgm4Nw== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a7b:c14f:: with SMTP id z15mr9137wmi.1.1599486074403; Mon, 07 Sep 2020 06:41:14 -0700 (PDT) Date: Mon, 7 Sep 2020 15:40:47 +0200 In-Reply-To: <20200907134055.2878499-1-elver@google.com> Message-Id: <20200907134055.2878499-3-elver@google.com> Mime-Version: 1.0 References: <20200907134055.2878499-1-elver@google.com> X-Mailer: git-send-email 2.28.0.526.ge36021eeef-goog Subject: [PATCH RFC 02/10] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, glider@google.com, akpm@linux-foundation.org, catalin.marinas@arm.com, cl@linux.com, rientjes@google.com, iamjoonsoo.kim@lge.com, mark.rutland@arm.com, penberg@kernel.org Cc: hpa@zytor.com, paulmck@kernel.org, andreyknvl@google.com, aryabinin@virtuozzo.com, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, dvyukov@google.com, edumazet@google.com, gregkh@linuxfoundation.org, mingo@redhat.com, jannh@google.com, corbet@lwn.net, keescook@chromium.org, peterz@infradead.org, cai@lca.pw, tglx@linutronix.de, will@kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: D12291826B6B4 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..e22dc722698c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..cf09e377faf9 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* The alignment should be at least a 4K page. */ +#define KFENCE_POOL_ALIGNMENT PAGE_SIZE + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_initialize_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLBs. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (!pte || level != PG_LEVEL_4K) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..423e15ad5eb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, } #endif + if (kfence_handle_page_fault(address)) + return; + /* * 32-bit: * -- 2.28.0.526.ge36021eeef-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 396A5C433E2 for ; Mon, 7 Sep 2020 14:46:10 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D510121532 for ; Mon, 7 Sep 2020 14:46:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="UatgwTX/"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="qrzGpgAI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D510121532 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5YsTTNGZKOCbOR/i+T/HiorJvJtKCGMy3xU9+YZe2TQ=; b=UatgwTX/HZ4xaOJ7R9xek2lDY uiuK6jwiiXLf9jyEkgM+LKil7JkEzxp5FKRLquuFo5lpIp0KU+EPeJG6lGhcZJ/cvAo07uC/UFFJM UBsh5laF6DS6sgmfpeRyvjs1L205n7iXW/Dag5XHmgRy5x9e2SxRtREWrjT6m8otxZ4wjoqg7pLVF xJT2xYUsezgzvvcDkI6g26w65RmYW00ySzbt8Xw4eMkTEeKJVB93cLDj5lNOLy8atPP9mDQxZ6pON Xq3Irn0Gcaa0UNDutdUkElGI0WpDBSbqXB8ebw7Zo/W2XrPHEyLMkYvs+pDWVd7tggZ86caiK+jAC TAYqE/7Vw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kFIOU-0000Iz-G6; Mon, 07 Sep 2020 14:45:18 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kFHOX-0003Wy-Qg for linux-arm-kernel@lists.infradead.org; Mon, 07 Sep 2020 13:41:23 +0000 Received: by mail-wr1-x44a.google.com with SMTP id 3so5751146wrm.4 for ; Mon, 07 Sep 2020 06:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5z/J5FmpUih5AunXUZcBxXMftgMmrnQQb5zqsYk89+U=; b=qrzGpgAIyNh1w1WAX6xz+D6eDKW3iGZQ1odGklPBY1aEhgeSVmHjibiYQHRsi5BhX7 pWGz513RdHCeDJcsAUIfZx/ErEkDGQP7Qh4xkOrC+juyvDNQJJuJEk3ym5W4ZN/odjpb 5eALUP8QxT/DyRhXS9wsBeQEG//vhpbeu9gkeqghAjUcgWpka7slfkhqVxqnO76Muijy eK/nm9eX2JWYZb1c5Ms682TiF/naUxEjIKpKgJElOceA1rM2RyBOGCviCKDiaxu5Hcgj KDBgU+km4pVvAncOD/zNAe7BpyvroPLAnErSVIbLQrPeBJ8PR0PyR53TNz6OZgIHFWjF oafw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5z/J5FmpUih5AunXUZcBxXMftgMmrnQQb5zqsYk89+U=; b=SeBSK9/tv4XnGZ90jzFEtrT3txiB9S5jUlLKCL98iedcZT9DXLrbkeFbOOye3ILoML Oe7hvii4h9S1RWHL4CA3V+SM2hE2S89E/xHVyIUqiwNkfKks3tGTwcH3PizaFSS+/P0m gR+roWBmzRzAh8mT6dDUTAsxmAgvzy9Wpyu6/qWPKjtfQuJ1qoC1kxNhAFRsQe2fYElq LaoQcdNxvrFsqMpz/v8PJnKZ581QHvx8nhDGoaLaaYWXouKBM6qWMOrFKRce6Gijzpp1 1b1KbHFdcNDvE5D+N0DuBJ1+dywd2cVcI2jzA11+1v7rZE0l03zqcE0LIKBnX89kom// Y8sw== X-Gm-Message-State: AOAM530tL6tuutOx2QrOtYDRhAnVSShO2bzJ1wL77ZVptcoNpq4rfWjl QCiUx5d8ZGQeym82BXPHzv4DFds6mg== X-Google-Smtp-Source: ABdhPJxhActcecvve95wvtZfEwTJHuDIR+XsqED/73NlG4rCv/rWU9g+NLwhYgxlkuyB59M81PWVbgm4Nw== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a7b:c14f:: with SMTP id z15mr9137wmi.1.1599486074403; Mon, 07 Sep 2020 06:41:14 -0700 (PDT) Date: Mon, 7 Sep 2020 15:40:47 +0200 In-Reply-To: <20200907134055.2878499-1-elver@google.com> Message-Id: <20200907134055.2878499-3-elver@google.com> Mime-Version: 1.0 References: <20200907134055.2878499-1-elver@google.com> X-Mailer: git-send-email 2.28.0.526.ge36021eeef-goog Subject: [PATCH RFC 02/10] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, glider@google.com, akpm@linux-foundation.org, catalin.marinas@arm.com, cl@linux.com, rientjes@google.com, iamjoonsoo.kim@lge.com, mark.rutland@arm.com, penberg@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200907_094117_941001_679F90CE X-CRM114-Status: GOOD ( 19.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-doc@vger.kernel.org, peterz@infradead.org, dave.hansen@linux.intel.com, linux-mm@kvack.org, edumazet@google.com, hpa@zytor.com, will@kernel.org, corbet@lwn.net, x86@kernel.org, kasan-dev@googlegroups.com, mingo@redhat.com, linux-arm-kernel@lists.infradead.org, aryabinin@virtuozzo.com, keescook@chromium.org, paulmck@kernel.org, jannh@google.com, andreyknvl@google.com, cai@lca.pw, luto@kernel.org, tglx@linutronix.de, dvyukov@google.com, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, bp@alien8.de Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..e22dc722698c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..cf09e377faf9 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* The alignment should be at least a 4K page. */ +#define KFENCE_POOL_ALIGNMENT PAGE_SIZE + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_initialize_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLBs. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (!pte || level != PG_LEVEL_4K) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..423e15ad5eb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, } #endif + if (kfence_handle_page_fault(address)) + return; + /* * 32-bit: * -- 2.28.0.526.ge36021eeef-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel