From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EC47C433E2 for ; Tue, 15 Sep 2020 23:25:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 632832068D for ; Tue, 15 Sep 2020 23:25:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Ic5gm/5J" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727305AbgIOXZ2 (ORCPT ); Tue, 15 Sep 2020 19:25:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727188AbgIOOiA (ORCPT ); Tue, 15 Sep 2020 10:38:00 -0400 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A765C06178B for ; Tue, 15 Sep 2020 06:21:10 -0700 (PDT) Received: by mail-qk1-x749.google.com with SMTP id r128so2842320qkc.9 for ; Tue, 15 Sep 2020 06:21:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=odW0Nb/rci20hhf4qlLrsuqbfMVKw2n0jytc4nSfGLw=; b=Ic5gm/5J7H57rrbgi2KUAbxYzDNz8YdSq5ZJ/v6u4Ac5dg8OrDRQ+Pt7M915w+yRSh pxiTvbk/H4XBY/WQD2Uq5EFNOh0nrFP9uzxLfzmg1rs+AopLvLhBPW/q3vEjThD99dL7 qrRJWuK823kI8mBCIvwjC68mmtOOJyAHLlvbSIybZDYGzy/xv8V3w6yjOsEBlCJM4KgJ vQRXFSOO1Zp2pI+L8K+dIx9YRoQbWq1vHeX5YUarWgO4WympBLwHXxV1p92l2ErDcbiS VWfBhw/z56liN0kg89Wfm6uq9cuGDYo3z5FUkd0Z3Noo9NtkiI8ggEYuytsJEvEGmROu 3PxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=odW0Nb/rci20hhf4qlLrsuqbfMVKw2n0jytc4nSfGLw=; b=Ce6rBVFLUjsPkFQVPzvDVyyXcIn52BNdjNmEvJwHWH2s/m8E/36697/RL+ehoC8FiQ WYmBaYiuiNo4hIIio/kHDqW3k/35xNfc4uLl3d3N4oEJzy7I5oGxUPhhwJdLD/mPBmiS UdSSpXxAyhHRuZUifkIIxT4IE2dTzXCC/PKFGohu31XuIJ9i+YH3D9AaJ8etODHO56uG llkC2ECAxTVT7UPCsrEP5OtBOaEoy1aD1yBiBQEJ2d+/V6A5WQZB6CGtNfz5HicLvv6S F3nYu6VEXqqyaL7pG5NisQLTLqHcNTUfjBNgZ+jRFq8Gg6M5ErF9LFS1ytQz4gjKkC9/ 3CXg== X-Gm-Message-State: AOAM533869qCXk3Kng35+eRc390dE/g4eBTQ9JlOInqv+HuiIiShfDro o9rZ7vb09ZM6/qdnUTPw6tCdStK11g== X-Google-Smtp-Source: ABdhPJwGw5MfQEtCAAY9VjPztZBNrmyYKXs2SVdQU3S5qk954oiAUcTvB0kWyGpJ9tJe35EZN6Q4i0DPPg== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a0c:b902:: with SMTP id u2mr17508718qvf.7.1600176063190; Tue, 15 Sep 2020 06:21:03 -0700 (PDT) Date: Tue, 15 Sep 2020 15:20:38 +0200 In-Reply-To: <20200915132046.3332537-1-elver@google.com> Message-Id: <20200915132046.3332537-3-elver@google.com> Mime-Version: 1.0 References: <20200915132046.3332537-1-elver@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [PATCH v2 02/10] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, akpm@linux-foundation.org, glider@google.com Cc: hpa@zytor.com, paulmck@kernel.org, andreyknvl@google.com, aryabinin@virtuozzo.com, luto@kernel.org, bp@alien8.de, catalin.marinas@arm.com, cl@linux.com, dave.hansen@linux.intel.com, rientjes@google.com, dvyukov@google.com, edumazet@google.com, gregkh@linuxfoundation.org, mingo@redhat.com, jannh@google.com, Jonathan.Cameron@huawei.com, corbet@lwn.net, iamjoonsoo.kim@lge.com, keescook@chromium.org, mark.rutland@arm.com, penberg@kernel.org, peterz@infradead.org, cai@lca.pw, tglx@linutronix.de, vbabka@suse.cz, will@kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..e22dc722698c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..cf09e377faf9 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* The alignment should be at least a 4K page. */ +#define KFENCE_POOL_ALIGNMENT PAGE_SIZE + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_initialize_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLBs. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (!pte || level != PG_LEVEL_4K) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..423e15ad5eb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, } #endif + if (kfence_handle_page_fault(address)) + return; + /* * 32-bit: * -- 2.28.0.618.gf4bc123cb7-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ADE7C433E2 for ; Tue, 15 Sep 2020 13:21:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BDDA721D24 for ; Tue, 15 Sep 2020 13:21:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Ic5gm/5J" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BDDA721D24 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 914DF900056; Tue, 15 Sep 2020 09:21:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C25D90004C; Tue, 15 Sep 2020 09:21:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B137900056; Tue, 15 Sep 2020 09:21:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id 64E8290004C for ; Tue, 15 Sep 2020 09:21:05 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CB02C363B for ; Tue, 15 Sep 2020 13:21:04 +0000 (UTC) X-FDA: 77265356448.22.trade73_111459027111 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id A93BA18038E67 for ; Tue, 15 Sep 2020 13:21:04 +0000 (UTC) X-HE-Tag: trade73_111459027111 X-Filterd-Recvd-Size: 7158 Received: from mail-qv1-f74.google.com (mail-qv1-f74.google.com [209.85.219.74]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Tue, 15 Sep 2020 13:21:04 +0000 (UTC) Received: by mail-qv1-f74.google.com with SMTP id y2so2155033qvs.14 for ; Tue, 15 Sep 2020 06:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=odW0Nb/rci20hhf4qlLrsuqbfMVKw2n0jytc4nSfGLw=; b=Ic5gm/5J7H57rrbgi2KUAbxYzDNz8YdSq5ZJ/v6u4Ac5dg8OrDRQ+Pt7M915w+yRSh pxiTvbk/H4XBY/WQD2Uq5EFNOh0nrFP9uzxLfzmg1rs+AopLvLhBPW/q3vEjThD99dL7 qrRJWuK823kI8mBCIvwjC68mmtOOJyAHLlvbSIybZDYGzy/xv8V3w6yjOsEBlCJM4KgJ vQRXFSOO1Zp2pI+L8K+dIx9YRoQbWq1vHeX5YUarWgO4WympBLwHXxV1p92l2ErDcbiS VWfBhw/z56liN0kg89Wfm6uq9cuGDYo3z5FUkd0Z3Noo9NtkiI8ggEYuytsJEvEGmROu 3PxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=odW0Nb/rci20hhf4qlLrsuqbfMVKw2n0jytc4nSfGLw=; b=pGGVS2QGdHtkWdyvuvT4APzIw8nhObXRiF/4RpwA1B+G1OLcazr0kGSPb6RmZsLSld 1q5lVuEC1pv2uuAJ/KMQFhQsJMTUbZUNWOSzwtcBVo+L6wuLOBgfeCTEAXFTQZvhIfJ+ 3d8qiXoz/TYfa7n8J2CwAn89s2woLYMAgl0u0KrHQ2NkmkSuJMWiGcf7XetWoQgbzdX4 eMsqpD7CNA1BIHYQEFdVWtPGZ0cTRAghu2Na31XoZOVKdQ+2rDEpWUqFUA6+//cZwyQn /iS3wryn8yoDnxmePqV9jFrOP7SuPIQgWYAluwiTjz1TUDsgXmuaO4DSxGMyQX99CGyo C84Q== X-Gm-Message-State: AOAM53194JjCzb6DLCUfrEXCOS48qSiea3R19RrgNC/qmzPxIUDWw0MR hUvUp/DWtfo2ojzOL3Hwr9dzt17Rvw== X-Google-Smtp-Source: ABdhPJwGw5MfQEtCAAY9VjPztZBNrmyYKXs2SVdQU3S5qk954oiAUcTvB0kWyGpJ9tJe35EZN6Q4i0DPPg== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a0c:b902:: with SMTP id u2mr17508718qvf.7.1600176063190; Tue, 15 Sep 2020 06:21:03 -0700 (PDT) Date: Tue, 15 Sep 2020 15:20:38 +0200 In-Reply-To: <20200915132046.3332537-1-elver@google.com> Message-Id: <20200915132046.3332537-3-elver@google.com> Mime-Version: 1.0 References: <20200915132046.3332537-1-elver@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [PATCH v2 02/10] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, akpm@linux-foundation.org, glider@google.com Cc: hpa@zytor.com, paulmck@kernel.org, andreyknvl@google.com, aryabinin@virtuozzo.com, luto@kernel.org, bp@alien8.de, catalin.marinas@arm.com, cl@linux.com, dave.hansen@linux.intel.com, rientjes@google.com, dvyukov@google.com, edumazet@google.com, gregkh@linuxfoundation.org, mingo@redhat.com, jannh@google.com, Jonathan.Cameron@huawei.com, corbet@lwn.net, iamjoonsoo.kim@lge.com, keescook@chromium.org, mark.rutland@arm.com, penberg@kernel.org, peterz@infradead.org, cai@lca.pw, tglx@linutronix.de, vbabka@suse.cz, will@kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: A93BA18038E67 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..e22dc722698c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..cf09e377faf9 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* The alignment should be at least a 4K page. */ +#define KFENCE_POOL_ALIGNMENT PAGE_SIZE + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_initialize_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLBs. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (!pte || level != PG_LEVEL_4K) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..423e15ad5eb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, } #endif + if (kfence_handle_page_fault(address)) + return; + /* * 32-bit: * -- 2.28.0.618.gf4bc123cb7-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78433C43461 for ; Tue, 15 Sep 2020 13:23:30 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0F8A820684 for ; Tue, 15 Sep 2020 13:23:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="UpspHGJK"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Ic5gm/5J" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0F8A820684 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ehiFdzld9nafZr/SoY9LRsx2eJFvPSWQDSpBrH3/FP8=; b=UpspHGJKUPCTRtlW2Gl/8gc1p zOIRiwFiKb7NymkO/1wclTrzEKFoa8aTa2GuWELON/v8MLjeTn535iRtkrM1B7/InMUzYjCHLjBdJ +nEAA7AMAgZhtbKXs3Vsz4sRvuTk0SHxOtPJrx+ieL+qWlvDMnU2pWxmkttGJFplaXme7AqS9pAsa +Lq1HbHUZQL4S2rRkl2WbvtVjhx7eAiqEUC4wrSnvM50DyQIdK1X/9eygxs1C+NrdERfE3T0N8Gw8 XWcrsVxtYXWz7pVG23jrAo861qvRadK64+kOuh1taxaK8CMDbl23Y9dr2vY362p1jgWs3vTxcWHBL SkmFNGbjQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kIAtt-0001l1-Kn; Tue, 15 Sep 2020 13:21:44 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kIAtP-0001Vq-F4 for linux-arm-kernel@lists.infradead.org; Tue, 15 Sep 2020 13:21:22 +0000 Received: by mail-qt1-x84a.google.com with SMTP id a26so2685792qtb.16 for ; Tue, 15 Sep 2020 06:21:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=odW0Nb/rci20hhf4qlLrsuqbfMVKw2n0jytc4nSfGLw=; b=Ic5gm/5J7H57rrbgi2KUAbxYzDNz8YdSq5ZJ/v6u4Ac5dg8OrDRQ+Pt7M915w+yRSh pxiTvbk/H4XBY/WQD2Uq5EFNOh0nrFP9uzxLfzmg1rs+AopLvLhBPW/q3vEjThD99dL7 qrRJWuK823kI8mBCIvwjC68mmtOOJyAHLlvbSIybZDYGzy/xv8V3w6yjOsEBlCJM4KgJ vQRXFSOO1Zp2pI+L8K+dIx9YRoQbWq1vHeX5YUarWgO4WympBLwHXxV1p92l2ErDcbiS VWfBhw/z56liN0kg89Wfm6uq9cuGDYo3z5FUkd0Z3Noo9NtkiI8ggEYuytsJEvEGmROu 3PxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=odW0Nb/rci20hhf4qlLrsuqbfMVKw2n0jytc4nSfGLw=; b=KPv1phtA9bNBbIwnNzAlGF3PRLERzUlaPQn6MBXAgud8YLcwMeg8fuJEQsGRBd6ljz FMhYsZyM+dmTa1ocdI77lzcrUvwJBdjpNnC7yKiHDsKzOV5Pj+ZUMdS6xb1RvSHaf8FK 86+q+XqvHA7hOpY9FHi8OQ6ncA/ncLl/JhW1llpsb9rmFhIUAmLkinAyxaKleT7uyAtC pqkKtFoMZcwP7vkQj59/g0bi5B9j3G+PP6Cldtl7Nysn4bdWosQUL0lGL3EB9QePfGae ST3A9TIYVtnhG7SHd8FzjvlkBf2OUoDoHXIUX4ndjmgqG+JZCAoPP1k9ZHRqDNkLc1x6 mCHA== X-Gm-Message-State: AOAM5317eD2V8n/UMlg/IAfsyoDR/ukOMfSqLB5vW50pdwhJLfxUK43L BLMPbN0AqDQZzr55YXRKdPObxPzutw== X-Google-Smtp-Source: ABdhPJwGw5MfQEtCAAY9VjPztZBNrmyYKXs2SVdQU3S5qk954oiAUcTvB0kWyGpJ9tJe35EZN6Q4i0DPPg== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a0c:b902:: with SMTP id u2mr17508718qvf.7.1600176063190; Tue, 15 Sep 2020 06:21:03 -0700 (PDT) Date: Tue, 15 Sep 2020 15:20:38 +0200 In-Reply-To: <20200915132046.3332537-1-elver@google.com> Message-Id: <20200915132046.3332537-3-elver@google.com> Mime-Version: 1.0 References: <20200915132046.3332537-1-elver@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [PATCH v2 02/10] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, akpm@linux-foundation.org, glider@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200915_092107_591625_1D15B60A X-CRM114-Status: GOOD ( 20.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, linux-doc@vger.kernel.org, peterz@infradead.org, catalin.marinas@arm.com, dave.hansen@linux.intel.com, linux-mm@kvack.org, edumazet@google.com, hpa@zytor.com, cl@linux.com, will@kernel.org, corbet@lwn.net, x86@kernel.org, kasan-dev@googlegroups.com, mingo@redhat.com, vbabka@suse.cz, rientjes@google.com, aryabinin@virtuozzo.com, keescook@chromium.org, paulmck@kernel.org, jannh@google.com, andreyknvl@google.com, bp@alien8.de, luto@kernel.org, Jonathan.Cameron@huawei.com, tglx@linutronix.de, dvyukov@google.com, linux-arm-kernel@lists.infradead.org, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, penberg@kernel.org, cai@lca.pw, iamjoonsoo.kim@lge.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..e22dc722698c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..cf09e377faf9 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* The alignment should be at least a 4K page. */ +#define KFENCE_POOL_ALIGNMENT PAGE_SIZE + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_initialize_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLBs. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (!pte || level != PG_LEVEL_4K) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..423e15ad5eb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, } #endif + if (kfence_handle_page_fault(address)) + return; + /* * 32-bit: * -- 2.28.0.618.gf4bc123cb7-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel