From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED7EDC433DB for ; Wed, 3 Mar 2021 14:27:29 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 773F160202 for ; Wed, 3 Mar 2021 14:27:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 773F160202 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-Id:Date:Subject:Cc:To:From:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=ZreU/qyKHhKJ3SjIArY+6G1uoV6WQBjSwA3a6AeOczI=; b=Fi3nI/z50rnIM3aBUqxCSUgPWP 6l6fTNaejehg6/35EljX+eKJtWtFM+0cNDftefRy17eHdCKskZNF0zvR/YT4c7p7SjpA6ChQHBZmW W8O6ByGDAmCKQS/b5d6iGwSVlu5pnzZcNxZvTuFmEHt8tvxMyRG5Z4/DSO8WORy1itCsL50d/7aR8 x/NL6VHtdwD11YSsl1csUge4MEEmxn617MZrU/tJ99rkJw0g5qYrV5O1o6tMXNW80H0zkJRcBPKj/ 73jdHGHdsITRpEUGbE1xtDwk1cVlOkRZUAWkPAZjeu+DVi2Y7Pn/6uu8o+t2K69oVOcXdER6PlppX 2zZDy1vA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHSQE-0057E4-E4; Wed, 03 Mar 2021 14:24:19 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHS7x-0052y1-Nd for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:05:27 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To:References; bh=zImApPV2C19e7LU9UMt3B/go8CSpsOkmSROGn9SRR4c=; b=o6CX3TOjy7K8lfdMYsnlz9+hx9 SemhkHK2WvtpUdDjM0oggq+k717XfdMfMN8ZVvIhoTvNlKtqteT001Z1Ptsfi9YebPVwpccWvD/u8 v8fO6Y3/19yC28kj9L7U4LQoNOHmJgt1+HSkOAFGLTGF/S5Fs0A7haP9knDJMcgt8Yut5zcfaVvqu 5KheHFzz+S4lEkyCYKiAOkpDiArGI1QK57TayasESgVs5/SHBZVVf95/wCFj6ZvhT4ArnU3o3QudC yw2az5EGPlYQ0UOeAkCzmkt2StscMoFxl6+w6MwnEUKiZZv3jqZjnRrj3r/+613ArNorfNNOWP1vl FBlP1z0A==; Received: from foss.arm.com ([217.140.110.172]) by casper.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lH3b2-00H2tB-GD for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 11:53:53 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9344D11D4; Tue, 2 Mar 2021 03:53:42 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8D8303F73B; Tue, 2 Mar 2021 03:53:41 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: cai@redhat.com, catalin.marinas@arm.com, elver@google.com, james.morse@arm.com, mark.rutland@arm.com, will@kernel.org Subject: [PATCHv3] arm64: initialize per-cpu offsets earlier Date: Tue, 2 Mar 2021 11:53:35 +0000 Message-Id: <20210302115335.42135-1-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_115353_153274_FBDEBFAB X-CRM114-Status: GOOD ( 18.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current initialization of the per-cpu offset register is difficult to follow and this initialization is not always early enough for upcoming instrumentation with KCSAN, where the instrumentation callbacks use the per-cpu offset. To make it possible to support KCSAN, and to simplify reasoning about early bringup code, let's initialize the per-cpu offset earlier, before we run any C code that may consume it. To do so, this patch adds a new init_this_cpu_offset() helper that's called before the usual primary/secondary start functions. For consistency, this is also used to re-initialize the per-cpu offset after the runtime per-cpu areas have been allocated (which can change CPU0's offset). So that init_this_cpu_offset() isn't subject to any instrumentation that might consume the per-cpu offset: * init_this_cpu_offset() is marked with noinstr to prevent any direct instrumentation. * set_my_cpu_offset() is marked __always_inline so that this cannot be compiled as a non-inline function subject to instrumentation. * We directly use `current->cpu` rather than `task_cpu()`, as the latter is subject to instrumentation. Note: as per_cpu_offset() is a macro, this will not be instrumented within a noinstr function. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Marco Elver Cc: Qian Cai Cc: Will Deacon --- arch/arm64/include/asm/cpu.h | 2 ++ arch/arm64/include/asm/percpu.h | 2 +- arch/arm64/kernel/head.S | 2 ++ arch/arm64/kernel/setup.c | 12 ++++++------ arch/arm64/kernel/smp.c | 13 ++++++++----- 5 files changed, 19 insertions(+), 12 deletions(-) Since v1 [1]: * Fix typos * Rebase atop v5.9-rc4 Since v2 [2]: * Prevent instrumentation issues reported by Qian Cai * Improve commit message and comments * Rebase to v5.12-rc1 I've built this with and without KASAN_INLINE, inspected the results with objdump, and given this basic boot testing, so I'm fairly confident we shouldn't hit the boot-time hand Qian Cai previously hit. There are still a few other things we'll need to sort out before we can enable KCSAN on arm64, but it'd be nice to get this out of way soon if possible. Mark. [1] https://lore.kernel.org/r/20200730163806.23053-1-mark.rutland@arm.com [2] https://lore.kernel.org/r/20201005164303.21389-1-mark.rutland@arm.com diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h index 7faae6ff3ab4..d9d60b18e811 100644 --- a/arch/arm64/include/asm/cpu.h +++ b/arch/arm64/include/asm/cpu.h @@ -68,4 +68,6 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info); void update_cpu_features(int cpu, struct cpuinfo_arm64 *info, struct cpuinfo_arm64 *boot); +void init_this_cpu_offset(void); + #endif /* __ASM_CPU_H */ diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 8f1661603b78..97cf71d18a7c 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -11,7 +11,7 @@ #include #include -static inline void set_my_cpu_offset(unsigned long off) +static __always_inline void set_my_cpu_offset(unsigned long off) { asm volatile(ALTERNATIVE("msr tpidr_el1, %0", "msr tpidr_el2, %0", diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 66b0e0b66e31..1db8a1f543fb 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -449,6 +449,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) add sp, sp, #16 mov x29, #0 mov x30, #0 + bl init_this_cpu_offset b start_kernel SYM_FUNC_END(__primary_switched) @@ -613,6 +614,7 @@ SYM_FUNC_START_LOCAL(__secondary_switched) ptrauth_keys_init_cpu x2, x3, x4, x5 #endif + bl init_this_cpu_offset b secondary_start_kernel SYM_FUNC_END(__secondary_switched) diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 61845c0821d9..46edcaa5e42b 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -87,12 +87,6 @@ void __init smp_setup_processor_id(void) u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK; set_cpu_logical_map(0, mpidr); - /* - * clear __my_cpu_offset on boot CPU to avoid hang caused by - * using percpu variable early, for example, lockdep will - * access percpu variable inside lock_release - */ - set_my_cpu_offset(0); pr_info("Booting Linux on physical CPU 0x%010lx [0x%08x]\n", (unsigned long)mpidr, read_cpuid_id()); } @@ -296,6 +290,12 @@ u64 cpu_logical_map(unsigned int cpu) return __cpu_logical_map[cpu]; } +void noinstr init_this_cpu_offset(void) +{ + unsigned int cpu = current->cpu; + set_my_cpu_offset(per_cpu_offset(cpu)); +} + void __init __no_sanitize_address setup_arch(char **cmdline_p) { init_mm.start_code = (unsigned long) _stext; diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 357590beaabb..0c906e61a7fc 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -201,10 +201,7 @@ asmlinkage notrace void secondary_start_kernel(void) u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK; struct mm_struct *mm = &init_mm; const struct cpu_operations *ops; - unsigned int cpu; - - cpu = task_cpu(current); - set_my_cpu_offset(per_cpu_offset(cpu)); + unsigned int cpu = smp_processor_id(); /* * All kernel threads share the same mm context; grab a @@ -451,7 +448,13 @@ void __init smp_cpus_done(unsigned int max_cpus) void __init smp_prepare_boot_cpu(void) { - set_my_cpu_offset(per_cpu_offset(smp_processor_id())); + /* + * Now that setup_per_cpu_areas() has allocated the runtime per-cpu + * areas we must reinitialize CPU0's offset to point to its runtime + * area before the boot area is freed. + */ + init_this_cpu_offset(); + cpuinfo_store_boot_cpu(); /* -- 2.11.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel