From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18E8AC3279D for ; Fri, 9 Oct 2020 19:43:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8DAD2222BA for ; Fri, 9 Oct 2020 19:43:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8DAD2222BA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1069D6B0072; Fri, 9 Oct 2020 15:43:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 092746B0073; Fri, 9 Oct 2020 15:43:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4ACC6B0074; Fri, 9 Oct 2020 15:43:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id B08286B0072 for ; Fri, 9 Oct 2020 15:43:26 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 49BCB181AE860 for ; Fri, 9 Oct 2020 19:43:26 +0000 (UTC) X-FDA: 77353411212.08.quill70_3315530271e3 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 288771819E623 for ; Fri, 9 Oct 2020 19:43:26 +0000 (UTC) X-HE-Tag: quill70_3315530271e3 X-Filterd-Recvd-Size: 16453 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Fri, 9 Oct 2020 19:43:24 +0000 (UTC) IronPort-SDR: ggxWze8ReD3lFM9kgt4LWp3iZ9y361Pkw+5Jmje4TeXw6gJQAVKZXAeCAz9qqtQAvWKPu/WS0q sDCMAig9w4mA== X-IronPort-AV: E=McAfee;i="6000,8403,9769"; a="144849960" X-IronPort-AV: E=Sophos;i="5.77,355,1596524400"; d="scan'208";a="144849960" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Oct 2020 12:43:22 -0700 IronPort-SDR: MkfnCLSSRwOBTQEfWdMVmXruOqSNup+ZdyDI/59QanK2/5c7cgi3y+Kyqitw3PbH47IJQ1/95p LM+Wp9uIp/MA== X-IronPort-AV: E=Sophos;i="5.77,355,1596524400"; d="scan'208";a="354957039" Received: from iweiny-desk2.sc.intel.com (HELO localhost) ([10.3.52.147]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Oct 2020 12:43:21 -0700 From: ira.weiny@intel.com To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Andy Lutomirski , Peter Zijlstra Cc: Ira Weiny , x86@kernel.org, Dave Hansen , Dan Williams , Andrew Morton , Fenghua Yu , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org Subject: [PATCH RFC V3 6/9] x86/entry: Pass irqentry_state_t by reference Date: Fri, 9 Oct 2020 12:42:55 -0700 Message-Id: <20201009194258.3207172-7-ira.weiny@intel.com> X-Mailer: git-send-email 2.28.0.rc0.12.gb6a658bd00c9 In-Reply-To: <20201009194258.3207172-1-ira.weiny@intel.com> References: <20201009194258.3207172-1-ira.weiny@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Ira Weiny In preparation for adding PKS information to struct irqentry_state_t change all call sites and usages to pass the struct by reference instead of by value. Signed-off-by: Ira Weiny --- arch/x86/entry/common.c | 16 +++++++--------- arch/x86/include/asm/idtentry.h | 29 +++++++++++++++++------------ arch/x86/kernel/kvm.c | 4 ++-- arch/x86/kernel/nmi.c | 7 ++++--- arch/x86/kernel/traps.c | 21 +++++++++++++-------- arch/x86/mm/fault.c | 4 ++-- include/linux/entry-common.h | 7 ++++--- kernel/entry/common.c | 20 ++++++++------------ 8 files changed, 57 insertions(+), 51 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 870efeec8bda..305da13770b6 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -209,9 +209,9 @@ SYSCALL_DEFINE0(ni_syscall) return -ENOSYS; } =20 -noinstr bool idtentry_enter_nmi(struct pt_regs *regs) +noinstr void idtentry_enter_nmi(struct pt_regs *regs, irqentry_state_t *= irq_state) { - bool irq_state =3D lockdep_hardirqs_enabled(); + irq_state->exit_rcu =3D lockdep_hardirqs_enabled(); =20 __nmi_enter(); lockdep_hardirqs_off(CALLER_ADDR0); @@ -222,15 +222,13 @@ noinstr bool idtentry_enter_nmi(struct pt_regs *reg= s) trace_hardirqs_off_finish(); ftrace_nmi_enter(); instrumentation_end(); - - return irq_state; } =20 -noinstr void idtentry_exit_nmi(struct pt_regs *regs, bool restore) +noinstr void idtentry_exit_nmi(struct pt_regs *regs, irqentry_state_t *i= rq_state) { instrumentation_begin(); ftrace_nmi_exit(); - if (restore) { + if (irq_state->exit_rcu) { trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); } @@ -238,7 +236,7 @@ noinstr void idtentry_exit_nmi(struct pt_regs *regs, = bool restore) =20 rcu_nmi_exit(); lockdep_hardirq_exit(); - if (restore) + if (irq_state->exit_rcu) lockdep_hardirqs_on(CALLER_ADDR0); __nmi_exit(); } @@ -295,7 +293,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct= pt_regs *regs) bool inhcall; irqentry_state_t state; =20 - state =3D irqentry_enter(regs); + irqentry_enter(regs, &state); old_regs =3D set_irq_regs(regs); =20 instrumentation_begin(); @@ -311,7 +309,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct= pt_regs *regs) instrumentation_end(); restore_inhcall(inhcall); } else { - irqentry_exit(regs, state); + irqentry_exit(regs, &state); } } #endif /* CONFIG_XEN_PV */ diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idten= try.h index a0638640f1ed..622889ba21d0 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -11,8 +11,8 @@ =20 #include =20 -bool idtentry_enter_nmi(struct pt_regs *regs); -void idtentry_exit_nmi(struct pt_regs *regs, bool irq_state); +void idtentry_enter_nmi(struct pt_regs *regs, irqentry_state_t *irq_stat= e); +void idtentry_exit_nmi(struct pt_regs *regs, irqentry_state_t *irq_state= ); =20 /** * DECLARE_IDTENTRY - Declare functions for simple IDT entry points @@ -52,12 +52,13 @@ static __always_inline void __##func(struct pt_regs *= regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state =3D irqentry_enter(regs); \ + irqentry_state_t state; \ \ + irqentry_enter(regs, &state); \ instrumentation_begin(); \ __##func (regs); \ instrumentation_end(); \ - irqentry_exit(regs, state); \ + irqentry_exit(regs, &state); \ } \ \ static __always_inline void __##func(struct pt_regs *regs) @@ -99,12 +100,13 @@ static __always_inline void __##func(struct pt_regs = *regs, \ __visible noinstr void func(struct pt_regs *regs, \ unsigned long error_code) \ { \ - irqentry_state_t state =3D irqentry_enter(regs); \ + irqentry_state_t state; \ \ + irqentry_enter(regs, &state); \ instrumentation_begin(); \ __##func (regs, error_code); \ instrumentation_end(); \ - irqentry_exit(regs, state); \ + irqentry_exit(regs, &state); \ } \ \ static __always_inline void __##func(struct pt_regs *regs, \ @@ -195,15 +197,16 @@ static __always_inline void __##func(struct pt_regs= *regs, u8 vector); \ __visible noinstr void func(struct pt_regs *regs, \ unsigned long error_code) \ { \ - irqentry_state_t state =3D irqentry_enter(regs); \ + irqentry_state_t state; \ \ + irqentry_enter(regs, &state); \ instrumentation_begin(); \ irq_enter_rcu(); \ kvm_set_cpu_l1tf_flush_l1d(); \ __##func (regs, (u8)error_code); \ irq_exit_rcu(); \ instrumentation_end(); \ - irqentry_exit(regs, state); \ + irqentry_exit(regs, &state); \ } \ \ static __always_inline void __##func(struct pt_regs *regs, u8 vector) @@ -237,15 +240,16 @@ static void __##func(struct pt_regs *regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state =3D irqentry_enter(regs); \ + irqentry_state_t state; \ \ + irqentry_enter(regs, &state); \ instrumentation_begin(); \ irq_enter_rcu(); \ kvm_set_cpu_l1tf_flush_l1d(); \ run_sysvec_on_irqstack_cond(__##func, regs); \ irq_exit_rcu(); \ instrumentation_end(); \ - irqentry_exit(regs, state); \ + irqentry_exit(regs, &state); \ } \ \ static noinline void __##func(struct pt_regs *regs) @@ -266,15 +270,16 @@ static __always_inline void __##func(struct pt_regs= *regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state =3D irqentry_enter(regs); \ + irqentry_state_t state; \ \ + irqentry_enter(regs, &state); \ instrumentation_begin(); \ __irq_enter_raw(); \ kvm_set_cpu_l1tf_flush_l1d(); \ __##func (regs); \ __irq_exit_raw(); \ instrumentation_end(); \ - irqentry_exit(regs, state); \ + irqentry_exit(regs, &state); \ } \ \ static __always_inline void __##func(struct pt_regs *regs) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 9663ba31347c..c6be0a54236f 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -241,7 +241,7 @@ noinstr bool __kvm_handle_async_pf(struct pt_regs *re= gs, u32 token) if (!flags) return false; =20 - state =3D irqentry_enter(regs); + irqentry_enter(regs, &state); instrumentation_begin(); =20 /* @@ -262,7 +262,7 @@ noinstr bool __kvm_handle_async_pf(struct pt_regs *re= gs, u32 token) } =20 instrumentation_end(); - irqentry_exit(regs, state); + irqentry_exit(regs, &state); return true; } =20 diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index 4fc9954a9560..68c07cad0150 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -33,6 +33,7 @@ #include #include #include +#include =20 #define CREATE_TRACE_POINTS #include @@ -475,7 +476,7 @@ static DEFINE_PER_CPU(unsigned long, nmi_dr7); =20 DEFINE_IDTENTRY_RAW(exc_nmi) { - bool irq_state; + irqentry_state_t irq_state =3D { }; =20 if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id())) return; @@ -490,14 +491,14 @@ DEFINE_IDTENTRY_RAW(exc_nmi) =20 this_cpu_write(nmi_dr7, local_db_save()); =20 - irq_state =3D idtentry_enter_nmi(regs); + idtentry_enter_nmi(regs, &irq_state); =20 inc_irq_stat(__nmi_count); =20 if (!ignore_nmis) default_do_nmi(regs); =20 - idtentry_exit_nmi(regs, irq_state); + idtentry_exit_nmi(regs, &irq_state); =20 local_db_restore(this_cpu_read(nmi_dr7)); =20 diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 81a2fb711091..daf7bc02fc99 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -254,11 +254,11 @@ DEFINE_IDTENTRY_RAW(exc_invalid_op) if (!user_mode(regs) && handle_bug(regs)) return; =20 - state =3D irqentry_enter(regs); + irqentry_enter(regs, &state); instrumentation_begin(); handle_invalid_op(regs); instrumentation_end(); - irqentry_exit(regs, state); + irqentry_exit(regs, &state); } =20 DEFINE_IDTENTRY(exc_coproc_segment_overrun) @@ -342,6 +342,7 @@ __visible void __noreturn handle_stack_overflow(const= char *message, */ DEFINE_IDTENTRY_DF(exc_double_fault) { + irqentry_state_t irq_state; static const char str[] =3D "double fault"; struct task_struct *tsk =3D current; =20 @@ -404,7 +405,7 @@ DEFINE_IDTENTRY_DF(exc_double_fault) } #endif =20 - idtentry_enter_nmi(regs); + idtentry_enter_nmi(regs, &irq_state); instrumentation_begin(); notify_die(DIE_TRAP, str, regs, error_code, X86_TRAP_DF, SIGSEGV); =20 @@ -650,12 +651,15 @@ DEFINE_IDTENTRY_RAW(exc_int3) instrumentation_end(); irqentry_exit_to_user_mode(regs); } else { - bool irq_state =3D idtentry_enter_nmi(regs); + irqentry_state_t irq_state; + + idtentry_enter_nmi(regs, &irq_state); + instrumentation_begin(); if (!do_int3(regs)) die("int3", regs, 0); instrumentation_end(); - idtentry_exit_nmi(regs, irq_state); + idtentry_exit_nmi(regs, &irq_state); } } =20 @@ -861,7 +865,9 @@ static __always_inline void exc_debug_kernel(struct p= t_regs *regs, * includes the entry stack is excluded for everything. */ unsigned long dr7 =3D local_db_save(); - bool irq_state =3D idtentry_enter_nmi(regs); + irqentry_state_t irq_state; + + idtentry_enter_nmi(regs, &irq_state); instrumentation_begin(); =20 /* @@ -880,8 +886,7 @@ static __always_inline void exc_debug_kernel(struct p= t_regs *regs, handle_debug(regs, dr6, false); =20 instrumentation_end(); - idtentry_exit_nmi(regs, irq_state); - + idtentry_exit_nmi(regs, &irq_state); local_db_restore(dr7); } =20 diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..e55bc4bff389 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1476,11 +1476,11 @@ DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault) * code reenabled RCU to avoid subsequent wreckage which helps * debugability. */ - state =3D irqentry_enter(regs); + irqentry_enter(regs, &state); =20 instrumentation_begin(); handle_page_fault(regs, error_code, address); instrumentation_end(); =20 - irqentry_exit(regs, state); + irqentry_exit(regs, &state); } diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index 159c7476b11b..de4f24c554ee 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -349,6 +349,7 @@ typedef struct irqentry_state { /** * irqentry_enter - Handle state tracking on ordinary interrupt entries * @regs: Pointer to pt_regs of interrupted context + * @state: Pointer to an object to store the irq state * * Invokes: * - lockdep irqflag state tracking as low level ASM entry disabled @@ -377,7 +378,7 @@ typedef struct irqentry_state { * * Returns: An opaque object that must be passed to idtentry_exit() */ -irqentry_state_t noinstr irqentry_enter(struct pt_regs *regs); +void noinstr irqentry_enter(struct pt_regs *regs, irqentry_state_t *stat= e); =20 /** * irqentry_exit_cond_resched - Conditionally reschedule on return from = interrupt @@ -389,7 +390,7 @@ void irqentry_exit_cond_resched(void); /** * irqentry_exit - Handle return from exception that used irqentry_enter= () * @regs: Pointer to pt_regs (exception entry regs) - * @state: Return value from matching call to irqentry_enter() + * @state: Reference to the value saved in irqentry_enter() * * Depending on the return target (kernel/user) this runs the necessary * preemption and work checks if possible and reguired and returns to @@ -400,6 +401,6 @@ void irqentry_exit_cond_resched(void); * * Counterpart to irqentry_enter(). */ -void noinstr irqentry_exit(struct pt_regs *regs, irqentry_state_t state)= ; +void noinstr irqentry_exit(struct pt_regs *regs, irqentry_state_t *state= ); =20 #endif diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 6fdb6105e6d6..21601993ad1b 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -282,15 +282,13 @@ noinstr void irqentry_exit_to_user_mode(struct pt_r= egs *regs) exit_to_user_mode(); } =20 -noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) +noinstr void irqentry_enter(struct pt_regs *regs, irqentry_state_t *stat= e) { - irqentry_state_t ret =3D { - .exit_rcu =3D false, - }; + state->exit_rcu =3D false; =20 if (user_mode(regs)) { irqentry_enter_from_user_mode(regs); - return ret; + return; } =20 /* @@ -328,8 +326,8 @@ noinstr irqentry_state_t irqentry_enter(struct pt_reg= s *regs) trace_hardirqs_off_finish(); instrumentation_end(); =20 - ret.exit_rcu =3D true; - return ret; + state->exit_rcu =3D true; + return; } =20 /* @@ -343,8 +341,6 @@ noinstr irqentry_state_t irqentry_enter(struct pt_reg= s *regs) /* Use the combo lockdep/tracing function */ trace_hardirqs_off(); instrumentation_end(); - - return ret; } =20 void irqentry_exit_cond_resched(void) @@ -359,7 +355,7 @@ void irqentry_exit_cond_resched(void) } } =20 -noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) +noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t *state= ) { lockdep_assert_irqs_disabled(); =20 @@ -372,7 +368,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqe= ntry_state_t state) * carefully and needs the same ordering of lockdep/tracing * and RCU as the return to user mode path. */ - if (state.exit_rcu) { + if (state->exit_rcu) { instrumentation_begin(); /* Tell the tracer that IRET will enable interrupts */ trace_hardirqs_on_prepare(); @@ -394,7 +390,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqe= ntry_state_t state) * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry. */ - if (state.exit_rcu) + if (state->exit_rcu) rcu_irq_exit(); } } --=20 2.28.0.rc0.12.gb6a658bd00c9