Re: [RFC patch 10/15] x86/entry: Move irq tracing to C code

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Sep 23, 2019 at 12:27:47PM +0200, Thomas Gleixner wrote:
> On Mon, 23 Sep 2019, Peter Zijlstra wrote:
> 
> > On Thu, Sep 19, 2019 at 05:03:24PM +0200, Thomas Gleixner wrote:
> > > To prepare for converting the exit to usermode code to the generic version,
> > > move the irqflags tracing into C code.
> > > 
> > > Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> > > ---
> > >  arch/x86/entry/common.c          |   10 ++++++++++
> > >  arch/x86/entry/entry_32.S        |   11 +----------
> > >  arch/x86/entry/entry_64.S        |   10 ++--------
> > >  arch/x86/entry/entry_64_compat.S |   21 ---------------------
> > >  4 files changed, 13 insertions(+), 39 deletions(-)
> > > 
> > > --- a/arch/x86/entry/common.c
> > > +++ b/arch/x86/entry/common.c
> > > @@ -102,6 +102,8 @@ static void exit_to_usermode_loop(struct
> > >  	struct thread_info *ti = current_thread_info();
> > >  	u32 cached_flags;
> > >  
> > > +	trace_hardirqs_off();
> > 
> > Bah.. so this gets called from:
> > 
> >  - C code, with IRQs disabled
> >  - entry_64.S:error_exit
> >  - entry_32.S:resume_userspace
> > 
> > The first obviously doesn't need this annotation, but this patch doesn't
> > remove the TRACE_IRQS_OFF from entry_64.S and only the 32bit case is
> > changed.
> > 
> > Is that entry_64.S case an oversight, or do we need an extensive comment
> > on this one?
> 
> Lemme stare at that again. At some point I probably lost track in that maze.

While walking the kids to school I wondered WTH we need to call
TRACE_IRQS_OFF in the first place. If this is the return from exception
path, interrupts had better be disabled already (in exception enter).

For entry_64.S we have:

  - idtentry_part; which does TRACE_IRQS_OFF at the start and error_exit
    at the end.

  - xen_do_hypervisor_callback, xen_failsafe_callback -- which are
    confusing.

So in the normal case, it appears we can simply do:

diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index b7c3ea4cb19d..e9cf59ac554e 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -1368,8 +1368,6 @@ END(error_entry)
 
 ENTRY(error_exit)
 	UNWIND_HINT_REGS
-	DISABLE_INTERRUPTS(CLBR_ANY)
-	TRACE_IRQS_OFF
 	testb	$3, CS(%rsp)
 	jz	retint_kernel
 	jmp	retint_user

and all should be well. This leaves Xen...

For entry_32.S it looks like nothing uses 'resume_userspace' so that
ENTRY can go away. Which then leaves:

 * ret_from_intr:
  - common_spurious: TRACE_IRQS_OFF
  - common_interrupt: TRACE_IRQS_OFF
  - BUILD_INTERRUPT3: TRACE_IRQS_OFF
  - xen_do_upcall: ...

 * ret_from_exception:
  - xen_failsafe_callback: ...
  - common_exception_read_cr2: TRACE_IRQS_OFF
  - common_exception: TRACE_IRQS_OFF
  - int3: TRACE_IRQS_OFF

Which again suggests:

diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S
index f83ca5aa8b77..7a19e7413a8e 100644
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -825,9 +825,6 @@ END(ret_from_fork)
 	cmpl	$USER_RPL, %eax
 	jb	restore_all_kernel		# not returning to v8086 or userspace
 
-ENTRY(resume_userspace)
-	DISABLE_INTERRUPTS(CLBR_ANY)
-	TRACE_IRQS_OFF
 	movl	%esp, %eax
 	call	prepare_exit_to_usermode
 	jmp	restore_all

with the notable exception (oh teh pun!) being Xen... _again_.

With these patchlets on, we'd want prepare_exit_to_usermode() to
validate the IRQ state, but AFAICT it _should_ all just 'work' (famous
last words).

Cc Juergen because Xen