Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

From: Steven Rostedt <rostedt@goodmis.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
	"H. Peter Anvin" <hpa@zytor.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@kernel.org>, Borislav Petkov <bp@alien8.de>,
	Andy Lutomirski <luto@amacapital.net>,
	Joel Fernandes <joel@joelfernandes.org>,
	He Zhe <zhe.he@windriver.com>,
	Linus Torvalds <torvalds@linux-foundation.org>
Subject: Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry
Date: Thu, 21 Mar 2019 09:04:22 -0400	[thread overview]
Message-ID: <20190321090422.067ab491@gandalf.local.home> (raw)
In-Reply-To: <20190321083317.GL6058@hirez.programming.kicks-ass.net>

On Thu, 21 Mar 2019 09:33:17 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> On Wed, Mar 20, 2019 at 10:15:34PM -0400, Steven Rostedt wrote:
> 
> > And it would crash similarly each time I tried it, but always at a
> > different place. After spending the day on this, I finally figured it
> > out. The bug is happening in entry_64.S right after error_entry.
> > There's two TRACE_IRQS_OFF in that code path, which if I comment out,
> > the bug goes away. Then it dawned on me that the crash always happens
> > when systemd does a normal page fault. We had this bug before, and it
> > was with the exception trace points.  
> 
> 0ac09f9f8cd1 ("x86, trace: Fix CR2 corruption when tracing page faults")
> d4078e232267 ("x86, trace: Further robustify CR2 handling vs tracing")

Probably these two, as I remember more about the discussions around
them, and not the actual commits. Although, I did take a look at the
do_page_fault() code that was added because of them. I just didn't do a
git blame to see what added it.

> 
> Or were you talking about:
> 
> 70fb74a5420f ("x86: Save cr2 in NMI in case NMIs take a page fault (for i386)")
> 
> > The issue is that a tracepoint can fault (reading vmalloc or whatever).
> > And doing a userspace stack trace most definitely will fault. But if we
> > are coming from a legitimate page fault, the address of that fault (in
> > the CR2 register) will be lost if we fault before we get to the page
> > fault handler. That's exactly what is happening.  
> 
> Shees, that could've been written much clearer. So you're saying:

I wrote this just before going to bed. It was the best I could come up
with at the time.

> 
> idtentry page_fault             do_page_fault           has_error_code=1
>   call error_entry
>     TRACE_IRQS_OFF
>       call trace_hardirqs_off*
>         <tracer stuff>
> 	  <fault> # modifies CR2
>   call do_page_fault
>     address = read_cr2(); /* whoopsie */
> 
> Right?

Yes.

> 
> > To solve this, a TRACE_IRQS_OFF_CR2 (and ON for consistency) was added
> > that saves the CR2 register. A new trace_hardirqs_off_thunk_cr2 is
> > created that stores the cr2 register, calls the
> > trace_hardirqs_off_caller, then on return restores the cr2 register if
> > it changed, before returning.  
> 
> Yuck.. also, not consistent with the actual patch. The thunk doesn't
> save/restore CR2.

Well, the thunk calls the caller_cr2 that does, which is just a helper
function for the thunk.

> 
> I really hate making this special TRACE_IRQS_OFF_CR2 thing, it feels far
> too fragile. I'd _much_ rather push the #PF CR2 read much earlier.
> 
> Also, argh I fscking hate context tracking. That makes all this so much
> more complicated. It if weren't for CALL_enter_from_user_mode, we could
> pull that TRACE_IRQS_OFF out of error_entry.

Yeah, and I didn't even test this with context tracking enabled yet.

-- Steve

> 
> Damn... Andy, any bright ideas?