Re: [REGRESSION] x86/cpu fsgsbase breaks TLS in 32 bit rr tracees on a 64 bit system

From: Andy Lutomirski <luto@kernel.org>
To: Kyle Huey <me@kylehuey.com>
Cc: Andy Lutomirski <luto@kernel.org>,
	"H. Peter Anvin" <hpa@zytor.com>,
	"Robert O'Callahan" <robert@ocallahan.org>,
	"Bae, Chang Seok" <chang.seok.bae@intel.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@kernel.org>, Andi Kleen <ak@linux.intel.com>,
	"Shankar, Ravi V" <ravi.v.shankar@intel.com>,
	LKML <linux-kernel@vger.kernel.org>,
	"Hansen, Dave" <dave.hansen@intel.com>
Subject: Re: [REGRESSION] x86/cpu fsgsbase breaks TLS in 32 bit rr tracees on a 64 bit system
Date: Tue, 25 Aug 2020 09:46:11 -0700	[thread overview]
Message-ID: <CALCETrX+TLB+w0X0jc9jq_U4SQezWXEmSpEmmdPobnbUuYfang@mail.gmail.com> (raw)
In-Reply-To: <CAP045Aqhox6YSdk0v_YZWY=y7Ps4ZfH779MG-W4a=gc+cYEY+Q@mail.gmail.com>

On Tue, Aug 25, 2020 at 9:32 AM Kyle Huey <me@kylehuey.com> wrote:
>
> On Tue, Aug 25, 2020 at 9:12 AM Andy Lutomirski <luto@amacapital.net> wrote:
> > I don’t like this at all. Your behavior really shouldn’t depend on
> > whether the new instructions are available.  Also, some day I would
> > like to change Linux to have the new behavior even if FSGSBASE
> > instructions are not available, and this will break rr again.  (The
> > current !FSGSBASE behavior is an ugly optimization of dubious value.
> > I would not go so far as to describe it as correct.)
>
> Ok.
>
> > I would suggest you do one of the following things:
> >
> > 1. Use int $0x80 directly to load 32-bit regs into a child.  This
> > might dramatically simplify your code and should just do the right
> > thing.
>
> I don't know what that means.

This is untested, but what I mean is:

static int ptrace32(int req, pid_t pid, int addr, int data) {
   int ret;
   /* new enough kernels won't clobber r8, etc. */
   asm volatile ("int $0x80" : "=a" (ret) : "a" (26 /* ptrace */), "b"
(req), "c" (pid), "d" (addr), "S" (data) : "flags", "r8", "r9", "r10",
"r11");
   return ret;
}

with a handful of caveats:

 - This won't compile with -fPIC, I think.  Instead you'll need to
write a little bit of asm to set up and restore ebx yourself.  gcc is
silly like this.

 - Note that addr is an int.  You'll need to mmap(..., MAP_32BIT, ...)
to get a buffer that can be pointed to with an int.

The advantage is that this should work on all kernels that support
32-bit mode at all.

>
> > 2. Something like your patch but make it unconditional.
> >
> > 3. Ask for, and receive, real kernel support for setting FS and GS in
> > the way that 32-bit code expects.
>
> I think the easiest way forward for us would be a PTRACE_GET/SETREGSET
> like operation that operates on the regsets according to the
> *tracee*'s bitness (rather than the tracer, as it works currently).
> Does that sound workable?
>

Strictly speaking, on Linux, there is no unified concept of a task's
bitness, so "set all these registers according to the target's
bitness" is not well defined.  We could easily give you a
PTRACE_SETREGS_X86_32, etc, though.