Can we break RDPID/RDTSCP ABI ASAP and see if it's okay?

Can we break RDPID/RDTSCP ABI ASAP and see if it's okay?

[Andy Lutomirski]

We currently do this on boot:

write_rdtscp_aux((node << 12) | cpu);

This *sucks*.  It means that, to very quickly obtain the CPU number
using RDPID, an ALU op is needed.  It also doesn't bloody work on
systems with more than 4096 CPUs.

IMO it should be ((u64)node << 32) | cpu.  Then getting the CPU number is just:

RDPID %rax
MOVL %eax, %eax

I'm thinking about this because rseq users could avoid ever *loading*
the rseq cacheline if they used RDPID to get the CPU number, and it
would be nice to give them a sane way to do it.

This won't break any existing RDPID users if we do it quickly because
there aren't any (the CPUs aren't available).  I would be a bit
surprised if anyone uses RDTSCP for this because it's absurdly slow.

We can change this without affecting the LSL hack, and I think there
are user programs that do the LSL hack.

--Andy

Re: Can we break RDPID/RDTSCP ABI ASAP and see if it's okay?

[Linus Torvalds]

On Sat, Oct 28, 2017 at 12:38 PM, Andy Lutomirski <luto@kernel.org> wrote:
>
> This won't break any existing RDPID users if we do it quickly because
> there aren't any (the CPUs aren't available).  I would be a bit
> surprised if anyone uses RDTSCP for this because it's absurdly slow.

Why do you think that?

Lots of people use rdtscp, I bet. They use it mainly for the TSC,
admittedly, and the example code I looked at all just ignored %rcx,
but maybe they *do* care.

And in particular, with rdtscp, the high 32 bits get lost, so your
suggestion will mean that the thing doesn't contain the node number at
all.

People may in fact use the node number for things like a fast
gettimeofday() - knowing that the TSC is synchronized within a node,
and doing per-node correction..

And maybe not. But it's by no means obvious that just because there
aren't any released processors that support RDPID that this value
isn't used at all.

So I'd be nervous. We could try it, perhaps, but there could be some
_very_ subtle breakage..

               Linus

Re: Can we break RDPID/RDTSCP ABI ASAP and see if it's okay?

[Thomas Gleixner]

On Sat, 28 Oct 2017, Andy Lutomirski wrote:

> We currently do this on boot:
> 
> write_rdtscp_aux((node << 12) | cpu);
> 
> This *sucks*.  It means that, to very quickly obtain the CPU number
> using RDPID, an ALU op is needed.  It also doesn't bloody work on
> systems with more than 4096 CPUs.
> 
> IMO it should be ((u64)node << 32) | cpu.  Then getting the CPU number is just:

That breaks 32bit

> RDPID %rax
> MOVL %eax, %eax
> 
> I'm thinking about this because rseq users could avoid ever *loading*
> the rseq cacheline if they used RDPID to get the CPU number, and it
> would be nice to give them a sane way to do it.
> 
> This won't break any existing RDPID users if we do it quickly because
> there aren't any (the CPUs aren't available).  I would be a bit
> surprised if anyone uses RDTSCP for this because it's absurdly slow.

What we can do on 64bit is:

     ((u64) cpu << 32) | (node << 12) | (cpu & 0xfff)

That does not solve the ALU op problem but works on both 32 and 64 bit and
on systems with more than 4096 CPUs.

Thanks,

	tglx

Re: Can we break RDPID/RDTSCP ABI ASAP and see if it's okay?

[Paolo Bonzini]

On 28/10/2017 21:38, Andy Lutomirski wrote:
> We currently do this on boot:
> 
> write_rdtscp_aux((node << 12) | cpu);
> 
> This *sucks*.  It means that, to very quickly obtain the CPU number
> using RDPID, an ALU op is needed.  It also doesn't bloody work on
> systems with more than 4096 CPUs.
> 
> IMO it should be ((u64)node << 32) | cpu.

MSR_TSC_AUX is still documented as reserving bits 32-63 in the October
2017 SDM, and indeed on a Haswell you get a #GP if you write a nonzero
value to it.

Has Intel quietly "unreserved" them on machines that have RDPID?

Thanks,

Paolo

> Then getting the CPU number is just:
> 
> RDPID %rax
> MOVL %eax, %eax
> 
> I'm thinking about this because rseq users could avoid ever *loading*
> the rseq cacheline if they used RDPID to get the CPU number, and it
> would be nice to give them a sane way to do it.
> 
> This won't break any existing RDPID users if we do it quickly because
> there aren't any (the CPUs aren't available).  I would be a bit
> surprised if anyone uses RDTSCP for this because it's absurdly slow.
> 
> We can change this without affecting the LSL hack, and I think there
> are user programs that do the LSL hack.
> 
> --Andy
> 

Re: Can we break RDPID/RDTSCP ABI ASAP and see if it's okay?

[Andy Lutomirski]

On Mon, Nov 13, 2017 at 5:54 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 28/10/2017 21:38, Andy Lutomirski wrote:
>> We currently do this on boot:
>>
>> write_rdtscp_aux((node << 12) | cpu);
>>
>> This *sucks*.  It means that, to very quickly obtain the CPU number
>> using RDPID, an ALU op is needed.  It also doesn't bloody work on
>> systems with more than 4096 CPUs.
>>
>> IMO it should be ((u64)node << 32) | cpu.
>
> MSR_TSC_AUX is still documented as reserving bits 32-63 in the October
> 2017 SDM, and indeed on a Haswell you get a #GP if you write a nonzero
> value to it.
>
> Has Intel quietly "unreserved" them on machines that have RDPID?
>

Dunno.  I don't have such a machine.

But this is a genuine problem on machines with >4096 CPUs.