Re: [XTF PATCH] XSA-186: Work around suspected Broadwell TLB erratum

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 28/10/16 13:49, Jan Beulich wrote:
>>>> On 28.10.16 at 14:39, <andrew.cooper3@citrix.com> wrote:
>> On 28/10/16 13:03, Jan Beulich wrote:
>>>>>> On 28.10.16 at 12:36, <andrew.cooper3@citrix.com> wrote:
>>>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>>> Reviewed-by: Jan Beulich <jbeulich@suse.com>
>>> (Maybe you want to drop the ...
>>>
>>>> --- a/tests/xsa-186/main.c
>>>> +++ b/tests/xsa-186/main.c
>>>> @@ -144,6 +144,29 @@ void test_main(void)
>>>>      memcpy(stub, insn_buf_start, insn_buf_end - insn_buf_start);
>>>>  
>>>>      /*
>>>> +     * Work around suspected Broadwell TLB Erratum
>>>> +     *
>>>> +     * Occasionally, this test failes with:
>>>> +     *
>>>> +     *   --- Xen Test Framework ---
>>>> +     *   Environment: HVM 64bit (Long mode 4 levels)
>>>> +     *   XSA-186 PoC
>>>> +     *   ******************************
>>>> +     *   PANIC: Unhandled exception at 0008:fffffffffffffffa
>>>> +     *   Vec 14 #PF[-I-sr-] %cr2 fffffffffffffffa
>>>> +     *   ******************************
>>>> +     *
>>>> +     * on Broadwell hardware.  The mapping is definitely present as the
>>>> +     * memcpy() has already succeeded.  Inserting an invlpg resolves the
>>>> +     * issue, sugguesting that there is a race conditon between dTLB/iTLB
>>> ... stray u which slipped into "suggesting".)
>>>
>>> Btw - would you mind trying something else: Instead of the INVLPG,
>>> put a CPUID or some other serializing instruction in here. ISTR that
>>> for self modifying code this is required, i.e. the CPU could have been
>>> fetching instructions ahead of the memcpy(), and nothing would be
>>> there to force it to drop what it has already executed speculatively,
>>> including the exception token.
>> That is an interesting point, but still doesn't explain the symptoms. 
>> If the icache wasn't flushed, we might get junk instructions and a #UD/#GP.
> No. As the processor speculates the call, it won't be able to fetch
> the target instruction and hence would insert an exception token
> into the queue. There would be junk instruction bytes only if there
> was a prior mapping for that page, but aiui a mapping for that
> address gets established exactly once.

Re-reading Intel Vol 3 11.6 "Self-Modifying Code".

* A write to a memory location in a code segment that is currently
cached in the processor causes the associated cache line (or lines) to
be invalidated. This check is based on the physical address of the
instruction.  If the write affects a prefetched instruction, the
prefetch queue is invalidated. This latter check is based on the linear
address of the instruction.

* Systems software, such as a debugger, that might possibly modify an
instruction using a different linear address than that used to fetch the
instruction, will execute a serializing operation, such as a CPUID
instruction, before the modified instruction is executed, which will
automatically resynchronize the instruction cache and prefetch queue.

As this is a single vcpu using a single flat address space, the memcpy()
should invalidate any speculative execution which has already happened.

>
>> However, in this case the fault is for an instruction fetch from a
>> non-present page, not a failure to execute what it found there.
>>
>> I expect a cpuid instruction would resolve the issue, but it also forces
>> a vmexit which complicates the microarchitectural interactions here. 
>> Something else, like executing an int3 will also serialise the pipeline,
>> but not vmexit.  I will try and find some time to experiment.
> You're in ring 0, aren't you? That gives you plenty of serializing
> instructions which don't directly interact with the TLBs. An LLDT
> with a zero selector might be the one with least side effects. And
> in case you're not in ring 0, make up an interrupt frame and
> execute an IRET.

Yes - all better options.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel