Re: [PATCH 62/65] x86/entry: Make IDT entrypoints CET-IBT compatible

[Andrew Cooper <amc96@srcf.net>]

On 06/12/2021 09:42, Jan Beulich wrote:
> On 03.12.2021 16:30, Andrew Cooper wrote:
>> On 03/12/2021 13:32, Jan Beulich wrote:
>>> On 26.11.2021 13:34, Andrew Cooper wrote:
>>>> Each IDT vector needs to land on an endbr64 instruction.  This is especially
>>>> important for the #CP handler, which will escalate to #DF if the endbr64 is
>>>> missing.
>>> One question here: How does this work?
>> Honestly, I'm not sure.
>>
>>>  I don't recall there being any "CET
>>> shadow" along the lines of "STI shadow" and "SS shadow", yet there's
>>> clearly an insn boundary here that gets "skipped" if the 2nd #CP gets
>>> converted to #DF. And fetching of the first handler insn also isn't part
>>> of exception delivery (and could cause other exceptions first, like #PF).
>> I can't make my observations of real hardware behaviour match the
>> description in the spec.
> I haven't been able to find a description at all of exception behavior
> when the exception occurs in wait-for-endbr state. There is text saying
> that #BP and #DB can occur this way, but I couldn't find anything about
> the tracker state changes in such cases. While I could see the state to
> remain engaged (requiring an ENDBR at the handler's entry point), I
> cannot see how the state would get re-engaged upon IRET from the
> exception handler, unless the return is back to CPL3.

Critically, there are two wait-for-endbr states.  One in MSR_U_CET and
one in MSR_S_CET, and the active one is dependent on CPL.

Interrupting CPL3 does leave the state visible (and frozen) in MSR_U_CET.

Interrupting CPL!=0 does not.  The interrupt/exception delivery
microcode forces the wait-for-endbr state which is why the entrypoints
need ENDBR64, and while I haven't confirmed yet, I'm pretty certain that
a WRMSR to MSR_S_CET which sets wait-for-endbr needs an ENDBR64
following it.

IRET does not alter the wait-for-endbr state, but does switch which of
the two trackers is active, as a side effect of changing CPL.


#CP is a decode-class fault, and takes priority over #UD and #NM. 
However, the #BP/#DB special cases are specific to the INT3/INT1
instructions, to specifically permit putting a breakpoint on an ENDBR
instruction and to take the breakpoint exception rather than #CP. 
Critically, #CP has higher priority than General Detect #DB.

(I've already got the beginnings of an XTF test case for shstk corner
cases, but it's got nothing on how complicated the IBT side of things
looks.)

>> Given what a mess it all is, I wouldn't be surprised if the exception
>> delivery microcode has a special case to escalate this to #DF.
> I am meanwhile wondering whether any exception in wait-for-endbr state
> at CPL < 3 would promote to #DF, for loss of state. Albeit there must
> still be a distinction between CALL/JMP induced state and that
> resulting from interrupt or exception delivery. Yet there's no
> architectural (or shadow) state expressing "first insn of an exception
> handler".

Architecturally, no, but there is internally by virtue of the fact that
all interrupt/exception delivery is organised by microcode.

> I'm not even convinced the aforementioned statements about #DB and #BP
> are actually meant to cover more than just CPL3, or at best ENDBR at
> normal CALL/JMP destinations.
>
> While for Xen's own use we may get away without knowing how all of this
> actually works (perhaps accepting the fact that one can't set breakpoints
> at exception handler entry points, depending on whether their delivery
> would promote to #DF), as soon as we were to support CET-IBT for guests
> we'd definitely need to know.

I'm afraid I don't follow the point you're trying to make.  INT3/INT1
are explicitly special, to let you breakpoint an ENDBR instruction, and
that really ought to mean everywhere.

Another possibility is that my observations are an errata in TGL, or
that I've made a mistake somewhere.

However, until we start getting to the point of having some real XTF
tests for behaviour, including that of the emulator, I doubt we'll get
much clarity.  Furthermore, there's a pile of work to do before that is
a possibility.  As a minimum, I need to fix up and repost my xsave
cleanup series, then implement XSAVES support for guests, before we can
make CET an opt-in feature pending completion of the emulation work.

~Andrew