Re: [RFC PATCH v2 2/2] KVM: s390: Extend the USER_SIGP capability

[David Hildenbrand <david@redhat.com>]

On 04.11.21 16:54, Eric Farman wrote:
> On Thu, 2021-11-04 at 15:59 +0100, David Hildenbrand wrote:
>>>> For example, we don't care about concurrent SIGP SENSE. We only
>>>> care
>>>> about "lightweight" SIGP orders with concurrent "heavy weight"
>>>> SIGP
>>>> orders.
>>>
>>> I very much care about concurrent SIGP SENSE (a "lightweight" order
>>> handled in-kernel) and how that interacts with the "heavy weight"
>>> SIGP
>>> orders (handled in userspace). SIGP SENSE might return CC0
>>> (accepted)
>>> if a vcpu is operating normally, or CC1 (status stored) with status
>>> bits indicating an external call is pending and/or the vcpu is
>>> stopped.
>>> This means that the actual response will depend on whether
>>> userspace
>>> has picked up the sigp order and processed it or not. Giving CC0
>>> when
>>> userspace is actively processing a SIGP STOP/STOP AND STORE STATUS
>>> would be misleading for the SIGP SENSE. (Did the STOP order get
>>> lost?
>>> Failed? Not yet dispatched? Blocked?)
>>
>> But that would only visible when concurrently SIGP STOP'ing from one
>> VCPU and SIGP SENSE'ing from another VCPU. But in that case, there
>> are
>> already no guarantees, because it's inherently racy:
>>
>> VCPU #2: SIGP STOP #3
>> VCPU #1: SIGP SENSE #3
>>
> 
> Is it inherently racy? QEMU has a global "one SIGP at a time,
> regardless of vcpu count" mechanism, so that it gets serialized at that
> level. POPS says an order is rejected (BUSY) if the "access path to a
> cpu is processing another order", and I would imagine that KVM is
> acting as that access path to the vcpu. The deliniation between
> kernelspace and userspace should be uninteresting on whether parallel
> orders are serialized (in QEMU via USER_SIGP) or not (!USER_SIGP or
> "lightweight" orders).

There is no real way for a guest to enforce the execution order of

VCPU #2: SIGP STOP #3
VCPU #1: SIGP SENSE #3

or

VCPU #1: SIGP SENSE #3
VCPU #2: SIGP STOP #3

without additional synchronization.

There could be random delays in the instruction execution at any point
in time. So the SENSE on #2 might observe "stopped" "not stopped" or
"busy" randomly, because it's inherently racy.


Of course, one could implement some synchronization on top:

VCPU #2: SIGP STOP #3
# VCPU #2 instructs #1 to SIGP SENSE #2
VCPU #1: SIGP SENSE #3
# VCPU #2 waits for SIGP SENSE #2 result from #1
VCPU #2: SIGP SENSE #3

Then, we have to make sure that it cannot happen that #1 observes "not
busy" and #2 observes "busy". But, to implement something like that, #2
has to execute additional instructions to perform the synchronization.

So after SIGP STOP returns on #2 and #2 was able to execute new
instructions, we have to make sure that SIGP SENSE of #3 returns "busy"
on all VCPUs until #3 finished the SIGP STOP.

> 
>> There is no guarantee who ends up first
>> a) In the kernel
>> b) On the final destination (SENSE -> kernel; STOP -> QEMU)
>>
>> They could be rescheduled/delayed in various ways.
>>
>>
>> The important part is that orders from the *same* CPU are properly
>> handled, right?
>>
>> VCPU #1: SIGP STOP #3
>> VCPU #1: SIGP SENSE #3
>>
>> SENSE must return BUSY in case the STOP was not successful yet,
>> correct?
> 
> It's not a matter of whether STOP is/not successful. If the vcpu is

Right, I meant "accepted but not fully processed yet".

> actively processing a STOP, then the SENSE gets a BUSY. But there's no
> code today to do that for the SENSE, which is of course why I'm here.
> :)

Right, and the only problematic SIGP orders are really SIGP STOP*,
because these are the only ones that will get processed asynchronously
-- the sending VCPU can return and execute new instructions without the
SIGP STOP order being fully processed.


-- 
Thanks,

David / dhildenb