Re: [RFC PATCH v2 2/2] KVM: s390: Extend the USER_SIGP capability

[Eric Farman <farman@linux.ibm.com>]

On Thu, 2021-11-04 at 15:59 +0100, David Hildenbrand wrote:
> > > For example, we don't care about concurrent SIGP SENSE. We only
> > > care
> > > about "lightweight" SIGP orders with concurrent "heavy weight"
> > > SIGP
> > > orders.
> > 
> > I very much care about concurrent SIGP SENSE (a "lightweight" order
> > handled in-kernel) and how that interacts with the "heavy weight"
> > SIGP
> > orders (handled in userspace). SIGP SENSE might return CC0
> > (accepted)
> > if a vcpu is operating normally, or CC1 (status stored) with status
> > bits indicating an external call is pending and/or the vcpu is
> > stopped.
> > This means that the actual response will depend on whether
> > userspace
> > has picked up the sigp order and processed it or not. Giving CC0
> > when
> > userspace is actively processing a SIGP STOP/STOP AND STORE STATUS
> > would be misleading for the SIGP SENSE. (Did the STOP order get
> > lost?
> > Failed? Not yet dispatched? Blocked?)
> 
> But that would only visible when concurrently SIGP STOP'ing from one
> VCPU and SIGP SENSE'ing from another VCPU. But in that case, there
> are
> already no guarantees, because it's inherently racy:
> 
> VCPU #2: SIGP STOP #3
> VCPU #1: SIGP SENSE #3
> 

Is it inherently racy? QEMU has a global "one SIGP at a time,
regardless of vcpu count" mechanism, so that it gets serialized at that
level. POPS says an order is rejected (BUSY) if the "access path to a
cpu is processing another order", and I would imagine that KVM is
acting as that access path to the vcpu. The deliniation between
kernelspace and userspace should be uninteresting on whether parallel
orders are serialized (in QEMU via USER_SIGP) or not (!USER_SIGP or
"lightweight" orders).

> There is no guarantee who ends up first
> a) In the kernel
> b) On the final destination (SENSE -> kernel; STOP -> QEMU)
> 
> They could be rescheduled/delayed in various ways.
> 
> 
> The important part is that orders from the *same* CPU are properly
> handled, right?
> 
> VCPU #1: SIGP STOP #3
> VCPU #1: SIGP SENSE #3
> 
> SENSE must return BUSY in case the STOP was not successful yet,
> correct?

It's not a matter of whether STOP is/not successful. If the vcpu is
actively processing a STOP, then the SENSE gets a BUSY. But there's no
code today to do that for the SENSE, which is of course why I'm here.
:)

> 
> And that can be achieved by setting the VCPU #3 busy when landing in
> user space to trigger the SIGP STOP, before returning to the kernel
> and
> processing the SIGP SENSE.
> 

I will try it, but I am not convinced.

> 
> Or am I missing something important?
> 
> > Meanwhile, the Principles of Operation (SA22-7832-12) page 4-95
> > describes a list of orders that would generate a CC2 (busy) when
> > the
> > order is still "active" in userspace:
> > 
> > """
> > A previously issued start, stop, restart, stop-
> > and-store-status, set-prefix, store-status-at-
> > address order, or store-additional-status-at-
> > address has been accepted by the
> > addressed CPU, and execution of the func-
> > tion requested by the order has not yet been
> > completed.
> 
> Right, but my take is that the order has not been accepted by the
> target
> CPU before we're actually in user space to e.g., trigger SIGP STOP.

Not accepted, yes, but also not rejected either. We're still trying to
figure out who's processing the order and getting it to the addressed
cpu.

> 
> > ...
> > If the currently specified order is sense, external
> > call, emergency signal, start, stop, restart, stop
> > and store status, set prefix, store status at
> > address, set architecture, set multithreading, or
> > store additional status at address, then the order
> > is rejected, and condition code 2 is set. If the cur-
> > rently specified order is one of the reset orders,
> > or an unassigned or not-implemented order, the
> > order code is interpreted as described in “Status
> > Bits” on page 4-96.
> > """
> > 
> > (There is another entry for the reset orders; not copied here for
> > sake
> > of keeping my novella manageable.)
> 
> Yes, these have to be special because we can have CPUs that never
> stop
> (endless program interruption stream).
> 
> > So, you're right that I could be more precise in terms how QEMU
> > handles
> > a SIGP order while it's already busy handling one, and only limit
> > the
> > CC2 from the kernel to those in-kernel orders. But I did say I took
> > this simplified approach in the cover letter. :)
> > 
> > Regardless, because of the above I really do want/need a way to
> > give
> > the kernel a clue that userspace is doing something, without
> > waiting
> > for userspace to say "hey, that order you kicked back to me? I'm
> > working on it now, I'll let you know when it's done!" Otherwise,
> > SIGP
> > SENSE (and other lightweight friends) is still racing with the
> > receipt
> > of a "start the sigp" ioctl.
> 
> And my point is that it's only visible when two VCPUs are involved
> and
> there are absolutely no guarantees regarding that. (see my first
> reply)
> 
>