Re: [Qemu-devel] [PATCH v7 4/7] qapi: remove COMMAND_DROPPED event

[Markus Armbruster <armbru@redhat.com>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Sep 03, 2018 at 09:30:52AM -0500, Eric Blake wrote:
>> On 09/03/2018 08:31 AM, Markus Armbruster wrote:
>> 
>> > Example:
>> > 
>> >      client sends in-band command #1
>> >      QEMU reads and queues
>> >      QEMU dequeues in-band command #1
>> >      in-band command #1 starts executing, but it's slooow
>> >      client sends in-band command #2
>> >      QEMU reads and queues
>> >      ...
>> >      client sends in-band command #8
>> >      QEMU reads, queues and suspends the monitor
>> >      client sends out-of-band command
>> > --> time passes...
>> >      in-band command #1 completes, QEMU sends reply
>> >      QEMU dequeues in-band command #2, resumes the monitor
>> >      in-band command #2 starts executing
>> >      QEMU reads and executes out-of-band command
>> >      out-of-band command completes, QEMU sends reply
>> >      in-band command #2 completes, QEMU sends reply
>> >      ... same for remaining in-band commands ...
>> > 
>> > The out-of-band command gets stuck behind the in-band commands.
>> > 
>> > The client can avoid this by managing the flow of in-band commands: have
>> > no more than 7 in flight, so the monitor never gets suspended.
>> > 
>> > This is a potentially useful thing to do for clients, isn't it?
>> > 
>> > Eric, Daniel, is it something libvirt would do?
>> 
>> Right now, libvirt serializes commands - it never sends a QMP command until
>> the previous command's response has been processed. But that may not help
>> much, since libvirt does not send OOB commands.
>
> Note that is not merely due to the QMP monitor restriction either.
>
> Libvirt serializes all its public APIs that can change state of a running
> domain.  It usually aims to allow read-only APIs to be run in parallel with
> APIs that change state.

Makes sense.  State changes are complex enough without concurrency.
Even permitting just concurrent queries can add non-trivial complexity.

However, pipelineing != concurrency.  "Serializing" as I understand it
implies no concurrency, it doesn't imply no pipelining.

Mind, I'm not telling you to pipeline, I'm just trying to understand the
constraints.

> The exception to the rule right now are some of the migration APIs which
> we allow to be invoked to manage the migration process. 

Can you give an example?

>> I guess when we are designing what libvirt should do, and deciding WHEN it
>> should send OOB commands, we have the luxury of designing libvirt to enforce
>> how many in-flight in-band commands it will ever have pending at once
>> (whether the current 'at most 1', or even if we make it more parallel to 'at
>> most 7'), so that we can still be ensured that the OOB command will be
>> processed without being stuck in the queue of suspended in-band commands.
>> If we never send more than one in-band at a time, then it's not a concern
>> how deep the qemu queue is; but if we do want libvirt to start parallel
>> in-band commands, then you are right that having a way to learn the qemu
>> queue depth is programmatically more precise than just guessing the maximum
>> depth.  But it's also hard to argue we need that complexity if we don't have
>> an immediate use envisioned for it.

True.

Options for the initial interface:

(1) Provide means for the client to determine the queue length limit
    (introspection or configuration).  Clients that need the monitory to
    remain available for out-of-band commands can keep limit - 1 in-band
    commands in flight.

(2) Make the queue length limit part of the documented interface.
    Clients that need the monitory to remain available for out-of-band
    commands can keep limit - 1 in-band commands in flight.  We can
    increase the limit later, but not decrease it.  We can also switch
    to (1) as needed.

(3) Treat the queue length limit as implementation detail (but tacitly
    assume its at least 2, since less makes no sense[*]).  Clients that
    need the monitory to remain available for out-of-band commands
    cannot safely keep more than one in-band command in flight.  We can
    switch to (2) or (1) as needed.

Opinions?

> In terms of what libvirt would want to parallelize, I think it is reasonable
> to consider any of the query-XXXX commands desirable. Other stuff is likely
> to remain serialized from libvirt's side.

QEMU doesn't care whether it queues in-band query commands or other
in-band commands.


[*] With a queue length of 1, the monitor suspends on receipt of an
in-band command, and resumes after it completes.  This renders
out-of-band commands useless: there is no way an out-of-band command can
overtake an in-band command.