Maximum QMP reply size

Maximum QMP reply size

[John Snow]

Hi, I suspect I have asked this before, but I didn't write it down in
a comment, so I forget my justification...

In the QMP lib, we need to set a buffering limit for how big a QMP
message can be -- In practice, I found that the largest possible
response was the QAPI schema reply, and I set the code to this:

    # Maximum allowable size of read buffer
    _limit = (64 * 1024)

However, I didn't document if this was a reasonable limit or just a
"worksforme" one. I assume that there's no hard limit for the protocol
or the implementation thereof in QEMU. Is there any kind of value here
that would be more sensible than another?

I'm worried that if replies get bigger in the future (possibly in some
degenerate case I am presently unaware of) that the library default
will become nonsensical.

Any pointers/tips?

--js

Re: Maximum QMP reply size

[Peter Maydell]

On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
> Hi, I suspect I have asked this before, but I didn't write it down in
> a comment, so I forget my justification...
>
> In the QMP lib, we need to set a buffering limit for how big a QMP
> message can be -- In practice, I found that the largest possible
> response was the QAPI schema reply, and I set the code to this:
>
>     # Maximum allowable size of read buffer
>     _limit = (64 * 1024)
>
> However, I didn't document if this was a reasonable limit or just a
> "worksforme" one. I assume that there's no hard limit for the protocol
> or the implementation thereof in QEMU. Is there any kind of value here
> that would be more sensible than another?
>
> I'm worried that if replies get bigger in the future (possibly in some
> degenerate case I am presently unaware of) that the library default
> will become nonsensical.

There are some QMP commands which return lists of things
where we put no inherent limit on how many things there
are in the list, like qom-list-types. We'd have to be getting
a bit enthusiastic about defining types for that to get
up towards 64K's worth of response, but it's not inherently
impossible. I think using human-monitor-command to send
an 'xp' HMP command is also a way to get back an arbitrarily
large string (just ask for a lot of memory to be dumped).

-- PMM

Re: Maximum QMP reply size

[Daniel P. Berrangé]

On Tue, Sep 06, 2022 at 03:38:54PM -0400, John Snow wrote:
> Hi, I suspect I have asked this before, but I didn't write it down in
> a comment, so I forget my justification...
> 
> In the QMP lib, we need to set a buffering limit for how big a QMP
> message can be -- In practice, I found that the largest possible
> response was the QAPI schema reply, and I set the code to this:
> 
>     # Maximum allowable size of read buffer
>     _limit = (64 * 1024)
> 
> However, I didn't document if this was a reasonable limit or just a
> "worksforme" one. I assume that there's no hard limit for the protocol
> or the implementation thereof in QEMU. Is there any kind of value here
> that would be more sensible than another?
> 
> I'm worried that if replies get bigger in the future (possibly in some
> degenerate case I am presently unaware of) that the library default
> will become nonsensical.
> 
> Any pointers/tips?

Beware of some of the blockdev commands, IIRC some of those could have
huge replies, scaling based on the depth of the backing chains. While
small in "normal" cases, people can do crazy things with 100's of
backing chains.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Maximum QMP reply size

[Daniel P. Berrangé]

On Tue, Sep 06, 2022 at 03:38:54PM -0400, John Snow wrote:
> Hi, I suspect I have asked this before, but I didn't write it down in
> a comment, so I forget my justification...
> 
> In the QMP lib, we need to set a buffering limit for how big a QMP
> message can be -- In practice, I found that the largest possible
> response was the QAPI schema reply, and I set the code to this:
> 
>     # Maximum allowable size of read buffer
>     _limit = (64 * 1024)
> 
> However, I didn't document if this was a reasonable limit or just a
> "worksforme" one. I assume that there's no hard limit for the protocol
> or the implementation thereof in QEMU. Is there any kind of value here
> that would be more sensible than another?

As a reference, libvirt arbitrarily chose 10 MB as the QMP reply
limit. It is huge enough it'll be hard to make a QMP reply exceed
that, but also still tiny in the context of managing VMs on a host
with GB's of RAM.  NB, this doesn't mean we allocate 10 MB every
time, it is just an upper bound - we only allocate what we actually
need.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Maximum QMP reply size

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Hi, I suspect I have asked this before, but I didn't write it down in
> a comment, so I forget my justification...
>
> In the QMP lib, we need to set a buffering limit for how big a QMP
> message can be -- In practice, I found that the largest possible
> response was the QAPI schema reply, and I set the code to this:
>
>     # Maximum allowable size of read buffer
>     _limit = (64 * 1024)
>
> However, I didn't document if this was a reasonable limit or just a
> "worksforme" one. I assume that there's no hard limit for the protocol
> or the implementation thereof in QEMU. Is there any kind of value here
> that would be more sensible than another?
>
> I'm worried that if replies get bigger in the future (possibly in some
> degenerate case I am presently unaware of) that the library default
> will become nonsensical.
>
> Any pointers/tips?

Peter and Daniel already provided some.  I can add a bit of insight into
how QMP output works in QEMU, which may or may not help you.

QEMU executes one command after the other.  A command's response
(success or failure) is a QDict.  Which is then formatted as JSON and
appended to the monitor's output buffer.

Events work similarly.

The conversion to JSON does not limit the resulting string's size.  If
it runs out of memory, QEMU dies.

The output buffer is also unbounded.  It drains into the monitor's
character device.

If the QMP client sends enough commands without reading their responses,
QEMU can run out of memory and die.

Now I'm ready to go back to your question, which is about a *single*
message (QMP command response or event): nothing in QEMU limits the size
of the QMP output message text.

Weak consolation: I guess QEMU is somewhat likely to run out of memory
and die before your client software does.  That's because QDict is a
pig: an empty one eats 4120 Bytes on my system.  Compares unfavourable
to its text representation "{}".

Re: Maximum QMP reply size

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
> > Hi, I suspect I have asked this before, but I didn't write it down in
> > a comment, so I forget my justification...
> >
> > In the QMP lib, we need to set a buffering limit for how big a QMP
> > message can be -- In practice, I found that the largest possible
> > response was the QAPI schema reply, and I set the code to this:
> >
> >     # Maximum allowable size of read buffer
> >     _limit = (64 * 1024)
> >
> > However, I didn't document if this was a reasonable limit or just a
> > "worksforme" one. I assume that there's no hard limit for the protocol
> > or the implementation thereof in QEMU. Is there any kind of value here
> > that would be more sensible than another?
> >
> > I'm worried that if replies get bigger in the future (possibly in some
> > degenerate case I am presently unaware of) that the library default
> > will become nonsensical.
> 
> There are some QMP commands which return lists of things
> where we put no inherent limit on how many things there
> are in the list, like qom-list-types. We'd have to be getting
> a bit enthusiastic about defining types for that to get
> up towards 64K's worth of response, but it's not inherently
> impossible. I think using human-monitor-command to send
> an 'xp' HMP command is also a way to get back an arbitrarily
> large string (just ask for a lot of memory to be dumped).

We could put size limits on xp; most Humans will only dump a few kB
maximum like that, any larger and you can dump to file.

Dave

> -- PMM
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Maximum QMP reply size

[Peter Maydell]

On Thu, 15 Sept 2022 at 16:21, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
>
> * Peter Maydell (peter.maydell@linaro.org) wrote:
> > On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
> > > Hi, I suspect I have asked this before, but I didn't write it down in
> > > a comment, so I forget my justification...
> > >
> > > In the QMP lib, we need to set a buffering limit for how big a QMP
> > > message can be -- In practice, I found that the largest possible
> > > response was the QAPI schema reply, and I set the code to this:
> > >
> > >     # Maximum allowable size of read buffer
> > >     _limit = (64 * 1024)
> > >
> > > However, I didn't document if this was a reasonable limit or just a
> > > "worksforme" one. I assume that there's no hard limit for the protocol
> > > or the implementation thereof in QEMU. Is there any kind of value here
> > > that would be more sensible than another?
> > >
> > > I'm worried that if replies get bigger in the future (possibly in some
> > > degenerate case I am presently unaware of) that the library default
> > > will become nonsensical.
> >
> > There are some QMP commands which return lists of things
> > where we put no inherent limit on how many things there
> > are in the list, like qom-list-types. We'd have to be getting
> > a bit enthusiastic about defining types for that to get
> > up towards 64K's worth of response, but it's not inherently
> > impossible. I think using human-monitor-command to send
> > an 'xp' HMP command is also a way to get back an arbitrarily
> > large string (just ask for a lot of memory to be dumped).
>
> We could put size limits on xp; most Humans will only dump a few kB
> maximum like that, any larger and you can dump to file.

Sure, we could, but why? It's not clear to me why a consumer
of QMP needs to impose a maximum message size limit on it:
I thought it was just JSON. Fixed buffer sizes are very 1980s :-)

If this is a common requirement then should we define something
in the protocol where the client says "I can support messages
up to this big" and then the server has to split things up?

-- PMM

Re: Maximum QMP reply size

[Daniel P. Berrangé]

On Fri, Sep 16, 2022 at 04:12:00PM +0100, Peter Maydell wrote:
> On Thu, 15 Sept 2022 at 16:21, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >
> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> > > On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
> > > > Hi, I suspect I have asked this before, but I didn't write it down in
> > > > a comment, so I forget my justification...
> > > >
> > > > In the QMP lib, we need to set a buffering limit for how big a QMP
> > > > message can be -- In practice, I found that the largest possible
> > > > response was the QAPI schema reply, and I set the code to this:
> > > >
> > > >     # Maximum allowable size of read buffer
> > > >     _limit = (64 * 1024)
> > > >
> > > > However, I didn't document if this was a reasonable limit or just a
> > > > "worksforme" one. I assume that there's no hard limit for the protocol
> > > > or the implementation thereof in QEMU. Is there any kind of value here
> > > > that would be more sensible than another?
> > > >
> > > > I'm worried that if replies get bigger in the future (possibly in some
> > > > degenerate case I am presently unaware of) that the library default
> > > > will become nonsensical.
> > >
> > > There are some QMP commands which return lists of things
> > > where we put no inherent limit on how many things there
> > > are in the list, like qom-list-types. We'd have to be getting
> > > a bit enthusiastic about defining types for that to get
> > > up towards 64K's worth of response, but it's not inherently
> > > impossible. I think using human-monitor-command to send
> > > an 'xp' HMP command is also a way to get back an arbitrarily
> > > large string (just ask for a lot of memory to be dumped).
> >
> > We could put size limits on xp; most Humans will only dump a few kB
> > maximum like that, any larger and you can dump to file.
> 
> Sure, we could, but why? It's not clear to me why a consumer
> of QMP needs to impose a maximum message size limit on it:
> I thought it was just JSON. Fixed buffer sizes are very 1980s :-)

Well even if they parse the JSON as it streams in, rather than
reading the whole doc and then parsing it in one go, you still
need to have limits on any sane QMP client.

The QEMU process is an untrusted component in the stack, talking
to a trusted mgmt layer. If the QEMU process sends a 1 TB JSON
message as a QMP reply, the mgmt layer must not try to parse
that as they'll let loose the kraken^H^H^H^H^H OOM killer.

To be robust against either a malicious or mis-behaving QEMU
they need to impose a limit on the size of QMP response they'll
be willing to process. The challenge is figuring out what limit
is big enough to handle any conceivable valid message, while
being small enough to minimize denial of service risks.

NB, that's not the only thing clients need todo to protect from
a bad QEMU. Rate limiting consumption is potentially important too
lest a bad QEMU inflict a DoS on the CPU by sending such frequent
messages that the QMP client is burning 100% CPU just parsing
them.  I've not seen any QMP client do this in practice though,
not even libvirt has attempted it.

> If this is a common requirement then should we define something
> in the protocol where the client says "I can support messages
> up to this big" and then the server has to split things up?

Splitting doesn't help protect against the DoS, because the QMP
client would have to reassemble the pieces afterwards to process
the reply / async event.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Maximum QMP reply size

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Fri, Sep 16, 2022 at 04:12:00PM +0100, Peter Maydell wrote:
>> On Thu, 15 Sept 2022 at 16:21, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> >
>> > * Peter Maydell (peter.maydell@linaro.org) wrote:
>> > > On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
>> > > > Hi, I suspect I have asked this before, but I didn't write it down in
>> > > > a comment, so I forget my justification...
>> > > >
>> > > > In the QMP lib, we need to set a buffering limit for how big a QMP
>> > > > message can be -- In practice, I found that the largest possible
>> > > > response was the QAPI schema reply, and I set the code to this:
>> > > >
>> > > >     # Maximum allowable size of read buffer
>> > > >     _limit = (64 * 1024)
>> > > >
>> > > > However, I didn't document if this was a reasonable limit or just a
>> > > > "worksforme" one. I assume that there's no hard limit for the protocol
>> > > > or the implementation thereof in QEMU. Is there any kind of value here
>> > > > that would be more sensible than another?
>> > > >
>> > > > I'm worried that if replies get bigger in the future (possibly in some
>> > > > degenerate case I am presently unaware of) that the library default
>> > > > will become nonsensical.
>> > >
>> > > There are some QMP commands which return lists of things
>> > > where we put no inherent limit on how many things there
>> > > are in the list, like qom-list-types. We'd have to be getting
>> > > a bit enthusiastic about defining types for that to get
>> > > up towards 64K's worth of response, but it's not inherently
>> > > impossible. I think using human-monitor-command to send
>> > > an 'xp' HMP command is also a way to get back an arbitrarily
>> > > large string (just ask for a lot of memory to be dumped).
>> >
>> > We could put size limits on xp; most Humans will only dump a few kB
>> > maximum like that, any larger and you can dump to file.
>> 
>> Sure, we could, but why? It's not clear to me why a consumer
>> of QMP needs to impose a maximum message size limit on it:
>> I thought it was just JSON. Fixed buffer sizes are very 1980s :-)
>
> Well even if they parse the JSON as it streams in, rather than
> reading the whole doc and then parsing it in one go, you still
> need to have limits on any sane QMP client.
>
> The QEMU process is an untrusted component in the stack, talking
> to a trusted mgmt layer. If the QEMU process sends a 1 TB JSON
> message as a QMP reply, the mgmt layer must not try to parse
> that as they'll let loose the kraken^H^H^H^H^H OOM killer.
>
> To be robust against either a malicious or mis-behaving QEMU
> they need to impose a limit on the size of QMP response they'll
> be willing to process. The challenge is figuring out what limit
> is big enough to handle any conceivable valid message, while
> being small enough to minimize denial of service risks.

Yes.

QEMU does this for QMP input.

Trying to defend against malicious QMP input would of course be
pointless; if you can send QMP input, you "own" QEMU anyway.  It's
defense against *accidents*.

The limits are extremely (overly?) generous: each command is limited to
1024 levels of nesting to protect the stack, 64MiB of total token size
and 2Mi[*] tokens to protect the heap.

> NB, that's not the only thing clients need todo to protect from
> a bad QEMU. Rate limiting consumption is potentially important too
> lest a bad QEMU inflict a DoS on the CPU by sending such frequent
> messages that the QMP client is burning 100% CPU just parsing
> them.  I've not seen any QMP client do this in practice though,
> not even libvirt has attempted it.

What could a management application do when it detects it can't /
doesn't want to keep up with QMP output?

>> If this is a common requirement then should we define something
>> in the protocol where the client says "I can support messages
>> up to this big" and then the server has to split things up?
>
> Splitting doesn't help protect against the DoS, because the QMP
> client would have to reassemble the pieces afterwards to process
> the reply / async event.

Yes.

Can we estimate limits that should suffice?  Documenting them could help
management applications.


[*] If you're curious about this value, see commit df649835fe "qjson:
Limit number of tokens in addition to total size".

Re: Maximum QMP reply size

[Daniel P. Berrangé]

On Mon, Sep 19, 2022 at 08:45:07AM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Fri, Sep 16, 2022 at 04:12:00PM +0100, Peter Maydell wrote:
> >> On Thu, 15 Sept 2022 at 16:21, Dr. David Alan Gilbert
> >> <dgilbert@redhat.com> wrote:
> >> >
> >> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> >> > > On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
> >> > > > Hi, I suspect I have asked this before, but I didn't write it down in
> >> > > > a comment, so I forget my justification...
> >> > > >
> >> > > > In the QMP lib, we need to set a buffering limit for how big a QMP
> >> > > > message can be -- In practice, I found that the largest possible
> >> > > > response was the QAPI schema reply, and I set the code to this:
> >> > > >
> >> > > >     # Maximum allowable size of read buffer
> >> > > >     _limit = (64 * 1024)
> >> > > >
> >> > > > However, I didn't document if this was a reasonable limit or just a
> >> > > > "worksforme" one. I assume that there's no hard limit for the protocol
> >> > > > or the implementation thereof in QEMU. Is there any kind of value here
> >> > > > that would be more sensible than another?
> >> > > >
> >> > > > I'm worried that if replies get bigger in the future (possibly in some
> >> > > > degenerate case I am presently unaware of) that the library default
> >> > > > will become nonsensical.
> >> > >
> >> > > There are some QMP commands which return lists of things
> >> > > where we put no inherent limit on how many things there
> >> > > are in the list, like qom-list-types. We'd have to be getting
> >> > > a bit enthusiastic about defining types for that to get
> >> > > up towards 64K's worth of response, but it's not inherently
> >> > > impossible. I think using human-monitor-command to send
> >> > > an 'xp' HMP command is also a way to get back an arbitrarily
> >> > > large string (just ask for a lot of memory to be dumped).
> >> >
> >> > We could put size limits on xp; most Humans will only dump a few kB
> >> > maximum like that, any larger and you can dump to file.
> >> 
> >> Sure, we could, but why? It's not clear to me why a consumer
> >> of QMP needs to impose a maximum message size limit on it:
> >> I thought it was just JSON. Fixed buffer sizes are very 1980s :-)
> >
> > Well even if they parse the JSON as it streams in, rather than
> > reading the whole doc and then parsing it in one go, you still
> > need to have limits on any sane QMP client.
> >
> > The QEMU process is an untrusted component in the stack, talking
> > to a trusted mgmt layer. If the QEMU process sends a 1 TB JSON
> > message as a QMP reply, the mgmt layer must not try to parse
> > that as they'll let loose the kraken^H^H^H^H^H OOM killer.
> >
> > To be robust against either a malicious or mis-behaving QEMU
> > they need to impose a limit on the size of QMP response they'll
> > be willing to process. The challenge is figuring out what limit
> > is big enough to handle any conceivable valid message, while
> > being small enough to minimize denial of service risks.
> 
> Yes.
> 
> QEMU does this for QMP input.
> 
> Trying to defend against malicious QMP input would of course be
> pointless; if you can send QMP input, you "own" QEMU anyway.  It's
> defense against *accidents*.
> 
> The limits are extremely (overly?) generous: each command is limited to
> 1024 levels of nesting to protect the stack, 64MiB of total token size
> and 2Mi[*] tokens to protect the heap.
> 
> > NB, that's not the only thing clients need todo to protect from
> > a bad QEMU. Rate limiting consumption is potentially important too
> > lest a bad QEMU inflict a DoS on the CPU by sending such frequent
> > messages that the QMP client is burning 100% CPU just parsing
> > them.  I've not seen any QMP client do this in practice though,
> > not even libvirt has attempted it.
> 
> What could a management application do when it detects it can't /
> doesn't want to keep up with QMP output?

Gratuitously killing the VM is an option, but likely too harsh for
most scenarios.

Another option is to disconnect from QMP and then immediately connect
again, in the hope it was just a transient confused QEMU.

Another option would be to simply mark the VM as "broken", leave it
running but stop talking to QMP entirely. THe guest owner can still
use their VM, but can't make mgmt app changes and loose some monitoring
features. Possibly allow user to attempt a "repair" - ie disconnect
and reconnect.

Another option is to have a dedicated OS thread for handling the QMP
client side, and ensure that thread is under the same CPU usage limits
as the VM as a whole, so any excess CPU usage is accounted to the VM.

> >> If this is a common requirement then should we define something
> >> in the protocol where the client says "I can support messages
> >> up to this big" and then the server has to split things up?
> >
> > Splitting doesn't help protect against the DoS, because the QMP
> > client would have to reassemble the pieces afterwards to process
> > the reply / async event.
> 
> Yes.
> 
> Can we estimate limits that should suffice?  Documenting them could help
> management applications.

FWIW, libvirt uses 10 MB as an arbitrarily picked large limit. I'm
not aware of us having bug reports complaining this is too low thus
far.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Maximum QMP reply size

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On Thu, 15 Sept 2022 at 16:21, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >
> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> > > On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
> > > > Hi, I suspect I have asked this before, but I didn't write it down in
> > > > a comment, so I forget my justification...
> > > >
> > > > In the QMP lib, we need to set a buffering limit for how big a QMP
> > > > message can be -- In practice, I found that the largest possible
> > > > response was the QAPI schema reply, and I set the code to this:
> > > >
> > > >     # Maximum allowable size of read buffer
> > > >     _limit = (64 * 1024)
> > > >
> > > > However, I didn't document if this was a reasonable limit or just a
> > > > "worksforme" one. I assume that there's no hard limit for the protocol
> > > > or the implementation thereof in QEMU. Is there any kind of value here
> > > > that would be more sensible than another?
> > > >
> > > > I'm worried that if replies get bigger in the future (possibly in some
> > > > degenerate case I am presently unaware of) that the library default
> > > > will become nonsensical.
> > >
> > > There are some QMP commands which return lists of things
> > > where we put no inherent limit on how many things there
> > > are in the list, like qom-list-types. We'd have to be getting
> > > a bit enthusiastic about defining types for that to get
> > > up towards 64K's worth of response, but it's not inherently
> > > impossible. I think using human-monitor-command to send
> > > an 'xp' HMP command is also a way to get back an arbitrarily
> > > large string (just ask for a lot of memory to be dumped).
> >
> > We could put size limits on xp; most Humans will only dump a few kB
> > maximum like that, any larger and you can dump to file.
> 
> Sure, we could, but why? It's not clear to me why a consumer
> of QMP needs to impose a maximum message size limit on it:

That was just a suggestion if someone did want to limit it; if 'xp'
sizing is the only thing holding us back.

> I thought it was just JSON. Fixed buffer sizes are very 1980s :-)

But generic JSON is awful for this, given there are no constraints on
field ordering, a parser has to read the whole JSON message before
being able to determine if anything in it makes sense.  You can't
generally 'stream' parse it, so do end up reading into buffers.

> If this is a common requirement then should we define something
> in the protocol where the client says "I can support messages
> up to this big" and then the server has to split things up?

I fear you'll have to do that for every command type that outputs
a lot of data; doing it generally could be tricky.

Dave

> -- PMM
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Maximum QMP reply size

[John Snow]

On Wed, Sep 7, 2022 at 7:54 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > Hi, I suspect I have asked this before, but I didn't write it down in
> > a comment, so I forget my justification...
> >
> > In the QMP lib, we need to set a buffering limit for how big a QMP
> > message can be -- In practice, I found that the largest possible
> > response was the QAPI schema reply, and I set the code to this:
> >
> >     # Maximum allowable size of read buffer
> >     _limit = (64 * 1024)
> >
> > However, I didn't document if this was a reasonable limit or just a
> > "worksforme" one. I assume that there's no hard limit for the protocol
> > or the implementation thereof in QEMU. Is there any kind of value here
> > that would be more sensible than another?
> >
> > I'm worried that if replies get bigger in the future (possibly in some
> > degenerate case I am presently unaware of) that the library default
> > will become nonsensical.
> >
> > Any pointers/tips?
>
> Peter and Daniel already provided some.  I can add a bit of insight into
> how QMP output works in QEMU, which may or may not help you.
>
> QEMU executes one command after the other.  A command's response
> (success or failure) is a QDict.  Which is then formatted as JSON and
> appended to the monitor's output buffer.
>
> Events work similarly.
>
> The conversion to JSON does not limit the resulting string's size.  If
> it runs out of memory, QEMU dies.
>
> The output buffer is also unbounded.  It drains into the monitor's
> character device.
>
> If the QMP client sends enough commands without reading their responses,
> QEMU can run out of memory and die.
>
> Now I'm ready to go back to your question, which is about a *single*
> message (QMP command response or event): nothing in QEMU limits the size
> of the QMP output message text.
>
> Weak consolation: I guess QEMU is somewhat likely to run out of memory
> and die before your client software does.  That's because QDict is a
> pig: an empty one eats 4120 Bytes on my system.  Compares unfavourable
> to its text representation "{}".
>

(Oops, I realize that my response was never sent, sending that now:)

Thanks for the responses, everyone.

I think I will leave it at 64KB for now, but the limit is absolutely
configurable; I will just document what the limit is and document how
to change it in the case you want to use QMP to do some really heavy
lifting. In practice, there's no unit test in our tree currently that
seems to blow through the 64KB, but I'll just make sure to pay some
attention to it in the docs.

... Or, maybe I'll set it to 10MB to match libvirt ("Well, it's good
enough for this other project" is always a fantastic justification),
but I need to profile how Python actually behaves in this case. If
it's just an upper-bound, I think that's no problem at all.

Thanks!

--js

PS: After reading further discussion that has happened since: The
problem is the readline buffer size in Python; since we don't have a
"streaming" JSON parser, we rely on readline to get "the next chunk of
data", and that buffer has a limit it adheres to. It is possible that
in the future if we switch to a different parsing method that this
limit would "go away", but it's not clear that this is a great idea.
In practice, it's probably reasonable that the client dies on input of
a certain size. Probably.

Re: Maximum QMP reply size

[Daniel P. Berrangé]

On Wed, Sep 07, 2022 at 01:54:05PM +0200, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
> > Hi, I suspect I have asked this before, but I didn't write it down in
> > a comment, so I forget my justification...
> >
> > In the QMP lib, we need to set a buffering limit for how big a QMP
> > message can be -- In practice, I found that the largest possible
> > response was the QAPI schema reply, and I set the code to this:
> >
> >     # Maximum allowable size of read buffer
> >     _limit = (64 * 1024)
> >
> > However, I didn't document if this was a reasonable limit or just a
> > "worksforme" one. I assume that there's no hard limit for the protocol
> > or the implementation thereof in QEMU. Is there any kind of value here
> > that would be more sensible than another?
> >
> > I'm worried that if replies get bigger in the future (possibly in some
> > degenerate case I am presently unaware of) that the library default
> > will become nonsensical.
> >
> > Any pointers/tips?
> 
> Peter and Daniel already provided some.  I can add a bit of insight into
> how QMP output works in QEMU, which may or may not help you.
> 
> QEMU executes one command after the other.  A command's response
> (success or failure) is a QDict.  Which is then formatted as JSON and
> appended to the monitor's output buffer.
> 
> Events work similarly.
> 
> The conversion to JSON does not limit the resulting string's size.  If
> it runs out of memory, QEMU dies.
> 
> The output buffer is also unbounded.  It drains into the monitor's
> character device.
> 
> If the QMP client sends enough commands without reading their responses,
> QEMU can run out of memory and die.
> 
> Now I'm ready to go back to your question, which is about a *single*
> message (QMP command response or event): nothing in QEMU limits the size
> of the QMP output message text.
> 
> Weak consolation: I guess QEMU is somewhat likely to run out of memory
> and die before your client software does.  That's because QDict is a
> pig: an empty one eats 4120 Bytes on my system.  Compares unfavourable
> to its text representation "{}".

A malicious QEMU that's trying to attack the mgmt software client
wouldn't need to use QDict, so that's only consolation against
accidents. An evil QEMU would just write JSON directly onto the
monitor chardev. It wouldn't even have to be well formed JSON,
as it could just start a string and never end it.

 {"blah..repeated for 1 TB for data...."

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Maximum QMP reply size

[Daniel P. Berrangé]

On Fri, Sep 23, 2022 at 03:51:48PM -0400, John Snow wrote:
> On Wed, Sep 7, 2022 at 7:54 AM Markus Armbruster <armbru@redhat.com> wrote:
> >
> > John Snow <jsnow@redhat.com> writes:
> >
> > > Hi, I suspect I have asked this before, but I didn't write it down in
> > > a comment, so I forget my justification...
> > >
> > > In the QMP lib, we need to set a buffering limit for how big a QMP
> > > message can be -- In practice, I found that the largest possible
> > > response was the QAPI schema reply, and I set the code to this:
> > >
> > >     # Maximum allowable size of read buffer
> > >     _limit = (64 * 1024)
> > >
> > > However, I didn't document if this was a reasonable limit or just a
> > > "worksforme" one. I assume that there's no hard limit for the protocol
> > > or the implementation thereof in QEMU. Is there any kind of value here
> > > that would be more sensible than another?
> > >
> > > I'm worried that if replies get bigger in the future (possibly in some
> > > degenerate case I am presently unaware of) that the library default
> > > will become nonsensical.
> > >
> > > Any pointers/tips?
> >
> > Peter and Daniel already provided some.  I can add a bit of insight into
> > how QMP output works in QEMU, which may or may not help you.
> >
> > QEMU executes one command after the other.  A command's response
> > (success or failure) is a QDict.  Which is then formatted as JSON and
> > appended to the monitor's output buffer.
> >
> > Events work similarly.
> >
> > The conversion to JSON does not limit the resulting string's size.  If
> > it runs out of memory, QEMU dies.
> >
> > The output buffer is also unbounded.  It drains into the monitor's
> > character device.
> >
> > If the QMP client sends enough commands without reading their responses,
> > QEMU can run out of memory and die.
> >
> > Now I'm ready to go back to your question, which is about a *single*
> > message (QMP command response or event): nothing in QEMU limits the size
> > of the QMP output message text.
> >
> > Weak consolation: I guess QEMU is somewhat likely to run out of memory
> > and die before your client software does.  That's because QDict is a
> > pig: an empty one eats 4120 Bytes on my system.  Compares unfavourable
> > to its text representation "{}".
> >
> 
> (Oops, I realize that my response was never sent, sending that now:)
> 
> Thanks for the responses, everyone.
> 
> I think I will leave it at 64KB for now, but the limit is absolutely
> configurable; I will just document what the limit is and document how
> to change it in the case you want to use QMP to do some really heavy
> lifting. In practice, there's no unit test in our tree currently that
> seems to blow through the 64KB, but I'll just make sure to pay some
> attention to it in the docs.
>
> ... Or, maybe I'll set it to 10MB to match libvirt ("Well, it's good
> enough for this other project" is always a fantastic justification),
> but I need to profile how Python actually behaves in this case. If
> it's just an upper-bound, I think that's no problem at all.

I feel like we should have a spec update that gives some guidance for
conforming impls in a few areas now. The current spec as written is
quite flexible, but in practice impls have made some assumptions to
simplify life, and we ought to try to include those as recomendations
to keep alignment. I feel this is especially valuable now that we're
talking about providing many official QEMU language bindings. On my
list so far I see

  1. Maximum message size
  2. Maximum structure nesting
  2. Messages terminated by a newline


> PS: After reading further discussion that has happened since: The
> problem is the readline buffer size in Python; since we don't have a
> "streaming" JSON parser, we rely on readline to get "the next chunk of
> data", and that buffer has a limit it adheres to. It is possible that
> in the future if we switch to a different parsing method that this
> limit would "go away", but it's not clear that this is a great idea.
> In practice, it's probably reasonable that the client dies on input of
> a certain size. Probably.

AFAIK, the readline method does not limit itself by default, it will
just read until newline. It is unsafe to use readline on untrusted
input sources unless you pass the 'limit' parameter to set an upper
limit for line length.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Maximum QMP reply size

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Sep 07, 2022 at 01:54:05PM +0200, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>> > Hi, I suspect I have asked this before, but I didn't write it down in
>> > a comment, so I forget my justification...
>> >
>> > In the QMP lib, we need to set a buffering limit for how big a QMP
>> > message can be -- In practice, I found that the largest possible
>> > response was the QAPI schema reply, and I set the code to this:
>> >
>> >     # Maximum allowable size of read buffer
>> >     _limit = (64 * 1024)
>> >
>> > However, I didn't document if this was a reasonable limit or just a
>> > "worksforme" one. I assume that there's no hard limit for the protocol
>> > or the implementation thereof in QEMU. Is there any kind of value here
>> > that would be more sensible than another?
>> >
>> > I'm worried that if replies get bigger in the future (possibly in some
>> > degenerate case I am presently unaware of) that the library default
>> > will become nonsensical.
>> >
>> > Any pointers/tips?
>> 
>> Peter and Daniel already provided some.  I can add a bit of insight into
>> how QMP output works in QEMU, which may or may not help you.
>> 
>> QEMU executes one command after the other.  A command's response
>> (success or failure) is a QDict.  Which is then formatted as JSON and
>> appended to the monitor's output buffer.
>> 
>> Events work similarly.
>> 
>> The conversion to JSON does not limit the resulting string's size.  If
>> it runs out of memory, QEMU dies.
>> 
>> The output buffer is also unbounded.  It drains into the monitor's
>> character device.
>> 
>> If the QMP client sends enough commands without reading their responses,
>> QEMU can run out of memory and die.
>> 
>> Now I'm ready to go back to your question, which is about a *single*
>> message (QMP command response or event): nothing in QEMU limits the size
>> of the QMP output message text.
>> 
>> Weak consolation: I guess QEMU is somewhat likely to run out of memory
>> and die before your client software does.  That's because QDict is a
>> pig: an empty one eats 4120 Bytes on my system.  Compares unfavourable
>> to its text representation "{}".
>
> A malicious QEMU that's trying to attack the mgmt software client
> wouldn't need to use QDict, so that's only consolation against
> accidents. An evil QEMU would just write JSON directly onto the
> monitor chardev. It wouldn't even have to be well formed JSON,
> as it could just start a string and never end it.
>
>  {"blah..repeated for 1 TB for data...."

Yes, a malicious QEMU should be able to flood the QMP socket with
constant memory and negligible CPU overhead.