* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-03-16 9:54 [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu Damien Hedde
@ 2022-03-16 10:24 ` Daniel P. Berrangé
2022-03-16 14:16 ` Damien Hedde
2022-04-05 5:41 ` Markus Armbruster
2022-04-04 20:34 ` John Snow
2022-05-25 16:06 ` Daniel P. Berrangé
2 siblings, 2 replies; 14+ messages in thread
From: Daniel P. Berrangé @ 2022-03-16 10:24 UTC (permalink / raw)
To: Damien Hedde
Cc: Markus Armbruster, Beraldo Leal, John Snow, qemu-devel, Cleber Rosa
On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
> It takes an input file containing raw qmp commands (concatenated json
> dicts) and send all commands one by one to a qmp server. When one
> command fails, it exits.
>
> As a convenience, it can also wrap the qemu process to avoid having
> to start qemu in background. When wrapping qemu, the program returns
> only when the qemu process terminates.
>
> Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
> ---
>
> Hi all,
>
> Following our discussion, I've started this. What do you think ?
>
> I tried to follow Daniel's qmp-shell-wrap. I think it is
> better to have similar options (eg: logging). There is also room
> for factorizing code if we want to keep them aligned and ease
> maintenance.
Having CLI similarity to the existing scripts is a good idea.
As a proof of usefulness, it might be worth trying to illustrate
this qmp-send command by converting an I/O test.
Quite a few I/O tests have code that look like:
do_run_qemu()
{
echo Testing: "$@" | _filter_imgfmt
$QEMU -nographic -qmp stdio -serial none "$@"
echo
}
run_qemu()
{
do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qemu | _filter_qmp | _filter_qemu_io
}
run_qemu <<EOF
{ "execute": "qmp_capabilities" }
{ "execute": "blockdev-add",
....
}
{ "execute": "quit" }
EOF
(eg iotests 71)
I would hope this qmp-send command to be able to satisfy that
use case by modifying do_run_qemu like this:
do_run_qemu()
{
echo Testing: "$@" | _filter_imgfmt
qmp-send --wrap $QEMU -nographic -serial none "$@"
echo
}
> There are still some pylint issues (too many branches in main and it
> does not like my context manager if else line). But it's kind of a
> mess to fix theses so I think it's enough for a first version.
>
> I name that qmp-send as Daniel proposed, maybe qmp-test matches better
> what I'm doing there ?
'qmp-test' is a use case specific name. I think it is better to
name it based on functionality provided rather than anticipated
use case, since use cases evolve over time, hence 'qmp-send'.
>
> Thanks,
> Damien
> ---
> python/qemu/aqmp/qmp_send.py | 229 +++++++++++++++++++++++++++++++++++
> scripts/qmp/qmp-send | 11 ++
> 2 files changed, 240 insertions(+)
> create mode 100644 python/qemu/aqmp/qmp_send.py
> create mode 100755 scripts/qmp/qmp-send
>
> diff --git a/python/qemu/aqmp/qmp_send.py b/python/qemu/aqmp/qmp_send.py
> new file mode 100644
> index 0000000000..cbca1d0205
> --- /dev/null
> +++ b/python/qemu/aqmp/qmp_send.py
> @@ -0,0 +1,229 @@
> +#
> +# Copyright (C) 2022 Greensocs
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +#
> +
> +"""
> +usage: qmp-send [-h] [-f FILE] [-s SOCKET] [-v] [-p] [--wrap ...]
> +
> +Send raw qmp commands to qemu as long as they succeed. It either connects to a
> +remote qmp server using the provided socket or wrap the qemu process. It stops
> +sending the provided commands when a command fails (disconnection or error
> +response).
> +
> +optional arguments:
> + -h, --help show this help message and exit
> + -f FILE, --file FILE Input file containing the commands
> + -s SOCKET, --socket SOCKET
> + < UNIX socket path | TCP address:port >
> + -v, --verbose Verbose (echo commands sent and received)
> + -p, --pretty Pretty-print JSON
> + --wrap ... QEMU command line to invoke
> +
> +When qemu wrap option is used, this script waits for qemu to terminate but
> +never send any quit or kill command. This needs to be done manually.
> +"""
> +
> +import argparse
> +import contextlib
> +import json
> +import logging
> +import os
> +from subprocess import Popen
> +import sys
> +from typing import List, TextIO
> +
> +from qemu.aqmp import ConnectError, QMPError, SocketAddrT
> +from qemu.aqmp.legacy import (
> + QEMUMonitorProtocol,
> + QMPBadPortError,
> + QMPMessage,
> +)
> +
> +
> +LOG = logging.getLogger(__name__)
> +
> +
> +class QmpRawDecodeError(Exception):
> + """
> + Exception for raw qmp decoding
> +
> + msg: exception message
> + lineno: input line of the error
> + colno: input column of the error
> + """
> + def __init__(self, msg: str, lineno: int, colno: int):
> + self.msg = msg
> + self.lineno = lineno
> + self.colno = colno
> + super().__init__(f"{msg}: line {lineno} column {colno}")
> +
> +
> +class QMPSendError(QMPError):
> + """
> + QMP Send Base error class.
> + """
> +
> +
> +class QMPSend(QEMUMonitorProtocol):
> + """
> + QMP Send class.
> + """
> + def __init__(self, address: SocketAddrT,
> + pretty: bool = False,
> + verbose: bool = False,
> + server: bool = False):
> + super().__init__(address, server=server)
> + self._verbose = verbose
> + self._pretty = pretty
> + self._server = server
> +
> + def setup_connection(self) -> None:
> + """Setup the connetion with the remote client/server."""
> + if self._server:
> + self.accept()
> + else:
> + self.connect()
> +
> + def _print(self, qmp_message: object) -> None:
> + jsobj = json.dumps(qmp_message,
> + indent=4 if self._pretty else None,
> + sort_keys=self._pretty)
> + print(str(jsobj))
> +
> + def execute_cmd(self, cmd: QMPMessage) -> None:
> + """Execute a qmp command."""
> + if self._verbose:
> + self._print(cmd)
> + resp = self.cmd_obj(cmd)
> + if resp is None:
> + raise QMPSendError("Disconnected")
> + if self._verbose:
> + self._print(resp)
> + if 'error' in resp:
> + raise QMPSendError(f"Command failed: {resp['error']}")
> +
> +
> +def raw_load(file: TextIO) -> List[QMPMessage]:
> + """parse a raw qmp command file.
> +
> + JSON formatted commands can expand on several lines but must
> + be separated by an end-of-line (two commands can not share the
> + same line).
> + File must not end with empty lines.
> + """
> + cmds: List[QMPMessage] = []
> + linecnt = 0
> + while True:
> + buf = file.readline()
> + if not buf:
> + return cmds
> + prev_err_pos = None
> + buf_linecnt = 1
> + while True:
> + try:
> + cmds.append(json.loads(buf))
> + break
> + except json.JSONDecodeError as err:
> + if prev_err_pos == err.pos:
> + # adding a line made no progress so
> + # + either we're at EOF and json data is truncated
> + # + or the parsing error is before
> + raise QmpRawDecodeError(err.msg, linecnt + err.lineno,
> + err.colno) from err
> + prev_err_pos = err.pos
> + buf += file.readline()
> + buf_linecnt += 1
> + linecnt += buf_linecnt
> +
> +
> +def report_error(msg: str) -> None:
> + """Write an error to stderr."""
> + sys.stderr.write('ERROR: %s\n' % msg)
> +
> +
> +def main() -> None:
> + """
> + qmp-send entry point: parse command line arguments and start the REPL.
> + """
> + parser = argparse.ArgumentParser(
> + description="""
> + Send raw qmp commands to qemu as long as they succeed. It either
> + connects to a remote qmp server using the provided socket or wrap
> + the qemu process. It stops sending the provided commands when a
> + command fails (disconnection or error response).
> + """,
> + epilog="""
> + When qemu wrap option is used, this script waits for qemu
> + to terminate but never send any quit or kill command. This
> + needs to be done manually.
> + """)
> +
> + parser.add_argument('-f', '--file', action='store',
> + help='Input file containing the commands')
> + parser.add_argument('-s', '--socket', action='store',
> + help='< UNIX socket path | TCP address:port >')
> + parser.add_argument('-v', '--verbose', action='store_true',
> + help='Verbose (echo commands sent and received)')
> + parser.add_argument('-p', '--pretty', action='store_true',
> + help='Pretty-print JSON')
> +
> + parser.add_argument('--wrap', nargs=argparse.REMAINDER,
> + help='QEMU command line to invoke')
> +
> + args = parser.parse_args()
> +
> + socket = args.socket
> + wrap_qemu = args.wrap is not None
> +
> + if wrap_qemu:
> + if len(args.wrap) != 0:
> + qemu_cmdline = args.wrap
> + else:
> + qemu_cmdline = ["qemu-system-x86_64"]
> + if socket is None:
> + socket = "qmp-send-wrap-%d" % os.getpid()
> + qemu_cmdline += ["-qmp", "unix:%s" % socket]
> +
> + try:
> + address = QMPSend.parse_address(socket)
> + except QMPBadPortError:
> + parser.error(f"Bad port number: {socket}")
> + return # pycharm doesn't know error() is noreturn
> +
> + try:
> + with open(args.file, mode='rt', encoding='utf8') as file:
> + qmp_cmds = raw_load(file)
> + except QmpRawDecodeError as err:
> + report_error(str(err))
> + sys.exit(1)
> +
> + try:
> + with QMPSend(address, args.pretty, args.verbose,
> + server=wrap_qemu) as qmp:
> + # starting with python 3.7 we could use contextlib.nullcontext
> + qemu = Popen(qemu_cmdline) if wrap_qemu else contextlib.suppress()
> + with qemu:
> + try:
> + qmp.setup_connection()
> + except ConnectError as err:
> + if isinstance(err.exc, OSError):
> + report_error(f"Couldn't connect to {socket}: {err!s}")
> + else:
> + report_error(str(err))
> + sys.exit(1)
> + try:
> + for cmd in qmp_cmds:
> + qmp.execute_cmd(cmd)
> + except QMPError as err:
> + report_error(str(err))
> + sys.exit(1)
> + finally:
> + if wrap_qemu:
> + os.unlink(socket)
> +
> +
> +if __name__ == '__main__':
> + main()
> diff --git a/scripts/qmp/qmp-send b/scripts/qmp/qmp-send
> new file mode 100755
> index 0000000000..8d3063797c
> --- /dev/null
> +++ b/scripts/qmp/qmp-send
> @@ -0,0 +1,11 @@
> +#!/usr/bin/env python3
> +
> +import os
> +import sys
> +
> +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
> +from qemu.aqmp import qmp_send
> +
> +
> +if __name__ == '__main__':
> + qmp_send.main()
> --
> 2.35.1
>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-03-16 10:24 ` Daniel P. Berrangé
@ 2022-03-16 14:16 ` Damien Hedde
2022-04-05 5:41 ` Markus Armbruster
1 sibling, 0 replies; 14+ messages in thread
From: Damien Hedde @ 2022-03-16 14:16 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Markus Armbruster, Beraldo Leal, John Snow, qemu-devel, Cleber Rosa
On 3/16/22 11:24, Daniel P. Berrangé wrote:
> On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
>> It takes an input file containing raw qmp commands (concatenated json
>> dicts) and send all commands one by one to a qmp server. When one
>> command fails, it exits.
>>
>> As a convenience, it can also wrap the qemu process to avoid having
>> to start qemu in background. When wrapping qemu, the program returns
>> only when the qemu process terminates.
>>
>> Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
>> ---
>>
>> Hi all,
>>
>> Following our discussion, I've started this. What do you think ?
>>
>> I tried to follow Daniel's qmp-shell-wrap. I think it is
>> better to have similar options (eg: logging). There is also room
>> for factorizing code if we want to keep them aligned and ease
>> maintenance.
>
> Having CLI similarity to the existing scripts is a good idea.
>
> As a proof of usefulness, it might be worth trying to illustrate
> this qmp-send command by converting an I/O test.
>
> Quite a few I/O tests have code that look like:
>
> do_run_qemu()
> {
> echo Testing: "$@" | _filter_imgfmt
> $QEMU -nographic -qmp stdio -serial none "$@"
> echo
> }
>
>
> run_qemu()
> {
> do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qemu | _filter_qmp | _filter_qemu_io
> }
>
> run_qemu <<EOF
> { "execute": "qmp_capabilities" }
> { "execute": "blockdev-add",
> ....
> }
> { "execute": "quit" }
> EOF
>
> (eg iotests 71)
>
> I would hope this qmp-send command to be able to satisfy that
> use case by modifying do_run_qemu like this:
>
> do_run_qemu()
> {
> echo Testing: "$@" | _filter_imgfmt
> qmp-send --wrap $QEMU -nographic -serial none "$@"
> echo
> }
I need to add stdin handling, but it should be straightforward.
I'm more worried by what should happen if there is a failure that makes
qemu hang, because then run_qemu won't exit. I'll take a look at the iotest.
I expect the test will be killed at some point, I need to ensure that
part is handled properly by qmp-send.
Thanks,
Damien
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-03-16 10:24 ` Daniel P. Berrangé
2022-03-16 14:16 ` Damien Hedde
@ 2022-04-05 5:41 ` Markus Armbruster
2022-04-05 12:45 ` Damien Hedde
1 sibling, 1 reply; 14+ messages in thread
From: Markus Armbruster @ 2022-04-05 5:41 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Damien Hedde, Beraldo Leal, John Snow, qemu-devel, Cleber Rosa
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
>> It takes an input file containing raw qmp commands (concatenated json
>> dicts) and send all commands one by one to a qmp server. When one
>> command fails, it exits.
>>
>> As a convenience, it can also wrap the qemu process to avoid having
>> to start qemu in background. When wrapping qemu, the program returns
>> only when the qemu process terminates.
>>
>> Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
[...]
>> I name that qmp-send as Daniel proposed, maybe qmp-test matches better
>> what I'm doing there ?
>
> 'qmp-test' is a use case specific name. I think it is better to
> name it based on functionality provided rather than anticipated
> use case, since use cases evolve over time, hence 'qmp-send'.
Well, it doesn't just send, it also receives.
qmpcat, like netcat and socat?
[...]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-04-05 5:41 ` Markus Armbruster
@ 2022-04-05 12:45 ` Damien Hedde
2022-04-19 17:18 ` Daniel P. Berrangé
0 siblings, 1 reply; 14+ messages in thread
From: Damien Hedde @ 2022-04-05 12:45 UTC (permalink / raw)
To: Markus Armbruster, Daniel P. Berrangé
Cc: Beraldo Leal, John Snow, qemu-devel, Cleber Rosa
On 4/5/22 07:41, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
>
>> On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
>>> It takes an input file containing raw qmp commands (concatenated json
>>> dicts) and send all commands one by one to a qmp server. When one
>>> command fails, it exits.
>>>
>>> As a convenience, it can also wrap the qemu process to avoid having
>>> to start qemu in background. When wrapping qemu, the program returns
>>> only when the qemu process terminates.
>>>
>>> Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
>
> [...]
>
>>> I name that qmp-send as Daniel proposed, maybe qmp-test matches better
>>> what I'm doing there ?
>>
>> 'qmp-test' is a use case specific name. I think it is better to
>> name it based on functionality provided rather than anticipated
>> use case, since use cases evolve over time, hence 'qmp-send'.
>
> Well, it doesn't just send, it also receives.
>
> qmpcat, like netcat and socat?
>
anyone against qmpcat ?
--
Damien
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-04-05 12:45 ` Damien Hedde
@ 2022-04-19 17:18 ` Daniel P. Berrangé
2022-04-20 6:28 ` Markus Armbruster
0 siblings, 1 reply; 14+ messages in thread
From: Daniel P. Berrangé @ 2022-04-19 17:18 UTC (permalink / raw)
To: Damien Hedde
Cc: Beraldo Leal, Cleber Rosa, John Snow, Markus Armbruster, qemu-devel
On Tue, Apr 05, 2022 at 02:45:14PM +0200, Damien Hedde wrote:
>
>
> On 4/5/22 07:41, Markus Armbruster wrote:
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> >
> > > On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
> > > > It takes an input file containing raw qmp commands (concatenated json
> > > > dicts) and send all commands one by one to a qmp server. When one
> > > > command fails, it exits.
> > > >
> > > > As a convenience, it can also wrap the qemu process to avoid having
> > > > to start qemu in background. When wrapping qemu, the program returns
> > > > only when the qemu process terminates.
> > > >
> > > > Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
> >
> > [...]
> >
> > > > I name that qmp-send as Daniel proposed, maybe qmp-test matches better
> > > > what I'm doing there ?
> > >
> > > 'qmp-test' is a use case specific name. I think it is better to
> > > name it based on functionality provided rather than anticipated
> > > use case, since use cases evolve over time, hence 'qmp-send'.
> >
> > Well, it doesn't just send, it also receives.
> >
> > qmpcat, like netcat and socat?
> >
>
> anyone against qmpcat ?
Fine with me[1], though I would have slight preference for 'qmp-cat'
to have a common tab-completion prefix with existing qmp-shell command.
With regards,
Daniel
[1] Especially if it displays a pretty ascii art cat when you pass
the --help flag ;-P
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-04-19 17:18 ` Daniel P. Berrangé
@ 2022-04-20 6:28 ` Markus Armbruster
0 siblings, 0 replies; 14+ messages in thread
From: Markus Armbruster @ 2022-04-20 6:28 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Damien Hedde, Beraldo Leal, Markus Armbruster, qemu-devel,
Cleber Rosa, John Snow
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Tue, Apr 05, 2022 at 02:45:14PM +0200, Damien Hedde wrote:
>>
>>
>> On 4/5/22 07:41, Markus Armbruster wrote:
>> > Daniel P. Berrangé <berrange@redhat.com> writes:
>> >
>> > > On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
>> > > > It takes an input file containing raw qmp commands (concatenated json
>> > > > dicts) and send all commands one by one to a qmp server. When one
>> > > > command fails, it exits.
>> > > >
>> > > > As a convenience, it can also wrap the qemu process to avoid having
>> > > > to start qemu in background. When wrapping qemu, the program returns
>> > > > only when the qemu process terminates.
>> > > >
>> > > > Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
>> >
>> > [...]
>> >
>> > > > I name that qmp-send as Daniel proposed, maybe qmp-test matches better
>> > > > what I'm doing there ?
>> > >
>> > > 'qmp-test' is a use case specific name. I think it is better to
>> > > name it based on functionality provided rather than anticipated
>> > > use case, since use cases evolve over time, hence 'qmp-send'.
>> >
>> > Well, it doesn't just send, it also receives.
>> >
>> > qmpcat, like netcat and socat?
>> >
>>
>> anyone against qmpcat ?
>
> Fine with me[1], though I would have slight preference for 'qmp-cat'
> to have a common tab-completion prefix with existing qmp-shell command.
No objections to the dash..
>
> With regards,
> Daniel
>
> [1] Especially if it displays a pretty ascii art cat when you pass
> the --help flag ;-P
A cat "singing" to an emu? Count me in!
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-03-16 9:54 [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu Damien Hedde
2022-03-16 10:24 ` Daniel P. Berrangé
@ 2022-04-04 20:34 ` John Snow
2022-04-05 9:02 ` Damien Hedde
2022-05-25 16:06 ` Daniel P. Berrangé
2 siblings, 1 reply; 14+ messages in thread
From: John Snow @ 2022-04-04 20:34 UTC (permalink / raw)
To: Damien Hedde
Cc: Markus Armbruster, Beraldo Leal, Daniel Berrange, qemu-devel,
Cleber Rosa
On Wed, Mar 16, 2022 at 5:55 AM Damien Hedde <damien.hedde@greensocs.com> wrote:
>
> It takes an input file containing raw qmp commands (concatenated json
> dicts) and send all commands one by one to a qmp server. When one
> command fails, it exits.
>
> As a convenience, it can also wrap the qemu process to avoid having
> to start qemu in background. When wrapping qemu, the program returns
> only when the qemu process terminates.
>
> Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
> ---
>
> Hi all,
>
> Following our discussion, I've started this. What do you think ?
>
> I tried to follow Daniel's qmp-shell-wrap. I think it is
> better to have similar options (eg: logging). There is also room
> for factorizing code if we want to keep them aligned and ease
> maintenance.
>
> There are still some pylint issues (too many branches in main and it
> does not like my context manager if else line). But it's kind of a
> mess to fix theses so I think it's enough for a first version.
Yeah, don't worry about these. You can just tell pylint to shut up
while you prototype. Sometimes it's just not worth spending more time
on a more beautiful factoring. Oh well.
>
> I name that qmp-send as Daniel proposed, maybe qmp-test matches better
> what I'm doing there ?
>
I think I agree with Dan's response.
> Thanks,
> Damien
> ---
> python/qemu/aqmp/qmp_send.py | 229 +++++++++++++++++++++++++++++++++++
I recommend putting this in qemu/util/qmp_send.py instead.
I'm in the process of pulling out the AQMP lib and hosting it
separately. Scripts like this I think should stay in the QEMU tree, so
moving it to util instead is probably best. Otherwise, I'll *really*
have to commit to the syntax, and that's probably a bigger hurdle than
you want to deal with.
> scripts/qmp/qmp-send | 11 ++
> 2 files changed, 240 insertions(+)
> create mode 100644 python/qemu/aqmp/qmp_send.py
> create mode 100755 scripts/qmp/qmp-send
>
> diff --git a/python/qemu/aqmp/qmp_send.py b/python/qemu/aqmp/qmp_send.py
> new file mode 100644
> index 0000000000..cbca1d0205
> --- /dev/null
> +++ b/python/qemu/aqmp/qmp_send.py
> @@ -0,0 +1,229 @@
> +#
> +# Copyright (C) 2022 Greensocs
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +#
> +
> +"""
> +usage: qmp-send [-h] [-f FILE] [-s SOCKET] [-v] [-p] [--wrap ...]
> +
> +Send raw qmp commands to qemu as long as they succeed. It either connects to a
> +remote qmp server using the provided socket or wrap the qemu process. It stops
> +sending the provided commands when a command fails (disconnection or error
> +response).
> +
> +optional arguments:
> + -h, --help show this help message and exit
> + -f FILE, --file FILE Input file containing the commands
> + -s SOCKET, --socket SOCKET
> + < UNIX socket path | TCP address:port >
> + -v, --verbose Verbose (echo commands sent and received)
> + -p, --pretty Pretty-print JSON
> + --wrap ... QEMU command line to invoke
> +
> +When qemu wrap option is used, this script waits for qemu to terminate but
> +never send any quit or kill command. This needs to be done manually.
> +"""
> +
> +import argparse
> +import contextlib
> +import json
> +import logging
> +import os
> +from subprocess import Popen
> +import sys
> +from typing import List, TextIO
> +
> +from qemu.aqmp import ConnectError, QMPError, SocketAddrT
> +from qemu.aqmp.legacy import (
> + QEMUMonitorProtocol,
> + QMPBadPortError,
> + QMPMessage,
> +)
> +
> +
> +LOG = logging.getLogger(__name__)
> +
> +
> +class QmpRawDecodeError(Exception):
> + """
> + Exception for raw qmp decoding
> +
> + msg: exception message
> + lineno: input line of the error
> + colno: input column of the error
> + """
> + def __init__(self, msg: str, lineno: int, colno: int):
> + self.msg = msg
> + self.lineno = lineno
> + self.colno = colno
> + super().__init__(f"{msg}: line {lineno} column {colno}")
> +
> +
> +class QMPSendError(QMPError):
> + """
> + QMP Send Base error class.
> + """
> +
> +
> +class QMPSend(QEMUMonitorProtocol):
> + """
> + QMP Send class.
> + """
> + def __init__(self, address: SocketAddrT,
> + pretty: bool = False,
> + verbose: bool = False,
> + server: bool = False):
> + super().__init__(address, server=server)
> + self._verbose = verbose
> + self._pretty = pretty
> + self._server = server
> +
> + def setup_connection(self) -> None:
> + """Setup the connetion with the remote client/server."""
> + if self._server:
> + self.accept()
> + else:
> + self.connect()
> +
> + def _print(self, qmp_message: object) -> None:
> + jsobj = json.dumps(qmp_message,
> + indent=4 if self._pretty else None,
> + sort_keys=self._pretty)
> + print(str(jsobj))
> +
> + def execute_cmd(self, cmd: QMPMessage) -> None:
> + """Execute a qmp command."""
> + if self._verbose:
> + self._print(cmd)
> + resp = self.cmd_obj(cmd)
> + if resp is None:
> + raise QMPSendError("Disconnected")
> + if self._verbose:
> + self._print(resp)
> + if 'error' in resp:
> + raise QMPSendError(f"Command failed: {resp['error']}")
> +
> +
> +def raw_load(file: TextIO) -> List[QMPMessage]:
> + """parse a raw qmp command file.
> +
> + JSON formatted commands can expand on several lines but must
> + be separated by an end-of-line (two commands can not share the
> + same line).
> + File must not end with empty lines.
> + """
> + cmds: List[QMPMessage] = []
> + linecnt = 0
> + while True:
> + buf = file.readline()
> + if not buf:
> + return cmds
> + prev_err_pos = None
> + buf_linecnt = 1
> + while True:
> + try:
> + cmds.append(json.loads(buf))
> + break
> + except json.JSONDecodeError as err:
> + if prev_err_pos == err.pos:
> + # adding a line made no progress so
> + # + either we're at EOF and json data is truncated
> + # + or the parsing error is before
> + raise QmpRawDecodeError(err.msg, linecnt + err.lineno,
> + err.colno) from err
> + prev_err_pos = err.pos
> + buf += file.readline()
> + buf_linecnt += 1
> + linecnt += buf_linecnt
> +
> +
> +def report_error(msg: str) -> None:
> + """Write an error to stderr."""
> + sys.stderr.write('ERROR: %s\n' % msg)
> +
> +
> +def main() -> None:
> + """
> + qmp-send entry point: parse command line arguments and start the REPL.
> + """
> + parser = argparse.ArgumentParser(
> + description="""
> + Send raw qmp commands to qemu as long as they succeed. It either
> + connects to a remote qmp server using the provided socket or wrap
> + the qemu process. It stops sending the provided commands when a
> + command fails (disconnection or error response).
> + """,
> + epilog="""
> + When qemu wrap option is used, this script waits for qemu
> + to terminate but never send any quit or kill command. This
> + needs to be done manually.
> + """)
> +
> + parser.add_argument('-f', '--file', action='store',
> + help='Input file containing the commands')
> + parser.add_argument('-s', '--socket', action='store',
> + help='< UNIX socket path | TCP address:port >')
> + parser.add_argument('-v', '--verbose', action='store_true',
> + help='Verbose (echo commands sent and received)')
> + parser.add_argument('-p', '--pretty', action='store_true',
> + help='Pretty-print JSON')
> +
> + parser.add_argument('--wrap', nargs=argparse.REMAINDER,
> + help='QEMU command line to invoke')
> +
> + args = parser.parse_args()
> +
> + socket = args.socket
> + wrap_qemu = args.wrap is not None
> +
> + if wrap_qemu:
> + if len(args.wrap) != 0:
> + qemu_cmdline = args.wrap
> + else:
> + qemu_cmdline = ["qemu-system-x86_64"]
> + if socket is None:
> + socket = "qmp-send-wrap-%d" % os.getpid()
> + qemu_cmdline += ["-qmp", "unix:%s" % socket]
> +
> + try:
> + address = QMPSend.parse_address(socket)
> + except QMPBadPortError:
> + parser.error(f"Bad port number: {socket}")
> + return # pycharm doesn't know error() is noreturn
> +
> + try:
> + with open(args.file, mode='rt', encoding='utf8') as file:
> + qmp_cmds = raw_load(file)
> + except QmpRawDecodeError as err:
> + report_error(str(err))
> + sys.exit(1)
> +
> + try:
> + with QMPSend(address, args.pretty, args.verbose,
> + server=wrap_qemu) as qmp:
> + # starting with python 3.7 we could use contextlib.nullcontext
> + qemu = Popen(qemu_cmdline) if wrap_qemu else contextlib.suppress()
> + with qemu:
> + try:
> + qmp.setup_connection()
> + except ConnectError as err:
> + if isinstance(err.exc, OSError):
> + report_error(f"Couldn't connect to {socket}: {err!s}")
> + else:
> + report_error(str(err))
> + sys.exit(1)
> + try:
> + for cmd in qmp_cmds:
> + qmp.execute_cmd(cmd)
> + except QMPError as err:
> + report_error(str(err))
> + sys.exit(1)
> + finally:
> + if wrap_qemu:
> + os.unlink(socket)
> +
> +
> +if __name__ == '__main__':
> + main()
> diff --git a/scripts/qmp/qmp-send b/scripts/qmp/qmp-send
> new file mode 100755
> index 0000000000..8d3063797c
> --- /dev/null
> +++ b/scripts/qmp/qmp-send
> @@ -0,0 +1,11 @@
> +#!/usr/bin/env python3
> +
> +import os
> +import sys
> +
> +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
> +from qemu.aqmp import qmp_send
> +
> +
> +if __name__ == '__main__':
> + qmp_send.main()
> --
> 2.35.1
>
Seems broadly fine to me, but I didn't review closely this time. If it
works for you, it works for me.
As for making QEMU hang: there's a few things you could do, take a
look at iotests and see how they handle timeout blocks in synchronous
code -- iotests.py line 696 or so, "class Timeout". When writing async
code, you can also do stuff like this:
async def foo():
await asyncio.wait_for(qmp.execute("some-command", args_etc), timeout=30)
See https://docs.python.org/3/library/asyncio-task.html#asyncio.wait_for
--js
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-04-04 20:34 ` John Snow
@ 2022-04-05 9:02 ` Damien Hedde
2022-04-05 9:45 ` Markus Armbruster
2022-04-05 18:08 ` John Snow
0 siblings, 2 replies; 14+ messages in thread
From: Damien Hedde @ 2022-04-05 9:02 UTC (permalink / raw)
To: John Snow
Cc: Markus Armbruster, Beraldo Leal, Daniel Berrange, qemu-devel,
Cleber Rosa
On 4/4/22 22:34, John Snow wrote:
> On Wed, Mar 16, 2022 at 5:55 AM Damien Hedde <damien.hedde@greensocs.com> wrote:
>>
>> It takes an input file containing raw qmp commands (concatenated json
>> dicts) and send all commands one by one to a qmp server. When one
>> command fails, it exits.
>>
>> As a convenience, it can also wrap the qemu process to avoid having
>> to start qemu in background. When wrapping qemu, the program returns
>> only when the qemu process terminates.
>>
>> Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
>> ---
>>
>> Hi all,
>>
>> Following our discussion, I've started this. What do you think ?
>>
>> I tried to follow Daniel's qmp-shell-wrap. I think it is
>> better to have similar options (eg: logging). There is also room
>> for factorizing code if we want to keep them aligned and ease
>> maintenance.
>>
>> There are still some pylint issues (too many branches in main and it
>> does not like my context manager if else line). But it's kind of a
>> mess to fix theses so I think it's enough for a first version.
>
> Yeah, don't worry about these. You can just tell pylint to shut up
> while you prototype. Sometimes it's just not worth spending more time
> on a more beautiful factoring. Oh well.
>
>>
>> I name that qmp-send as Daniel proposed, maybe qmp-test matches better
>> what I'm doing there ?
>>
>
> I think I agree with Dan's response.
>
>> Thanks,
>> Damien
>> ---
>> python/qemu/aqmp/qmp_send.py | 229 +++++++++++++++++++++++++++++++++++
>
> I recommend putting this in qemu/util/qmp_send.py instead.
>
> I'm in the process of pulling out the AQMP lib and hosting it
> separately. Scripts like this I think should stay in the QEMU tree, so
> moving it to util instead is probably best. Otherwise, I'll *really*
> have to commit to the syntax, and that's probably a bigger hurdle than
> you want to deal with.
If it stays in QEMU tree, what licensing should I use ? LGPL does not
hurt, no ?
>
>> scripts/qmp/qmp-send | 11 ++
>> 2 files changed, 240 insertions(+)
>> create mode 100644 python/qemu/aqmp/qmp_send.py
>> create mode 100755 scripts/qmp/qmp-send
>>
>> diff --git a/python/qemu/aqmp/qmp_send.py b/python/qemu/aqmp/qmp_send.py
>> new file mode 100644
>> index 0000000000..cbca1d0205
>> --- /dev/null
>> +++ b/python/qemu/aqmp/qmp_send.py
>
> Seems broadly fine to me, but I didn't review closely this time. If it
> works for you, it works for me.
>
> As for making QEMU hang: there's a few things you could do, take a
> look at iotests and see how they handle timeout blocks in synchronous
> code -- iotests.py line 696 or so, "class Timeout". When writing async
> code, you can also do stuff like this:
>
> async def foo():
> await asyncio.wait_for(qmp.execute("some-command", args_etc), timeout=30)
>
> See https://docs.python.org/3/library/asyncio-task.html#asyncio.wait_for
>
> --js
>
Thanks for the tip,
--
Damien
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-04-05 9:02 ` Damien Hedde
@ 2022-04-05 9:45 ` Markus Armbruster
2022-04-05 18:08 ` John Snow
1 sibling, 0 replies; 14+ messages in thread
From: Markus Armbruster @ 2022-04-05 9:45 UTC (permalink / raw)
To: Damien Hedde
Cc: Daniel Berrange, Beraldo Leal, John Snow, qemu-devel, Cleber Rosa
Damien Hedde <damien.hedde@greensocs.com> writes:
> On 4/4/22 22:34, John Snow wrote:
>> On Wed, Mar 16, 2022 at 5:55 AM Damien Hedde <damien.hedde@greensocs.com> wrote:
[...]
>> I recommend putting this in qemu/util/qmp_send.py instead.
>> I'm in the process of pulling out the AQMP lib and hosting it
>> separately. Scripts like this I think should stay in the QEMU tree, so
>> moving it to util instead is probably best. Otherwise, I'll *really*
>> have to commit to the syntax, and that's probably a bigger hurdle than
>> you want to deal with.
>
> If it stays in QEMU tree, what licensing should I use ? LGPL does not
> hurt, no ?
GPLv2+ is the default, and for a reason.
[...]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-04-05 9:02 ` Damien Hedde
2022-04-05 9:45 ` Markus Armbruster
@ 2022-04-05 18:08 ` John Snow
2022-04-06 5:18 ` Markus Armbruster
1 sibling, 1 reply; 14+ messages in thread
From: John Snow @ 2022-04-05 18:08 UTC (permalink / raw)
To: Damien Hedde
Cc: Markus Armbruster, Beraldo Leal, Daniel Berrange, qemu-devel,
Cleber Rosa
[-- Attachment #1: Type: text/plain, Size: 4043 bytes --]
On Tue, Apr 5, 2022, 5:03 AM Damien Hedde <damien.hedde@greensocs.com>
wrote:
>
>
> On 4/4/22 22:34, John Snow wrote:
> > On Wed, Mar 16, 2022 at 5:55 AM Damien Hedde <damien.hedde@greensocs.com>
> wrote:
> >>
> >> It takes an input file containing raw qmp commands (concatenated json
> >> dicts) and send all commands one by one to a qmp server. When one
> >> command fails, it exits.
> >>
> >> As a convenience, it can also wrap the qemu process to avoid having
> >> to start qemu in background. When wrapping qemu, the program returns
> >> only when the qemu process terminates.
> >>
> >> Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
> >> ---
> >>
> >> Hi all,
> >>
> >> Following our discussion, I've started this. What do you think ?
> >>
> >> I tried to follow Daniel's qmp-shell-wrap. I think it is
> >> better to have similar options (eg: logging). There is also room
> >> for factorizing code if we want to keep them aligned and ease
> >> maintenance.
> >>
> >> There are still some pylint issues (too many branches in main and it
> >> does not like my context manager if else line). But it's kind of a
> >> mess to fix theses so I think it's enough for a first version.
> >
> > Yeah, don't worry about these. You can just tell pylint to shut up
> > while you prototype. Sometimes it's just not worth spending more time
> > on a more beautiful factoring. Oh well.
> >
> >>
> >> I name that qmp-send as Daniel proposed, maybe qmp-test matches better
> >> what I'm doing there ?
> >>
> >
> > I think I agree with Dan's response.
> >
> >> Thanks,
> >> Damien
> >> ---
> >> python/qemu/aqmp/qmp_send.py | 229 +++++++++++++++++++++++++++++++++++
> >
> > I recommend putting this in qemu/util/qmp_send.py instead.
> >
> > I'm in the process of pulling out the AQMP lib and hosting it
> > separately. Scripts like this I think should stay in the QEMU tree, so
> > moving it to util instead is probably best. Otherwise, I'll *really*
> > have to commit to the syntax, and that's probably a bigger hurdle than
> > you want to deal with.
>
> If it stays in QEMU tree, what licensing should I use ? LGPL does not
> hurt, no ?
>
Whichever you please. GPLv2+ would be convenient and harmonizes well with
other tools. LGPL is only something I started doing so that the "qemu.qmp"
package would be LGPL. Licensing the tools as LGPL was just a sin of
convenience so I could claim a single license for the whole wheel/egg/tgz.
(I didn't want to make separate qmp and qmp-tools packages.)
Go with what you feel is best.
> >
> >> scripts/qmp/qmp-send | 11 ++
> >> 2 files changed, 240 insertions(+)
> >> create mode 100644 python/qemu/aqmp/qmp_send.py
> >> create mode 100755 scripts/qmp/qmp-send
> >>
> >> diff --git a/python/qemu/aqmp/qmp_send.py b/python/qemu/aqmp/qmp_send.py
> >> new file mode 100644
> >> index 0000000000..cbca1d0205
> >> --- /dev/null
> >> +++ b/python/qemu/aqmp/qmp_send.py
> >
> > Seems broadly fine to me, but I didn't review closely this time. If it
> > works for you, it works for me.
> >
> > As for making QEMU hang: there's a few things you could do, take a
> > look at iotests and see how they handle timeout blocks in synchronous
> > code -- iotests.py line 696 or so, "class Timeout". When writing async
> > code, you can also do stuff like this:
> >
> > async def foo():
> > await asyncio.wait_for(qmp.execute("some-command", args_etc),
> timeout=30)
> >
> > See https://docs.python.org/3/library/asyncio-task.html#asyncio.wait_for
> >
> > --js
> >
>
> Thanks for the tip,
> --
> Damien
>
Oh, and one more. the legacy.py bindings for AQMP also support a
configurable timeout that applies to most API calls by default.
see https://gitlab.com/jsnow/qemu.qmp/-/blob/main/qemu/qmp/legacy.py#L285
(Branch still in limbo here, but it should still be close to the same in
qemu.git)
I believe this is used by iotests.py when it sets up its machine.py
subclass ("VM", iirc) so that most qmp invocations in iotests have a
default timeout and won't hang tests indefinitely.
--js
>
[-- Attachment #2: Type: text/html, Size: 6186 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-04-05 18:08 ` John Snow
@ 2022-04-06 5:18 ` Markus Armbruster
0 siblings, 0 replies; 14+ messages in thread
From: Markus Armbruster @ 2022-04-06 5:18 UTC (permalink / raw)
To: John Snow
Cc: Damien Hedde, Beraldo Leal, Daniel Berrange, qemu-devel, Cleber Rosa
John Snow <jsnow@redhat.com> writes:
> On Tue, Apr 5, 2022, 5:03 AM Damien Hedde <damien.hedde@greensocs.com>
> wrote:
[...]
>> If it stays in QEMU tree, what licensing should I use ? LGPL does not
>> hurt, no ?
>>
>
> Whichever you please. GPLv2+ would be convenient and harmonizes well with
> other tools. LGPL is only something I started doing so that the "qemu.qmp"
> package would be LGPL. Licensing the tools as LGPL was just a sin of
> convenience so I could claim a single license for the whole wheel/egg/tgz.
>
> (I didn't want to make separate qmp and qmp-tools packages.)
>
> Go with what you feel is best.
Any license other than GPLv2+ needs justification in the commit message.
[...]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-03-16 9:54 [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu Damien Hedde
2022-03-16 10:24 ` Daniel P. Berrangé
2022-04-04 20:34 ` John Snow
@ 2022-05-25 16:06 ` Daniel P. Berrangé
2022-05-30 7:12 ` Damien Hedde
2 siblings, 1 reply; 14+ messages in thread
From: Daniel P. Berrangé @ 2022-05-25 16:06 UTC (permalink / raw)
To: Damien Hedde
Cc: qemu-devel, John Snow, Cleber Rosa, Beraldo Leal, Markus Armbruster
On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
> +def raw_load(file: TextIO) -> List[QMPMessage]:
> + """parse a raw qmp command file.
> +
> + JSON formatted commands can expand on several lines but must
> + be separated by an end-of-line (two commands can not share the
> + same line).
> + File must not end with empty lines.
> + """
> + cmds: List[QMPMessage] = []
> + linecnt = 0
> + while True:
> + buf = file.readline()
> + if not buf:
> + return cmds
If you change this to 'break'...
> + prev_err_pos = None
> + buf_linecnt = 1
> + while True:
> + try:
> + cmds.append(json.loads(buf))
...and this to
yield json.loads(buf)
then....
> + break
> + except json.JSONDecodeError as err:
> + if prev_err_pos == err.pos:
> + # adding a line made no progress so
> + # + either we're at EOF and json data is truncated
> + # + or the parsing error is before
> + raise QmpRawDecodeError(err.msg, linecnt + err.lineno,
> + err.colno) from err
> + prev_err_pos = err.pos
> + buf += file.readline()
> + buf_linecnt += 1
> + linecnt += buf_linecnt
> +
> +
> +def report_error(msg: str) -> None:
> + """Write an error to stderr."""
> + sys.stderr.write('ERROR: %s\n' % msg)
> +
> +
> +def main() -> None:
> + """
> + qmp-send entry point: parse command line arguments and start the REPL.
> + """
> + parser = argparse.ArgumentParser(
> + description="""
> + Send raw qmp commands to qemu as long as they succeed. It either
> + connects to a remote qmp server using the provided socket or wrap
> + the qemu process. It stops sending the provided commands when a
> + command fails (disconnection or error response).
> + """,
> + epilog="""
> + When qemu wrap option is used, this script waits for qemu
> + to terminate but never send any quit or kill command. This
> + needs to be done manually.
> + """)
> +
> + parser.add_argument('-f', '--file', action='store',
> + help='Input file containing the commands')
> + parser.add_argument('-s', '--socket', action='store',
> + help='< UNIX socket path | TCP address:port >')
> + parser.add_argument('-v', '--verbose', action='store_true',
> + help='Verbose (echo commands sent and received)')
> + parser.add_argument('-p', '--pretty', action='store_true',
> + help='Pretty-print JSON')
> +
> + parser.add_argument('--wrap', nargs=argparse.REMAINDER,
> + help='QEMU command line to invoke')
> +
> + args = parser.parse_args()
> +
> + socket = args.socket
> + wrap_qemu = args.wrap is not None
> +
> + if wrap_qemu:
> + if len(args.wrap) != 0:
> + qemu_cmdline = args.wrap
> + else:
> + qemu_cmdline = ["qemu-system-x86_64"]
> + if socket is None:
> + socket = "qmp-send-wrap-%d" % os.getpid()
> + qemu_cmdline += ["-qmp", "unix:%s" % socket]
> +
> + try:
> + address = QMPSend.parse_address(socket)
> + except QMPBadPortError:
> + parser.error(f"Bad port number: {socket}")
> + return # pycharm doesn't know error() is noreturn
> +
> + try:
> + with open(args.file, mode='rt', encoding='utf8') as file:
> + qmp_cmds = raw_load(file)
> + except QmpRawDecodeError as err:
> + report_error(str(err))
> + sys.exit(1)
...change this to
fh = sys.stdin
if args.file is not None and args.file != '-':
fh = open(args.file, mode='rt', encoding='utf8')
....
> +
> + try:
> + with QMPSend(address, args.pretty, args.verbose,
> + server=wrap_qemu) as qmp:
> + # starting with python 3.7 we could use contextlib.nullcontext
> + qemu = Popen(qemu_cmdline) if wrap_qemu else contextlib.suppress()
> + with qemu:
> + try:
> + qmp.setup_connection()
> + except ConnectError as err:
> + if isinstance(err.exc, OSError):
> + report_error(f"Couldn't connect to {socket}: {err!s}")
> + else:
> + report_error(str(err))
> + sys.exit(1)
> + try:
> + for cmd in qmp_cmds:
...finally this to
for cmd in raw_load(fh)
This means we can use qmp-send in a pipeline with commands
sent to QEMU on the fly as they arrive, rather than having
to read all the commands upfront before QEMU is started.
BTW, as an example usage I was trying your impl here in the following
way to extract information about CPUs that are deprecated
echo -e '{ "execute": "query-cpu-definitions"}\n{"execute": "quit"}' | \
qmp-send -v -p --wrap ./build/qemu-system-x86_64 -nodefaults -vnc :1 | \
jq -r --slurp '.[1].return[] | [.name, .deprecated] | @csv'
> + qmp.execute_cmd(cmd)
> + except QMPError as err:
> + report_error(str(err))
> + sys.exit(1)
> + finally:
> + if wrap_qemu:
> + os.unlink(socket)
> +
> +
> +if __name__ == '__main__':
> + main()
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH] python: add qmp-send program to send raw qmp commands to qemu
2022-05-25 16:06 ` Daniel P. Berrangé
@ 2022-05-30 7:12 ` Damien Hedde
0 siblings, 0 replies; 14+ messages in thread
From: Damien Hedde @ 2022-05-30 7:12 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, John Snow, Cleber Rosa, Beraldo Leal, Markus Armbruster
On 5/25/22 18:06, Daniel P. Berrangé wrote:
> On Wed, Mar 16, 2022 at 10:54:55AM +0100, Damien Hedde wrote:
>
>
>> +def raw_load(file: TextIO) -> List[QMPMessage]:
>> + """parse a raw qmp command file.
>> +
>> + JSON formatted commands can expand on several lines but must
>> + be separated by an end-of-line (two commands can not share the
>> + same line).
>> + File must not end with empty lines.
>> + """
>> + cmds: List[QMPMessage] = []
>> + linecnt = 0
>> + while True:
>> + buf = file.readline()
>> + if not buf:
>> + return cmds
>
> If you change this to 'break'...
>
>> + prev_err_pos = None
>> + buf_linecnt = 1
>> + while True:
>> + try:
>> + cmds.append(json.loads(buf))
>
> ...and this to
>
> yield json.loads(buf)
>
> then....
>
>> + break
>> + except json.JSONDecodeError as err:
>> + if prev_err_pos == err.pos:
>> + # adding a line made no progress so
>> + # + either we're at EOF and json data is truncated
>> + # + or the parsing error is before
>> + raise QmpRawDecodeError(err.msg, linecnt + err.lineno,
>> + err.colno) from err
>> + prev_err_pos = err.pos
>> + buf += file.readline()
>> + buf_linecnt += 1
>> + linecnt += buf_linecnt
>> +
>> +
>> +def report_error(msg: str) -> None:
>> + """Write an error to stderr."""
>> + sys.stderr.write('ERROR: %s\n' % msg)
>> +
>> +
>> +def main() -> None:
>> + """
>> + qmp-send entry point: parse command line arguments and start the REPL.
>> + """
>> + parser = argparse.ArgumentParser(
>> + description="""
>> + Send raw qmp commands to qemu as long as they succeed. It either
>> + connects to a remote qmp server using the provided socket or wrap
>> + the qemu process. It stops sending the provided commands when a
>> + command fails (disconnection or error response).
>> + """,
>> + epilog="""
>> + When qemu wrap option is used, this script waits for qemu
>> + to terminate but never send any quit or kill command. This
>> + needs to be done manually.
>> + """)
>> +
>> + parser.add_argument('-f', '--file', action='store',
>> + help='Input file containing the commands')
>> + parser.add_argument('-s', '--socket', action='store',
>> + help='< UNIX socket path | TCP address:port >')
>> + parser.add_argument('-v', '--verbose', action='store_true',
>> + help='Verbose (echo commands sent and received)')
>> + parser.add_argument('-p', '--pretty', action='store_true',
>> + help='Pretty-print JSON')
>> +
>> + parser.add_argument('--wrap', nargs=argparse.REMAINDER,
>> + help='QEMU command line to invoke')
>> +
>> + args = parser.parse_args()
>> +
>> + socket = args.socket
>> + wrap_qemu = args.wrap is not None
>> +
>> + if wrap_qemu:
>> + if len(args.wrap) != 0:
>> + qemu_cmdline = args.wrap
>> + else:
>> + qemu_cmdline = ["qemu-system-x86_64"]
>> + if socket is None:
>> + socket = "qmp-send-wrap-%d" % os.getpid()
>> + qemu_cmdline += ["-qmp", "unix:%s" % socket]
>> +
>> + try:
>> + address = QMPSend.parse_address(socket)
>> + except QMPBadPortError:
>> + parser.error(f"Bad port number: {socket}")
>> + return # pycharm doesn't know error() is noreturn
>> +
>> + try:
>> + with open(args.file, mode='rt', encoding='utf8') as file:
>> + qmp_cmds = raw_load(file)
>> + except QmpRawDecodeError as err:
>> + report_error(str(err))
>> + sys.exit(1)
>
> ...change this to
>
> fh = sys.stdin
> if args.file is not None and args.file != '-':
> fh = open(args.file, mode='rt', encoding='utf8')
>
> ....
>
>> +
>> + try:
>> + with QMPSend(address, args.pretty, args.verbose,
>> + server=wrap_qemu) as qmp:
>> + # starting with python 3.7 we could use contextlib.nullcontext
>> + qemu = Popen(qemu_cmdline) if wrap_qemu else contextlib.suppress()
>> + with qemu:
>> + try:
>> + qmp.setup_connection()
>> + except ConnectError as err:
>> + if isinstance(err.exc, OSError):
>> + report_error(f"Couldn't connect to {socket}: {err!s}")
>> + else:
>> + report_error(str(err))
>> + sys.exit(1)
>> + try:
>> + for cmd in qmp_cmds:
>
> ...finally this to
>
> for cmd in raw_load(fh)
>
>
> This means we can use qmp-send in a pipeline with commands
> sent to QEMU on the fly as they arrive, rather than having
> to read all the commands upfront before QEMU is started.
Yes. I was not sure which way was "better" between reading on the fly or
buffering everything before. In we want pipelining, we don't have much
choice.
>
> BTW, as an example usage I was trying your impl here in the following
> way to extract information about CPUs that are deprecated
>
> echo -e '{ "execute": "query-cpu-definitions"}\n{"execute": "quit"}' | \
> qmp-send -v -p --wrap ./build/qemu-system-x86_64 -nodefaults -vnc :1 | \
> jq -r --slurp '.[1].return[] | [.name, .deprecated] | @csv'
>
>
>> + qmp.execute_cmd(cmd)
>> + except QMPError as err:
>> + report_error(str(err))
>> + sys.exit(1)
>> + finally:
>> + if wrap_qemu:
>> + os.unlink(socket)
>> +
>> +
>> +if __name__ == '__main__':
>> + main()
>
>
> With regards,
> Daniel
^ permalink raw reply [flat|nested] 14+ messages in thread