[Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

This file contains detailed QMP description and definitions.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 180 insertions(+), 0 deletions(-)
 create mode 100644 monitor-protocol-spec.txt

diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
new file mode 100644
index 0000000..d20e3f9
--- /dev/null
+++ b/monitor-protocol-spec.txt
@@ -0,0 +1,180 @@
+               QEMU Monitor Protocol Specification - Version 0.1
+
+                                Luiz Capitulino
+                            <lcapitulino@redhat.com>
+
+1. Introduction
+===============
+
+This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol
+which is available for applications to control QEMU at the machine-level.
+
+For a detailed list of supported commands, please, refer to file
+monitor-protocol-commands.txt.
+
+2. Control Mode
+===============
+
+In order to enable QMP support, QEMU must start its Monitor subsystem in
+"control mode" rather than the default "user mode".
+
+Currently, this is done by passing the following command-line option to
+QEMU:
+
+-monitor control,<device>
+
+Please, refer to QEMU manpage or user-manual for more details.
+
+3. Protocol Specification
+=========================
+
+This section details the protocol format. For the purpose of this document
+"client" is any application that is communication with QEMU in control mode,
+and "server" is QEMU itself.
+
+3.1 General definition
+----------------------
+
+ o Only ASCII is permitted
+ o Case-insensitive
+ o All lines are terminated with CRLF
+
+3.2 Client Issued Commands
+--------------------------
+
+ o Arguments are separated by single space
+ o Arguments, responses and errors are defined separately by each command
+
+3.3 Server responses
+--------------------
+
+ There are four types of responses the server may send to the client:
+greeting, command completion, command data and asynchronous messages.
+
+3.3.1 Server Greeting
+---------------------
+
+Sent when a new connection is opened.
+
+Format: + OK QEMU <version> QMP <version>
+Example: + OK QEMU 0.10.50 QMP 0.1
+
+3.3.2 Server Command Completion
+-------------------------------
+
+Indicates that the execution of the last command has finished, there are
+three possible responses, as outlined below.
+
+ o Command completion was successful
+
+    Format: + OK <command> completed
+    Example: + OK stop completed
+
+ o Command completion failed
+
+    Format: - ERR <reason>
+    Example: - ERR could not find network device 'foo'
+
+ o Protocol error
+
+    Format: - BAD <reason>
+    Example: - BAD unknown command
+
+3.3.3 Server Command Data
+-------------------------
+
+Output issued by commands will always be prefixed by '='. If it is a
+multiline response, then each line is prefixed by '='.
+
+For example, the output of 'info network' would be:
+
+= VLAN 0 devices:
+= user.0:
+= ne2k_pci.0: model=ne2k_pci,macaddr=52:54:00:12:34:56
+
+3.3.4 Asynchronous Messages
+---------------------------
+
+Messages that can be sent unilaterally by the server, at any time. They are
+always prefixed by '*'.
+
+Currently there is only one type of asynchronous messages supported,
+known as 'events'.
+
+Format: * EVENT <event>
+Example: * EVENT reboot
+
+4. Examples
+===========
+
+This section provides some examples of the protocol in action, in all of
+them 'C' stands for 'Client' and 'S' stands for 'Server'.
+
+4.1 Simple 'cont' Execution
+---------------------------
+
+Client started QEMU with the "-S" option, which makes QEMU do not start
+the CPU at start up. Then Client issues the "cont" command.
+
+S: + OK QEMU 0.10.50 QMP 0.1
+C: cont
+S: + OK cont completed
+
+4.2 Multiline Response
+----------------------
+
+Client queries the Server about the network.
+
+S: + OK QEMU 0.10.50 QMP 0.1
+C: info network
+S: = VLAN 0 devices:
+S: = user.0:
+S: = ne2k_pci.0: model=ne2k_pci,macaddr=52:54:00:12:34:56
+S: + OK info network completed
+
+4.3 Events
+----------
+
+Client queries the Server about memory, but QEMU reboots.
+
+S: + OK QEMU 0.10.50 QMP 0.1
+C: info balloon
+S: * EVENT reboot
+
+4.4 Protocol Error
+------------------
+
+Client issues an invalid command.
+
+S: + OK QEMU 0.10.50 QMP 0.1
+C: foo
+S: - BAD unknown command
+
+4.5 Command Error
+-----------------
+
+Client tries to change link status for a non existing device. 
+
+S: + OK QEMU 0.10.50 QMP 0.1
+C: set_link foo down
+S: - ERR could not find network device 'foo'
+
+5. Client Considerations
+========================
+
+This section contains some considerations for Client implementation.
+
+ o If events are not supported, then lines starting with '*' should be
+   ignored and not considered errors
+
+ o If events are supported, the Client must be prepared to deal with
+   events added in future protocol versions. The best way to do this is
+   to handle known events and ignore the others
+
+6. Protocol History
+===================
+
+v0.1 - 2009-06-19
+-----------------
+
+    o Initial version
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 02/11] QMP: Introduce MONITOR_USE_CONTROL flag

[Luiz Capitulino]

It will be used to tell Monitor we are in control mode, also
introduce a function to do the check.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c |    6 ++++++
 monitor.h |    1 +
 2 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/monitor.c b/monitor.c
index 6b45f6c..514db00 100644
--- a/monitor.c
+++ b/monitor.c
@@ -93,6 +93,12 @@ Monitor *cur_mon = NULL;
 static void monitor_command_cb(Monitor *mon, const char *cmdline,
                                void *opaque);
 
+/* Return true if in control mode, false otherwise */
+static inline int monitor_ctrl_mode(Monitor *mon)
+{
+    return (mon->flags & MONITOR_USE_CONTROL);
+}
+
 static void monitor_read_command(Monitor *mon, int show_prompt)
 {
     readline_start(mon->rs, "(qemu) ", 0, monitor_command_cb, NULL);
diff --git a/monitor.h b/monitor.h
index 13e8cc7..48bc056 100644
--- a/monitor.h
+++ b/monitor.h
@@ -10,6 +10,7 @@ extern Monitor *cur_mon;
 /* flags for monitor_init */
 #define MONITOR_IS_DEFAULT    0x01
 #define MONITOR_USE_READLINE  0x02
+#define MONITOR_USE_CONTROL   0x04
 
 void monitor_init(CharDriverState *chr, int flags);
 
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 03/11] QMP: Introduce protocol print functions

[Luiz Capitulino]

This change introduces the functions that will be used by the Monitor
to output data in the format defined by the protocol specification.

There are four functions:

 o monitor_printf_bad()  signals a protocol error
 o monitor_print_ok()    signals successful execution
 o monitor_printf_err()  used by commands, to signal execution errors
 o monitor_printf_data() used by commands, to output data

For now monitor_print_ok() compilation is disabled to avoid breaking
the build as the function is not currently used. It will be enabled
by a later commit.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c   |   63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 monitor.h   |    3 ++
 qemu-tool.c |   12 +++++++++++
 3 files changed, 78 insertions(+), 0 deletions(-)

diff --git a/monitor.c b/monitor.c
index 514db00..dfa777d 100644
--- a/monitor.c
+++ b/monitor.c
@@ -199,6 +199,69 @@ static int monitor_fprintf(FILE *stream, const char *fmt, ...)
     return 0;
 }
 
+/*
+ * QEMU Monitor Control print functions
+ */
+
+/* Protocol errors */
+void monitor_printf_bad(Monitor *mon, const char *fmt, ...)
+{
+    if (monitor_ctrl_mode(mon)) {
+        va_list ap;
+
+        monitor_puts(mon, "- BAD ");
+
+        va_start(ap, fmt);
+        monitor_vprintf(mon, fmt, ap);
+        va_end(ap);
+    }
+}
+
+#if 0
+/* OK command completion, 'info' commands are special */
+static void monitor_print_ok(Monitor *mon, const char *cmd, const char *arg)
+{
+    if (!monitor_ctrl_mode(mon))
+        return;
+
+    monitor_printf(mon, "+ OK %s", cmd);
+    if (!strcmp(cmd, "info"))
+        monitor_printf(mon, " %s", arg);
+    monitor_puts(mon, " completed\n");
+}
+#endif
+
+static void monitor_ctrl_pprintf(Monitor *mon, const char *prefix,
+                                 const char *fmt, va_list ap)
+{
+    if (monitor_ctrl_mode(mon))
+        monitor_puts(mon, prefix);
+
+    monitor_vprintf(mon, fmt, ap);
+}
+
+/* Should be used by commands to signal errors, will add the prefix
+ * '- ERR ' if in control mode */
+void monitor_printf_err(Monitor *mon, const char *fmt, ...)
+{
+    va_list ap;
+
+    va_start(ap, fmt);
+    monitor_ctrl_pprintf(mon, "- ERR ", fmt, ap);
+    va_end(ap);
+}
+
+/* Should be used by commands to print any output which is not
+ * an error, will add the prefix '= ' if in control mode */
+void monitor_printf_data(Monitor *mon, const char *fmt, ...)
+{
+    va_list ap;
+
+    va_start(ap, fmt);
+    monitor_ctrl_pprintf(mon, "= ", fmt, ap);
+    va_end(ap);
+}
+
 static int compare_cmd(const char *name, const char *list)
 {
     const char *p, *pstart;
diff --git a/monitor.h b/monitor.h
index 48bc056..8b054eb 100644
--- a/monitor.h
+++ b/monitor.h
@@ -24,6 +24,9 @@ void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs,
 void monitor_vprintf(Monitor *mon, const char *fmt, va_list ap);
 void monitor_printf(Monitor *mon, const char *fmt, ...)
     __attribute__ ((__format__ (__printf__, 2, 3)));
+void monitor_printf_bad(Monitor *mon, const char *fmt, ...);
+void monitor_printf_err(Monitor *mon, const char *fmt, ...);
+void monitor_printf_data(Monitor *mon, const char *fmt, ...);
 void monitor_print_filename(Monitor *mon, const char *filename);
 void monitor_flush(Monitor *mon);
 
diff --git a/qemu-tool.c b/qemu-tool.c
index c08f061..2ba9983 100644
--- a/qemu-tool.c
+++ b/qemu-tool.c
@@ -40,6 +40,18 @@ void monitor_print_filename(Monitor *mon, const char *filename)
 {
 }
 
+void monitor_printf_bad(Monitor *mon, const char *fmt, ...)
+{
+}
+
+void monitor_printf_err(Monitor *mon, const char *fmt, ...)
+{
+}
+
+void monitor_printf_data(Monitor *mon, const char *fmt, ...)
+{
+}
+
 QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque)
 {
     QEMUBH *bh;
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 04/11] QMP: Make monitor_handle_command() QMP aware

[Luiz Capitulino]

Two changes are needed in monitor_handle_command() so that it
knows how to work in control mode.

First, it has to know whether a given command is part of the
protocol or not. Second, it has to print the correct server
response when a command execution finishes.

A good approach to do this would be:

1. Add a new member to struct mon_cmd_t, which will tell if
the command is already part of the protocol.

2. Change handler_* functions to return exit status, so that
monitor_handle_command() can print "+ OK" or "- ERR"
accordling.

However, to make the prototype simpler I have chosen a quick & dirty
approach, which is:

1. Introduce valid_control_cmd(). This function has a listing of
commands that have already been ported and thus are part of the
protocol.

2. Always print "+ OK", as current commands will probably not
print "- ERR" :). This requires enabling monitor_print_ok()
compilation.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c |   35 +++++++++++++++++++++++++++++++++--
 1 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/monitor.c b/monitor.c
index dfa777d..6d40b9b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -217,7 +217,6 @@ void monitor_printf_bad(Monitor *mon, const char *fmt, ...)
     }
 }
 
-#if 0
 /* OK command completion, 'info' commands are special */
 static void monitor_print_ok(Monitor *mon, const char *cmd, const char *arg)
 {
@@ -229,7 +228,6 @@ static void monitor_print_ok(Monitor *mon, const char *cmd, const char *arg)
         monitor_printf(mon, " %s", arg);
     monitor_puts(mon, " completed\n");
 }
-#endif
 
 static void monitor_ctrl_pprintf(Monitor *mon, const char *prefix,
                                  const char *fmt, va_list ap)
@@ -2476,6 +2474,35 @@ static const char *get_command_name(const char *cmdline,
     return p;
 }
 
+/* When  in control mode, return true if 'cmd' is part of the protocol,
+ * return false otherwise */
+static int valid_control_cmd(Monitor *mon, const char *cmd,
+                             const char *cmdline)
+{
+    int i;
+    const char *valid_cmds[] = { NULL };
+    const char *valid_infos[] = { NULL };
+
+    if (!monitor_ctrl_mode(mon)) {
+        /* all commands are valid in user-mode mode */
+        return 1;
+    }
+
+    for (i = 0; valid_cmds[i]; i++)
+        if (compare_cmd(valid_cmds[i], cmd))
+            return 1;
+
+    /* info is special */
+    if (compare_cmd(cmd, "info")) {
+        for (i = 0; valid_infos[i]; i++)
+            if (strstr(cmdline, valid_infos[i]))
+                return 1;
+    }
+
+    monitor_printf_bad(mon, "unknown command\n");
+    return 0;
+}
+
 static int default_fmt_format = 'x';
 static int default_fmt_size = 4;
 
@@ -2512,6 +2539,9 @@ static void monitor_handle_command(Monitor *mon, const char *cmdline)
     if (!p)
         return;
 
+    if (valid_control_cmd(mon, cmdname, cmdline) == 0)
+        return;
+
     /* find the command */
     for(cmd = mon_cmds; cmd->name != NULL; cmd++) {
         if (compare_cmd(cmdname, cmd->name))
@@ -2794,6 +2824,7 @@ static void monitor_handle_command(Monitor *mon, const char *cmdline)
         monitor_printf(mon, "unsupported number of arguments: %d\n", nb_args);
         goto fail;
     }
+    monitor_print_ok(mon, cmdname, args[0]);
  fail:
     for(i = 0; i < MAX_ARGS; i++)
         qemu_free(str_allocated[i]);
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 05/11] QMP: Introduce control mode chardev handling

[Luiz Capitulino]

Control mode disables readline support and handles Monitor's
chardev by itself.

In order to this this change adds:

 o MonitorControl type: used to store input from the client
 o monitor_control_read(): read input from the chardev
 o monitor_control_event(): prints our greeting message

Support to control mode is also added to monitor_init().

Also note that monitor_control_read() is a bit simple right now,
will be improved in future versions.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/monitor.c b/monitor.c
index 6d40b9b..462f60b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -70,6 +70,11 @@ typedef struct mon_cmd_t {
     const char *help;
 } mon_cmd_t;
 
+typedef struct MonitorControl {
+    uint8_t buf[128];
+    int size;
+} MonitorControl;
+
 struct Monitor {
     CharDriverState *chr;
     int flags;
@@ -77,6 +82,7 @@ struct Monitor {
     uint8_t outbuf[1024];
     int outbuf_index;
     ReadLineState *rs;
+    MonitorControl *mc;
     CPUState *mon_cpu;
     BlockDriverCompletionFunc *password_completion_cb;
     void *password_opaque;
@@ -3038,6 +3044,31 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 128 : 0;
 }
 
+static void monitor_control_read(void *opaque, const uint8_t *buf, int size)
+{
+    Monitor *old_mon = cur_mon;
+    int i;
+
+    cur_mon = opaque;
+
+    for (i = 0; i < size; i++) {
+        if (buf[i] == '\r' || buf[i] == '\n') {
+            cur_mon->mc->buf[cur_mon->mc->size] = '\0';
+            cur_mon->mc->size = 0;
+            monitor_handle_command(cur_mon, (char *)cur_mon->mc->buf);
+        } else {
+            cur_mon->mc->buf[cur_mon->mc->size++] = buf[i];
+            if (cur_mon->mc->size == sizeof(cur_mon->mc->buf)) {
+                monitor_printf_bad(cur_mon, "command too long\n");
+                cur_mon->mc->size = 0;
+                return;
+            }
+        }
+    }
+
+    cur_mon = old_mon;
+}
+
 static void monitor_read(void *opaque, const uint8_t *buf, int size)
 {
     Monitor *old_mon = cur_mon;
@@ -3081,6 +3112,15 @@ void monitor_resume(Monitor *mon)
         readline_show_prompt(mon->rs);
 }
 
+static void monitor_control_event(void *opaque, int event)
+{
+    if (event == CHR_EVENT_RESET) {
+        Monitor *mon = opaque;
+        mon->mc->size = 0;
+        monitor_printf(mon, "+ QEMU %s QMP 0.1\n", QEMU_VERSION);
+    }
+}
+
 static void monitor_event(void *opaque, int event)
 {
     Monitor *mon = opaque;
@@ -3138,8 +3178,15 @@ void monitor_init(CharDriverState *chr, int flags)
         monitor_read_command(mon, 0);
     }
 
-    qemu_chr_add_handlers(chr, monitor_can_read, monitor_read, monitor_event,
-                          mon);
+    if (monitor_ctrl_mode(mon)) {
+        mon->mc = qemu_mallocz(sizeof(MonitorControl));
+        /* Control mode requires special handlers */
+        qemu_chr_add_handlers(chr, monitor_can_read, monitor_control_read,
+                              monitor_control_event, mon);
+    } else {
+        qemu_chr_add_handlers(chr, monitor_can_read, monitor_read,
+                              monitor_event, mon);
+    }
 
     LIST_INSERT_HEAD(&mon_list, mon, entry);
     if (!cur_mon || (flags & MONITOR_IS_DEFAULT))
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Luiz Capitulino]

It is just an exported function that will be used by other subsystems
to print specific events to the output buffer.

This is based in ideas by Amit Shah <amit.shah@redhat.com>.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c   |   18 ++++++++++++++++++
 monitor.h   |    6 ++++++
 qemu-tool.c |    4 ++++
 3 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/monitor.c b/monitor.c
index 462f60b..df58bdd 100644
--- a/monitor.c
+++ b/monitor.c
@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)
     va_end(ap);
 }
 
+/* Asynchronous events main function */
+void monitor_notify_event(MonitorEvent event)
+{
+    if (!monitor_ctrl_mode(cur_mon))
+        return;
+
+    assert(event < EVENT_MAX);
+    monitor_puts(cur_mon, "* EVENT ");
+
+    switch (event) {
+    case EVENT_MAX:
+        // Avoid gcc warning, will never get here
+        break;
+    }
+
+    monitor_puts(cur_mon, "\n");
+}
+
 static int compare_cmd(const char *name, const char *list)
 {
     const char *p, *pstart;
diff --git a/monitor.h b/monitor.h
index 8b054eb..7cc88cc 100644
--- a/monitor.h
+++ b/monitor.h
@@ -12,6 +12,11 @@ extern Monitor *cur_mon;
 #define MONITOR_USE_READLINE  0x02
 #define MONITOR_USE_CONTROL   0x04
 
+/* QMP events */
+typedef enum MonitorEvent {
+    EVENT_MAX,
+} MonitorEvent;
+
 void monitor_init(CharDriverState *chr, int flags);
 
 int monitor_suspend(Monitor *mon);
@@ -28,6 +33,7 @@ void monitor_printf_bad(Monitor *mon, const char *fmt, ...);
 void monitor_printf_err(Monitor *mon, const char *fmt, ...);
 void monitor_printf_data(Monitor *mon, const char *fmt, ...);
 void monitor_print_filename(Monitor *mon, const char *filename);
+void monitor_notify_event(MonitorEvent event);
 void monitor_flush(Monitor *mon);
 
 #endif /* !MONITOR_H */
diff --git a/qemu-tool.c b/qemu-tool.c
index 2ba9983..4798665 100644
--- a/qemu-tool.c
+++ b/qemu-tool.c
@@ -40,6 +40,10 @@ void monitor_print_filename(Monitor *mon, const char *filename)
 {
 }
 
+void monitor_notify_event(MonitorEvent event)
+{
+}
+
 void monitor_printf_bad(Monitor *mon, const char *fmt, ...)
 {
 }
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 07/11] QMP: Enable simple commands

[Luiz Capitulino]

'cont', 'quit', 'stop', 'system_reset' and 'system_powerdown' are
so simple that enabling them in valid_control_cmd() is the only
step requered to port them.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/monitor.c b/monitor.c
index df58bdd..ce836a9 100644
--- a/monitor.c
+++ b/monitor.c
@@ -2504,7 +2504,12 @@ static int valid_control_cmd(Monitor *mon, const char *cmd,
                              const char *cmdline)
 {
     int i;
-    const char *valid_cmds[] = { NULL };
+    const char *valid_cmds[] = { "quit",
+                                 "cont",
+                                 "stop",
+                                 "system_reset",
+                                 "system_powerdown",
+                                 NULL };
     const char *valid_infos[] = { NULL };
 
     if (!monitor_ctrl_mode(mon)) {
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 08/11] QMP: Port balloon command

[Luiz Capitulino]

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index ce836a9..84d4399 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1662,12 +1662,12 @@ static void do_info_balloon(Monitor *mon)
 
     actual = qemu_balloon_status();
     if (kvm_enabled() && !kvm_has_sync_mmu())
-        monitor_printf(mon, "Using KVM without synchronous MMU, "
+        monitor_printf_err(mon, "Using KVM without synchronous MMU, "
                        "ballooning disabled\n");
     else if (actual == 0)
-        monitor_printf(mon, "Ballooning not activated in VM\n");
+        monitor_printf_err(mon, "Ballooning not activated in VM\n");
     else
-        monitor_printf(mon, "balloon: actual=%d\n", (int)(actual >> 20));
+        monitor_printf_data(mon, "balloon: actual=%d\n", (int)(actual >> 20));
 }
 
 static void do_acl(Monitor *mon,
@@ -2509,8 +2509,10 @@ static int valid_control_cmd(Monitor *mon, const char *cmd,
                                  "stop",
                                  "system_reset",
                                  "system_powerdown",
+                                 "balloon",
+                                 NULL };
+    const char *valid_infos[] = { "balloon",
                                  NULL };
-    const char *valid_infos[] = { NULL };
 
     if (!monitor_ctrl_mode(mon)) {
         /* all commands are valid in user-mode mode */
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 09/11] QMP: Port 'info blockstats' command

[Luiz Capitulino]

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 block.c   |   18 +++++++++---------
 monitor.c |    3 ++-
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/block.c b/block.c
index aca5a6d..3b4af27 100644
--- a/block.c
+++ b/block.c
@@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)
     BlockDriverState *bs;
 
     for (bs = bdrv_first; bs != NULL; bs = bs->next) {
-        monitor_printf(mon, "%s:"
-                       " rd_bytes=%" PRIu64
-                       " wr_bytes=%" PRIu64
-                       " rd_operations=%" PRIu64
-                       " wr_operations=%" PRIu64
-                       "\n",
-                       bs->device_name,
-                       bs->rd_bytes, bs->wr_bytes,
-                       bs->rd_ops, bs->wr_ops);
+        monitor_printf_data(mon, "%s:"
+                            " rd_bytes=%" PRIu64
+                            " wr_bytes=%" PRIu64
+                            " rd_operations=%" PRIu64
+                            " wr_operations=%" PRIu64
+                            "\n",
+                            bs->device_name,
+                            bs->rd_bytes, bs->wr_bytes,
+                            bs->rd_ops, bs->wr_ops);
     }
 }
 
diff --git a/monitor.c b/monitor.c
index 84d4399..378a3ae 100644
--- a/monitor.c
+++ b/monitor.c
@@ -2512,7 +2512,8 @@ static int valid_control_cmd(Monitor *mon, const char *cmd,
                                  "balloon",
                                  NULL };
     const char *valid_infos[] = { "balloon",
-                                 NULL };
+                                  "blockstats",
+                                  NULL };
 
     if (!monitor_ctrl_mode(mon)) {
         /* all commands are valid in user-mode mode */
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 10/11] QMP: Introduce basic events

[Luiz Capitulino]

Reboot, shutdown and powerdown are very basic events and can be
added together.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c |    9 +++++++++
 monitor.h |    3 +++
 vl.c      |    3 +++
 3 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/monitor.c b/monitor.c
index 378a3ae..7ebf0ea 100644
--- a/monitor.c
+++ b/monitor.c
@@ -276,6 +276,15 @@ void monitor_notify_event(MonitorEvent event)
     monitor_puts(cur_mon, "* EVENT ");
 
     switch (event) {
+    case EVENT_REBOOT:
+        monitor_puts(cur_mon, "reboot");
+        break;
+    case EVENT_SHUTDOWN:
+        monitor_puts(cur_mon, "shutdown");
+        break;
+    case EVENT_POWERDOWN:
+        monitor_puts(cur_mon, "powerdown");
+        break;
     case EVENT_MAX:
         // Avoid gcc warning, will never get here
         break;
diff --git a/monitor.h b/monitor.h
index 7cc88cc..579b80b 100644
--- a/monitor.h
+++ b/monitor.h
@@ -14,6 +14,9 @@ extern Monitor *cur_mon;
 
 /* QMP events */
 typedef enum MonitorEvent {
+    EVENT_REBOOT,
+    EVENT_SHUTDOWN,
+    EVENT_POWERDOWN,
     EVENT_MAX,
 } MonitorEvent;
 
diff --git a/vl.c b/vl.c
index 60a00e1..7dc5954 100644
--- a/vl.c
+++ b/vl.c
@@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void)
         reset_requested = 1;
     }
     qemu_notify_event();
+    monitor_notify_event(EVENT_REBOOT);
 }
 
 void qemu_system_shutdown_request(void)
 {
     shutdown_requested = 1;
     qemu_notify_event();
+    monitor_notify_event(EVENT_SHUTDOWN);
 }
 
 void qemu_system_powerdown_request(void)
 {
     powerdown_requested = 1;
     qemu_notify_event();
+    monitor_notify_event(EVENT_POWERDOWN);
 }
 
 #ifdef CONFIG_IOTHREAD
-- 
1.6.3.GIT

[Qemu-devel] [PATCH 11/11] QMP: Command-line flag to enable control mode

[Luiz Capitulino]

This change adds a flag called 'control' to the already existing
'-monitor' command-line option. This flag can be used to enable
control mode.

Its syntax is:

qemu [...] -monitor control,<device>

Where <device> is a chardev (excluding 'vc', for obvious reasons).

For example:

$ qemu [...] -monitor control,tcp:localhost:4444,server

Will run QEMU in control mode, waiting for a client TCP connection
on localhost port 4444.

Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
---
 monitor.c |   19 +++++++++++++++++++
 monitor.h |    1 +
 vl.c      |    5 +++--
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/monitor.c b/monitor.c
index 7ebf0ea..c1baeac 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3192,6 +3192,25 @@ static void monitor_event(void *opaque, int event)
  * End:
  */
 
+const char *monitor_cmdline_parse(const char *cmdline, int *flags)
+{
+    const char *dev;
+
+    *flags = MONITOR_IS_DEFAULT;
+    if (strstart(cmdline, "control,", &dev)) {
+        if (strstart(dev, "vc", NULL)) {
+            fprintf(stderr, "qemu: control mode is for low-level interaction ");
+            fprintf(stderr, "cannot be used with device 'vc'\n");
+            exit(1);
+        }
+        *flags |= MONITOR_USE_CONTROL;
+        return dev;
+    }
+
+    *flags |= MONITOR_USE_READLINE;
+    return cmdline;
+}
+
 void monitor_init(CharDriverState *chr, int flags)
 {
     static int is_first_init = 1;
diff --git a/monitor.h b/monitor.h
index 579b80b..9ddd264 100644
--- a/monitor.h
+++ b/monitor.h
@@ -20,6 +20,7 @@ typedef enum MonitorEvent {
     EVENT_MAX,
 } MonitorEvent;
 
+const char *monitor_cmdline_parse(const char *cmdline, int *flags);
 void monitor_init(CharDriverState *chr, int flags);
 
 int monitor_suspend(Monitor *mon);
diff --git a/vl.c b/vl.c
index 7dc5954..bd83dac 100644
--- a/vl.c
+++ b/vl.c
@@ -4977,6 +4977,7 @@ int main(int argc, char **argv, char **envp)
     const char *r, *optarg;
     CharDriverState *monitor_hd = NULL;
     const char *monitor_device;
+    int monitor_flags = MONITOR_IS_DEFAULT | MONITOR_USE_READLINE;
     const char *serial_devices[MAX_SERIAL_PORTS];
     int serial_device_index;
     const char *parallel_devices[MAX_PARALLEL_PORTS];
@@ -5462,7 +5463,7 @@ int main(int argc, char **argv, char **envp)
                     break;
                 }
             case QEMU_OPTION_monitor:
-                monitor_device = optarg;
+                monitor_device = monitor_cmdline_parse(optarg, &monitor_flags);
                 break;
             case QEMU_OPTION_serial:
                 if (serial_device_index >= MAX_SERIAL_PORTS) {
@@ -6150,7 +6151,7 @@ int main(int argc, char **argv, char **envp)
     qemu_chr_initial_reset();
 
     if (monitor_device && monitor_hd)
-        monitor_init(monitor_hd, MONITOR_USE_READLINE | MONITOR_IS_DEFAULT);
+        monitor_init(monitor_hd, monitor_flags);
 
     for(i = 0; i < MAX_SERIAL_PORTS; i++) {
         const char *devname = serial_devices[i];
-- 
1.6.3.GIT

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 07:28 AM, Luiz Capitulino wrote:
> This file contains detailed QMP description and definitions.
>
> Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
> ---
>   monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
>   1 files changed, 180 insertions(+), 0 deletions(-)
>   create mode 100644 monitor-protocol-spec.txt
>
> diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
> new file mode 100644
> index 0000000..d20e3f9
> --- /dev/null
> +++ b/monitor-protocol-spec.txt
> @@ -0,0 +1,180 @@
> +               QEMU Monitor Protocol Specification - Version 0.1
> +
> +                                Luiz Capitulino
> +<lcapitulino@redhat.com>
> +
> +1. Introduction
> +===============
> +
> +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol
> +which is available for applications to control QEMU at the machine-level.
>    

Without a doubt, this is a the most important file of this patchset.  
There's a huge difference between working with an implementation and 
working with a specification.

> +
> +For a detailed list of supported commands, please, refer to file
> +monitor-protocol-commands.txt.
>    

I don't see you update that file anywhere.  In any case, my preference 
would be to have everything in one file.

> +3.1 General definition
> +----------------------
> +
> + o Only ASCII is permitted
>    

Since the some commands contain user-specified strings, UTF-8 is needed.

I think it's worthwhile to define a quoted string format, to be used 
both in commands and responses.

> +
> +3.3.1 Server Greeting
> +---------------------
> +
> +Sent when a new connection is opened.
> +
> +Format: + OK QEMU<version>  QMP<version>
> +Example: + OK QEMU 0.10.50 QMP 0.1
>    

Clients should never make decisions based on the qemu or qmp version.  
Rather, we should provide a facility to query the availability of features.
> + o Command completion failed
> +
> +    Format: - ERR<reason>
> +    Example: - ERR could not find network device 'foo'
> +
>    

Maybe add a numeric error code (to be defined by individual commands).

> +4.3 Events
> +----------
> +
> +Client queries the Server about memory, but QEMU reboots.
> +
> +S: + OK QEMU 0.10.50 QMP 0.1
> +C: info balloon
> +S: * EVENT reboot
>    

The guest reboots (actually, resets), not qemu.  And 'info balloon' will 
eventually print its response, no?

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Jan Kiszka]

Luiz Capitulino wrote:
> It is just an exported function that will be used by other subsystems
> to print specific events to the output buffer.
> 
> This is based in ideas by Amit Shah <amit.shah@redhat.com>.
> 
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  monitor.c   |   18 ++++++++++++++++++
>  monitor.h   |    6 ++++++
>  qemu-tool.c |    4 ++++
>  3 files changed, 28 insertions(+), 0 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 462f60b..df58bdd 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)
>      va_end(ap);
>  }
>  
> +/* Asynchronous events main function */
> +void monitor_notify_event(MonitorEvent event)
> +{
> +    if (!monitor_ctrl_mode(cur_mon))
> +        return;
> +
> +    assert(event < EVENT_MAX);
> +    monitor_puts(cur_mon, "* EVENT ");
> +
> +    switch (event) {
> +    case EVENT_MAX:
> +        // Avoid gcc warning, will never get here
> +        break;
> +    }
> +
> +    monitor_puts(cur_mon, "\n");
> +}
> +

You shouldn't use cur_mon here. It's for legacy code that still assumes
there can be only one monitor terminal. Instead, iterate over all
monitors and distribute the event to those that are in control mode.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Luiz Capitulino wrote:
> This change adds a flag called 'control' to the already existing
> '-monitor' command-line option. This flag can be used to enable
> control mode.
> 
> Its syntax is:
> 
> qemu [...] -monitor control,<device>
> 
> Where <device> is a chardev (excluding 'vc', for obvious reasons).
> 
> For example:
> 
> $ qemu [...] -monitor control,tcp:localhost:4444,server
> 
> Will run QEMU in control mode, waiting for a client TCP connection
> on localhost port 4444.
> 
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>

At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
instances". I think Anthony posted such a patch before. Now we should
really include this as your extension may block the stand-alone -monitor
switch for the control channel, preventing to additionally set up one
(or more) for debugging purposes.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Avi Kivity]

On 06/23/2009 07:29 AM, Luiz Capitulino wrote:
>       else if (actual == 0)
> -        monitor_printf(mon, "Ballooning not activated in VM\n");
> +        monitor_printf_err(mon, "Ballooning not activated in VM\n");
>       else
> -        monitor_printf(mon, "balloon: actual=%d\n", (int)(actual>>  20));
> +        monitor_printf_data(mon, "balloon: actual=%d\n", (int)(actual>>  20));
>   }
>    

Control mode should always use bytes and seconds (and this should be 
described in the spec).  You avoid rounding, and more importantly, 
ambiguity and a source of unit conversion errors.

Patched that add a command to machine mode without updating the spec 
should be automatically NACKed.

We also need a way to discover that the command is available:

info commands
= info
= balloon
+ OK go for it

info command balloon
+ OK (no extensions yet)

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command

[Avi Kivity]

On 06/23/2009 07:29 AM, Luiz Capitulino wrote:
> Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
> ---
>   block.c   |   18 +++++++++---------
>   monitor.c |    3 ++-
>   2 files changed, 11 insertions(+), 10 deletions(-)
>
> diff --git a/block.c b/block.c
> index aca5a6d..3b4af27 100644
> --- a/block.c
> +++ b/block.c
> @@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)
>       BlockDriverState *bs;
>
>       for (bs = bdrv_first; bs != NULL; bs = bs->next) {
> -        monitor_printf(mon, "%s:"
> -                       " rd_bytes=%" PRIu64
> -                       " wr_bytes=%" PRIu64
> -                       " rd_operations=%" PRIu64
> -                       " wr_operations=%" PRIu64
> -                       "\n",
> -                       bs->device_name,
> -                       bs->rd_bytes, bs->wr_bytes,
> -                       bs->rd_ops, bs->wr_ops);
> +        monitor_printf_data(mon, "%s:"
> +                            " rd_bytes=%" PRIu64
> +                            " wr_bytes=%" PRIu64
> +                            " rd_operations=%" PRIu64
> +                            " wr_operations=%" PRIu64
> +                            "\n",
> +                            bs->device_name,
> +                            bs->rd_bytes, bs->wr_bytes,
> +                            bs->rd_ops, bs->wr_ops);
>    

It may be easier for control mode to drop the variable= prefixes (they 
are described in the spec anyway).

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 10/11] QMP: Introduce basic events

[Avi Kivity]

On 06/23/2009 07:29 AM, Luiz Capitulino wrote:
> Reboot, shutdown and powerdown are very basic events and can be
> added together.
>
>    

Reboot only exists within a guest.  qemu sees a reset (there are reboots 
not associated with a reset, for example kexec).
>
> diff --git a/vl.c b/vl.c
> index 60a00e1..7dc5954 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void)
>           reset_requested = 1;
>       }
>       qemu_notify_event();
> +    monitor_notify_event(EVENT_REBOOT);
>   }
>
>   void qemu_system_shutdown_request(void)
>   {
>       shutdown_requested = 1;
>       qemu_notify_event();
> +    monitor_notify_event(EVENT_SHUTDOWN);
>   }
>
>   void qemu_system_powerdown_request(void)
>   {
>       powerdown_requested = 1;
>       qemu_notify_event();
> +    monitor_notify_event(EVENT_POWERDOWN);
>   }
>    

Maybe it's better to signal the event when the reset etc. actually takes 
place, to avoid subtle races.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 11:55:56AM +0300, Avi Kivity wrote:
> On 06/23/2009 07:28 AM, Luiz Capitulino wrote:
> >This file contains detailed QMP description and definitions.
> >
> >Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
> >---
> >  monitor-protocol-spec.txt |  180 
> >  +++++++++++++++++++++++++++++++++++++++++++++
> >  1 files changed, 180 insertions(+), 0 deletions(-)
> >  create mode 100644 monitor-protocol-spec.txt
> >
> >diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
> >new file mode 100644
> >index 0000000..d20e3f9
> >--- /dev/null
> >+++ b/monitor-protocol-spec.txt
> >@@ -0,0 +1,180 @@
> >+               QEMU Monitor Protocol Specification - Version 0.1
> >+
> >+                                Luiz Capitulino
> >+<lcapitulino@redhat.com>
> >+
> >+1. Introduction
> >+===============
> >+
> >+This document specifies the QEMU Monitor Protocol (QMP), a text-based 
> >protocol
> >+which is available for applications to control QEMU at the machine-level.
> >   
> 
> Without a doubt, this is a the most important file of this patchset.  
> There's a huge difference between working with an implementation and 
> working with a specification.
> 
> >+
> >+For a detailed list of supported commands, please, refer to file
> >+monitor-protocol-commands.txt.
> >   
> 
> I don't see you update that file anywhere.  In any case, my preference 
> would be to have everything in one file.
> 
> >+3.1 General definition
> >+----------------------
> >+
> >+ o Only ASCII is permitted
> >   
> 
> Since the some commands contain user-specified strings, UTF-8 is needed.

This would imply that QEMU does charset conversion of all data going in
or out of the monitor, to cope with the case of being run in a non-UTF8
locale. This is probably worth it though, because the only viable
alternative would be to have the monitor always work in the locale the
QEMU is launched under which pushes the problem off to every client.

> I think it's worthwhile to define a quoted string format, to be used 
> both in commands and responses.

Definitely need a quoted string format, space delimited is not sufficient
to cope with filenames with spaces in them

> 
> >+
> >+3.3.1 Server Greeting
> >+---------------------
> >+
> >+Sent when a new connection is opened.
> >+
> >+Format: + OK QEMU<version>  QMP<version>
> >+Example: + OK QEMU 0.10.50 QMP 0.1
> >   
> 
> Clients should never make decisions based on the qemu or qmp version.  
> Rather, we should provide a facility to query the availability of features.

True, but having QEMU print full version info when connecting is still 
a useful thing for clients to log for developer debugging.


> >+ o Command completion failed
> >+
> >+    Format: - ERR<reason>
> >+    Example: - ERR could not find network device 'foo'
> >+
> >   
> 
> Maybe add a numeric error code (to be defined by individual commands).

I think that would be particularly useful to allow clients to distinguish
the error from a command which does not exist, vs a command that exists
but failed. There's probably a handful of common errors that you want
to detect from all commands with the same error handling logic. 

> 
> >+4.3 Events
> >+----------
> >+
> >+Client queries the Server about memory, but QEMU reboots.
> >+
> >+S: + OK QEMU 0.10.50 QMP 0.1
> >+C: info balloon
> >+S: * EVENT reboot
> >   
> 
> The guest reboots (actually, resets), not qemu.  And 'info balloon' will 
> eventually print its response, no?

Might we need to have timestamp assoicated with each response, or will we
define that responses are explicitly ordered wrt events,to reflect the order
in which they're handled. eg When the 'info balloon' response arrives, we
want to know whether the data it contains is reflecting state before or
after the reboot. 

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command

[Jan Kiszka]

Avi Kivity wrote:
> On 06/23/2009 07:29 AM, Luiz Capitulino wrote:
>> Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
>> ---
>>   block.c   |   18 +++++++++---------
>>   monitor.c |    3 ++-
>>   2 files changed, 11 insertions(+), 10 deletions(-)
>>
>> diff --git a/block.c b/block.c
>> index aca5a6d..3b4af27 100644
>> --- a/block.c
>> +++ b/block.c
>> @@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)
>>       BlockDriverState *bs;
>>
>>       for (bs = bdrv_first; bs != NULL; bs = bs->next) {
>> -        monitor_printf(mon, "%s:"
>> -                       " rd_bytes=%" PRIu64
>> -                       " wr_bytes=%" PRIu64
>> -                       " rd_operations=%" PRIu64
>> -                       " wr_operations=%" PRIu64
>> -                       "\n",
>> -                       bs->device_name,
>> -                       bs->rd_bytes, bs->wr_bytes,
>> -                       bs->rd_ops, bs->wr_ops);
>> +        monitor_printf_data(mon, "%s:"
>> +                            " rd_bytes=%" PRIu64
>> +                            " wr_bytes=%" PRIu64
>> +                            " rd_operations=%" PRIu64
>> +                            " wr_operations=%" PRIu64
>> +                            "\n",
>> +                            bs->device_name,
>> +                            bs->rd_bytes, bs->wr_bytes,
>> +                            bs->rd_ops, bs->wr_ops);
>>    
> 
> It may be easier for control mode to drop the variable= prefixes (they
> are described in the spec anyway).
> 

Isn't monitor_printf_data also used for pretty-printing in case of
human-operated monitors? I think we need to keep this - or provide two
output formats which are filtered by the monitor based on the the target
terminal's mode. But that sounds like a lot of work /wrt conversion and
long-term maintenance.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 11:03:09AM +0200, Jan Kiszka wrote:
> Luiz Capitulino wrote:
> > This change adds a flag called 'control' to the already existing
> > '-monitor' command-line option. This flag can be used to enable
> > control mode.
> > 
> > Its syntax is:
> > 
> > qemu [...] -monitor control,<device>
> > 
> > Where <device> is a chardev (excluding 'vc', for obvious reasons).
> > 
> > For example:
> > 
> > $ qemu [...] -monitor control,tcp:localhost:4444,server
> > 
> > Will run QEMU in control mode, waiting for a client TCP connection
> > on localhost port 4444.
> > 
> > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> 
> At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
> instances". I think Anthony posted such a patch before. Now we should
> really include this as your extension may block the stand-alone -monitor
> switch for the control channel, preventing to additionally set up one
> (or more) for debugging purposes.

If we support multiple monitors, it might be desirable to allow the
app owning the main 'control' monitor channel to be able to indicate
that additional monitor channels are read-only. eg, so libvirt could
allow the user to connect to the monitor to run 'info' commands for
debug support without risk of having state changed behind its back

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 10/11] QMP: Introduce basic events

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 01:29:55AM -0300, Luiz Capitulino wrote:
> Reboot, shutdown and powerdown are very basic events and can be
> added together.
> 
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  monitor.c |    9 +++++++++
>  monitor.h |    3 +++
>  vl.c      |    3 +++
>  3 files changed, 15 insertions(+), 0 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 378a3ae..7ebf0ea 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -276,6 +276,15 @@ void monitor_notify_event(MonitorEvent event)
>      monitor_puts(cur_mon, "* EVENT ");
>  
>      switch (event) {
> +    case EVENT_REBOOT:
> +        monitor_puts(cur_mon, "reboot");
> +        break;
> +    case EVENT_SHUTDOWN:
> +        monitor_puts(cur_mon, "shutdown");
> +        break;
> +    case EVENT_POWERDOWN:
> +        monitor_puts(cur_mon, "powerdown");
> +        break;
>      case EVENT_MAX:
>          // Avoid gcc warning, will never get here
>          break;
> diff --git a/monitor.h b/monitor.h
> index 7cc88cc..579b80b 100644
> --- a/monitor.h
> +++ b/monitor.h
> @@ -14,6 +14,9 @@ extern Monitor *cur_mon;
>  
>  /* QMP events */
>  typedef enum MonitorEvent {
> +    EVENT_REBOOT,
> +    EVENT_SHUTDOWN,
> +    EVENT_POWERDOWN,
>      EVENT_MAX,
>  } MonitorEvent;
>  
> diff --git a/vl.c b/vl.c
> index 60a00e1..7dc5954 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void)
>          reset_requested = 1;
>      }
>      qemu_notify_event();
> +    monitor_notify_event(EVENT_REBOOT);
>  }
>  
>  void qemu_system_shutdown_request(void)
>  {
>      shutdown_requested = 1;
>      qemu_notify_event();
> +    monitor_notify_event(EVENT_SHUTDOWN);
>  }
>  
>  void qemu_system_powerdown_request(void)
>  {
>      powerdown_requested = 1;
>      qemu_notify_event();
> +    monitor_notify_event(EVENT_POWERDOWN);
>  }

I could be mis-reading this code, but this implies that the mgmt app 
would get these events at the time the action is requested, but not
neccessarily actually acted upon. Wouldn't you want to be receiving
the events immediately after the action has been completed ?

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 12:57 PM, Daniel P. Berrange wrote:
>> Maybe add a numeric error code (to be defined by individual commands).
>>      
>
> I think that would be particularly useful to allow clients to distinguish
> the error from a command which does not exist, vs a command that exists
> but failed. There's probably a handful of common errors that you want
> to detect from all commands with the same error handling logic.
>    

There should be mechanisms to detect which commands are available.  Not 
all commands will have fallbacks, so the user might want to query which 
commands (or features) exist before actually launching the guest.

>> The guest reboots (actually, resets), not qemu.  And 'info balloon' will
>> eventually print its response, no?
>>      
>
> Might we need to have timestamp assoicated with each response, or will we
> define that responses are explicitly ordered wrt events,to reflect the order
> in which they're handled. eg When the 'info balloon' response arrives, we
> want to know whether the data it contains is reflecting state before or
> after the reboot.
>    

Responses are implicitly ordered.  But a timestamp for events might be 
good for other reasons (when the event happened vs. when management got 
around to processing it; might be good to correlate with system events).

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 01:29:11AM -0300, Luiz Capitulino wrote:
> It is just an exported function that will be used by other subsystems
> to print specific events to the output buffer.
> 
> This is based in ideas by Amit Shah <amit.shah@redhat.com>.
> 
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>

As per my mail to the lead message in this thread, I'm thinking it
might be valuable to include a timestamp with all events, and command
replies. A 64-bit seconds-since-epoch prefix on all replies, no need
for a human-format timestamp

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command

[Avi Kivity]

On 06/23/2009 12:59 PM, Jan Kiszka wrote:
>>> --- a/block.c
>>> +++ b/block.c
>>> @@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)
>>>        BlockDriverState *bs;
>>>
>>>        for (bs = bdrv_first; bs != NULL; bs = bs->next) {
>>> -        monitor_printf(mon, "%s:"
>>> -                       " rd_bytes=%" PRIu64
>>> -                       " wr_bytes=%" PRIu64
>>> -                       " rd_operations=%" PRIu64
>>> -                       " wr_operations=%" PRIu64
>>> -                       "\n",
>>> -                       bs->device_name,
>>> -                       bs->rd_bytes, bs->wr_bytes,
>>> -                       bs->rd_ops, bs->wr_ops);
>>> +        monitor_printf_data(mon, "%s:"
>>> +                            " rd_bytes=%" PRIu64
>>> +                            " wr_bytes=%" PRIu64
>>> +                            " rd_operations=%" PRIu64
>>> +                            " wr_operations=%" PRIu64
>>> +                            "\n",
>>> +                            bs->device_name,
>>> +                            bs->rd_bytes, bs->wr_bytes,
>>> +                            bs->rd_ops, bs->wr_ops);
>>>
>>>        
>> It may be easier for control mode to drop the variable= prefixes (they
>> are described in the spec anyway).
>>
>>      
>
> Isn't monitor_printf_data also used for pretty-printing in case of
> human-operated monitors?

It is.

> I think we need to keep this - or provide two
> output formats which are filtered by the monitor based on the the target
> terminal's mode. But that sounds like a lot of work /wrt conversion and
> long-term maintenance.
>    

I prefer separate printfs.  We don't want to allow choices made for the 
human protocol to influence the machine protocol, the whole point of the 
machine protocol was to have a clean break.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command

[Jan Kiszka]

Avi Kivity wrote:
> On 06/23/2009 12:59 PM, Jan Kiszka wrote:
>>>> --- a/block.c
>>>> +++ b/block.c
>>>> @@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)
>>>>        BlockDriverState *bs;
>>>>
>>>>        for (bs = bdrv_first; bs != NULL; bs = bs->next) {
>>>> -        monitor_printf(mon, "%s:"
>>>> -                       " rd_bytes=%" PRIu64
>>>> -                       " wr_bytes=%" PRIu64
>>>> -                       " rd_operations=%" PRIu64
>>>> -                       " wr_operations=%" PRIu64
>>>> -                       "\n",
>>>> -                       bs->device_name,
>>>> -                       bs->rd_bytes, bs->wr_bytes,
>>>> -                       bs->rd_ops, bs->wr_ops);
>>>> +        monitor_printf_data(mon, "%s:"
>>>> +                            " rd_bytes=%" PRIu64
>>>> +                            " wr_bytes=%" PRIu64
>>>> +                            " rd_operations=%" PRIu64
>>>> +                            " wr_operations=%" PRIu64
>>>> +                            "\n",
>>>> +                            bs->device_name,
>>>> +                            bs->rd_bytes, bs->wr_bytes,
>>>> +                            bs->rd_ops, bs->wr_ops);
>>>>
>>>>        
>>> It may be easier for control mode to drop the variable= prefixes (they
>>> are described in the spec anyway).
>>>
>>>      
>>
>> Isn't monitor_printf_data also used for pretty-printing in case of
>> human-operated monitors?
> 
> It is.
> 
>> I think we need to keep this - or provide two
>> output formats which are filtered by the monitor based on the the target
>> terminal's mode. But that sounds like a lot of work /wrt conversion and
>> long-term maintenance.
>>    
> 
> I prefer separate printfs.  We don't want to allow choices made for the
> human protocol to influence the machine protocol, the whole point of the
> machine protocol was to have a clean break.

OK... yeah, makes sense. Then the above hunk must not remove the
existing monitor_printf, just add new monitor_printf_data.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Daniel P. Berrange wrote:
> On Tue, Jun 23, 2009 at 11:03:09AM +0200, Jan Kiszka wrote:
>> Luiz Capitulino wrote:
>>> This change adds a flag called 'control' to the already existing
>>> '-monitor' command-line option. This flag can be used to enable
>>> control mode.
>>>
>>> Its syntax is:
>>>
>>> qemu [...] -monitor control,<device>
>>>
>>> Where <device> is a chardev (excluding 'vc', for obvious reasons).
>>>
>>> For example:
>>>
>>> $ qemu [...] -monitor control,tcp:localhost:4444,server
>>>
>>> Will run QEMU in control mode, waiting for a client TCP connection
>>> on localhost port 4444.
>>>
>>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>> At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
>> instances". I think Anthony posted such a patch before. Now we should
>> really include this as your extension may block the stand-alone -monitor
>> switch for the control channel, preventing to additionally set up one
>> (or more) for debugging purposes.
> 
> If we support multiple monitors, it might be desirable to allow the
> app owning the main 'control' monitor channel to be able to indicate
> that additional monitor channels are read-only. eg, so libvirt could
> allow the user to connect to the monitor to run 'info' commands for
> debug support without risk of having state changed behind its back

Couldn't libvirt deal with update given we provide them as events?
Otherwise, your suggestion makes sense, definitely as long as it would
wreck libvirt's internal house keeping or for commands that are not
synchronizable.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Avi Kivity]

On 06/23/2009 01:08 PM, Daniel P. Berrange wrote:
> On Tue, Jun 23, 2009 at 01:29:11AM -0300, Luiz Capitulino wrote:
>    
>> It is just an exported function that will be used by other subsystems
>> to print specific events to the output buffer.
>>
>> This is based in ideas by Amit Shah<amit.shah@redhat.com>.
>>
>> Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
>>      
>
> As per my mail to the lead message in this thread, I'm thinking it
> might be valuable to include a timestamp with all events, and command
> replies. A 64-bit seconds-since-epoch prefix on all replies, no need
> for a human-format timestamp
>    

A bit of an overkill... seconds.nanoseconds.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Avi Kivity]

On 06/23/2009 01:11 PM, Jan Kiszka wrote:
>> If we support multiple monitors, it might be desirable to allow the
>> app owning the main 'control' monitor channel to be able to indicate
>> that additional monitor channels are read-only. eg, so libvirt could
>> allow the user to connect to the monitor to run 'info' commands for
>> debug support without risk of having state changed behind its back
>>      
>
> Couldn't libvirt deal with update given we provide them as events?
> Otherwise, your suggestion makes sense, definitely as long as it would
> wreck libvirt's internal house keeping or for commands that are not
> synchronizable.
>    

It would be a nightmare, consider both libvirt and a user hotplugging 
something into the same pci slot, or a user starting migration, or 
quitting, or ...

Having -monitor readonly makes sense.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 12:11:09PM +0200, Jan Kiszka wrote:
> Daniel P. Berrange wrote:
> > On Tue, Jun 23, 2009 at 11:03:09AM +0200, Jan Kiszka wrote:
> >> Luiz Capitulino wrote:
> >>> This change adds a flag called 'control' to the already existing
> >>> '-monitor' command-line option. This flag can be used to enable
> >>> control mode.
> >>>
> >>> Its syntax is:
> >>>
> >>> qemu [...] -monitor control,<device>
> >>>
> >>> Where <device> is a chardev (excluding 'vc', for obvious reasons).
> >>>
> >>> For example:
> >>>
> >>> $ qemu [...] -monitor control,tcp:localhost:4444,server
> >>>
> >>> Will run QEMU in control mode, waiting for a client TCP connection
> >>> on localhost port 4444.
> >>>
> >>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> >> At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
> >> instances". I think Anthony posted such a patch before. Now we should
> >> really include this as your extension may block the stand-alone -monitor
> >> switch for the control channel, preventing to additionally set up one
> >> (or more) for debugging purposes.
> > 
> > If we support multiple monitors, it might be desirable to allow the
> > app owning the main 'control' monitor channel to be able to indicate
> > that additional monitor channels are read-only. eg, so libvirt could
> > allow the user to connect to the monitor to run 'info' commands for
> > debug support without risk of having state changed behind its back
> 
> Couldn't libvirt deal with update given we provide them as events?
> Otherwise, your suggestion makes sense, definitely as long as it would
> wreck libvirt's internal house keeping or for commands that are not
> synchronizable.

If the mgmt app knows about & supports all features in the QEMU its 
talking to, and we generate events for all possible changes, then it
could detect & cope with changes. In reality though I don't think
you can expect mgmt apps to have complete coverage of all features,
so it would be safer for them to be able to designate additional 
monitor channels readonly.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 01:29:11AM -0300, Luiz Capitulino wrote:
> It is just an exported function that will be used by other subsystems
> to print specific events to the output buffer.
> 
> This is based in ideas by Amit Shah <amit.shah@redhat.com>.
> 
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  monitor.c   |   18 ++++++++++++++++++
>  monitor.h   |    6 ++++++
>  qemu-tool.c |    4 ++++
>  3 files changed, 28 insertions(+), 0 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 462f60b..df58bdd 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)
>      va_end(ap);
>  }
>  
> +/* Asynchronous events main function */
> +void monitor_notify_event(MonitorEvent event)
> +{
> +    if (!monitor_ctrl_mode(cur_mon))
> +        return;
> +
> +    assert(event < EVENT_MAX);
> +    monitor_puts(cur_mon, "* EVENT ");
> +
> +    switch (event) {
> +    case EVENT_MAX:
> +        // Avoid gcc warning, will never get here
> +        break;
> +    }
> +
> +    monitor_puts(cur_mon, "\n");
> +}
> +

If a client is not reading from the monitor channel quickly enough, then
would this cause QEMU to block once the FD buffer is full ? A QEMU level
buffer might give more leyway, but we don't want to grow unbounded, so
ultimately we'll end up having to drop events. At which point you'd want
to send an event to the client indicating that the event queue overflowed,
so it can take remedial action to re-sync its state.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Avi Kivity wrote:
> On 06/23/2009 01:11 PM, Jan Kiszka wrote:
>>> If we support multiple monitors, it might be desirable to allow the
>>> app owning the main 'control' monitor channel to be able to indicate
>>> that additional monitor channels are read-only. eg, so libvirt could
>>> allow the user to connect to the monitor to run 'info' commands for
>>> debug support without risk of having state changed behind its back
>>>      
>>
>> Couldn't libvirt deal with update given we provide them as events?
>> Otherwise, your suggestion makes sense, definitely as long as it would
>> wreck libvirt's internal house keeping or for commands that are not
>> synchronizable.
>>    
> 
> It would be a nightmare, consider both libvirt and a user hotplugging
> something into the same pci slot, or a user starting migration, or
> quitting, or ...

Migration, yes, but hot-plugging is resolvable - given config update events.

> 
> Having -monitor readonly makes sense.

But not for all features, only those libvirt is capable to handle (I
think there are quite a few qemu specifics libvirt does not bother
about) and only as long as there is no proper synchronization. Again,
migration and save/restore will continue to require exclusive access,
but the rest is just a question of proper synchronization IMHO.

See, I don't want to kill my management app just because I attached to a
guest via gdb and start injecting reconfiguration events for testing
purposes (e.g. attach/detach a USB device for which I develop a driver).

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Daniel P. Berrange wrote:
> On Tue, Jun 23, 2009 at 12:11:09PM +0200, Jan Kiszka wrote:
>> Daniel P. Berrange wrote:
>>> On Tue, Jun 23, 2009 at 11:03:09AM +0200, Jan Kiszka wrote:
>>>> Luiz Capitulino wrote:
>>>>> This change adds a flag called 'control' to the already existing
>>>>> '-monitor' command-line option. This flag can be used to enable
>>>>> control mode.
>>>>>
>>>>> Its syntax is:
>>>>>
>>>>> qemu [...] -monitor control,<device>
>>>>>
>>>>> Where <device> is a chardev (excluding 'vc', for obvious reasons).
>>>>>
>>>>> For example:
>>>>>
>>>>> $ qemu [...] -monitor control,tcp:localhost:4444,server
>>>>>
>>>>> Will run QEMU in control mode, waiting for a client TCP connection
>>>>> on localhost port 4444.
>>>>>
>>>>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>>>> At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
>>>> instances". I think Anthony posted such a patch before. Now we should
>>>> really include this as your extension may block the stand-alone -monitor
>>>> switch for the control channel, preventing to additionally set up one
>>>> (or more) for debugging purposes.
>>> If we support multiple monitors, it might be desirable to allow the
>>> app owning the main 'control' monitor channel to be able to indicate
>>> that additional monitor channels are read-only. eg, so libvirt could
>>> allow the user to connect to the monitor to run 'info' commands for
>>> debug support without risk of having state changed behind its back
>> Couldn't libvirt deal with update given we provide them as events?
>> Otherwise, your suggestion makes sense, definitely as long as it would
>> wreck libvirt's internal house keeping or for commands that are not
>> synchronizable.
> 
> If the mgmt app knows about & supports all features in the QEMU its 
> talking to, and we generate events for all possible changes, then it
> could detect & cope with changes. In reality though I don't think
> you can expect mgmt apps to have complete coverage of all features,
> so it would be safer for them to be able to designate additional 
> monitor channels readonly.

Also reading your valid concerns regarding event queue overflows, I
think making it flexible /wrt asynchronous changes will actually make
the whole think much more robust than simply trying to block every
competing change.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Avi Kivity]

On 06/23/2009 01:32 PM, Daniel P. Berrange wrote:
>> +/* Asynchronous events main function */
>> +void monitor_notify_event(MonitorEvent event)
>> +{
>> +    if (!monitor_ctrl_mode(cur_mon))
>> +        return;
>> +
>> +    assert(event<  EVENT_MAX);
>> +    monitor_puts(cur_mon, "* EVENT ");
>> +
>> +    switch (event) {
>> +    case EVENT_MAX:
>> +        // Avoid gcc warning, will never get here
>> +        break;
>> +    }
>> +
>> +    monitor_puts(cur_mon, "\n");
>> +}
>> +
>>      
>
> If a client is not reading from the monitor channel quickly enough, then
> would this cause QEMU to block once the FD buffer is full ? A QEMU level
> buffer might give more leyway, but we don't want to grow unbounded, so
> ultimately we'll end up having to drop events. At which point you'd want
> to send an event to the client indicating that the event queue overflowed,
> so it can take remedial action to re-sync its state.
>    

IMO relying on kernel buffers is sufficient here.  An 8k buffer will 
hold hundreds of events.  If the system is so congested that management 
cannot consume events rapidly enough, then it will also be so congested 
that guest vcpus will be starved.

Blocking on buffers full is acceptable IMO.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Avi Kivity]

On 06/23/2009 01:54 PM, Jan Kiszka wrote:
>> It would be a nightmare, consider both libvirt and a user hotplugging
>> something into the same pci slot, or a user starting migration, or
>> quitting, or ...
>>      
>
> Migration, yes, but hot-plugging is resolvable - given config update events.
>    

I'm not saying it's impossible, just that we don't want to do it.

>> Having -monitor readonly makes sense.
>>      
>
> But not for all features, only those libvirt is capable to handle (I
> think there are quite a few qemu specifics libvirt does not bother
> about) and only as long as there is no proper synchronization. Again,
> migration and save/restore will continue to require exclusive access,
> but the rest is just a question of proper synchronization IMHO.
>    

If libvirt "doesn't bother" about something useful, that's a libvirt 
bug.  It's wierd to have something controlled by two parallel management 
channels.

> See, I don't want to kill my management app just because I attached to a
> guest via gdb and start injecting reconfiguration events for testing
> purposes (e.g. attach/detach a USB device for which I develop a driver).
>    

You're talking about a usb developer, not a qemu developer, working in 
some virtual lab environment?  In that case, libvirt and the management 
app should certainly support usb.  A random usb developer shouldn't need 
to ever talk to the qemu monitor.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Avi Kivity wrote:
> On 06/23/2009 01:54 PM, Jan Kiszka wrote:
>>> It would be a nightmare, consider both libvirt and a user hotplugging
>>> something into the same pci slot, or a user starting migration, or
>>> quitting, or ...
>>>      
>>
>> Migration, yes, but hot-plugging is resolvable - given config update
>> events.
>>    
> 
> I'm not saying it's impossible, just that we don't want to do it.

I agree it's not something that is mandatory for the first versions, but
disagree that we should not strive for achieving this level of support
mid- to long-term. No design decision made today should prevent it, but
rather keep that door open. Also for robustness reasons (if you loose
contact to your qemu backend and need to resync the management frontend).

> 
>>> Having -monitor readonly makes sense.
>>>      
>>
>> But not for all features, only those libvirt is capable to handle (I
>> think there are quite a few qemu specifics libvirt does not bother
>> about) and only as long as there is no proper synchronization. Again,
>> migration and save/restore will continue to require exclusive access,
>> but the rest is just a question of proper synchronization IMHO.
>>    
> 
> If libvirt "doesn't bother" about something useful, that's a libvirt
> bug.  It's wierd to have something controlled by two parallel management
> channels.

I think the libvirt policy is that things which are not generic enough
to be supported by >1 hypervisor are left alone.

> 
>> See, I don't want to kill my management app just because I attached to a
>> guest via gdb and start injecting reconfiguration events for testing
>> purposes (e.g. attach/detach a USB device for which I develop a driver).
>>    
> 
> You're talking about a usb developer, not a qemu developer, working in
> some virtual lab environment?  In that case, libvirt and the management
> app should certainly support usb.  A random usb developer shouldn't need
> to ever talk to the qemu monitor.

I'm talking about, just as one example, sitting in front of my gdb
[frontend], doing guest kernel [driver] debugging and issuing "monitor
whatever" commands from one place, ie. not having to switch between
management app and debugging interface back and forth.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 02:28:38PM +0300, Avi Kivity wrote:
> On 06/23/2009 01:54 PM, Jan Kiszka wrote:
> >>It would be a nightmare, consider both libvirt and a user hotplugging
> >>something into the same pci slot, or a user starting migration, or
> >>quitting, or ...
> >>     
> >
> >Migration, yes, but hot-plugging is resolvable - given config update 
> >events.
> >   
> 
> I'm not saying it's impossible, just that we don't want to do it.
> 
> >>Having -monitor readonly makes sense.
> >>     
> >
> >But not for all features, only those libvirt is capable to handle (I
> >think there are quite a few qemu specifics libvirt does not bother
> >about) and only as long as there is no proper synchronization. Again,
> >migration and save/restore will continue to require exclusive access,
> >but the rest is just a question of proper synchronization IMHO.
> >   
> 
> If libvirt "doesn't bother" about something useful, that's a libvirt 
> bug.  It's wierd to have something controlled by two parallel management 
> channels.

It is not neccessaryily a 'bug'. The issue is that there is always a
potential time lag between a feature appearing in QEMU, and it being
fully supported in libvirt. So it is not a case of "doesnt' bother",
but rather 'not yet known about'. I think we should make it possible
for apps to be robust in this scenario, by allowing them to lockdown
2ndary monitor channels readonly. 

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Avi Kivity]

On 06/23/2009 02:44 PM, Jan Kiszka wrote:
>> I'm not saying it's impossible, just that we don't want to do it.
>>      
>
> I agree it's not something that is mandatory for the first versions, but
> disagree that we should not strive for achieving this level of support
> mid- to long-term. No design decision made today should prevent it, but
> rather keep that door open. Also for robustness reasons (if you loose
> contact to your qemu backend and need to resync the management frontend).
>    

No objections here.  Certainly reconnect is important.
>> If libvirt "doesn't bother" about something useful, that's a libvirt
>> bug.  It's wierd to have something controlled by two parallel management
>> channels.
>>      
>
> I think the libvirt policy is that things which are not generic enough
> to be supported by>1 hypervisor are left alone.
>    

That's a bit offtopic here, but it appears to me this dooms libvirt to 
be useless in all but the most basic use cases.  Hypervisor writers try 
to new features to differentiate their products, and forcing users off 
libvirt in order to use them seems to be counterproductive.

>>> See, I don't want to kill my management app just because I attached to a
>>> guest via gdb and start injecting reconfiguration events for testing
>>> purposes (e.g. attach/detach a USB device for which I develop a driver).
>>>
>>>        
>> You're talking about a usb developer, not a qemu developer, working in
>> some virtual lab environment?  In that case, libvirt and the management
>> app should certainly support usb.  A random usb developer shouldn't need
>> to ever talk to the qemu monitor.
>>      
>
> I'm talking about, just as one example, sitting in front of my gdb
> [frontend], doing guest kernel [driver] debugging and issuing "monitor
> whatever" commands from one place, ie. not having to switch between
> management app and debugging interface back and forth.
>    

Can gdb issue monitor commands?

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Avi Kivity wrote:
> On 06/23/2009 02:44 PM, Jan Kiszka wrote:
>>> I'm not saying it's impossible, just that we don't want to do it.
>>>      
>>
>> I agree it's not something that is mandatory for the first versions, but
>> disagree that we should not strive for achieving this level of support
>> mid- to long-term. No design decision made today should prevent it, but
>> rather keep that door open. Also for robustness reasons (if you loose
>> contact to your qemu backend and need to resync the management frontend).
>>    
> 
> No objections here.  Certainly reconnect is important.
>>> If libvirt "doesn't bother" about something useful, that's a libvirt
>>> bug.  It's wierd to have something controlled by two parallel management
>>> channels.
>>>      
>>
>> I think the libvirt policy is that things which are not generic enough
>> to be supported by>1 hypervisor are left alone.
>>    
> 
> That's a bit offtopic here, but it appears to me this dooms libvirt to
> be useless in all but the most basic use cases.  Hypervisor writers try
> to new features to differentiate their products, and forcing users off
> libvirt in order to use them seems to be counterproductive.

That was my personal interpretation, maybe based on incomplete information.

> 
>>>> See, I don't want to kill my management app just because I attached
>>>> to a
>>>> guest via gdb and start injecting reconfiguration events for testing
>>>> purposes (e.g. attach/detach a USB device for which I develop a
>>>> driver).
>>>>
>>>>        
>>> You're talking about a usb developer, not a qemu developer, working in
>>> some virtual lab environment?  In that case, libvirt and the management
>>> app should certainly support usb.  A random usb developer shouldn't need
>>> to ever talk to the qemu monitor.
>>>      
>>
>> I'm talking about, just as one example, sitting in front of my gdb
>> [frontend], doing guest kernel [driver] debugging and issuing "monitor
>> whatever" commands from one place, ie. not having to switch between
>> management app and debugging interface back and forth.
>>    
> 
> Can gdb issue monitor commands?

Yes.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 02:51:18PM +0300, Avi Kivity wrote:
> On 06/23/2009 02:44 PM, Jan Kiszka wrote:
> >>If libvirt "doesn't bother" about something useful, that's a libvirt
> >>bug.  It's wierd to have something controlled by two parallel management
> >>channels.
> >>     
> >
> >I think the libvirt policy is that things which are not generic enough
> >to be supported by>1 hypervisor are left alone.
> >   
> 
> That's a bit offtopic here, but it appears to me this dooms libvirt to 
> be useless in all but the most basic use cases.  Hypervisor writers try 
> to new features to differentiate their products, and forcing users off 
> libvirt in order to use them seems to be counterproductive.

libvirt is *not* restricted to 'lowest common denominator'. We are fine
with supporting features that are only supported in 1 hypervisor. Our
goal is to specify a generic representation of the feature, rather than
directly exposing the underlying syntax of the HV. 


Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Avi Kivity]

On 06/23/2009 02:53 PM, Jan Kiszka wrote:
>>> I'm talking about, just as one example, sitting in front of my gdb
>>> [frontend], doing guest kernel [driver] debugging and issuing "monitor
>>> whatever" commands from one place, ie. not having to switch between
>>> management app and debugging interface back and forth.
>>>
>>>        
>> Can gdb issue monitor commands?
>>      
>
> Yes.
>    

How does one do that?  seems a very useful feature.

I could say that management can proxy the gdb packets and thus support 
its own cli (in fact, it must if it wants to support live migration), 
but that's really an edge case and I doubt anyone will do that, so you 
have a good point.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Avi Kivity]

On 06/23/2009 02:48 PM, Daniel P. Berrange wrote:
>>> But not for all features, only those libvirt is capable to handle (I
>>> think there are quite a few qemu specifics libvirt does not bother
>>> about) and only as long as there is no proper synchronization. Again,
>>> migration and save/restore will continue to require exclusive access,
>>> but the rest is just a question of proper synchronization IMHO.
>>>
>>>        
>> If libvirt "doesn't bother" about something useful, that's a libvirt
>> bug.  It's wierd to have something controlled by two parallel management
>> channels.
>>      
>
> It is not neccessaryily a 'bug'. The issue is that there is always a
> potential time lag between a feature appearing in QEMU, and it being
> fully supported in libvirt. So it is not a case of "doesnt' bother",
> but rather 'not yet known about'. I think we should make it possible
> for apps to be robust in this scenario, by allowing them to lockdown
> 2ndary monitor channels readonly.
>    

In this case I don't see much of an issue, since the time to implement a 
feature in libvirt is probably much shorter than the time needed to 
implement a proper GUI, database support, etc. that's needed for most 
features.  Short cutting through another interface is likely to cost 
more time than it saves.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Avi Kivity wrote:
> On 06/23/2009 02:53 PM, Jan Kiszka wrote:
>>>> I'm talking about, just as one example, sitting in front of my gdb
>>>> [frontend], doing guest kernel [driver] debugging and issuing "monitor
>>>> whatever" commands from one place, ie. not having to switch between
>>>> management app and debugging interface back and forth.
>>>>
>>>>        
>>> Can gdb issue monitor commands?
>>>      
>>
>> Yes.
>>    
> 
> How does one do that?  seems a very useful feature.

By spawning an additional monitor terminal, but switching off its
readline support (as that is handled by gdb).

In case there is still someone out there not being aware of this ;) : we
_do_ have support for multiple monitors in qemu already. Just try
"-monitor X -serial mon:Y" with X!=Y.

> 
> I could say that management can proxy the gdb packets and thus support
> its own cli (in fact, it must if it wants to support live migration),
> but that's really an edge case and I doubt anyone will do that, so you
> have a good point.

Proxying would mean interpreting all "classic" monitor commands to catch
those that may interfere with the mgmt app state. I also don't think
that is worth the effort.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Avi Kivity]

On 06/23/2009 03:39 PM, Jan Kiszka wrote:
> Avi Kivity wrote:
>    
>> On 06/23/2009 02:53 PM, Jan Kiszka wrote:
>>      
>>>>> I'm talking about, just as one example, sitting in front of my gdb
>>>>> [frontend], doing guest kernel [driver] debugging and issuing "monitor
>>>>> whatever" commands from one place, ie. not having to switch between
>>>>> management app and debugging interface back and forth.
>>>>>
>>>>>
>>>>>            
>>>> Can gdb issue monitor commands?
>>>>
>>>>          
>>> Yes.
>>>
>>>        
>> How does one do that?  seems a very useful feature.
>>      
>
> By spawning an additional monitor terminal, but switching off its
> readline support (as that is handled by gdb).
>
> In case there is still someone out there not being aware of this ;) : we
> _do_ have support for multiple monitors in qemu already. Just try
> "-monitor X -serial mon:Y" with X!=Y.
>
>    
>> I could say that management can proxy the gdb packets and thus support
>> its own cli (in fact, it must if it wants to support live migration),
>> but that's really an edge case and I doubt anyone will do that, so you
>> have a good point.
>>      
>
> Proxying would mean interpreting all "classic" monitor commands to catch
> those that may interfere with the mgmt app state. I also don't think
> that is worth the effort.
>    

No, you would use the management system's cli which naturally enforces 
all of a user's permissions and is integrated with the GUI:

gui ----\
cli ----+---[ management server ] ---- [ libvirt ] ---- qemu
gdb ----/

That's the only setup that can support migration without dropping 
sessions.  That doesn't change the "I don't think it's worth the effort" 
state though.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Avi Kivity]

On 06/23/2009 03:39 PM, Jan Kiszka wrote:
>>>> Can gdb issue monitor commands?
>>>>
>>>>          
>>> Yes.
>>>
>>>        
>> How does one do that?  seems a very useful feature.
>>      
>
> By spawning an additional monitor terminal, but switching off its
> readline support (as that is handled by gdb).
>    

Within gdb, how do you tell it to send a line to the monitor?

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

On Tue, 23 Jun 2009 11:55:56 +0300
Avi Kivity <avi@redhat.com> wrote:

> On 06/23/2009 07:28 AM, Luiz Capitulino wrote:
> > This file contains detailed QMP description and definitions.
> >
> > Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
> > ---
> >   monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
> >   1 files changed, 180 insertions(+), 0 deletions(-)
> >   create mode 100644 monitor-protocol-spec.txt
> >
> > diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
> > new file mode 100644
> > index 0000000..d20e3f9
> > --- /dev/null
> > +++ b/monitor-protocol-spec.txt
> > @@ -0,0 +1,180 @@
> > +               QEMU Monitor Protocol Specification - Version 0.1
> > +
> > +                                Luiz Capitulino
> > +<lcapitulino@redhat.com>
> > +
> > +1. Introduction
> > +===============
> > +
> > +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol
> > +which is available for applications to control QEMU at the machine-level.
> >    
> 
> Without a doubt, this is a the most important file of this patchset.  
> There's a huge difference between working with an implementation and 
> working with a specification.

 Exactly.

> > +
> > +For a detailed list of supported commands, please, refer to file
> > +monitor-protocol-commands.txt.
> >    
> 
> I don't see you update that file anywhere.  In any case, my preference 
> would be to have everything in one file.

 Sorry, I forgot to mention that I wouldn't include this file
in the patchset.

 But its format will be like POP3 or IMAP specs, something like:

"""
6.2 cont

Arguments: none.

Responses: + OK cont completed

<description>

<example>
"""

 We can have everything in the same file then.

> > +3.1 General definition
> > +----------------------
> > +
> > + o Only ASCII is permitted
> >    
> 
> Since the some commands contain user-specified strings, UTF-8 is needed.
> 
> I think it's worthwhile to define a quoted string format, to be used 
> both in commands and responses.

 Ok, as Daniel also said we will need this.

 The only problem is that I was trying to avoid changing commands'
output too much, because this can turn into a maintenance nightmare.

 Note that we will have to maintain two protocols: the current existing
(user mode) and this new control mode.

> > +
> > +3.3.1 Server Greeting
> > +---------------------
> > +
> > +Sent when a new connection is opened.
> > +
> > +Format: + OK QEMU<version>  QMP<version>
> > +Example: + OK QEMU 0.10.50 QMP 0.1
> >    
> 
> Clients should never make decisions based on the qemu or qmp version.  
> Rather, we should provide a facility to query the availability of features.

 Right. What about changing 'help' output to only list command names?

> > + o Command completion failed
> > +
> > +    Format: - ERR<reason>
> > +    Example: - ERR could not find network device 'foo'
> > +
> >    
> 
> Maybe add a numeric error code (to be defined by individual commands).

 Well, numeric error codes seem redundant in text protocols. At least,
POP3 and IMAP don't use them.

 This may seem contradictory, but although this a low-level protocol,
I feel that text protocols tend to be very descriptive.

> > +4.3 Events
> > +----------
> > +
> > +Client queries the Server about memory, but QEMU reboots.
> > +
> > +S: + OK QEMU 0.10.50 QMP 0.1
> > +C: info balloon
> > +S: * EVENT reboot
> >    
> 
> The guest reboots (actually, resets), not qemu.  And 'info balloon' will 
> eventually print its response, no?

Yes, it will. I will fix this.

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 10:06:15AM -0300, Luiz Capitulino wrote:
> On Tue, 23 Jun 2009 11:55:56 +0300
> Avi Kivity <avi@redhat.com> wrote:
> 
> > On 06/23/2009 07:28 AM, Luiz Capitulino wrote:
> > > This file contains detailed QMP description and definitions.
> > >
> > > Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
> > > ---
> > >   monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
> > >   1 files changed, 180 insertions(+), 0 deletions(-)
> > >   create mode 100644 monitor-protocol-spec.txt
> > >
> > > diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
> > > new file mode 100644
> > > index 0000000..d20e3f9
> > > --- /dev/null
> > > +++ b/monitor-protocol-spec.txt
> > > @@ -0,0 +1,180 @@
> > > +               QEMU Monitor Protocol Specification - Version 0.1
> > > +
> > > +                                Luiz Capitulino
> > > +<lcapitulino@redhat.com>
> > > +
> > > +1. Introduction
> > > +===============
> > > +
> > > +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol
> > > +which is available for applications to control QEMU at the machine-level.
> > >    
> > 
> > Without a doubt, this is a the most important file of this patchset.  
> > There's a huge difference between working with an implementation and 
> > working with a specification.
> 
>  Exactly.
> 
> > > +
> > > +For a detailed list of supported commands, please, refer to file
> > > +monitor-protocol-commands.txt.
> > >    
> > 
> > I don't see you update that file anywhere.  In any case, my preference 
> > would be to have everything in one file.
> 
>  Sorry, I forgot to mention that I wouldn't include this file
> in the patchset.
> 
>  But its format will be like POP3 or IMAP specs, something like:
> 
> """
> 6.2 cont
> 
> Arguments: none.
> 
> Responses: + OK cont completed
> 
> <description>
> 
> <example>
> """
> 
>  We can have everything in the same file then.
> 
> > > +3.1 General definition
> > > +----------------------
> > > +
> > > + o Only ASCII is permitted
> > >    
> > 
> > Since the some commands contain user-specified strings, UTF-8 is needed.
> > 
> > I think it's worthwhile to define a quoted string format, to be used 
> > both in commands and responses.
> 
>  Ok, as Daniel also said we will need this.
> 
>  The only problem is that I was trying to avoid changing commands'
> output too much, because this can turn into a maintenance nightmare.
> 
>  Note that we will have to maintain two protocols: the current existing
> (user mode) and this new control mode.
> 
> > > +
> > > +3.3.1 Server Greeting
> > > +---------------------
> > > +
> > > +Sent when a new connection is opened.
> > > +
> > > +Format: + OK QEMU<version>  QMP<version>
> > > +Example: + OK QEMU 0.10.50 QMP 0.1
> > >    
> > 
> > Clients should never make decisions based on the qemu or qmp version.  
> > Rather, we should provide a facility to query the availability of features.
> 
>  Right. What about changing 'help' output to only list command names?
> 
> > > + o Command completion failed
> > > +
> > > +    Format: - ERR<reason>
> > > +    Example: - ERR could not find network device 'foo'
> > > +
> > >    
> > 
> > Maybe add a numeric error code (to be defined by individual commands).
> 
>  Well, numeric error codes seem redundant in text protocols. At least,
> POP3 and IMAP don't use them.

I counter with FTP and HTTP, which do :-P

eg  http://www.faqs.org/rfcs/rfc959.html

         200 Command okay.
         500 Syntax error, command unrecognized.
             This may include errors such as command line too long.
         501 Syntax error in parameters or arguments.
         202 Command not implemented, superfluous at this site.
         502 Command not implemented.
         503 Bad sequence of commands.
         504 Command not implemented for that parameter.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Hi Luiz,

The specification looks pretty good.


>> +
>> +3.3.1 Server Greeting
>> +---------------------
>> +
>> +Sent when a new connection is opened.
>> +
>> +Format: + OK QEMU<version>  QMP<version>
>> +Example: + OK QEMU 0.10.50 QMP 0.1
>>    
>
> Clients should never make decisions based on the qemu or qmp version.  
> Rather, we should provide a facility to query the availability of 
> features.

I agree, but I'd suggest leaving the QMP version in there for insurance 
purposes in case we really screw up and need to bump the version.  In 
fact, having the client also negotiate the QMP version isn't a bad idea.

>> + o Command completion failed
>> +
>> +    Format: - ERR<reason>
>> +    Example: - ERR could not find network device 'foo'
>> +
>>    
>
> Maybe add a numeric error code (to be defined by individual commands).

I think copying HTTP makes sense here.

How would asynchronous commands work?  Could you give an example of 
doing live migration through QMP?

Regards,

Anthony Liguori

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

On Tue, 23 Jun 2009 10:57:54 +0100
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> > >+ o Command completion failed
> > >+
> > >+    Format: - ERR<reason>
> > >+    Example: - ERR could not find network device 'foo'
> > >+
> > >   
> > 
> > Maybe add a numeric error code (to be defined by individual commands).
> 
> I think that would be particularly useful to allow clients to distinguish
> the error from a command which does not exist, vs a command that exists
> but failed. There's probably a handful of common errors that you want
> to detect from all commands with the same error handling logic. 

 I have followed IMAP's design on this, which specifies that commands
execution errors are "- ERR" and protocol errors are "- BAD".

 So, commands that don't exist are protocol errors and applications
can distinguish them.

> > >+4.3 Events
> > >+----------
> > >+
> > >+Client queries the Server about memory, but QEMU reboots.
> > >+
> > >+S: + OK QEMU 0.10.50 QMP 0.1
> > >+C: info balloon
> > >+S: * EVENT reboot
> > >   
> > 
> > The guest reboots (actually, resets), not qemu.  And 'info balloon' will 
> > eventually print its response, no?
> 
> Might we need to have timestamp assoicated with each response, or will we
> define that responses are explicitly ordered wrt events,to reflect the order
> in which they're handled. eg When the 'info balloon' response arrives, we
> want to know whether the data it contains is reflecting state before or
> after the reboot. 

 They will be ordered, but I liked the timestamp idea. Do you think it's
going to useful only for events, as suggested by Avi?

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 04:06 PM, Luiz Capitulino wrote:
>> I don't see you update that file anywhere.  In any case, my preference
>> would be to have everything in one file.
>>      
>
>   Sorry, I forgot to mention that I wouldn't include this file
> in the patchset.
>    

You mean now, or forever?

>   Ok, as Daniel also said we will need this.
>
>   The only problem is that I was trying to avoid changing commands'
> output too much, because this can turn into a maintenance nightmare.
>
>   Note that we will have to maintain two protocols: the current existing
> (user mode) and this new control mode.
>    

I don't see a way out.  The current protocol is not machine consumable, 
but we can't change it.  We'll just have to live with it.

>> Clients should never make decisions based on the qemu or qmp version.
>> Rather, we should provide a facility to query the availability of features.
>>      
>
>   Right. What about changing 'help' output to only list command names?
>    

Also command subfeatures, when new subfeatures are added.

>>
>> Maybe add a numeric error code (to be defined by individual commands).
>>      
>
>   Well, numeric error codes seem redundant in text protocols. At least,
> POP3 and IMAP don't use them.
>
>   This may seem contradictory, but although this a low-level protocol,
> I feel that text protocols tend to be very descriptive.
>    

Well, the strings are not single words, so the key for the application 
is "all words until the end of the line".  Not too machine friendly.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 04:12 PM, Anthony Liguori wrote:
> Hi Luiz,
>
> The specification looks pretty good.
>
>
>>> +
>>> +3.3.1 Server Greeting
>>> +---------------------
>>> +
>>> +Sent when a new connection is opened.
>>> +
>>> +Format: + OK QEMU<version>  QMP<version>
>>> +Example: + OK QEMU 0.10.50 QMP 0.1
>>
>> Clients should never make decisions based on the qemu or qmp 
>> version.  Rather, we should provide a facility to query the 
>> availability of features.
>
> I agree, but I'd suggest leaving the QMP version in there for 
> insurance purposes in case we really screw up and need to bump the 
> version.  In fact, having the client also negotiate the QMP version 
> isn't a bad idea.

Agreed.

> How would asynchronous commands work? 

IMO, there aren't any.  All commands are synchronous (but may cause 
events to be generated later).

> Could you give an example of doing live migration through QMP?
>

migrate tcp:blah:4444
+OK don't you love us anymore?
* EVENT stopped (migration)
* EVENT migration completed


-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 04:22 PM, Luiz Capitulino wrote:
>   They will be ordered, but I liked the timestamp idea. Do you think it's
> going to useful only for events, as suggested by Avi?
>
>    

Might be useful for synchronous responses as well.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Luiz Capitulino]

On Tue, 23 Jun 2009 10:59:27 +0200
Jan Kiszka <jan.kiszka@siemens.com> wrote:

> Luiz Capitulino wrote:
> > It is just an exported function that will be used by other subsystems
> > to print specific events to the output buffer.
> > 
> > This is based in ideas by Amit Shah <amit.shah@redhat.com>.
> > 
> > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> > ---
> >  monitor.c   |   18 ++++++++++++++++++
> >  monitor.h   |    6 ++++++
> >  qemu-tool.c |    4 ++++
> >  3 files changed, 28 insertions(+), 0 deletions(-)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index 462f60b..df58bdd 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)
> >      va_end(ap);
> >  }
> >  
> > +/* Asynchronous events main function */
> > +void monitor_notify_event(MonitorEvent event)
> > +{
> > +    if (!monitor_ctrl_mode(cur_mon))
> > +        return;
> > +
> > +    assert(event < EVENT_MAX);
> > +    monitor_puts(cur_mon, "* EVENT ");
> > +
> > +    switch (event) {
> > +    case EVENT_MAX:
> > +        // Avoid gcc warning, will never get here
> > +        break;
> > +    }
> > +
> > +    monitor_puts(cur_mon, "\n");
> > +}
> > +
> 
> You shouldn't use cur_mon here. It's for legacy code that still assumes
> there can be only one monitor terminal. Instead, iterate over all
> monitors and distribute the event to those that are in control mode.

 Thanks. I was really in doubt on what to do here. :)

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Eduardo Habkost]

On Tue, Jun 23, 2009 at 04:30:32PM +0300, Avi Kivity wrote:
> On 06/23/2009 04:12 PM, Anthony Liguori wrote:
>> Hi Luiz,
>>
>> The specification looks pretty good.
>>
>>
>>>> +
>>>> +3.3.1 Server Greeting
>>>> +---------------------
>>>> +
>>>> +Sent when a new connection is opened.
>>>> +
>>>> +Format: + OK QEMU<version>  QMP<version>
>>>> +Example: + OK QEMU 0.10.50 QMP 0.1
>>>
>>> Clients should never make decisions based on the qemu or qmp  
>>> version.  Rather, we should provide a facility to query the  
>>> availability of features.
>>
>> I agree, but I'd suggest leaving the QMP version in there for  
>> insurance purposes in case we really screw up and need to bump the  
>> version.  In fact, having the client also negotiate the QMP version  
>> isn't a bad idea.
>
> Agreed.
>
>> How would asynchronous commands work? 
>
> IMO, there aren't any.  All commands are synchronous (but may cause  
> events to be generated later).

And if there is any, we can simply convert them to synchronous commands
that cause events to be generated.

I think we could make this clear on the specification, specifying that
all commands are synchronous.

-- 
Eduardo

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Luiz Capitulino]

On Tue, 23 Jun 2009 11:32:51 +0100
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Tue, Jun 23, 2009 at 01:29:11AM -0300, Luiz Capitulino wrote:
> > It is just an exported function that will be used by other subsystems
> > to print specific events to the output buffer.
> > 
> > This is based in ideas by Amit Shah <amit.shah@redhat.com>.
> > 
> > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> > ---
> >  monitor.c   |   18 ++++++++++++++++++
> >  monitor.h   |    6 ++++++
> >  qemu-tool.c |    4 ++++
> >  3 files changed, 28 insertions(+), 0 deletions(-)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index 462f60b..df58bdd 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)
> >      va_end(ap);
> >  }
> >  
> > +/* Asynchronous events main function */
> > +void monitor_notify_event(MonitorEvent event)
> > +{
> > +    if (!monitor_ctrl_mode(cur_mon))
> > +        return;
> > +
> > +    assert(event < EVENT_MAX);
> > +    monitor_puts(cur_mon, "* EVENT ");
> > +
> > +    switch (event) {
> > +    case EVENT_MAX:
> > +        // Avoid gcc warning, will never get here
> > +        break;
> > +    }
> > +
> > +    monitor_puts(cur_mon, "\n");
> > +}
> > +
> 
> If a client is not reading from the monitor channel quickly enough, then
> would this cause QEMU to block once the FD buffer is full ? A QEMU level
> buffer might give more leyway, but we don't want to grow unbounded, so
> ultimately we'll end up having to drop events. At which point you'd want
> to send an event to the client indicating that the event queue overflowed,
> so it can take remedial action to re-sync its state.

 Wouldn't a buffer flush right at the beginning of this function solve this?
At least for QEMU?

 I'm not sure if it's possible to get QEMU blocked.

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> On 06/23/2009 04:12 PM, Anthony Liguori wrote:
>> Hi Luiz,
>>
>> The specification looks pretty good.
>>
>>
>>>> +
>>>> +3.3.1 Server Greeting
>>>> +---------------------
>>>> +
>>>> +Sent when a new connection is opened.
>>>> +
>>>> +Format: + OK QEMU<version>  QMP<version>
>>>> +Example: + OK QEMU 0.10.50 QMP 0.1
>>>
>>> Clients should never make decisions based on the qemu or qmp 
>>> version.  Rather, we should provide a facility to query the 
>>> availability of features.
>>
>> I agree, but I'd suggest leaving the QMP version in there for 
>> insurance purposes in case we really screw up and need to bump the 
>> version.  In fact, having the client also negotiate the QMP version 
>> isn't a bad idea.
>
> Agreed.
>
>> How would asynchronous commands work? 
>
> IMO, there aren't any.  All commands are synchronous (but may cause 
> events to be generated later).

At least today, there are quite a few commands that are synchronous and 
block.  They would all have to be converted to events.  For instance:

commit
change
screendump
savevm
loadvm
delvm
stop
cont
memsave
pmemsave
migrate
balloon

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 03/11] QMP: Introduce protocol print functions

[Anthony Liguori]

Luiz Capitulino wrote:
> This change introduces the functions that will be used by the Monitor
> to output data in the format defined by the protocol specification.
>
> There are four functions:
>
>  o monitor_printf_bad()  signals a protocol error
>  o monitor_print_ok()    signals successful execution
>  o monitor_printf_err()  used by commands, to signal execution errors
>  o monitor_printf_data() used by commands, to output data
>
> For now monitor_print_ok() compilation is disabled to avoid breaking
> the build as the function is not currently used. It will be enabled
> by a later commit.
>
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  monitor.c   |   63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  monitor.h   |    3 ++
>  qemu-tool.c |   12 +++++++++++
>  3 files changed, 78 insertions(+), 0 deletions(-)
>
> diff --git a/monitor.c b/monitor.c
> index 514db00..dfa777d 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -199,6 +199,69 @@ static int monitor_fprintf(FILE *stream, const char *fmt, ...)
>      return 0;
>  }
>  
> +/*
> + * QEMU Monitor Control print functions
> + */
> +
> +/* Protocol errors */
> +void monitor_printf_bad(Monitor *mon, const char *fmt, ...)
> +{
> +    if (monitor_ctrl_mode(mon)) {
> +        va_list ap;
> +
> +        monitor_puts(mon, "- BAD ");
> +
> +        va_start(ap, fmt);
> +        monitor_vprintf(mon, fmt, ap);
> +        va_end(ap);
> +    }
> +}
> +
> +#if 0
> +/* OK command completion, 'info' commands are special */
> +static void monitor_print_ok(Monitor *mon, const char *cmd, const char *arg)
> +{
> +    if (!monitor_ctrl_mode(mon))
> +        return;
> +
> +    monitor_printf(mon, "+ OK %s", cmd);
> +    if (!strcmp(cmd, "info"))
> +        monitor_printf(mon, " %s", arg);
> +    monitor_puts(mon, " completed\n");
> +}
> +#endif
> +
> +static void monitor_ctrl_pprintf(Monitor *mon, const char *prefix,
> +                                 const char *fmt, va_list ap)
> +{
> +    if (monitor_ctrl_mode(mon))
> +        monitor_puts(mon, prefix);
> +
> +    monitor_vprintf(mon, fmt, ap);
> +}
> +
> +/* Should be used by commands to signal errors, will add the prefix
> + * '- ERR ' if in control mode */
> +void monitor_printf_err(Monitor *mon, const char *fmt, ...)
> +{
> +    va_list ap;
> +
> +    va_start(ap, fmt);
> +    monitor_ctrl_pprintf(mon, "- ERR ", fmt, ap);
> +    va_end(ap);
> +}
> +
> +/* Should be used by commands to print any output which is not
> + * an error, will add the prefix '= ' if in control mode */
> +void monitor_printf_data(Monitor *mon, const char *fmt, ...)
> +{
> +    va_list ap;
> +
> +    va_start(ap, fmt);
> +    monitor_ctrl_pprintf(mon, "= ", fmt, ap);
> +    va_end(ap);
> +}
> +
>  static int compare_cmd(const char *name, const char *list)
>  {
>      const char *p, *pstart;
> diff --git a/monitor.h b/monitor.h
> index 48bc056..8b054eb 100644
> --- a/monitor.h
> +++ b/monitor.h
> @@ -24,6 +24,9 @@ void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs,
>  void monitor_vprintf(Monitor *mon, const char *fmt, va_list ap);
>  void monitor_printf(Monitor *mon, const char *fmt, ...)
>      __attribute__ ((__format__ (__printf__, 2, 3)));
> +void monitor_printf_bad(Monitor *mon, const char *fmt, ...);
> +void monitor_printf_err(Monitor *mon, const char *fmt, ...);
> +void monitor_printf_data(Monitor *mon, const char *fmt, ...);
>   

Probably could just introduce a flag to monitor_printf().  In fact, you 
could go kernel style and have:

monitor_printf(mon, MON_BAD "bad monitor command\n");
monitor_printf(mon, MON_DATA "some monitor data\n");

With MON_DATA being explicit.

If not, make sure to add the format attribute to the printfs.

Regards,

Anthony Liguori

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Eduardo Habkost]

On Tue, Jun 23, 2009 at 10:36:16AM -0300, Luiz Capitulino wrote:
> On Tue, 23 Jun 2009 11:32:51 +0100
> "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > On Tue, Jun 23, 2009 at 01:29:11AM -0300, Luiz Capitulino wrote:
> > > It is just an exported function that will be used by other subsystems
> > > to print specific events to the output buffer.
> > > 
> > > This is based in ideas by Amit Shah <amit.shah@redhat.com>.
> > > 
> > > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> > > ---
> > >  monitor.c   |   18 ++++++++++++++++++
> > >  monitor.h   |    6 ++++++
> > >  qemu-tool.c |    4 ++++
> > >  3 files changed, 28 insertions(+), 0 deletions(-)
> > > 
> > > diff --git a/monitor.c b/monitor.c
> > > index 462f60b..df58bdd 100644
> > > --- a/monitor.c
> > > +++ b/monitor.c
> > > @@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)
> > >      va_end(ap);
> > >  }
> > >  
> > > +/* Asynchronous events main function */
> > > +void monitor_notify_event(MonitorEvent event)
> > > +{
> > > +    if (!monitor_ctrl_mode(cur_mon))
> > > +        return;
> > > +
> > > +    assert(event < EVENT_MAX);
> > > +    monitor_puts(cur_mon, "* EVENT ");
> > > +
> > > +    switch (event) {
> > > +    case EVENT_MAX:
> > > +        // Avoid gcc warning, will never get here
> > > +        break;
> > > +    }
> > > +
> > > +    monitor_puts(cur_mon, "\n");
> > > +}
> > > +
> > 
> > If a client is not reading from the monitor channel quickly enough, then
> > would this cause QEMU to block once the FD buffer is full ? A QEMU level
> > buffer might give more leyway, but we don't want to grow unbounded, so
> > ultimately we'll end up having to drop events. At which point you'd want
> > to send an event to the client indicating that the event queue overflowed,
> > so it can take remedial action to re-sync its state.
> 
>  Wouldn't a buffer flush right at the beginning of this function solve this?
> At least for QEMU?

No, if the messages aren't being consumed fast enough by the other side.
If the consumer isn't consuming the messages fast enough, we would need
to either drop messages or block on the channel (that's the code code
above would do: block on write()).

I agree with Avi: if the system is so busy that the client doesn't
consume the monitor messages fast enough, we are already screwed.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 04/11] QMP: Make monitor_handle_command() QMP aware

[Anthony Liguori]

Luiz Capitulino wrote:
> Two changes are needed in monitor_handle_command() so that it
> knows how to work in control mode.
>
> First, it has to know whether a given command is part of the
> protocol or not. Second, it has to print the correct server
> response when a command execution finishes.
>
> A good approach to do this would be:
>
> 1. Add a new member to struct mon_cmd_t, which will tell if
> the command is already part of the protocol.
>
> 2. Change handler_* functions to return exit status, so that
> monitor_handle_command() can print "+ OK" or "- ERR"
> accordling.
>   
These changes are a requirement for merging.

Regards,

Anthony Liguori

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Jan Kiszka]

Eduardo Habkost wrote:
> On Tue, Jun 23, 2009 at 10:36:16AM -0300, Luiz Capitulino wrote:
>> On Tue, 23 Jun 2009 11:32:51 +0100
>> "Daniel P. Berrange" <berrange@redhat.com> wrote:
>>
>>> On Tue, Jun 23, 2009 at 01:29:11AM -0300, Luiz Capitulino wrote:
>>>> It is just an exported function that will be used by other subsystems
>>>> to print specific events to the output buffer.
>>>>
>>>> This is based in ideas by Amit Shah <amit.shah@redhat.com>.
>>>>
>>>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>>>> ---
>>>>  monitor.c   |   18 ++++++++++++++++++
>>>>  monitor.h   |    6 ++++++
>>>>  qemu-tool.c |    4 ++++
>>>>  3 files changed, 28 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/monitor.c b/monitor.c
>>>> index 462f60b..df58bdd 100644
>>>> --- a/monitor.c
>>>> +++ b/monitor.c
>>>> @@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)
>>>>      va_end(ap);
>>>>  }
>>>>  
>>>> +/* Asynchronous events main function */
>>>> +void monitor_notify_event(MonitorEvent event)
>>>> +{
>>>> +    if (!monitor_ctrl_mode(cur_mon))
>>>> +        return;
>>>> +
>>>> +    assert(event < EVENT_MAX);
>>>> +    monitor_puts(cur_mon, "* EVENT ");
>>>> +
>>>> +    switch (event) {
>>>> +    case EVENT_MAX:
>>>> +        // Avoid gcc warning, will never get here
>>>> +        break;
>>>> +    }
>>>> +
>>>> +    monitor_puts(cur_mon, "\n");
>>>> +}
>>>> +
>>> If a client is not reading from the monitor channel quickly enough, then
>>> would this cause QEMU to block once the FD buffer is full ? A QEMU level
>>> buffer might give more leyway, but we don't want to grow unbounded, so
>>> ultimately we'll end up having to drop events. At which point you'd want
>>> to send an event to the client indicating that the event queue overflowed,
>>> so it can take remedial action to re-sync its state.
>>  Wouldn't a buffer flush right at the beginning of this function solve this?
>> At least for QEMU?
> 
> No, if the messages aren't being consumed fast enough by the other side.
> If the consumer isn't consuming the messages fast enough, we would need
> to either drop messages or block on the channel (that's the code code
> above would do: block on write()).
> 
> I agree with Avi: if the system is so busy that the client doesn't
> consume the monitor messages fast enough, we are already screwed.

Can't mgmt app and qemu also run on different hosts?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [PATCH 03/11] QMP: Introduce protocol print functions

[Eduardo Habkost]

On Tue, Jun 23, 2009 at 08:45:06AM -0500, Anthony Liguori wrote:
> Luiz Capitulino wrote:
>> This change introduces the functions that will be used by the Monitor
>> to output data in the format defined by the protocol specification.
>>
>> There are four functions:
>>
>>  o monitor_printf_bad()  signals a protocol error
>>  o monitor_print_ok()    signals successful execution
>>  o monitor_printf_err()  used by commands, to signal execution errors
>>  o monitor_printf_data() used by commands, to output data
>>
>> For now monitor_print_ok() compilation is disabled to avoid breaking
>> the build as the function is not currently used. It will be enabled
>> by a later commit.
>>
>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>> ---
>>  monitor.c   |   63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  monitor.h   |    3 ++
>>  qemu-tool.c |   12 +++++++++++
>>  3 files changed, 78 insertions(+), 0 deletions(-)
>>
>> diff --git a/monitor.c b/monitor.c
>> index 514db00..dfa777d 100644
>> --- a/monitor.c
>> +++ b/monitor.c
>> @@ -199,6 +199,69 @@ static int monitor_fprintf(FILE *stream, const char *fmt, ...)
>>      return 0;
>>  }
>>  +/*
>> + * QEMU Monitor Control print functions
>> + */
>> +
>> +/* Protocol errors */
>> +void monitor_printf_bad(Monitor *mon, const char *fmt, ...)
>> +{
>> +    if (monitor_ctrl_mode(mon)) {
>> +        va_list ap;
>> +
>> +        monitor_puts(mon, "- BAD ");
>> +
>> +        va_start(ap, fmt);
>> +        monitor_vprintf(mon, fmt, ap);
>> +        va_end(ap);
>> +    }
>> +}
>> +
>> +#if 0
>> +/* OK command completion, 'info' commands are special */
>> +static void monitor_print_ok(Monitor *mon, const char *cmd, const char *arg)
>> +{
>> +    if (!monitor_ctrl_mode(mon))
>> +        return;
>> +
>> +    monitor_printf(mon, "+ OK %s", cmd);
>> +    if (!strcmp(cmd, "info"))
>> +        monitor_printf(mon, " %s", arg);
>> +    monitor_puts(mon, " completed\n");
>> +}
>> +#endif
>> +
>> +static void monitor_ctrl_pprintf(Monitor *mon, const char *prefix,
>> +                                 const char *fmt, va_list ap)
>> +{
>> +    if (monitor_ctrl_mode(mon))
>> +        monitor_puts(mon, prefix);
>> +
>> +    monitor_vprintf(mon, fmt, ap);
>> +}
>> +
>> +/* Should be used by commands to signal errors, will add the prefix
>> + * '- ERR ' if in control mode */
>> +void monitor_printf_err(Monitor *mon, const char *fmt, ...)
>> +{
>> +    va_list ap;
>> +
>> +    va_start(ap, fmt);
>> +    monitor_ctrl_pprintf(mon, "- ERR ", fmt, ap);
>> +    va_end(ap);
>> +}
>> +
>> +/* Should be used by commands to print any output which is not
>> + * an error, will add the prefix '= ' if in control mode */
>> +void monitor_printf_data(Monitor *mon, const char *fmt, ...)
>> +{
>> +    va_list ap;
>> +
>> +    va_start(ap, fmt);
>> +    monitor_ctrl_pprintf(mon, "= ", fmt, ap);
>> +    va_end(ap);
>> +}
>> +
>>  static int compare_cmd(const char *name, const char *list)
>>  {
>>      const char *p, *pstart;
>> diff --git a/monitor.h b/monitor.h
>> index 48bc056..8b054eb 100644
>> --- a/monitor.h
>> +++ b/monitor.h
>> @@ -24,6 +24,9 @@ void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs,
>>  void monitor_vprintf(Monitor *mon, const char *fmt, va_list ap);
>>  void monitor_printf(Monitor *mon, const char *fmt, ...)
>>      __attribute__ ((__format__ (__printf__, 2, 3)));
>> +void monitor_printf_bad(Monitor *mon, const char *fmt, ...);
>> +void monitor_printf_err(Monitor *mon, const char *fmt, ...);
>> +void monitor_printf_data(Monitor *mon, const char *fmt, ...);
>>   
>
> Probably could just introduce a flag to monitor_printf().  In fact, you  
> could go kernel style and have:
>
> monitor_printf(mon, MON_BAD "bad monitor command\n");
> monitor_printf(mon, MON_DATA "some monitor data\n");
>
> With MON_DATA being explicit.
>
> If not, make sure to add the format attribute to the printfs.

I think different functions make sense because some messages may have
additional attributes (not only a text message), that may be arguments
to the function. e.g.:

void monitor_printf_event(Monitor *mon, qemu_event_t event, const char *fmt, ...);

void monitor_printf_err(Monitor *mon, int status_code, const char *fmt, ...);

(or maybe we can just call it monitor_printf_result(), being applicable
to both success and failure, depending on the status code).

-- 
Eduardo

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Luiz Capitulino]

On Tue, 23 Jun 2009 11:03:09 +0200
Jan Kiszka <jan.kiszka@siemens.com> wrote:

> Luiz Capitulino wrote:
> > This change adds a flag called 'control' to the already existing
> > '-monitor' command-line option. This flag can be used to enable
> > control mode.
> > 
> > Its syntax is:
> > 
> > qemu [...] -monitor control,<device>
> > 
> > Where <device> is a chardev (excluding 'vc', for obvious reasons).
> > 
> > For example:
> > 
> > $ qemu [...] -monitor control,tcp:localhost:4444,server
> > 
> > Will run QEMU in control mode, waiting for a client TCP connection
> > on localhost port 4444.
> > 
> > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> 
> At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
> instances". I think Anthony posted such a patch before. Now we should
> really include this as your extension may block the stand-alone -monitor
> switch for the control channel, preventing to additionally set up one
> (or more) for debugging purposes.

 Do you really think that this is needed for the first version? I would
rather prefer to concentrate on protocol specification and command
porting, which will require a lot of effort.

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Anthony Liguori]

Avi Kivity wrote:
> On 06/23/2009 07:29 AM, Luiz Capitulino wrote:
>>       else if (actual == 0)
>> -        monitor_printf(mon, "Ballooning not activated in VM\n");
>> +        monitor_printf_err(mon, "Ballooning not activated in VM\n");
>>       else
>> -        monitor_printf(mon, "balloon: actual=%d\n", (int)(actual>>  
>> 20));
>> +        monitor_printf_data(mon, "balloon: actual=%d\n", 
>> (int)(actual>>  20));
>>   }
>>    
>
> Control mode should always use bytes and seconds (and this should be 
> described in the spec).  You avoid rounding, and more importantly, 
> ambiguity and a source of unit conversion errors.

I'd actually like to see a lot more structure in this sort of output.  
For instance:

monitor_printf_list(mon, "balloon",
                               "actual", MON_TIME, actual,
                               NULL);

How this gets output can then be conditional on control mode vs. human 
mode.  In human mode, we can use human-friendly units like MBs.  In 
control mode, we would always use bytes.

> Patched that add a command to machine mode without updating the spec 
> should be automatically NACKed.
>
> We also need a way to discover that the command is available:

I think we want to version each command too.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 09/11] QMP: Port 'info blockstats' command

[Anthony Liguori]

Luiz Capitulino wrote:
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  block.c   |   18 +++++++++---------
>  monitor.c |    3 ++-
>  2 files changed, 11 insertions(+), 10 deletions(-)
>
> diff --git a/block.c b/block.c
> index aca5a6d..3b4af27 100644
> --- a/block.c
> +++ b/block.c
> @@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)
>      BlockDriverState *bs;
>  
>      for (bs = bdrv_first; bs != NULL; bs = bs->next) {
> -        monitor_printf(mon, "%s:"
> -                       " rd_bytes=%" PRIu64
> -                       " wr_bytes=%" PRIu64
> -                       " rd_operations=%" PRIu64
> -                       " wr_operations=%" PRIu64
> -                       "\n",
> -                       bs->device_name,
> -                       bs->rd_bytes, bs->wr_bytes,
> -                       bs->rd_ops, bs->wr_ops);
> +        monitor_printf_data(mon, "%s:"
> +                            " rd_bytes=%" PRIu64
> +                            " wr_bytes=%" PRIu64
> +                            " rd_operations=%" PRIu64
> +                            " wr_operations=%" PRIu64
> +                            "\n",
> +                            bs->device_name,
> +                            bs->rd_bytes, bs->wr_bytes,
> +                            bs->rd_ops, bs->wr_ops);
>   

monitor_printf_list(mon, bs->device_name,
                              "rd_bytes", MON_BYTES, bs->rd_bytes,
                              "wr_bytes", MON_BYTES, bs->wr_bytes,
                              "rd_operations", MON_BYTES, bs->rd_ops,
                              "wr_operations", MON_BYTES, bs->wr_ops,
                              NULL);

Regards,

Anthony Liguori

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Jan Kiszka]

Luiz Capitulino wrote:
> On Tue, 23 Jun 2009 11:03:09 +0200
> Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
>> Luiz Capitulino wrote:
>>> This change adds a flag called 'control' to the already existing
>>> '-monitor' command-line option. This flag can be used to enable
>>> control mode.
>>>
>>> Its syntax is:
>>>
>>> qemu [...] -monitor control,<device>
>>>
>>> Where <device> is a chardev (excluding 'vc', for obvious reasons).
>>>
>>> For example:
>>>
>>> $ qemu [...] -monitor control,tcp:localhost:4444,server
>>>
>>> Will run QEMU in control mode, waiting for a client TCP connection
>>> on localhost port 4444.
>>>
>>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>> At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
>> instances". I think Anthony posted such a patch before. Now we should
>> really include this as your extension may block the stand-alone -monitor
>> switch for the control channel, preventing to additionally set up one
>> (or more) for debugging purposes.
> 
>  Do you really think that this is needed for the first version? I would
> rather prefer to concentrate on protocol specification and command
> porting, which will require a lot of effort.

You may postpone it if it's too much work, but please have a look at
http://permalink.gmane.org/gmane.comp.emulators.libvirt/12895 first as I
think it should be easy to integrate.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Anthony Liguori]

Avi Kivity wrote:
> On 06/23/2009 01:11 PM, Jan Kiszka wrote:
>>> If we support multiple monitors, it might be desirable to allow the
>>> app owning the main 'control' monitor channel to be able to indicate
>>> that additional monitor channels are read-only. eg, so libvirt could
>>> allow the user to connect to the monitor to run 'info' commands for
>>> debug support without risk of having state changed behind its back
>>>      
>>
>> Couldn't libvirt deal with update given we provide them as events?
>> Otherwise, your suggestion makes sense, definitely as long as it would
>> wreck libvirt's internal house keeping or for commands that are not
>> synchronizable.
>>    
>
> It would be a nightmare, consider both libvirt and a user hotplugging 
> something into the same pci slot, or a user starting migration, or 
> quitting, or ...
>
> Having -monitor readonly makes sense.

I don't like such a concept.  If a user goes to the trouble to create a 
second monitor, then they need to know what they're doing.

libvirt can simply not expose the option to the user.  Clever users can 
change the emulator tag to point to a script that they can use to create 
a second monitor.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Tue, Jun 23, 2009 at 01:28:11AM -0300, Luiz Capitulino wrote:
> This file contains detailed QMP description and definitions.
> 
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 180 insertions(+), 0 deletions(-)
>  create mode 100644 monitor-protocol-spec.txt
> 
> diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
> new file mode 100644
> index 0000000..d20e3f9
> --- /dev/null
> +++ b/monitor-protocol-spec.txt
> @@ -0,0 +1,180 @@
> +               QEMU Monitor Protocol Specification - Version 0.1

I think this is extremelly needed however inventing yet another RPC protocol
make no sense in 2009, specially with all the limitations (no UTF-8, no quoted
string ...).

You should really look at what's already out there, so you don't have to 
concentrate on the marshalled protocol, but just the format.

At this point my personal suggestion would be jsonrpc:

- it's easy to parse (even in C)
- it's text with markup
- easy to generate
- support utf-8
- quoted string
- it's not ugly to read (compare to xml)
- can be embedded directly (no more library dependancy for qemu) in the code
- lots of language have json library making the interaction with the monitor
  easy from the other side

However i'm sure other RPC "format" exists that suit all of these too, but
again please don't do something from scratch ..

-- 
Vincent

Re: [Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Paul Brook]

On Tuesday 23 June 2009, Daniel P. Berrange wrote:
> On Tue, Jun 23, 2009 at 01:29:11AM -0300, Luiz Capitulino wrote:
> > It is just an exported function that will be used by other subsystems
> > to print specific events to the output buffer.
>
> As per my mail to the lead message in this thread, I'm thinking it
> might be valuable to include a timestamp with all events, and command
> replies. A 64-bit seconds-since-epoch prefix on all replies, no need
> for a human-format timestamp

I'm reluctant to add timestamps just because they seem like a cool feature.  I 
think it's important to first understand why you want timestamps, and exactly 
where the timestamp comes from. What are you expecting to gain from recording 
the timestamp in qemu rather than the management application?

I'd expect nearest-second timestamps to be pretty much useless for any 
practical purpose.

Also, should the timestamps be real time or virtual time?

Paul

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 04:40 PM, Anthony Liguori wrote:
>> IMO, there aren't any.  All commands are synchronous (but may cause 
>> events to be generated later).
>
>
> At least today, there are quite a few commands that are synchronous 
> and block.  They would all have to be converted to events.  For instance:
>

That's what I meant, though I phrased it poorly.

> commit

That's a biggie to make truly async.

> change
> screendump

I think we can keep those synchronous.

> savevm
> loadvm
> delvm

Like commit, very difficult to make async.

> stop
> cont

These certainly seem very synchronous to me.

> memsave
> pmemsave

Can keep these synchronous IMO.  You certainly don't want the guest to 
progress during memsave.

> migrate
> balloon 

Isn't balloon synchronous?  It sets the target and exits immediately.  
We don't want command completion to depend on the guest.

-- 

error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Blue Swirl]

On 6/23/09, Luiz Capitulino <lcapitulino@redhat.com> wrote:
> This file contains detailed QMP description and definitions.
>
>  Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>  ---
>   monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
>   1 files changed, 180 insertions(+), 0 deletions(-)
>   create mode 100644 monitor-protocol-spec.txt

I'd rather have the description in texi format and include it as an
appendix from qemu-tech.texi.

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
>> stop
>> cont
>
> These certainly seem very synchronous to me.

Not with KVM.  You have to wait for the VCPU threads to get notified and 
stop.  Maybe it's not a big deal to block during this.

>> migrate
>> balloon 
>
> Isn't balloon synchronous?  It sets the target and exits immediately.  
> We don't want command completion to depend on the guest.

I can see a use-case for having balloon only return when the target has 
been reached by the guest.

I think I can buy the event stuff.  As long as we version QMP protocol, 
we can potentially fix this down the road.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 05:45 PM, Vincent Hanquez wrote:
> On Tue, Jun 23, 2009 at 01:28:11AM -0300, Luiz Capitulino wrote:
>    
>> This file contains detailed QMP description and definitions.
>>
>> Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
>> ---
>>   monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
>>   1 files changed, 180 insertions(+), 0 deletions(-)
>>   create mode 100644 monitor-protocol-spec.txt
>>
>> diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
>> new file mode 100644
>> index 0000000..d20e3f9
>> --- /dev/null
>> +++ b/monitor-protocol-spec.txt
>> @@ -0,0 +1,180 @@
>> +               QEMU Monitor Protocol Specification - Version 0.1
>>      
>
> I think this is extremelly needed however inventing yet another RPC protocol
> make no sense in 2009, specially with all the limitations (no UTF-8, no quoted
> string ...).
>
> You should really look at what's already out there, so you don't have to
> concentrate on the marshalled protocol, but just the format.
>
> At this point my personal suggestion would be jsonrpc:
>
> - it's easy to parse (even in C)
> - it's text with markup
> - easy to generate
> - support utf-8
> - quoted string
> - it's not ugly to read (compare to xml)
> - can be embedded directly (no more library dependancy for qemu) in the code
> - lots of language have json library making the interaction with the monitor
>    easy from the other side
>    

- supports passing structure
- supports notifications
- supports asynchronous commands

seems quite nice.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> On 06/23/2009 05:45 PM, Vincent Hanquez wrote:
>>
>> At this point my personal suggestion would be jsonrpc:
>>
>> - it's easy to parse (even in C)
>> - it's text with markup
>> - easy to generate
>> - support utf-8
>> - quoted string
>> - it's not ugly to read (compare to xml)
>> - can be embedded directly (no more library dependancy for qemu) in 
>> the code
>> - lots of language have json library making the interaction with the 
>> monitor
>>    easy from the other side
>>    
>
> - supports passing structure
> - supports notifications
> - supports asynchronous commands
>
> seems quite nice.

I'd rather not do json.

As they stand, I think the current patch set is very close to being 
mergable.  I'd rather not go off on a new tangent.

Regards,

Anthony Liguori


-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 06:47 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>>> stop
>>> cont
>>
>> These certainly seem very synchronous to me.
>
> Not with KVM.  You have to wait for the VCPU threads to get notified 
> and stop.  Maybe it's not a big deal to block during this.

If a command returns immediately, we shouldn't try to make it asynchronous.

>>> migrate
>>> balloon 
>>
>> Isn't balloon synchronous?  It sets the target and exits 
>> immediately.  We don't want command completion to depend on the guest.
>
> I can see a use-case for having balloon only return when the target 
> has been reached by the guest.

There's 'info balloon' for that, and maybe request a notification.  
Waiting for the guest is asking for trouble.  What if you want to set 
the balloon to a different target? you have two different balloon 
commands running?

>
> I think I can buy the event stuff.  As long as we version QMP 
> protocol, we can potentially fix this down the road.
>

Alternatively we can introduce a tag, and so make all commands asynchronous:

   @tag1 commit
   @tag1 = committed
   @tag1 + OK have fun

Or go with the jsonrpc thing that has all this built in.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 06:56 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>> On 06/23/2009 05:45 PM, Vincent Hanquez wrote:
>>>
>>> At this point my personal suggestion would be jsonrpc:
>>>
>>> - it's easy to parse (even in C)
>>> - it's text with markup
>>> - easy to generate
>>> - support utf-8
>>> - quoted string
>>> - it's not ugly to read (compare to xml)
>>> - can be embedded directly (no more library dependancy for qemu) in 
>>> the code
>>> - lots of language have json library making the interaction with the 
>>> monitor
>>>    easy from the other side
>>
>> - supports passing structure
>> - supports notifications
>> - supports asynchronous commands
>>
>> seems quite nice.
>
> I'd rather not do json.
>

Why not?

> As they stand, I think the current patch set is very close to being 
> mergable.  I'd rather not go off on a new tangent.

It fulfils all are requirements and more.  There's a C implementation 
available.  We can concentrate on the commands and ignore the plumbing.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
>> I'd rather not do json.
>>
>
> Why not?

Because it's too much complexity at this level in the stack.

>> As they stand, I think the current patch set is very close to being 
>> mergable.  I'd rather not go off on a new tangent.
>
> It fulfils all are requirements and more.  There's a C implementation 
> available.  We can concentrate on the commands and ignore the plumbing.

The same is true for XML-RPC, CIM, dbus, etc.  RPC grow like weeds.  In 
5 years, it will be shocking that qemu uses jsonrpc instead of the next 
super-neat RPC protocol.

If we did a real RPC mechanism, I'd want to stick with something 
tried-and-true like XDR.  But an RPC mechanism is not the monitor.  If 
we were going to do that, there's no point in morphing the monitor into 
that.

What we're doing here is adding a computer-mode to the existing 
monitor.  This will be easy for current apps to consume and for us to 
transition the code base to.  It's a safe and incremental improvement.

I don't want to invent Xen-API here and go through that pain in QEMU.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 07:09 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>>> I'd rather not do json.
>>>
>>
>> Why not?
>
> Because it's too much complexity at this level in the stack.

I don't see that.  Just another library.

>>> As they stand, I think the current patch set is very close to being 
>>> mergable.  I'd rather not go off on a new tangent.
>>
>> It fulfils all are requirements and more.  There's a C implementation 
>> available.  We can concentrate on the commands and ignore the plumbing.
>
> The same is true for XML-RPC, CIM, dbus, etc.  RPC grow like weeds.  
> In 5 years, it will be shocking that qemu uses jsonrpc instead of the 
> next super-neat RPC protocol.

The only one that actually competes here is xml-rpc, which is 
ridiculously verbose.

> If we did a real RPC mechanism, I'd want to stick with something 
> tried-and-true like XDR.

Having some experience with xdr, I'd call it tried-and-wrong.  It's 
simply horrible.

>   But an RPC mechanism is not the monitor.  If we were going to do 
> that, there's no point in morphing the monitor into that.
>
> What we're doing here is adding a computer-mode to the existing 
> monitor.  This will be easy for current apps to consume and for us to 
> transition the code base to.  It's a safe and incremental improvement.

An RPC makes it a lot easier for clients to parse, there's no need for a 
regular expression per command, or to bother with line continuations 
etc.  You just focus on the data.

We have an opportunity to make a clean break, let's make use of it.

> I don't want to invent Xen-API here and go through that pain in QEMU.

I don't see the parallel.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 07:09 PM, Anthony Liguori wrote:
> In 5 years, it will be shocking that qemu uses jsonrpc instead of the 
> next super-neat RPC protocol.

But, we'll be able to control qemu from our browsers!

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Avi Kivity]

On 06/23/2009 04:51 PM, Jan Kiszka wrote:
> Can't mgmt app and qemu also run on different hosts?
>
>    

In theory, yes.  In practice you need someone to launch qemu and wait 
for it if it dies, so you'll have a daemon on the host.  Typically it 
will be an intermediate daemon proxying all the guests for the real 
management server somewhere else.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Avi Kivity]

On 06/23/2009 04:59 PM, Anthony Liguori wrote:
>> Control mode should always use bytes and seconds (and this should be 
>> described in the spec).  You avoid rounding, and more importantly, 
>> ambiguity and a source of unit conversion errors.
>
>
> I'd actually like to see a lot more structure in this sort of output.  
> For instance:
>
> monitor_printf_list(mon, "balloon",
>                               "actual", MON_TIME, actual,
>                               NULL);
>
> How this gets output can then be conditional on control mode vs. human 
> mode.  In human mode, we can use human-friendly units like MBs.  In 
> control mode, we would always use bytes.

That actually works well with rpc.  You have a structure that contains 
the data.  For machine mode, you send that away.  For humans, you 
unit-convert, format and print.

>
>> Patched that add a command to machine mode without updating the spec 
>> should be automatically NACKed.
>>
>> We also need a way to discover that the command is available:
>
> I think we want to version each command too.

Not version, discover supported features.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure

[Daniel P. Berrange]

On Tue, Jun 23, 2009 at 03:51:29PM +0200, Jan Kiszka wrote:
> Eduardo Habkost wrote:
> > On Tue, Jun 23, 2009 at 10:36:16AM -0300, Luiz Capitulino wrote:
> >> On Tue, 23 Jun 2009 11:32:51 +0100
> >> "Daniel P. Berrange" <berrange@redhat.com> wrote:
> >>
> >>>> +
> >>> If a client is not reading from the monitor channel quickly enough, then
> >>> would this cause QEMU to block once the FD buffer is full ? A QEMU level
> >>> buffer might give more leyway, but we don't want to grow unbounded, so
> >>> ultimately we'll end up having to drop events. At which point you'd want
> >>> to send an event to the client indicating that the event queue overflowed,
> >>> so it can take remedial action to re-sync its state.
> >>  Wouldn't a buffer flush right at the beginning of this function solve this?
> >> At least for QEMU?
> > 
> > No, if the messages aren't being consumed fast enough by the other side.
> > If the consumer isn't consuming the messages fast enough, we would need
> > to either drop messages or block on the channel (that's the code code
> > above would do: block on write()).
> > 
> > I agree with Avi: if the system is so busy that the client doesn't
> > consume the monitor messages fast enough, we are already screwed.
> 
> Can't mgmt app and qemu also run on different hosts?

If you happened to run the monitor using a TCP backend, then yes the
mgmt app could be remote. This would be just  insane though, since
there's no security there whatsoever. I don't think the QEMU monitor
wants to be in the business of encrypting, authenticating & authorizing
monitor clients. Rather it should just be local only, and any mgmt tool
using QEMU can provide secure remote mgmt transports at a higher level.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Anthony Liguori]

Avi Kivity wrote:
>>
>>> Patched that add a command to machine mode without updating the spec 
>>> should be automatically NACKed.
>>>
>>> We also need a way to discover that the command is available:
>>
>> I think we want to version each command too.
>
> Not version, discover supported features.

That's fine too.  capabilities and versions are equivalent AFAICT so as 
long as one or the other is supported for each command, I'm happy.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Luiz Capitulino]

On Tue, 23 Jun 2009 08:59:42 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> Avi Kivity wrote:
> > On 06/23/2009 07:29 AM, Luiz Capitulino wrote:
> >>       else if (actual == 0)
> >> -        monitor_printf(mon, "Ballooning not activated in VM\n");
> >> +        monitor_printf_err(mon, "Ballooning not activated in VM\n");
> >>       else
> >> -        monitor_printf(mon, "balloon: actual=%d\n", (int)(actual>>  
> >> 20));
> >> +        monitor_printf_data(mon, "balloon: actual=%d\n", 
> >> (int)(actual>>  20));
> >>   }
> >>    
> >
> > Control mode should always use bytes and seconds (and this should be 
> > described in the spec).  You avoid rounding, and more importantly, 
> > ambiguity and a source of unit conversion errors.
> 
> I'd actually like to see a lot more structure in this sort of output.  
> For instance:
> 
> monitor_printf_list(mon, "balloon",
>                                "actual", MON_TIME, actual,
>                                NULL);
> 
> How this gets output can then be conditional on control mode vs. human 
> mode.  In human mode, we can use human-friendly units like MBs.  In 
> control mode, we would always use bytes.

 For me, this is one of the hardest aspects of the project.

 It is more general than unit conversion, the problem is: how should
commands output (specially multline) look like?

 I have talked with Eduardo about this, and I have chosen the simplest
approach, which is prefixing commands output by '='. That is, the
protocol doesn't change commands output, just adds prefixes.

 This approach has the advantage of making the protocol merge very
easy and I believe it will be less painful to maintain as well.

 On the other hand, the format of most commands are not good/meant
to be parsed, this gives the possibility of:

> > We also need a way to discover that the command is available:
> 
> I think we want to version each command too.

 making it quite complicated. :)

 If we go this way (of defining an output for each command or even
defining a version for each command) I fear we will take forever
to get in agreement and merge the whole series.

 Another related question is: should we port 100% of the command
set up front or can we merge small sets, defining a new version of
the protocol each time?

 What I would like to do for QMP V1 is:

1. Keep what this series is doing, that is, keep commands output
unchanged and only add prefixes (unit conversion is ok though)

2. Define a set of commands to be part of V1, and port them

 We can try to port the whole set as fast as we can (say, until
V3). Right after that, we could start improving commands output.

[Qemu-devel] Re: [PATCH 10/11] QMP: Introduce basic events

[Luiz Capitulino]

On Tue, 23 Jun 2009 12:46:06 +0300
Avi Kivity <avi@redhat.com> wrote:

> On 06/23/2009 07:29 AM, Luiz Capitulino wrote:
> > Reboot, shutdown and powerdown are very basic events and can be
> > added together.
> >
> >    
> 
> Reboot only exists within a guest.  qemu sees a reset (there are reboots 
> not associated with a reset, for example kexec).

 Yeah, will fix.

> > diff --git a/vl.c b/vl.c
> > index 60a00e1..7dc5954 100644
> > --- a/vl.c
> > +++ b/vl.c
> > @@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void)
> >           reset_requested = 1;
> >       }
> >       qemu_notify_event();
> > +    monitor_notify_event(EVENT_REBOOT);
> >   }
> >
> >   void qemu_system_shutdown_request(void)
> >   {
> >       shutdown_requested = 1;
> >       qemu_notify_event();
> > +    monitor_notify_event(EVENT_SHUTDOWN);
> >   }
> >
> >   void qemu_system_powerdown_request(void)
> >   {
> >       powerdown_requested = 1;
> >       qemu_notify_event();
> > +    monitor_notify_event(EVENT_POWERDOWN);
> >   }
> >    
> 
> Maybe it's better to signal the event when the reset etc. actually takes 
> place, to avoid subtle races.

 This concern was raised by Daniel as well, will change
this too.

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

On Tue, 23 Jun 2009 14:10:34 +0100
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Tue, Jun 23, 2009 at 10:06:15AM -0300, Luiz Capitulino wrote:
> > On Tue, 23 Jun 2009 11:55:56 +0300
> > Avi Kivity <avi@redhat.com> wrote:
> > 
> > > On 06/23/2009 07:28 AM, Luiz Capitulino wrote:
> > > > This file contains detailed QMP description and definitions.
> > > >
> > > > Signed-off-by: Luiz Capitulino<lcapitulino@redhat.com>
> > > > ---
> > > >   monitor-protocol-spec.txt |  180 +++++++++++++++++++++++++++++++++++++++++++++
> > > >   1 files changed, 180 insertions(+), 0 deletions(-)
> > > >   create mode 100644 monitor-protocol-spec.txt
> > > >
> > > > diff --git a/monitor-protocol-spec.txt b/monitor-protocol-spec.txt
> > > > new file mode 100644
> > > > index 0000000..d20e3f9
> > > > --- /dev/null
> > > > +++ b/monitor-protocol-spec.txt
> > > > @@ -0,0 +1,180 @@
> > > > +               QEMU Monitor Protocol Specification - Version 0.1
> > > > +
> > > > +                                Luiz Capitulino
> > > > +<lcapitulino@redhat.com>
> > > > +
> > > > +1. Introduction
> > > > +===============
> > > > +
> > > > +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol
> > > > +which is available for applications to control QEMU at the machine-level.
> > > >    
> > > 
> > > Without a doubt, this is a the most important file of this patchset.  
> > > There's a huge difference between working with an implementation and 
> > > working with a specification.
> > 
> >  Exactly.
> > 
> > > > +
> > > > +For a detailed list of supported commands, please, refer to file
> > > > +monitor-protocol-commands.txt.
> > > >    
> > > 
> > > I don't see you update that file anywhere.  In any case, my preference 
> > > would be to have everything in one file.
> > 
> >  Sorry, I forgot to mention that I wouldn't include this file
> > in the patchset.
> > 
> >  But its format will be like POP3 or IMAP specs, something like:
> > 
> > """
> > 6.2 cont
> > 
> > Arguments: none.
> > 
> > Responses: + OK cont completed
> > 
> > <description>
> > 
> > <example>
> > """
> > 
> >  We can have everything in the same file then.
> > 
> > > > +3.1 General definition
> > > > +----------------------
> > > > +
> > > > + o Only ASCII is permitted
> > > >    
> > > 
> > > Since the some commands contain user-specified strings, UTF-8 is needed.
> > > 
> > > I think it's worthwhile to define a quoted string format, to be used 
> > > both in commands and responses.
> > 
> >  Ok, as Daniel also said we will need this.
> > 
> >  The only problem is that I was trying to avoid changing commands'
> > output too much, because this can turn into a maintenance nightmare.
> > 
> >  Note that we will have to maintain two protocols: the current existing
> > (user mode) and this new control mode.
> > 
> > > > +
> > > > +3.3.1 Server Greeting
> > > > +---------------------
> > > > +
> > > > +Sent when a new connection is opened.
> > > > +
> > > > +Format: + OK QEMU<version>  QMP<version>
> > > > +Example: + OK QEMU 0.10.50 QMP 0.1
> > > >    
> > > 
> > > Clients should never make decisions based on the qemu or qmp version.  
> > > Rather, we should provide a facility to query the availability of features.
> > 
> >  Right. What about changing 'help' output to only list command names?
> > 
> > > > + o Command completion failed
> > > > +
> > > > +    Format: - ERR<reason>
> > > > +    Example: - ERR could not find network device 'foo'
> > > > +
> > > >    
> > > 
> > > Maybe add a numeric error code (to be defined by individual commands).
> > 
> >  Well, numeric error codes seem redundant in text protocols. At least,
> > POP3 and IMAP don't use them.
> 
> I counter with FTP and HTTP, which do :-P
> 
> eg  http://www.faqs.org/rfcs/rfc959.html
> 
>          200 Command okay.
>          500 Syntax error, command unrecognized.
>              This may include errors such as command line too long.
>          501 Syntax error in parameters or arguments.
>          202 Command not implemented, superfluous at this site.
>          502 Command not implemented.
>          503 Bad sequence of commands.
>          504 Command not implemented for that parameter.

 There goes my argument. :)

 Anthony seems to prefer this way too, although I still think
this is a bit redundant, I don't have a strong opinion here.

 So, we can do this way then.

[Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

On Tue, 23 Jun 2009 16:27:16 +0300
Avi Kivity <avi@redhat.com> wrote:

> On 06/23/2009 04:06 PM, Luiz Capitulino wrote:
> >> I don't see you update that file anywhere.  In any case, my preference
> >> would be to have everything in one file.
> >>      
> >
> >   Sorry, I forgot to mention that I wouldn't include this file
> > in the patchset.
> >    
> 
> You mean now, or forever?

 Only for now, as you noted in another email command porting/additions
should be NACKed if not properly documented.

> >   Ok, as Daniel also said we will need this.
> >
> >   The only problem is that I was trying to avoid changing commands'
> > output too much, because this can turn into a maintenance nightmare.
> >
> >   Note that we will have to maintain two protocols: the current existing
> > (user mode) and this new control mode.
> >    
> 
> I don't see a way out.  The current protocol is not machine consumable, 
> but we can't change it.  We'll just have to live with it.

 Yes.

> >> Clients should never make decisions based on the qemu or qmp version.
> >> Rather, we should provide a facility to query the availability of features.
> >>      
> >
> >   Right. What about changing 'help' output to only list command names?
> >    
> 
> Also command subfeatures, when new subfeatures are added.

 Ok, you suggested a format I'm probably going to use.

> >> Maybe add a numeric error code (to be defined by individual commands).
> >>      
> >
> >   Well, numeric error codes seem redundant in text protocols. At least,
> > POP3 and IMAP don't use them.
> >
> >   This may seem contradictory, but although this a low-level protocol,
> > I feel that text protocols tend to be very descriptive.
> >    
> 
> Well, the strings are not single words, so the key for the application 
> is "all words until the end of the line".  Not too machine friendly.
> 

Re: [Qemu-devel] Re: [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

On Tue, 23 Jun 2009 08:12:33 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> The specification looks pretty good.

 Thanks, used previous discussions as my main reference.

> >> +
> >> +3.3.1 Server Greeting
> >> +---------------------
> >> +
> >> +Sent when a new connection is opened.
> >> +
> >> +Format: + OK QEMU<version>  QMP<version>
> >> +Example: + OK QEMU 0.10.50 QMP 0.1
> >>    
> >
> > Clients should never make decisions based on the qemu or qmp version.  
> > Rather, we should provide a facility to query the availability of 
> > features.
> 
> I agree, but I'd suggest leaving the QMP version in there for insurance 
> purposes in case we really screw up and need to bump the version.  In 
> fact, having the client also negotiate the QMP version isn't a bad idea.

 Will keep.

> >> + o Command completion failed
> >> +
> >> +    Format: - ERR<reason>
> >> +    Example: - ERR could not find network device 'foo'
> >> +
> >>    
> >
> > Maybe add a numeric error code (to be defined by individual commands).
> 
> I think copying HTTP makes sense here.

 Ok, as I said in others emails I was a bit agains't it but have
changed my mind after three being in favor. :)

> How would asynchronous commands work?  Could you give an example of 
> doing live migration through QMP?

 Avi has already explained this part. To be honest, I didn't have this
clear in my mind, but Avi's suggestion is what makes sense to me.

Re: [Qemu-devel] [PATCH 04/11] QMP: Make monitor_handle_command() QMP aware

[Luiz Capitulino]

On Tue, 23 Jun 2009 08:51:16 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> Luiz Capitulino wrote:
> > Two changes are needed in monitor_handle_command() so that it
> > knows how to work in control mode.
> >
> > First, it has to know whether a given command is part of the
> > protocol or not. Second, it has to print the correct server
> > response when a command execution finishes.
> >
> > A good approach to do this would be:
> >
> > 1. Add a new member to struct mon_cmd_t, which will tell if
> > the command is already part of the protocol.
> >
> > 2. Change handler_* functions to return exit status, so that
> > monitor_handle_command() can print "+ OK" or "- ERR"
> > accordling.
> >   
> These changes are a requirement for merging.

 Yes, sure. Would never ask you to merge such a hack.

[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode

[Luiz Capitulino]

On Tue, 23 Jun 2009 16:06:14 +0200
Jan Kiszka <jan.kiszka@siemens.com> wrote:

> Luiz Capitulino wrote:
> > On Tue, 23 Jun 2009 11:03:09 +0200
> > Jan Kiszka <jan.kiszka@siemens.com> wrote:
> > 
> >> Luiz Capitulino wrote:
> >>> This change adds a flag called 'control' to the already existing
> >>> '-monitor' command-line option. This flag can be used to enable
> >>> control mode.
> >>>
> >>> Its syntax is:
> >>>
> >>> qemu [...] -monitor control,<device>
> >>>
> >>> Where <device> is a chardev (excluding 'vc', for obvious reasons).
> >>>
> >>> For example:
> >>>
> >>> $ qemu [...] -monitor control,tcp:localhost:4444,server
> >>>
> >>> Will run QEMU in control mode, waiting for a client TCP connection
> >>> on localhost port 4444.
> >>>
> >>> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> >> At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor
> >> instances". I think Anthony posted such a patch before. Now we should
> >> really include this as your extension may block the stand-alone -monitor
> >> switch for the control channel, preventing to additionally set up one
> >> (or more) for debugging purposes.
> > 
> >  Do you really think that this is needed for the first version? I would
> > rather prefer to concentrate on protocol specification and command
> > porting, which will require a lot of effort.
> 
> You may postpone it if it's too much work, but please have a look at
> http://permalink.gmane.org/gmane.comp.emulators.libvirt/12895 first as I
> think it should be easy to integrate.

 I will have a look before deciding, if it doesn't seem that complicated
I will add the patch.

 Thanks for the pointer!

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Tue, Jun 23, 2009 at 11:09:21AM -0500, Anthony Liguori wrote:
> Avi Kivity wrote:
>>> I'd rather not do json.
>>>
>>
>> Why not?
>
> Because it's too much complexity at this level in the stack.

It's not.

instead of replying the OK, ERROR .. you reply a JsonRPC OK|ERROR structure
with the same string (quoted and utf-8) as the data payload.

instead of parsing each command line with the text protocol thing, you'll get
every parameter from a json array of string (can contain any characters
compared to QMP so far).

if you want you can make things generic at the marshalling level, by using
string everywhere .. which is what QMP is doing.

>>> As they stand, I think the current patch set is very close to being  
>>> mergable.  I'd rather not go off on a new tangent.
>>
>> It fulfils all are requirements and more.  There's a C implementation  
>> available.  We can concentrate on the commands and ignore the plumbing.
>
> The same is true for XML-RPC, CIM, dbus, etc.  RPC grow like weeds.  In  
> 5 years, it will be shocking that qemu uses jsonrpc instead of the next  
> super-neat RPC protocol.
>
> If we did a real RPC mechanism, I'd want to stick with something  
> tried-and-true like XDR.  But an RPC mechanism is not the monitor.  If  
> we were going to do that, there's no point in morphing the monitor into  
> that.

CIM, dbus are binary protocol. same apparently for XDR (which i've never used).
only XMLRPC is fairly similar (but much more verbose and xml is ugly)

Jsonrpc is just one "standard" marshalling protocol, that I think would fit
quite well, but the point is using something already out there not something
that looks like the evil cousin of the IRC protocol.

Usually too, if the rpc has been correctly bolted on something, changing one to
the other is usually trivial (for example xmlrpc to jsonrpc or the other way).
So maybe in 5 years, people will push for a new protocol XYZ .. but that doesn't
mean it's not going to happen to QMP too anyway...

> What we're doing here is adding a computer-mode to the existing monitor.  
> This will be easy for current apps to consume and for us to transition 
> the code base to.  It's a safe and incremental improvement.

i don't see my "proposal", as going against this statement. you could bolt
jsonrpc or xmlrpc or <insert your known protocol> in an incremental fashion too.

> I don't want to invent Xen-API here and go through that pain in QEMU.

This is really *not* comparable to Xen-API. QEMU manage VM and only VM.
Xen-API goes much further.

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
>
> An RPC makes it a lot easier for clients to parse, there's no need for 
> a regular expression per command, or to bother with line continuations 
> etc.  You just focus on the data.
>
> We have an opportunity to make a clean break, let's make use of it.

If we structure things internally correctly, by having a flow that 
basically looks like:

int <monitor_command>(Monitor *mon, <args>)
{
   // process args
   if (success) {
       monitor_print_data(mon, <highly structured output>);
    } else {
       record_extended_error(<message>);
       return error_code;
    }
}

Then it that works well for both the human monitor and the computer 
monitor.  It's an RPC though.

Should we support other types of RPCs?  I don't know.  I'd think that 
you could create a computer monitor -> jsonrpc bridge easily enough.

If we implement this correctly, then an RPC interface could just be 
another option to see the computer monitor mode implemented as a "module".

>> I don't want to invent Xen-API here and go through that pain in QEMU.
>
> I don't see the parallel.

Xen-API was the "clean break" for Xend.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Anthony Liguori]

Luiz Capitulino wrote:
> On Tue, 23 Jun 2009 08:59:42 -0500
> Anthony Liguori <anthony@codemonkey.ws> wrote:
>
>   
>> How this gets output can then be conditional on control mode vs. human 
>> mode.  In human mode, we can use human-friendly units like MBs.  In 
>> control mode, we would always use bytes.
>>     
>
>  For me, this is one of the hardest aspects of the project.
>
>  It is more general than unit conversion, the problem is: how should
> commands output (specially multline) look like?
>
>  I have talked with Eduardo about this, and I have chosen the simplest
> approach, which is prefixing commands output by '='. That is, the
> protocol doesn't change commands output, just adds prefixes.
>   

I'm not saying you should change command output.  Most commands follow a 
similar format quite intentionally.  That's:

<header>: <param>=<value>[,<param>=<value>[...]]

I'm just suggesting that you codify that instead of leaving it open 
coded everywhere.  Nothing more than that.

>  This approach has the advantage of making the protocol merge very
> easy and I believe it will be less painful to maintain as well.
>   

FWIW, you could introduce monitor_printf_list() as your first patch.

>>> We also need a way to discover that the command is available:
>>>       
>> I think we want to version each command too.
>>     
>
>  making it quite complicated. :)
>
>  If we go this way (of defining an output for each command or even
> defining a version for each command) I fear we will take forever
> to get in agreement and merge the whole series.
>
>  Another related question is: should we port 100% of the command
> set up front or can we merge small sets, defining a new version of
> the protocol each time?
>
>  What I would like to do for QMP V1 is:
>
> 1. Keep what this series is doing, that is, keep commands output
> unchanged and only add prefixes (unit conversion is ok though)
>
> 2. Define a set of commands to be part of V1, and port them
>
>  We can try to port the whole set as fast as we can (say, until
> V3). Right after that, we could start improving commands output.
>   

I wanted to respond to the top level, but here's what I see as the 
potential merge plan:

1) Make all commands return a value
2) Add a mechanism to mark commands as being "structured"
3) Get the structured printfs in order.
4) Merge

I won't want QMP to be exposed as a usable interface until all commands 
are converted.  This means holding off on the last patch I think.  I 
don't think we'll get QMP for 0.11.  I think it's likely going to be a 
0.12 feature.  However, I'd like to start merging this series as soon as 
humanly possible.  I think it's already pretty close.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 09:32 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>>
>> An RPC makes it a lot easier for clients to parse, there's no need 
>> for a regular expression per command, or to bother with line 
>> continuations etc.  You just focus on the data.
>>
>> We have an opportunity to make a clean break, let's make use of it.
>
> If we structure things internally correctly, by having a flow that 
> basically looks like:
>
> int <monitor_command>(Monitor *mon, <args>)
> {
>   // process args
>   if (success) {
>       monitor_print_data(mon, <highly structured output>);
>    } else {
>       record_extended_error(<message>);
>       return error_code;
>    }
> }
>
> Then it that works well for both the human monitor and the computer 
> monitor.  It's an RPC though.

It looks brittle to me.  What if you want to add an array?  Some 
structure that contains a nested structure?

> Should we support other types of RPCs?  I don't know.  I'd think that 
> you could create a computer monitor -> jsonrpc bridge easily enough.

No doubt, but I look at jsonrpc as a way of reducing the effort in 
qemu.  Not so much interested in it for its own sake.

> If we implement this correctly, then an RPC interface could just be 
> another option to see the computer monitor mode implemented as a 
> "module".

This is deep into overkill territory IMO.  We should pick some format 
that is good enough and has implementations for a variety of languages.  
Luiz' RFC was well done, but the subject is complex enough to be 
difficult to get right.  And as far as I could tell, all the issues are 
addressed by jsonrpc.  The only downside I can see is that jsonrpc is 
some kind of ajax-kiddie web-2.0 script dom thingie while we are a 
serious UNIX style daemon that complies with RFC 123456.


>>> I don't want to invent Xen-API here and go through that pain in QEMU.
>>
>> I don't see the parallel.
>
> Xen-API was the "clean break" for Xend.
>

Anything can be screwed up.  Normally I'm very much against rewrites and 
totally for incremental progress, but when it comes to protocols, you 
have to get them right first shot.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> It looks brittle to me.  What if you want to add an array?  Some 
> structure that contains a nested structure?

Can you give me concrete examples?

The typical way this would be handled in an RPC library in C is to 
introduce complex types.  For instance, you could have a MonitorList and 
a corresponding MON_LIST type.  FWIW, is there a jsonrpc library for C 
that's actually maintained?

I can't find one that even has a functioning website.

>> If we implement this correctly, then an RPC interface could just be 
>> another option to see the computer monitor mode implemented as a 
>> "module".
>
> This is deep into overkill territory IMO.  We should pick some format 
> that is good enough and has implementations for a variety of 
> languages.  Luiz' RFC was well done, but the subject is complex enough 
> to be difficult to get right.  And as far as I could tell, all the 
> issues are addressed by jsonrpc.  The only downside I can see is that 
> jsonrpc is some kind of ajax-kiddie web-2.0 script dom thingie while 
> we are a serious UNIX style daemon that complies with RFC 123456.

There are many reasons why JSON is a bad representation.  For starters, 
there is no way to encode NULL values in a JSON string.  This means you 
cannot send binary data unless you base64 encode it first.  The bounds 
of integers are also not well defined but most implementations restrict 
integers to int32.  You basically cannot send 64-bit integers without 
encoding them first.  It gets really hairy.

For jsonrpc, there's no standard way to enumerate methods like there is 
in xml-rpc and jsonrpc isn't self-describing.  Having an RPC be 
self-describing is really useful when dealing with a dynamic language 
because you can then avoid IDL compilation.

So as far as RPCs go, jsonrpc is actually a pretty bad one.  The odd 
thing about it is you still have to parse json in JavaScript to avoid 
security problems.  Parsing json in js seems to eliminate 99% of the 
useness of it.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 10:00 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>> It looks brittle to me.  What if you want to add an array?  Some 
>> structure that contains a nested structure?
>
> Can you give me concrete examples?

You want statistics for all network interfaces in one go.  In RPC you 
return [ { name: blah stats: { rx: { packets: 123, bytes: 456, errors: 
789 }, tx: etc.  In a text protocol you have more work to write, parse, 
and most importantly document the protocol.

> The typical way this would be handled in an RPC library in C is to 
> introduce complex types.  For instance, you could have a MonitorList 
> and a corresponding MON_LIST type.  FWIW, is there a jsonrpc library 
> for C that's actually maintained?
>
> I can't find one that even has a functioning website.

I think a json C library would be sufficient, there likely are several.  
The RPC part is really trivial for a server.

>> This is deep into overkill territory IMO.  We should pick some format 
>> that is good enough and has implementations for a variety of 
>> languages.  Luiz' RFC was well done, but the subject is complex 
>> enough to be difficult to get right.  And as far as I could tell, all 
>> the issues are addressed by jsonrpc.  The only downside I can see is 
>> that jsonrpc is some kind of ajax-kiddie web-2.0 script dom thingie 
>> while we are a serious UNIX style daemon that complies with RFC 123456.
>
> There are many reasons why JSON is a bad representation.  For 
> starters, there is no way to encode NULL values in a JSON string.  
> This means you cannot send binary data unless you base64 encode it 
> first.  The bounds of integers are also not well defined but most 
> implementations restrict integers to int32.  You basically cannot send 
> 64-bit integers without encoding them first.  It gets really hairy.

These look like show stoppers to me.

> For jsonrpc, there's no standard way to enumerate methods like there 
> is in xml-rpc and jsonrpc isn't self-describing.  Having an RPC be 
> self-describing is really useful when dealing with a dynamic language 
> because you can then avoid IDL compilation.

I think you can avoid IDL compilation even without a self describing RPC.

> So as far as RPCs go, jsonrpc is actually a pretty bad one.  The odd 
> thing about it is you still have to parse json in JavaScript to avoid 
> security problems.  Parsing json in js seems to eliminate 99% of the 
> useness of it.

Are you referring to the fact that eval() on json is insecure?  We don't 
have eval() in C.

Well, if not json, let's find some established RPC (or just data 
serialization) that is not XML-RPC or Sun RPC and use it, so we can 
concentrate on adding commands easily rather than writing formats, 
formatters, and parsers.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 10:00 PM, Anthony Liguori wrote:
> For starters, there is no way to encode NULL values in a JSON string.  
> This means you cannot send binary data unless you base64 encode it first.

That seems to have been solved:


js> a = 'x\0y'
xy
js> a.length
3

I don't know enough javascript to verify that it's indeed a NUL, but I'm 
guessing that it is.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> On 06/23/2009 10:00 PM, Anthony Liguori wrote:
>> For starters, there is no way to encode NULL values in a JSON 
>> string.  This means you cannot send binary data unless you base64 
>> encode it first.
>
> That seems to have been solved:
>
>
> js> a = 'x\0y'
> xy
> js> a.length
> 3
>
> I don't know enough javascript to verify that it's indeed a NUL, but 
> I'm guessing that it is.

This is a browser restriction.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/23/2009 11:07 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>> On 06/23/2009 10:00 PM, Anthony Liguori wrote:
>>> For starters, there is no way to encode NULL values in a JSON 
>>> string.  This means you cannot send binary data unless you base64 
>>> encode it first.
>>
>> That seems to have been solved:
>>
>>
>> js> a = 'x\0y'
>> xy
>> js> a.length
>> 3
>>
>> I don't know enough javascript to verify that it's indeed a NUL, but 
>> I'm guessing that it is.
>
> This is a browser restriction.
>

So this will only hit us is we port qemu to javascript and run it in a 
web broswer.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Tue, Jun 23, 2009 at 02:00:41PM -0500, Anthony Liguori wrote:
> Avi Kivity wrote:
>> It looks brittle to me.  What if you want to add an array?  Some  
>> structure that contains a nested structure?
>
> Can you give me concrete examples?
>
> The typical way this would be handled in an RPC library in C is to  
> introduce complex types.  For instance, you could have a MonitorList and  
> a corresponding MON_LIST type.  FWIW, is there a jsonrpc library for C  
> that's actually maintained?
>

this one is mentioned on the website and has been changed recently (may 2009):

http://fara.cs.uni-potsdam.de/~jsg/json_parser/

> There are many reasons why JSON is a bad representation.  For starters,  
> there is no way to encode NULL values in a JSON string.  This means you  
> cannot send binary data unless you base64 encode it first.  The bounds  
> of integers are also not well defined but most implementations restrict  
> integers to int32.  You basically cannot send 64-bit integers without  
> encoding them first.  It gets really hairy.

i'm not sure why do you think that, but the JSON string are completly escaped
and there's no restricted value (compared to xml) since everything can be as
a last measure be escaped as decimal \1234 .. in a case of a null \0000

for Int, they don't have any precision (infinite number of
digits), so at the C level they should be represented as string of digits. So
that some library represent the int as a C int, doesn't mean the protocol
enforce that on you.

If that would be a problem and you couldn't change the implementation to do the
right thing, you could represent your 64 bits int as a json string, so that the
implementation on the other side represent it adequately.

but again the protocol can handle any length of integer, and there's no
encoding issue because it's just like a human would write an integer of any
precision (i.e. 123218478219742718)

> For jsonrpc, there's no standard way to enumerate methods like there is  
> in xml-rpc and jsonrpc isn't self-describing.  Having an RPC be  
> self-describing is really useful when dealing with a dynamic language  
> because you can then avoid IDL compilation.

does QMP solve the problem here ?

> So as far as RPCs go, jsonrpc is actually a pretty bad one.  The odd  
> thing about it is you still have to parse json in JavaScript to avoid  
> security problems.  Parsing json in js seems to eliminate 99% of the  
> useness of it.

but C is not javascript. it's doesn't have eval, so your argument here is
bogus. also JSon doesn't imply to have any dependancy whatsoever on javascript.

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> So this will only hit us is we port qemu to javascript and run it in a 
> web broswer.

You assume that client implementations are going to always accept nulls 
in strings.  Considering that the most popular clients (i.e. browsers) 
don't support this, I don't think it's a safe assumption.

Regards,

Anthony Liguori


-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Vincent Hanquez wrote:
> On Tue, Jun 23, 2009 at 02:00:41PM -0500, Anthony Liguori wrote:
>   
>> Avi Kivity wrote:
>>     
>>> It looks brittle to me.  What if you want to add an array?  Some  
>>> structure that contains a nested structure?
>>>       
>> Can you give me concrete examples?
>>
>> The typical way this would be handled in an RPC library in C is to  
>> introduce complex types.  For instance, you could have a MonitorList and  
>> a corresponding MON_LIST type.  FWIW, is there a jsonrpc library for C  
>> that's actually maintained?
>>
>>     
>
> this one is mentioned on the website and has been changed recently (may 2009):
>
> http://fara.cs.uni-potsdam.de/~jsg/json_parser/
>   

That's not a jsonrpc library, that's just a json parser.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Tue, Jun 23, 2009 at 05:50:24PM -0500, Anthony Liguori wrote:
>> this one is mentioned on the website and has been changed recently (may 2009):
>>
>> http://fara.cs.uni-potsdam.de/~jsg/json_parser/
>>   
>
> That's not a jsonrpc library, that's just a json parser.

but this is completly trivial at this point. jsonrpc is a json message put in a
certain way.

now providing there was a maintained library out there, would you use it ?
or do you have more concerns that were unanswered ?

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[James]

> now providing there was a maintained library out there, would you use it ?
> or do you have more concerns that were unanswered ?

I have exactly one concern. If Qemu grows an RPC protocol, it will be used 
by a large number of other programs to control qemu. This is a good thing.
But it is important that such a system 

1) either supports some sort of introspection so that the functions and
   their arguments can be divined by the external program without knowing
   which particular version of qemu it is talking to.

2) and/or supports an IDL so that the interface is properly documented in
   something other than an under-edited text file.

	
To this end I tentitatively suggest d-bus as a possible choice. It fills 1 and 2
nicely and also allows each module to have its own interfaces and objects. 
The IDL will also generate C bindings for both ends.

I should add I'm not a fan of d-bus, but it seems a better fit than JSON-RPC

James.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Wed, Jun 24, 2009 at 12:55:37PM +0100, James wrote:
> > now providing there was a maintained library out there, would you use it ?
> > or do you have more concerns that were unanswered ?
> 
> I have exactly one concern. If Qemu grows an RPC protocol, it will be used 
> by a large number of other programs to control qemu. This is a good thing.
> But it is important that such a system 
> 
> 1) either supports some sort of introspection so that the functions and
>    their arguments can be divined by the external program without knowing
>    which particular version of qemu it is talking to.
> 
> 2) and/or supports an IDL so that the interface is properly documented in
>    something other than an under-edited text file.
> 
> 	
> To this end I tentitatively suggest d-bus as a possible choice. It fills 1 and 2
> nicely and also allows each module to have its own interfaces and objects. 
> The IDL will also generate C bindings for both ends.

It depends what you mean by 'dbus' here. I don't think managing QEMU over
the dbus bus provides the right architecture - I don't think you want
every QEMU appearing on the bus.5B You could do direct peer-to-peer comms
so you're just using libdbus for message encapsulation / processing, but
there's not really much above XDR. XDR is nice because its a portable
library that works on any OS, including OS-X and WIndows, and solely
concerns itself with data encoding, not data transport.  

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[James]

> It depends what you mean by 'dbus' here. I don't think managing QEMU over
> the dbus bus provides the right architecture - I don't think you want
> every QEMU appearing on the bus.  5B You could do direct peer-to-peer comms
> so you're just using libdbus for message encapsulation / processing, but
> there's not really much above XDR. XDR is nice because its a portable
> library that works on any OS, including OS-X and WIndows, and solely
> concerns itself with data encoding, not data transport.  

That leaves the rendezvous, and security issues to be re-invented wheels,
you could use SUNRPC and XDR. That has an excellent IDL. 

I don't see why you wouldn't want it on the 'bus', and more than you'd want it
bound to a tcp port on localhost. It would make things a lot simpler: something
like a little command line utility to connect a CDROM or ISO image to a Qemu
instance which would be identified by some uuid or name would be very easy to
implement. 

James.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Vincent Hanquez wrote:
> On Tue, Jun 23, 2009 at 05:50:24PM -0500, Anthony Liguori wrote:
>   
>>> this one is mentioned on the website and has been changed recently (may 2009):
>>>
>>> http://fara.cs.uni-potsdam.de/~jsg/json_parser/
>>>   
>>>       
>> That's not a jsonrpc library, that's just a json parser.
>>     
>
> but this is completly trivial at this point. jsonrpc is a json message put in a
> certain way.
>
> now providing there was a maintained library out there, would you use it ?
> or do you have more concerns that were unanswered ?

There are two questions to resolve.  The first is whether we should 
continue with the current direction (line-based protocol) or whether we 
should switch to an RPC.  The second question is which RPC we should use.

I'm not at all convinced that we should switch to an RPC mechanism in 
the first place.  Perhaps someone could summarize the advantages of 
doing this because right now, I don't see many.

With respect to RPC choice, if we did go that route, I'd be very 
concerned about using jsonrpc verses a more well established rpc.  I 
would honestly prefer xml-rpc over jsonrpc.

One reason to choose an RPC is based on the adoption of it.  You want to 
use something that has a vibrant community with well established client 
libraries to make writing clients as easy as possible.  Without an 
active jsonrpc C library, it's hard to argue that jsonrpc has that.  
xml-rpc certainly does.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Wed, Jun 24, 2009 at 01:32:12PM +0100, James wrote:
> > It depends what you mean by 'dbus' here. I don't think managing QEMU over
> > the dbus bus provides the right architecture - I don't think you want
> > every QEMU appearing on the bus.  5B You could do direct peer-to-peer comms
> > so you're just using libdbus for message encapsulation / processing, but
> > there's not really much above XDR. XDR is nice because its a portable
> > library that works on any OS, including OS-X and WIndows, and solely
> > concerns itself with data encoding, not data transport.  
> 
> That leaves the rendezvous, and security issues to be re-invented wheels,
> you could use SUNRPC and XDR. That has an excellent IDL. 
> 
> I don't see why you wouldn't want it on the 'bus', and more than you'd want it
> bound to a tcp port on localhost. It would make things a lot simpler: something
> like a little command line utility to connect a CDROM or ISO image to a Qemu
> instance which would be identified by some uuid or name would be very easy to
> implement. 

This is exactly why I don't want it on the bus. I don't think the monitor
interface is the general end-developer/user administrative interface. It
is a control channel by which a mgmt tool will control QEMU, with the mgmt
tool providing the end-developer/user admin interface. If someone wants
to write a management app that controls QEMU instances and exposes an
API on dbus that's fine, but the individual QEMU instances should not be
directly exposed for use.


Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Wed, Jun 24, 2009 at 07:46:04AM -0500, Anthony Liguori wrote:
> Vincent Hanquez wrote:
> >On Tue, Jun 23, 2009 at 05:50:24PM -0500, Anthony Liguori wrote:
> >  
> >>>this one is mentioned on the website and has been changed recently (may 
> >>>2009):
> >>>
> >>>http://fara.cs.uni-potsdam.de/~jsg/json_parser/
> >>>  
> >>>      
> >>That's not a jsonrpc library, that's just a json parser.
> >>    
> >
> >but this is completly trivial at this point. jsonrpc is a json message put 
> >in a
> >certain way.
> >
> >now providing there was a maintained library out there, would you use it ?
> >or do you have more concerns that were unanswered ?
> 
> There are two questions to resolve.  The first is whether we should 
> continue with the current direction (line-based protocol) or whether we 
> should switch to an RPC.  The second question is which RPC we should use.
> 
> I'm not at all convinced that we should switch to an RPC mechanism in 
> the first place.  Perhaps someone could summarize the advantages of 
> doing this because right now, I don't see many.

It really depends what you consider the purpose of the monitor to be.
If the monitor is the "API" application developers use for management,
then a full blown RPC mechaism makes sense. If the monitor is an
internal control mechanism, to enable the building mgmt tools, then
RPC is overkill. I consider QEMU's monitor to be the latter, and 
interface for a single application to use (and developer debugging). 
If we try to turn the monitor into the general end-app developer mgmt
interface, then you open the floodgates to vastly more functionality 
being pushed into the QEMU codebase. I don't think we want that, QEMU
should be lean & mean, and if you want fancy RPC systems then do it in
a tool layered above, be it libvirt, XenD, a CIM agent, or something 
else.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/24/2009 03:46 PM, Anthony Liguori wrote:
> There are two questions to resolve.  The first is whether we should 
> continue with the current direction (line-based protocol) or whether 
> we should switch to an RPC.  The second question is which RPC we 
> should use.
>
> I'm not at all convinced that we should switch to an RPC mechanism in 
> the first place.  Perhaps someone could summarize the advantages of 
> doing this because right now, I don't see many.

Less effort for us.

Less effort for clients.

Less documentation effort.

Less likelihood of design and implementation holes (cf. UTF-8, quoting).

Richer data types (arrays, structures, nesting).

> With respect to RPC choice, if we did go that route, I'd be very 
> concerned about using jsonrpc verses a more well established rpc.  I 
> would honestly prefer xml-rpc over jsonrpc.

I agree xml-rpc is a more rational choice than jsonrpc, but I cannot 
find it in my heart to say something nice about xml.  Additionally, XML 
parsers are pretty heavy.

> One reason to choose an RPC is based on the adoption of it.  You want 
> to use something that has a vibrant community with well established 
> client libraries to make writing clients as easy as possible.  Without 
> an active jsonrpc C library, it's hard to argue that jsonrpc has 
> that.  xml-rpc certainly does.

jsonrpc really is a trivial addition over json, and json is pretty 
widespread.  I think I saw a Python jsonrpc implementation.

The important thing however is to reuse, it doesn't really matter what 
we reuse really as long as it's good enough.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/24/2009 04:02 PM, Daniel P. Berrange wrote:
>
> It really depends what you consider the purpose of the monitor to be.
> If the monitor is the "API" application developers use for management,
> then a full blown RPC mechaism makes sense. If the monitor is an
> internal control mechanism, to enable the building mgmt tools, then
> RPC is overkill.

I don't understand the difference.  Every controllable feature should be 
exposed by the monitor, and nothing else.  In both cases.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Ian Jackson]

Anthony Liguori writes ("Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file"):
> I'm not at all convinced that we should switch to an RPC mechanism in 
> the first place.  Perhaps someone could summarize the advantages of 
> doing this because right now, I don't see many.

I'm keen on RPC-like systems.  The main advantage is that you don't
have to invent and write a whole lot of marshalling and parsing code;
the RPC system will provide all that for you.  The more sophisticated
systems deal with message transport as well, so you don't need to mess
about with sockets either.

Parsers and formatters are one of the most tedious bits of code to
write and they are also critical for security, reliability and data
transparency.  It really doesn't make sense to roll your own for any
nontrivial problem.

If you pick the right RPC system you can get, entirely for free (well,
entirely piggybacking on other people's work) compact and efficiently
parseable binary wire formats.  Debugging trace output facilities,
which produce in nice human readable formats, are often provided as a
standard feature for no added effort.  Bindings for a wide variety of
programming languages can be easily found.  If you pick a bus- or
broker-based system (dbus, XDR, corba even) you get discovery and
rendezvous for free too.  Many systems have builtin facilities for
cross-version compatibility, of varying sophistication.

I would go further and say that we should use an RPC system with a
machine-readable IDL (Interface Description Language).  A formal IDL
provides an number of very significant advantages.  If you use an IDL,
the protocol is documented not primarily in a human-only-readable text
file, but primarily in the IDL, which means that the documentation
cannot get out of date.  Introspection and debugging features are
easier to provide because all of the programs have actual knowledge of
the message format and structure.  The wire formats can be much more
compact and efficient, because they do not need to tag every value
with its type and format - the receiving end knows what the format is
supposed to be and can get straight to unpicking it.

You can have an argument about which specific RPC mechanism to use,
but it's an embarrassment of riches compared to designing and writing
your own textual protocol, particularly if that protocol needs to be
quite complex and asynchronous.

I think dbus would be a good choice, but even XML-RPC (which, frankly,
is flabby, overcomplex, slow, hard to read, etc.), JSON (which doesn't
have an IDL that can even do checking), and ASN.1/DER/BER (which is a
complexity nightmare) would be an improvement over no automatic
parsing machinery.

> One reason to choose an RPC is based on the adoption of it.  You want to 
> use something that has a vibrant community with well established client 
> libraries to make writing clients as easy as possible.  Without an 
> active jsonrpc C library, it's hard to argue that jsonrpc has that.  
> xml-rpc certainly does.

Yes.

Ian.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Wed, Jun 24, 2009 at 2:46 PM, Anthony Liguori<anthony@codemonkey.ws> wrote:
> Vincent Hanquez wrote:
>>
>> On Tue, Jun 23, 2009 at 05:50:24PM -0500, Anthony Liguori wrote:
>>
>>>>
>>>> this one is mentioned on the website and has been changed recently (may
>>>> 2009):
>>>>
>>>> http://fara.cs.uni-potsdam.de/~jsg/json_parser/
>>>>
>>>
>>> That's not a jsonrpc library, that's just a json parser.
>>>
>>
>> but this is completly trivial at this point. jsonrpc is a json message put
>> in a
>> certain way.
>>
>> now providing there was a maintained library out there, would you use it ?
>> or do you have more concerns that were unanswered ?
>
> There are two questions to resolve.  The first is whether we should continue
> with the current direction (line-based protocol) or whether we should switch
> to an RPC.  The second question is which RPC we should use.

I've spent the last 5 years working on different projects that
involved text/line-based protocols. In fact my current day job
involves working on a program that works with wide variety of such
protocols (POP3, SMTP, IMAP, ACAP, HTTP to name a few) and based on my
experience I would seriously recommend using an existing RPC protocol
instead. While it's simple to design a synchronous protocol like the
original POP3 it gets a lot harder with asynchronous protocols like
IMAP. Even if you decide for a custom line-level protocol I'd suggest
to learn from the past mistakes, more details below.

The current draft specification used a mix of semantics of the
forementioned protocols - error handling based on POP3 (+OK/-ERR),
asynchronous events based on IMAP (* prefix). IMAP-like tagging of
commands was also suggested (to deal with asynchronous commands) and
issues have been raised concerning multi-line responses. Note that all
this gets a bit hairy when you combine it. Just a few issues that come
to mind:

* An event happens while a multi-line response is currently written to
the stream. The behavior has to be defined somehow - either by
specifying that the multi-line response mustn't be interrupted by
asynchronous events or that it can be interrupted, but only on line
breaks. This issue don't arise with message-based protocols.

* Should the asynchronous replies (* EVENT) be tagged (ie. C: "<tag1>
reboot", S: "* <tag1> reset") when they result from a command issued
by the manager? If they are not tagged then it's impossible to find
out what (if any) of the commands caused the event when multiple
asynchronous commands are in progress. That's what basic IMAP does (as
specified by RFC 3501). What IMAP did wrong [1] is that even the basic
responses to commands like SEARCH are returned as untagged
asynchronous notifications, which makes the client implementation
hard. Several extensions were later developed to mitigate it and the
protocol is now a messy combination of both tagged and untagged
responses with no universal syntax. On the other hand, if you tag the
events then it's hard for the server (in this case QEMU) to keep track
of it and a special syntax still has to be present for truly
asynchronous events.

* It is necessary to separate the syntax of the data protocol from the
semantics of the commands itself. The syntax should be generic enough
to support transferring all the necessary information (integers,
strings, lists) and simple enough to be parsable even if you receive
the data in chunks (assuming a text-based protocol). This is where
IMAP went really wrong, since parsing the server responses requires
knowning the semantics of ALL the commands.

* Many of you asked for facility to negotiate capabilities. I'd
suggest the following - the server would lists capabilities by name as
a list on connect and the client would be able to specifically enable
those that change the semantics of commands. Example:

S: + OK QEMU 0.10.50 QMP 0.1 (pci-addr, x-some-extension)
C: enable pci-addr
S: + OK pci-addr extension enabled

The "enable" command would optionally allow for a list to be
specified, to enable more extensions at once.

Best regards,
Filip Navara

[1] ...according to large amount of people with the exception of Mark Crispin.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Wed, Jun 24, 2009 at 07:46:04AM -0500, Anthony Liguori wrote:
> There are two questions to resolve.  The first is whether we should  
> continue with the current direction (line-based protocol) or whether we  
> should switch to an RPC.  The second question is which RPC we should use.

well, when choosing simple RPC like xmlrpc or jsonrpc you still need to
do the transport. so it could well be a line-based protocol that happens
to contains query in a JSONRPC or XMLRPC format ... as long as you don't
pretty print JSONRPC or XMLRPC (guaranteed to not contain \n). a bit clunky
but possible.

> I'm not at all convinced that we should switch to an RPC mechanism in  
> the first place.  Perhaps someone could summarize the advantages of  
> doing this because right now, I don't see many.
>
> With respect to RPC choice, if we did go that route, I'd be very  
> concerned about using jsonrpc verses a more well established rpc.  I  
> would honestly prefer xml-rpc over jsonrpc.

while i have a profound distaste of xmlrpc ... i still think that would
be a improvement against the current situation .. or even to QMP.

however you have to keep in mind that xmlrpc means no binary data,
and depending on an XML library (compare to beeing able to pull a very tiny
code base directly in qemu tree)

> One reason to choose an RPC is based on the adoption of it.  You want to  
> use something that has a vibrant community with well established client  
> libraries to make writing clients as easy as possible.  Without an  
> active jsonrpc C library, it's hard to argue that jsonrpc has that.   
> xml-rpc certainly does.

it might be missing for C, but lots of language got a maintained and used
library nowadays. We're using it in ocaml, and i have to say we haven't 
been disappointed using it, compared to the number of disappointments with
xmlrpc (and mostly xml in general).

That said, I'll make a C library even if qemu doesn't end up using it.

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

Daniel P. Berrange wrote:

> On Wed, Jun 24, 2009 at 01:32:12PM +0100, James wrote:
>>> It depends what you mean by 'dbus' here. I don't think managing QEMU over
>>> the dbus bus provides the right architecture - I don't think you want
>>> every QEMU appearing on the bus.  5B You could do direct peer-to-peer comms
>>> so you're just using libdbus for message encapsulation / processing, but
>>> there's not really much above XDR. XDR is nice because its a portable
>>> library that works on any OS, including OS-X and WIndows, and solely
>>> concerns itself with data encoding, not data transport.  
>> That leaves the rendezvous, and security issues to be re-invented wheels,
>> you could use SUNRPC and XDR. That has an excellent IDL. 
>>
>> I don't see why you wouldn't want it on the 'bus', and more than you'd want it
>> bound to a tcp port on localhost. It would make things a lot simpler: something
>> like a little command line utility to connect a CDROM or ISO image to a Qemu
>> instance which would be identified by some uuid or name would be very easy to
>> implement. 
> 
> This is exactly why I don't want it on the bus. I don't think the monitor
> interface is the general end-developer/user administrative interface. It
> is a control channel by which a mgmt tool will control QEMU, with the mgmt
> tool providing the end-developer/user admin interface. If someone wants
> to write a management app that controls QEMU instances and exposes an
> API on dbus that's fine, but the individual QEMU instances should not be
> directly exposed for use.
> 

Why qemu shouldn't provide a monitor interface on dbus?
The mgmt tool could still control qemu and provide the
end-developer/user admin interface anyway.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

Anthony Liguori wrote:

> Vincent Hanquez wrote:
>> On Tue, Jun 23, 2009 at 05:50:24PM -0500, Anthony Liguori wrote:
>>   
>>>> this one is mentioned on the website and has been changed recently (may 2009):
>>>>
>>>> http://fara.cs.uni-potsdam.de/~jsg/json_parser/
>>>>   
>>>>       
>>> That's not a jsonrpc library, that's just a json parser.
>>>     
>> but this is completly trivial at this point. jsonrpc is a json message put in a
>> certain way.
>>
>> now providing there was a maintained library out there, would you use it ?
>> or do you have more concerns that were unanswered ?
> 
> There are two questions to resolve.  The first is whether we should 
> continue with the current direction (line-based protocol) or whether we 
> should switch to an RPC.  The second question is which RPC we should use.
> 
> I'm not at all convinced that we should switch to an RPC mechanism in 
> the first place.  Perhaps someone could summarize the advantages of 
> doing this because right now, I don't see many.
> 

Using an IDL it would be easier to generate boilerplate code for
different languages, easier to document and evolve the interface.
If the rpc library is good enough we would get marshalling, data
transport, rendezvous, introspection and security already solved
for us.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Chris Webb]

Anthony Liguori <anthony@codemonkey.ws> writes:

> There are two questions to resolve.  The first is whether we should  
> continue with the current direction (line-based protocol) or whether we  
> should switch to an RPC.  The second question is which RPC we should use.
>
> I'm not at all convinced that we should switch to an RPC mechanism in  
> the first place.  Perhaps someone could summarize the advantages of  
> doing this because right now, I don't see many.
>
> With respect to RPC choice, if we did go that route, I'd be very  
> concerned about using jsonrpc verses a more well established rpc.  I  
> would honestly prefer xml-rpc over jsonrpc.

We are an example of a end user of qemu (or more specifically qemu-kvm) that
doesn't go via a management layer like libvirt. Our management shell scripts
directly control the virtual machines using the current line-oriented 'human
friendly' monitor. This is a bit of a pain, but not too bad in practice.

A more regular and well defined line-based protocol would be a big plus for
us, whereas something like jsonrpc or xml-rpc would be a complete disaster
to call from the shell---less useful than the current human oriented monitor
rather than more so.

(Aside from the practical question, personally I share your gut feeling that
it seems like a sledgehammer to crack a nut and not a good fit at all for
this layer of the stack.)

Cheers,

Chris.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Wed, Jun 24, 2009 at 2:46 PM, Anthony Liguori<anthony@codemonkey.ws> wrote:
[snip]
> With respect to RPC choice, if we did go that route, I'd be very concerned
> about using jsonrpc verses a more well established rpc.  I would honestly
> prefer xml-rpc over jsonrpc.
>
> One reason to choose an RPC is based on the adoption of it.  You want to use
> something that has a vibrant community with well established client
> libraries to make writing clients as easy as possible.  Without an active
> jsonrpc C library, it's hard to argue that jsonrpc has that.  xml-rpc
> certainly does.

After reading the XDR specification I realized that the basic blocks
are already implemented in QEMU.

Data type mapping:

Integer <-> qemu_put_sbe32 / qemu_get_sbe32
Unsigned Integer <-> qemu_put_be32 / qemu_get_be32
Enumeration - same as signed integer
Boolean - same as signed interger, limited to values 0 and 1
Hyper Integer <-> qemu_put_sbe64 / qemu_get_sbe64
Unsigned Hyper Integer <-> qemu_put_be64 / qemu_get_be64
Floating-point, Double-precision Floating-point, Quadruple-precision
Floating-point <-> ? (uses IEEE notation)
Fixed-length Opaque Data <->

void qemu_put_xdr_buffer(QEMUFile *f, const uint8_t *buf, int size)
{
    int reminder = size % 4;
    qemu_put_buffer(f, buf, size);
    while (--reminder >= 0)
        qemu_put_byte(f, 0);
}

Variable-length Opaque Data <->

void qemu_put_xdr_var_buffer(QEMUFile *f, const uint8_t *buf, int size)
{
    int reminder = size % 4;
    qemu_put_be32(f, size);
    qemu_put_buffer(f, buf, size);
    while (--reminder >= 0)
        qemu_put_byte(f, 0);
}

String <-> same as qemu_put_xdr_var_buffer, ASCII bytes (*not
characters*, the encoding of characters is not part of the RFC, NFSv4
uses UTF-8 for example), null-terminator is not sent
Fixed-length Array <-> for loop with the elements described using the
basic types above
Variable-length Array <-> qemu_put_be32 + for loop / qemu_get_be32 +
allocation + for loop
Structure, Discriminated Union <-> built from the basic data types, same as in C
Void <-> nothing :)

Given the fact that we practically have the library for XDR it would
be very easy to build Sun RPC server on top of it. So my vote is for
SunRPC.

Best regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Anthony Liguori wrote:
> The odd thing about it is you still have to parse json in JavaScript
> to avoid security problems.  Parsing json in js seems to eliminate
> 99% of the useness of it.

I think this is mistaken.  Last time I looked, you would check the
whole string with a simple regexp to look for badness, and that
confirms it's safe to eval the json.  Good json always passes the
regexp.  Not that I've ever done any json :-)

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Ian Jackson wrote:
> I think dbus would be a good choice, but even XML-RPC (which, frankly,
> is flabby, overcomplex, slow, hard to read, etc.), JSON (which doesn't
> have an IDL that can even do checking), and ASN.1/DER/BER (which is a
> complexity nightmare) would be an improvement over no automatic
> parsing machinery.

I strongly dislike D-Bus, having been forced to use it to manage Linux
Bluetooth connections.  (The old way with a nice traditional
unix-style executable command to control them was far nicer and
actually worked).

D-Bus is impossible to use properly from shell with the available
shell client (some value types aren't supported), and you have to read
application source code to figure out what calls to use, if the
applications RPCs aren't well documented (which they aren't for Linux
Bluetooth).  There's no useful introspection.

D-Bus seems to be ok for asynchronous notifications, such as what's
used to update icons on a desktop in response to background events.
But it's not at all transactional so you often have temporary
display inconsistencies in a GUI using it.

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Avi Kivity wrote:
> >With respect to RPC choice, if we did go that route, I'd be very 
> >concerned about using jsonrpc verses a more well established rpc.  I 
> >would honestly prefer xml-rpc over jsonrpc.
> 
> I agree xml-rpc is a more rational choice than jsonrpc, but I cannot 
> find it in my heart to say something nice about xml.  Additionally, XML 
> parsers are pretty heavy.

I'm not defending XML here, but:

You can code a minimal XML parser in straight C quite easily, if it's
a restricted subset.

That is, if you specify the XML won't use any of the irrelevant
features like PIs, DTDs, namespaces and named entities and know it's
all UTF-8.  Maybe even no comments.  Then it's just nested elements,
attributes and strings, with < to escape a few characters.

But the XML-RPC spec does not appear to restrict the type of XML, so
you have to handle every XML feature to be XML-RPC compliant,
unfortunately.

Some other "wire" protocols do specify a reduced subset of XML, which
means you don't have to use a heavyweight library for them, which is
nice.

I think feature-restricted XML is quite good for this sort of thing.
It's a bit more self-documenting than JSON.  But for some reason, lots
of XML users use it very badly, doing stupid things like
<array><element><type><typename>int</typename></type><subscript>0</subscript><element-value>0</element-value></element>...</array>.
That's an exaggeration but I'm sure you all recognise the pattern.

XML and JSON both have the same ugly problem with binary data: they
can't carry it.  It's usually base64 encoded.  Then again the QEMU
monitor is no better this respect :-)

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/24/2009 06:57 PM, Filip Navara wrote:
> Given the fact that we practically have the library for XDR it would
> be very easy to build Sun RPC server on top of it. So my vote is for
> SunRPC.
>    

My experience with SunRPC (NFS) has been pretty bad.  It may have been 
due to implementation details, not to the spec itself, but it definitely 
left a bad taste in my mouth.  Also, I don't see a SunRPC implementation 
for Python, which (along with similar languages) is an ideal platform 
for controlling qemu.  I'd rather have something more modern.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/24/2009 07:22 PM, Jamie Lokier wrote:
> Avi Kivity wrote:
>    
>>> With respect to RPC choice, if we did go that route, I'd be very
>>> concerned about using jsonrpc verses a more well established rpc.  I
>>> would honestly prefer xml-rpc over jsonrpc.
>>>        
>> I agree xml-rpc is a more rational choice than jsonrpc, but I cannot
>> find it in my heart to say something nice about xml.  Additionally, XML
>> parsers are pretty heavy.
>>      
>
> I'm not defending XML here, but:
>
> You can code a minimal XML parser in straight C quite easily,

One of the points of using an existing RPC implementation is not to have 
to code it up.

> if it's a restricted subset.
>    

You can't control what's coming in from the other side.


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Wed, Jun 24, 2009 at 6:22 PM, Avi Kivity<avi@redhat.com> wrote:
> On 06/24/2009 06:57 PM, Filip Navara wrote:
>>
>> Given the fact that we practically have the library for XDR it would
>> be very easy to build Sun RPC server on top of it. So my vote is for
>> SunRPC.
>>
>
> My experience with SunRPC (NFS) has been pretty bad.  It may have been due
> to implementation details, not to the spec itself, but it definitely left a
> bad taste in my mouth.

I have no experience with it, but the specifications looked simple and sane.

> Also, I don't see a SunRPC implementation for
> Python, which (along with similar languages) is an ideal platform for
> controlling qemu.  I'd rather have something more modern.

http://svn.python.org/projects/python/trunk/Demo/rpc/
http://www.cs.umd.edu/~gaburici/rpc/

Libraries are available for vast amount of languages, including Java and .NET.

Best regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Wed, Jun 24, 2009 at 05:22:07PM +0100, Jamie Lokier wrote:
> You can code a minimal XML parser in straight C quite easily, if it's
> a restricted subset.

even the restricted subset is not as straighforward as a json parser. and
usually using a subset means you can't interact correctly with the one that
does the full spec.

> XML and JSON both have the same ugly problem with binary data: they
> can't carry it.  It's usually base64 encoded.  Then again the QEMU
> monitor is no better this respect :-)

JSon ***DOES*** do binary data.

C String "abc\0\xff" -> Json String "abc\0000\00ff"

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Wed, Jun 24, 2009 at 7:39 PM, Vincent Hanquez<tab@snarc.org> wrote:
> On Wed, Jun 24, 2009 at 05:22:07PM +0100, Jamie Lokier wrote:
>> You can code a minimal XML parser in straight C quite easily, if it's
>> a restricted subset.
>
> even the restricted subset is not as straighforward as a json parser. and
> usually using a subset means you can't interact correctly with the one that
> does the full spec.
>
>> XML and JSON both have the same ugly problem with binary data: they
>> can't carry it.  It's usually base64 encoded.  Then again the QEMU
>> monitor is no better this respect :-)
>
> JSon ***DOES*** do binary data.
>
> C String "abc\0\xff" -> Json String "abc\0000\00ff"

I find the Json representation problematic. In C you have two distinct
data types - null-terminated string where the length is implicitly
known from the content (char *string) and a binary data blob (char
*buffer, int size). If you encode them into the same JSON data type
and don't supply "out-of-band" information about which one of the C
types is it, the receiver has no way to decide what to decode it into.
JSONRPC allows supplying this "out-of-band" information only for the
JSON data types which is very limiting.

For text based protocols it's vital to separate the syntax from
semantics and decoding the above would require knowning the specific
context and semantics.

A more natural representation of binary blob in JSON would be array of
numbers, but that would have a big overhead.

Best regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

> On Wed, Jun 24, 2009 at 7:39 PM, Vincent Hanquez<tab@snarc.org> wrote:
> > On Wed, Jun 24, 2009 at 05:22:07PM +0100, Jamie Lokier wrote:
> >> You can code a minimal XML parser in straight C quite easily, if it's
> >> a restricted subset.
> >
> > even the restricted subset is not as straighforward as a json parser. and
> > usually using a subset means you can't interact correctly with the one that
> > does the full spec.
> >
> >> XML and JSON both have the same ugly problem with binary data: they
> >> can't carry it.  It's usually base64 encoded.  Then again the QEMU
> >> monitor is no better this respect :-)
> >
> > JSon ***DOES*** do binary data.
> >
> > C String "abc\0\xff" -> Json String "abc\0000\00ff"

Any reason you can't simply put the UTF-8 encoding of U+0000 and
U+00FF directly in the string?

(Let's ignore Java, which encodes U+0000 in "Java-UTF-8" differently
from everyone else's UTF-8!)

Filip Navara wrote:
> I find the Json representation problematic. In C you have two distinct
> data types - null-terminated string where the length is implicitly
> known from the content (char *string) and a binary data blob (char
> *buffer, int size). If you encode them into the same JSON data type
> and don't supply "out-of-band" information about which one of the C
> types is it, the receiver has no way to decide what to decode it into.
> JSONRPC allows supplying this "out-of-band" information only for the
> JSON data types which is very limiting.
> 
> For text based protocols it's vital to separate the syntax from
> semantics and decoding the above would require knowning the specific
> context and semantics.
> 
> A more natural representation of binary blob in JSON would be array of
> numbers, but that would have a big overhead.

Actually, an array doesn't add much more overhead :-)

Binary has a big overhead in JSON as a string.

Encode the blob [0,1,2,3,252,253,254,255]:

    JSON: "\0\1\2\3\xfc\xfd\xfe\xff"

That's fine if you've just got a few non-ASCII bytes.  But for general
binary, base64 is much more compact.

Base64 expands by about 4/3, whereas a JSON \x-encoded string expands
by up to 4 times.  Even a hex string is more compact :-)

A further complication is that a JSON string carries Unicode (which is
good for text), so at some point you have to know to signal a parse
error when any characters are outside the range 0-255.

But I don't know why we're talking about binary, as the monitor (old
or proposed) doesn't handle binary either, and no particular need for
binary has come up...

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Wed, Jun 24, 2009 at 08:23:06PM +0200, Filip Navara wrote:
> On Wed, Jun 24, 2009 at 7:39 PM, Vincent Hanquez<tab@snarc.org> wrote:
> > On Wed, Jun 24, 2009 at 05:22:07PM +0100, Jamie Lokier wrote:
> >> You can code a minimal XML parser in straight C quite easily, if it's
> >> a restricted subset.
> >
> > even the restricted subset is not as straighforward as a json parser. and
> > usually using a subset means you can't interact correctly with the one that
> > does the full spec.
> >
> >> XML and JSON both have the same ugly problem with binary data: they
> >> can't carry it.  It's usually base64 encoded.  Then again the QEMU
> >> monitor is no better this respect :-)
> >
> > JSon ***DOES*** do binary data.
> >
> > C String "abc\0\xff" -> Json String "abc\0000\00ff"

btw, sorry i meant \\ instead of \ in the json string.
as in: 'a' 'b' 'c' '\\' '0' '0' '0' '0' '\\' '0' '0' 'f' 'f'

> I find the Json representation problematic. In C you have two distinct
> data types - null-terminated string where the length is implicitly
> known from the content (char *string) and a binary data blob (char
> *buffer, int size). If you encode them into the same JSON data type
> and don't supply "out-of-band" information about which one of the C
> types is it, the receiver has no way to decide what to decode it into.
> JSONRPC allows supplying this "out-of-band" information only for the
> JSON data types which is very limiting.

it's a problem related to the representation of your string, not of JSon.

in most case it doesn't matter though, since if your string contains null that you
want to send you need to use the correct "json_append_string" in the first
place which will take the lenght as parameter and on the receiving end you can
expect that every characters received were meant to be sent and need to use the proper
accessor to get the string of the proper lenght.

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Avi Kivity wrote:
> On 06/24/2009 07:22 PM, Jamie Lokier wrote:
> >Avi Kivity wrote:
> >   
> >>>With respect to RPC choice, if we did go that route, I'd be very
> >>>concerned about using jsonrpc verses a more well established rpc.  I
> >>>would honestly prefer xml-rpc over jsonrpc.
> >>>       
> >>I agree xml-rpc is a more rational choice than jsonrpc, but I cannot
> >>find it in my heart to say something nice about xml.  Additionally, XML
> >>parsers are pretty heavy.
> >>     
> >
> >I'm not defending XML here, but:
> >
> >You can code a minimal XML parser in straight C quite easily,
> 
> One of the points of using an existing RPC implementation is not to have 
> to code it up.

In my experience, using some existing RPC implementations tends to
involve quite a lot of code, and implementing from scratch only a
little more, when we're talking about *simple* RPC protocols.  Like
QEMU's monitor for example.  You still reuse code of course, but it's
things like sockets, buffering, select and dispatch loops.

Obviously if it's something larger it's good to start with a good
library.  One which not only exists, but doesn't need a lot of
application boilerplate.  RPC libraries vary a lot in how much
application code they need you to write.

I don't know whether there are good XML-RPC libraries for C.
There doesn't seem to be much for JSON-RPC services.

> >if it's a restricted subset.
> 
> You can't control what's coming in from the other side.

You can reject it if the protocol specification says it's not allowed.

Some protocols do that.  Otherwise you have to process it correctly.
It's trivial to reject byte values that shouldn't be there.

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Wed, Jun 24, 2009 at 8:42 PM, Vincent Hanquez<tab@snarc.org> wrote:
> On Wed, Jun 24, 2009 at 08:23:06PM +0200, Filip Navara wrote:
>> On Wed, Jun 24, 2009 at 7:39 PM, Vincent Hanquez<tab@snarc.org> wrote:
>> > On Wed, Jun 24, 2009 at 05:22:07PM +0100, Jamie Lokier wrote:
>> >> You can code a minimal XML parser in straight C quite easily, if it's
>> >> a restricted subset.
>> >
>> > even the restricted subset is not as straighforward as a json parser. and
>> > usually using a subset means you can't interact correctly with the one that
>> > does the full spec.
>> >
>> >> XML and JSON both have the same ugly problem with binary data: they
>> >> can't carry it.  It's usually base64 encoded.  Then again the QEMU
>> >> monitor is no better this respect :-)
>> >
>> > JSon ***DOES*** do binary data.
>> >
>> > C String "abc\0\xff" -> Json String "abc\0000\00ff"
>
> btw, sorry i meant \\ instead of \ in the json string.
> as in: 'a' 'b' 'c' '\\' '0' '0' '0' '0' '\\' '0' '0' 'f' 'f'
>
>> I find the Json representation problematic. In C you have two distinct
>> data types - null-terminated string where the length is implicitly
>> known from the content (char *string) and a binary data blob (char
>> *buffer, int size). If you encode them into the same JSON data type
>> and don't supply "out-of-band" information about which one of the C
>> types is it, the receiver has no way to decide what to decode it into.
>> JSONRPC allows supplying this "out-of-band" information only for the
>> JSON data types which is very limiting.
>
> it's a problem related to the representation of your string, not of JSon.
>
> in most case it doesn't matter though, since if your string contains null that you
> want to send you need to use the correct "json_append_string" in the first
> place which will take the lenght as parameter and on the receiving end you can
> expect that every characters received were meant to be sent and need to use the proper
> accessor to get the string of the proper lenght.
>

*need to use the proper accessor to get the string of the proper length*

This is precisely what I am talking about when I'm talking about
mixing syntax and semantics. You'd require the parser to know the
semantics based on context. That's bad enough of itself, but
fortunately JSONRPC supports "service descriptions" which allow the
parser to be fed with some generic description of the semantics.
Unfortunately this service description doesn't cover all C data types
and can't even describe the basic integer sizes, which would allow for
syntax and semantic validation of the RPC payload by the parser.

Best regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Vincent Hanquez wrote:
> On Wed, Jun 24, 2009 at 08:23:06PM +0200, Filip Navara wrote:
> > On Wed, Jun 24, 2009 at 7:39 PM, Vincent Hanquez<tab@snarc.org> wrote:
> > > On Wed, Jun 24, 2009 at 05:22:07PM +0100, Jamie Lokier wrote:
> > >> You can code a minimal XML parser in straight C quite easily, if it's
> > >> a restricted subset.
> > >
> > > even the restricted subset is not as straighforward as a json parser. and
> > > usually using a subset means you can't interact correctly with the one that
> > > does the full spec.
> > >
> > >> XML and JSON both have the same ugly problem with binary data: they
> > >> can't carry it.  It's usually base64 encoded.  Then again the QEMU
> > >> monitor is no better this respect :-)
> > >
> > > JSon ***DOES*** do binary data.
> > >
> > > C String "abc\0\xff" -> Json String "abc\0000\00ff"
> 
> btw, sorry i meant \\ instead of \ in the json string.
> as in: 'a' 'b' 'c' '\\' '0' '0' '0' '0' '\\' '0' '0' 'f' 'f'

Wow.  So a JSON string is actually _larger_ than an array of numbers,
for some binary data?

As in:

    "abc\\0000\\00ff"

versus

    [97,98,99,0,255]

Heh :-)

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Chris Webb wrote:
> Anthony Liguori <anthony@codemonkey.ws> writes:
> 
> > There are two questions to resolve.  The first is whether we should  
> > continue with the current direction (line-based protocol) or whether we  
> > should switch to an RPC.  The second question is which RPC we should use.
> >
> > I'm not at all convinced that we should switch to an RPC mechanism in  
> > the first place.  Perhaps someone could summarize the advantages of  
> > doing this because right now, I don't see many.
> >
> > With respect to RPC choice, if we did go that route, I'd be very  
> > concerned about using jsonrpc verses a more well established rpc.  I  
> > would honestly prefer xml-rpc over jsonrpc.
> 
> We are an example of a end user of qemu (or more specifically qemu-kvm) that
> doesn't go via a management layer like libvirt. Our management shell scripts
> directly control the virtual machines using the current line-oriented 'human
> friendly' monitor. This is a bit of a pain, but not too bad in practice.

I do the same.  I even have a QEMU-monitor-multiplexor script, which
accepts multiple monitor connections and issues the commands over the
real monitor.  So that I can have one script managing (basically
insert/remove disk images, trigger reboots etc.), while still having
monitor accesss from the command line for those special extras.

> A more regular and well defined line-based protocol would be a big plus for
> us, whereas something like jsonrpc or xml-rpc would be a complete disaster
> to call from the shell---less useful than the current human oriented monitor
> rather than more so.

It's fine as long as there's a good shell-friendly utility.

Unfortunately my experience of using D-Bus through it's shell utility
was not positive.  You need the utility to do more than just pass the
whole request and response back and forth.  It needs to combine and
extract fields in compound structures, and escape/unescape when you
need that.

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Avi Kivity wrote:
> On 06/24/2009 06:57 PM, Filip Navara wrote:
> >Given the fact that we practically have the library for XDR it would
> >be very easy to build Sun RPC server on top of it. So my vote is for
> >SunRPC.
> >   
> 
> My experience with SunRPC (NFS) has been pretty bad.  It may have been 
> due to implementation details, not to the spec itself, but it definitely 
> left a bad taste in my mouth.  Also, I don't see a SunRPC implementation 
> for Python, which (along with similar languages) is an ideal platform 
> for controlling qemu.  I'd rather have something more modern.

SunRPC works, and it's fine if you always access it through utilities,
like say a C command-line client.  Hiding the SunRPC aspect :-)

It's pretty fast too, so good for things like network filesystems.

But it's a real pain to use from scripts if you don't have a pre-built
utility which implements all application RPC functions already.

The basic problem is needing to create and compile C code - for all
clients as well as the server - each time an RPC function is added or
changed.

QEMU's monitor is already much easier to use than that.

When a new monitor command is added, you can use it from scripts as
trivial one-liner text commands, assuming you already have a way to
connect to the monitor.

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Wed, Jun 24, 2009 at 9:05 PM, Jamie Lokier<jamie@shareable.org> wrote:
> Avi Kivity wrote:
>> On 06/24/2009 06:57 PM, Filip Navara wrote:
>> >Given the fact that we practically have the library for XDR it would
>> >be very easy to build Sun RPC server on top of it. So my vote is for
>> >SunRPC.
>> >
>>
>> My experience with SunRPC (NFS) has been pretty bad.  It may have been
>> due to implementation details, not to the spec itself, but it definitely
>> left a bad taste in my mouth.  Also, I don't see a SunRPC implementation
>> for Python, which (along with similar languages) is an ideal platform
>> for controlling qemu.  I'd rather have something more modern.
>
> SunRPC works, and it's fine if you always access it through utilities,
> like say a C command-line client.  Hiding the SunRPC aspect :-)
>
> It's pretty fast too, so good for things like network filesystems.
>
> But it's a real pain to use from scripts if you don't have a pre-built
> utility which implements all application RPC functions already.

True. Text-based RPCs as JSON-RPC and XML-RPC are half as painful, but
still painful.

> The basic problem is needing to create and compile C code - for all
> clients as well as the server - each time an RPC function is added or
> changed.

You don't need rebuild the client if the server interface stays
compatible and if you don't need the new functionality. Also my
initial idea was to use QEMU functions for building the XDR binary
payloads inside QEMU. Not sure if it's good idea, but it would work
and it's easy to implement for the proposed monitor_print_data
function.

> QEMU's monitor is already much easier to use than that.

For human? Definitely! For shell scripts? Yes. For libraries that need
to parse the data reliably? No.

> When a new monitor command is added, you can use it from scripts as
> trivial one-liner text commands, assuming you already have a way to
> connect to the monitor.

Since when was calling from scripts added as requirement? Don't get me
wrong, I would be happy to cover this use case as well. It's a diverse
use case from the libvirt usage though, in my humble opinion. I'd
prefer killing one bird to knocking two of them and leaving them
half-dead.

Best regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Filip Navara wrote:
> > QEMU's monitor is already much easier to use than that.
> 
> For human? Definitely! For shell scripts? Yes. For libraries that need
> to parse the data reliably? No.

Yes, it could do with a more well-defined formatting.

Documented escaping of unusual characters in names would be a start :-)

> > When a new monitor command is added, you can use it from scripts as
> > trivial one-liner text commands, assuming you already have a way to
> > connect to the monitor.
> 
> Since when was calling from scripts added as requirement? Don't get me
> wrong, I would be happy to cover this use case as well. It's a diverse
> use case from the libvirt usage though, in my humble opinion.

Aren't most QEMU management programs (other than libvirt) scripts?

Looks to me like "works with libvirt and other management programs"
implies that you can use it from scripts, because many management
programs are, in fact, scripts.

If the requirement is simply "works with libvirt", then closer
integration with libvirt would be better than trying to cleanly RPC
everything.  After all, libvirt already has it's own RPC layer for
everything connecting to it, and libvirt people want all QEMU
interaction to happen via libvirt anyway.  There's no point having
multiple different RPC layers carrying exactly the same commands in
different ways.

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Wed, Jun 24, 2009 at 11:13 PM, Jamie Lokier<jamie@shareable.org> wrote:
> Filip Navara wrote:
>> > QEMU's monitor is already much easier to use than that.
>>
>> For human? Definitely! For shell scripts? Yes. For libraries that need
>> to parse the data reliably? No.
>
> Yes, it could do with a more well-defined formatting.
>
> Documented escaping of unusual characters in names would be a start :-)
>
>> > When a new monitor command is added, you can use it from scripts as
>> > trivial one-liner text commands, assuming you already have a way to
>> > connect to the monitor.
>>
>> Since when was calling from scripts added as requirement? Don't get me
>> wrong, I would be happy to cover this use case as well. It's a diverse
>> use case from the libvirt usage though, in my humble opinion.
>
> Aren't most QEMU management programs (other than libvirt) scripts?

I don't feel competent to answer that.

> Looks to me like "works with libvirt and other management programs"
> implies that you can use it from scripts, because many management
> programs are, in fact, scripts.

If that was true then certainly a text/line-based protocol would make
sense (possibly in addition to a simple RPC one, if desired). The
thing I am worried about is that several corner cases are currently
not defined in the proposed protocol and it introduces asynchronous
events, which is actually harder to get right in shell scripts than
the human protocol.

> If the requirement is simply "works with libvirt", then closer
> integration with libvirt would be better than trying to cleanly RPC
> everything.  After all, libvirt already has it's own RPC layer for
> everything connecting to it, and libvirt people want all QEMU
> interaction to happen via libvirt anyway.  There's no point having
> multiple different RPC layers carrying exactly the same commands in
> different ways.
>
> -- Jamie
>

Best regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Jamie Lokier]

Filip Navara wrote:
> > Looks to me like "works with libvirt and other management programs"
> > implies that you can use it from scripts, because many management
> > programs are, in fact, scripts.
> 
> If that was true then certainly a text/line-based protocol would make
> sense (possibly in addition to a simple RPC one, if desired). The
> thing I am worried about is that several corner cases are currently
> not defined in the proposed protocol and it introduces asynchronous
> events, which is actually harder to get right in shell scripts than
> the human protocol.

I generally agree, and to be honest the most useful API for scripts is
probably a "qemu-control [OPTIONS] CMD [ARGS...]" _program_ which they
can run, along the same lines as apachectl for Apache, smbcontrol for
Samba, ntpdc for NTPd, etc.

Such a program can be third party or not, once the monitor behaviour
is reliable for a program to use.

-- Jamie

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Dor Laor]

On 06/23/2009 09:38 PM, Anthony Liguori wrote:
>
> I wanted to respond to the top level, but here's what I see as the 
> potential merge plan:
>
> 1) Make all commands return a value
> 2) Add a mechanism to mark commands as being "structured"

This seems like a good idea.
My original intent was to make the monitor interface a library.
Various qemu users can link it with their favorite option: 
json/dbus/rpm/xml/soap ;)

So, we should first harden up the monitor commands like 1),2), and 
refactor the current
monitor interface as a library above it.

A second or parallel phase is to implement a machine readable interface 
library.

> 3) Get the structured printfs in order.
> 4) Merge
>
> I won't want QMP to be exposed as a usable interface until all 
> commands are converted.  This means holding off on the last patch I 
> think.  I don't think we'll get QMP for 0.11.  I think it's likely 
> going to be a 0.12 feature.  However, I'd like to start merging this 
> series as soon as humanly possible.  I think it's already pretty close.
>
If we push 'monitor standardization' into 0.11, the version will support 
future
implementations of the protocol.

Regards,
Dor

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

Jamie Lokier wrote:

> Aren't most QEMU management programs (other than libvirt) scripts?


Probably because we didn't provide an rpc interface before. Chichen and
egg problem here...

 
> Looks to me like "works with libvirt and other management programs"
> implies that you can use it from scripts, because many management
> programs are, in fact, scripts.

Having the possibility of using it from scripts would be nice but I
wouldn't make it a requirement.
With an rpc interface and the right scripting language you can still use
the rpc interface from a very very  short script.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/25/2009 04:07 PM, Stefano Stabellini wrote:
>> Looks to me like "works with libvirt and other management programs"
>> implies that you can use it from scripts, because many management
>> programs are, in fact, scripts.
>>      
>
> Having the possibility of using it from scripts would be nice but I
> wouldn't make it a requirement.
> With an rpc interface and the right scripting language you can still use
> the rpc interface from a very very  short script.
>    

I'd go further and make shell script support a non-goal.  Maybe the 
majority of qemu control programs are scripts, but only a tiny minority 
of VMs are controlled by shell scripts.  It is nearly impossible to 
write a correct nontrivial program in bash; and scripts will be slow due 
to the need to constantly fork/exec/connect.

Serious management programs will be written in real languages, and 
that's what we should optimize for.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> On 06/25/2009 04:07 PM, Stefano Stabellini wrote:
>>> Looks to me like "works with libvirt and other management programs"
>>> implies that you can use it from scripts, because many management
>>> programs are, in fact, scripts.
>>>      
>>
>> Having the possibility of using it from scripts would be nice but I
>> wouldn't make it a requirement.
>> With an rpc interface and the right scripting language you can still use
>> the rpc interface from a very very  short script.
>>    
>
> I'd go further and make shell script support a non-goal.  Maybe the 
> majority of qemu control programs are scripts, but only a tiny 
> minority of VMs are controlled by shell scripts.  It is nearly 
> impossible to write a correct nontrivial program in bash; and scripts 
> will be slow due to the need to constantly fork/exec/connect.
>
> Serious management programs will be written in real languages, and 
> that's what we should optimize for.

I'm still inclined to stick with the text-based protocol.  libvirt likes 
it.  If Serious Management Programs want an RPC interface they can use 
libvirt.

An RPC is overkill unless you also introduce things like life cycle 
management.  Otherwise, you could never realistically interact with QEMU 
without a intermediary that handled life cycle management (e.g. libvirt).

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/25/2009 06:10 PM, Anthony Liguori wrote:
>> I'd go further and make shell script support a non-goal.  Maybe the 
>> majority of qemu control programs are scripts, but only a tiny 
>> minority of VMs are controlled by shell scripts.  It is nearly 
>> impossible to write a correct nontrivial program in bash; and scripts 
>> will be slow due to the need to constantly fork/exec/connect.
>>
>> Serious management programs will be written in real languages, and 
>> that's what we should optimize for.
>
> I'm still inclined to stick with the text-based protocol.  libvirt 
> likes it.  If Serious Management Programs want an RPC interface they 
> can use libvirt.

That's really an onerous requirement.  What if libvirt lags behind 
qemu?  What if libvirt doesn't want to support some feature?

> An RPC is overkill unless you also introduce things like life cycle 
> management.  Otherwise, you could never realistically interact with 
> QEMU without a intermediary that handled life cycle management (e.g. 
> libvirt).

QMP is an RPC implemented over a text based protocol, with ad-hoc 
marshalling.

I agree that you need something to reap dead qemus, but I don't think we 
can force libvirt on people.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

Avi Kivity wrote:

> On 06/25/2009 06:10 PM, Anthony Liguori wrote:
>>> I'd go further and make shell script support a non-goal.  Maybe the 
>>> majority of qemu control programs are scripts, but only a tiny 
>>> minority of VMs are controlled by shell scripts.  It is nearly 
>>> impossible to write a correct nontrivial program in bash; and scripts 
>>> will be slow due to the need to constantly fork/exec/connect.
>>>
>>> Serious management programs will be written in real languages, and 
>>> that's what we should optimize for.
>> I'm still inclined to stick with the text-based protocol.  libvirt 
>> likes it.  If Serious Management Programs want an RPC interface they 
>> can use libvirt.
> 
> That's really an onerous requirement.  What if libvirt lags behind 
> qemu?  What if libvirt doesn't want to support some feature?
> 
>> An RPC is overkill unless you also introduce things like life cycle 
>> management.  Otherwise, you could never realistically interact with 
>> QEMU without a intermediary that handled life cycle management (e.g. 
>> libvirt).
> 
> QMP is an RPC implemented over a text based protocol, with ad-hoc 
> marshalling.
> 
> I agree that you need something to reap dead qemus, but I don't think we 
> can force libvirt on people.
> 


Clearly I agree with Avi.
I am thinking for example that we could use the RPC protocol directly
from Xend and I am sure other people will find it useful too.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
>> An RPC is overkill unless you also introduce things like life cycle 
>> management.  Otherwise, you could never realistically interact with 
>> QEMU without a intermediary that handled life cycle management (e.g. 
>> libvirt).
>
> QMP is an RPC implemented over a text based protocol, with ad-hoc 
> marshalling.
>
> I agree that you need something to reap dead qemus, but I don't think 
> we can force libvirt on people.

QMP is an incremental improvement to the current monitor which people 
are consuming today.  It will have a pretty immediate and high rate of 
return.  libvirt can make use of it with probably very little change and 
it will immediately provide them with new features (async notifications) 
and better information about commands (error return values).

I have a really hard time not thinking this is the right thing to do 
because for relatively little churn we'll get a large return.  If you 
just exposed the monitor as an RPC, I don't think you really get 
anything more useful than QMP and you force a lot more change in libvirt 
which hurt the adoption of this protocol.

This is exactly what happened with Xend's transition from SExpr/HTTP 
(custom RPC) -> XML-RPC -> XenAPI.  libvirt still uses SExpr/HTTP 
because the cost of switching to the new way of doing things didn't 
warrant the return on investment.

s/libvirt/any other management tool that works today/g

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Stefano Stabellini wrote:
>
> Clearly I agree with Avi.
> I am thinking for example that we could use the RPC protocol directly
> from Xend and I am sure other people will find it useful too.
>   

But you can also use QMP from Xend.  In fact, it should be pretty easy 
to convert the current code in Xend to use QMP.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/25/2009 09:09 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>>> An RPC is overkill unless you also introduce things like life cycle 
>>> management.  Otherwise, you could never realistically interact with 
>>> QEMU without a intermediary that handled life cycle management (e.g. 
>>> libvirt).
>>
>> QMP is an RPC implemented over a text based protocol, with ad-hoc 
>> marshalling.
>>
>> I agree that you need something to reap dead qemus, but I don't think 
>> we can force libvirt on people.
>
> QMP is an incremental improvement to the current monitor which people 
> are consuming today.  It will have a pretty immediate and high rate of 
> return.  libvirt can make use of it with probably very little change 
> and it will immediately provide them with new features (async 
> notifications) and better information about commands (error return 
> values).
>

The incremental cost of porting a monitor command, documenting its 
format, improving QMP infrastructure where necessary, and implementing 
that command across all clients is much greater than RPC.  For a new 
client implemented in a high level language, using QMP will be 
substantially more difficult than using RPC, and the probability of 
getting things right will be much greater.

Targeting libvirt alone is a mistake.  Of course libvirt is a very 
important user of the qemu monitor, but not doing the right thing 
because libvirt has some prior art is shortsighted IMO.  RPC offers the 
best choice for growth, in adding new commands, and in supporting more 
clients.

> I have a really hard time not thinking this is the right thing to do 
> because for relatively little churn we'll get a large return.  If you 
> just exposed the monitor as an RPC, I don't think you really get 
> anything more useful than QMP and you force a lot more change in 
> libvirt which hurt the adoption of this protocol.

Adopting an RPC should be easier than adopting QMP, since all you have 
to do is compile the IDL.  Again I point to the incremental cost of 
adding a command.

> This is exactly what happened with Xend's transition from SExpr/HTTP 
> (custom RPC) -> XML-RPC -> XenAPI.  libvirt still uses SExpr/HTTP 
> because the cost of switching to the new way of doing things didn't 
> warrant the return on investment.

I think this points to libvirt doing something wrong.  I suspect it's 
the implementation language.

> s/libvirt/any other management tool that works today/g

Anything written in a high level language (say, Python) will be trivial 
to port to an RPC.  RPC clients in Python can be typed in with one 
finger in a couple of minutes.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Thu, Jun 25, 2009 at 01:10:36PM -0500, Anthony Liguori wrote:
> Stefano Stabellini wrote:
> >
> >Clearly I agree with Avi.
> >I am thinking for example that we could use the RPC protocol directly
> >from Xend and I am sure other people will find it useful too.
> >  
> 
> But you can also use QMP from Xend.  In fact, it should be pretty easy 
> to convert the current code in Xend to use QMP.

AFAIK, the current XenD code doesn't talk to the QEMU monitor at all,
instead having a add-on to QEMU code which pulls info from XenStored.
That doesn't alter your point though, it would be pretty easy to make
XenD talk to the proposed QMP monitor if desired.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Luiz Capitulino]

On Thu, 25 Jun 2009 14:27:05 +0300
Dor Laor <dlaor@redhat.com> wrote:

> On 06/23/2009 09:38 PM, Anthony Liguori wrote:
> >
> > I wanted to respond to the top level, but here's what I see as the 
> > potential merge plan:
> >
> > 1) Make all commands return a value
> > 2) Add a mechanism to mark commands as being "structured"
> 
> This seems like a good idea.
> My original intent was to make the monitor interface a library.
> Various qemu users can link it with their favorite option: 
> json/dbus/rpm/xml/soap ;)

 Yes, having a library was suggested by Amit some months ago. The
problem is that it has various issues wrt maintainability.

 For example, libvirt is able to run two instances of different
versions of qemu at the same time. How to handle this if you
update libmonitor.so?

 I think people have agreed that having a protocol is good here, but
as Anthonhy puts it we need:

1. Choose between QMP and some kind of RPC
2. If RPC is choosen, what kind of RPC to use

> So, we should first harden up the monitor commands like 1),2), and 
> refactor the current
> monitor interface as a library above it.

 If I'm not mistaken, "structured" commands only makes sense with
QMP. The plan will be different if we choose RPC.

> > 3) Get the structured printfs in order.
> > 4) Merge
> >
> > I won't want QMP to be exposed as a usable interface until all 
> > commands are converted.  This means holding off on the last patch I 
> > think.  I don't think we'll get QMP for 0.11.  I think it's likely 
> > going to be a 0.12 feature.  However, I'd like to start merging this 
> > series as soon as humanly possible.  I think it's already pretty close.
> >
> If we push 'monitor standardization' into 0.11, the version will support 
> future
> implementations of the protocol.
> 
> Regards,
> Dor

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

On Wed, 24 Jun 2009 16:03:13 +0200
Filip Navara <filip.navara@gmail.com> wrote:

> The current draft specification used a mix of semantics of the
> forementioned protocols - error handling based on POP3 (+OK/-ERR),
> asynchronous events based on IMAP (* prefix). IMAP-like tagging of
> commands was also suggested (to deal with asynchronous commands) and
> issues have been raised concerning multi-line responses. Note that all
> this gets a bit hairy when you combine it. Just a few issues that come
> to mind:

 Although I think QMP is very close to what QEMU needs, there is no
doubt it's not complete.

 I appreciate your feedback, but I will only go through it if we
decide using QMP.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> Adopting an RPC should be easier than adopting QMP, since all you have 
> to do is compile the IDL.  Again I point to the incremental cost of 
> adding a command.

Having written clients for XML-RPC, SExpr/HTTP, and the QEMU monitor, in 
both C and in Python, along with having spent way too much time with 
various IDL compilers, I really don't agree with you here.

It is relatively easy to parse a line based protocol.  Certainly easier 
than most poorly designed RPCs.  For RPCs that are not self-described, 
you need an IDL compiler to make any sense out of the packets.  IDL 
compilers are a huge problem because you need to build the compiler and 
run it on the same platform unless you choose from the handful of common 
compilers.  Those compilers/RPCs tend to produce absolutely terrible C 
code so you need to do all sorts of tricks to avoid excessive warnings.

For self-describing RPCs (like XML-RPC), you can avoid IDL which is 
good.  You still end up with pretty funky C code though and the type 
system almost always comes back to haunt you.

>> This is exactly what happened with Xend's transition from SExpr/HTTP 
>> (custom RPC) -> XML-RPC -> XenAPI.  libvirt still uses SExpr/HTTP 
>> because the cost of switching to the new way of doing things didn't 
>> warrant the return on investment.
>
> I think this points to libvirt doing something wrong.  I suspect it's 
> the implementation language.
>
>> s/libvirt/any other management tool that works today/g
>
> Anything written in a high level language (say, Python) will be 
> trivial to port to an RPC.  RPC clients in Python can be typed in with 
> one finger in a couple of minutes.

Reality is less rosy.  Take a look at Xend's XML-RPC server/client.  
Python doesn't provide anything but the most dumb transports.  
Implementing custom transports requires poking internal bits of the 
class library.  It gets ugly really quickly.

Also, typing really hurts with dynamic languages because the type 
semantics never match up well which causes unexpected failures.  A 
classic example with XML-RPC is None.  There is no standard way to 
transport None in XML-RPC although there are common extensions to 
support it.  Of course, not all clients support it and there isn't 
really a standard way to negotiate it.  It's really a mess.

At this point, we have a close-to-mergable set of patches that provide 
real value.  I don't see a reason not to merge them once they've been 
resubmitted.

I don't believe an RPC is going to be a dramatic simplification.  It 
could be if there were a high quality RPC out there that had well 
maintained client/server libraries.  I don't believe such a thing exists 
though.  If you want to keep arguing this, let's focus on actual RPC 
implementations.  I think we've agreed that JSON-RPC isn't going to work 
because of the lack of client/server library availability.  XML-RPC 
falls into the same boat for C as far as I'm concerned.  If there are 
any other ones you want to consider, let's look at them so that we can 
make progress in this discussion.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Anthony Liguori]

Dor Laor wrote:
>> 3) Get the structured printfs in order.
>> 4) Merge
>>
>> I won't want QMP to be exposed as a usable interface until all 
>> commands are converted.  This means holding off on the last patch I 
>> think.  I don't think we'll get QMP for 0.11.  I think it's likely 
>> going to be a 0.12 feature.  However, I'd like to start merging this 
>> series as soon as humanly possible.  I think it's already pretty close.
>>
> If we push 'monitor standardization' into 0.11, the version will 
> support future
> implementations of the protocol.

We feature freeze in 2 weeks.  There's no way we can have it ready by then.

Regards,

Anthony Liguori

> Regards,
> Dor

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/25/2009 10:03 PM, Daniel P. Berrange wrote:
> On Thu, Jun 25, 2009 at 01:10:36PM -0500, Anthony Liguori wrote:
>    
>> Stefano Stabellini wrote:
>>      
>>> Clearly I agree with Avi.
>>> I am thinking for example that we could use the RPC protocol directly
>>>        
>> >from Xend and I am sure other people will find it useful too.
>>      
>>>
>>>        
>> But you can also use QMP from Xend.  In fact, it should be pretty easy
>> to convert the current code in Xend to use QMP.
>>      
>
> AFAIK, the current XenD code doesn't talk to the QEMU monitor at all,
> instead having a add-on to QEMU code which pulls info from XenStored.
> That doesn't alter your point though, it would be pretty easy to make
> XenD talk to the proposed QMP monitor if desired.
>    

All it takes is implementing QMP, and an emitter/parser for each qemu 
command.  On the other hand, things like xml-rpc require around one line 
of code in Python per command.

Let's turn this around.  IIRC libvirt uses an RPC interface for its 
clients.  How would you estimate the effort to port this interface to 
QMP, and adapt all its clients?

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Fri, Jun 26, 2009 at 12:00:54PM +0300, Avi Kivity wrote:
> On 06/25/2009 10:03 PM, Daniel P. Berrange wrote:
> >On Thu, Jun 25, 2009 at 01:10:36PM -0500, Anthony Liguori wrote:
> >   
> >>Stefano Stabellini wrote:
> >>     
> >>>Clearly I agree with Avi.
> >>>I am thinking for example that we could use the RPC protocol directly
> >>>       
> >>>from Xend and I am sure other people will find it useful too.
> >>     
> >>>
> >>>       
> >>But you can also use QMP from Xend.  In fact, it should be pretty easy
> >>to convert the current code in Xend to use QMP.
> >>     
> >
> >AFAIK, the current XenD code doesn't talk to the QEMU monitor at all,
> >instead having a add-on to QEMU code which pulls info from XenStored.
> >That doesn't alter your point though, it would be pretty easy to make
> >XenD talk to the proposed QMP monitor if desired.
> >   
> 
> All it takes is implementing QMP, and an emitter/parser for each qemu 
> command.  On the other hand, things like xml-rpc require around one line 
> of code in Python per command.
> 
> Let's turn this around.  IIRC libvirt uses an RPC interface for its 
> clients.  How would you estimate the effort to port this interface to 
> QMP, and adapt all its clients?

The libvirt RPC interface is a binary message based protocol, using XDR
to encode arguments/return values, and emit signals. The libvirt for
handling this is quite complex because it has to deal with TLS, and 
SASL for data encryption, as well as alllowing overlapping RPC requests
on a single TCP connection ( to allow multi-threaded clients to have
parallelized method calls without opening multiple connections). So
it would be a fairly significant amount of work, in comparison to the
current QMP proposal.

The existing libvirt code to talk to QEMU's monitor is actually rather
simple by comparison to our native RPC code. The problems we have 
historically had with the QEMU monitor were, ill defined data printed
when an error occurs,  the changing of monitor commands between QEMU
releases in incompatible ways, the lack of quoting for arguments and
no support for async events. 

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/25/2009 10:54 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>> Adopting an RPC should be easier than adopting QMP, since all you 
>> have to do is compile the IDL.  Again I point to the incremental cost 
>> of adding a command.
>
> Having written clients for XML-RPC, SExpr/HTTP, and the QEMU monitor, 
> in both C and in Python, along with having spent way too much time 
> with various IDL compilers, I really don't agree with you here.

IDL does suck, but dynamic languages generally don't need it.

> It is relatively easy to parse a line based protocol.  Certainly 
> easier than most poorly designed RPCs. 

The idea is that the parsers have already been written.  And I doubt 
that our RPC masquerading as a line based protocol wouldn't earn the 
"poorly designed" epithet after a few years of cruft have accumulated.

In Python, adding a new RPC procedure call amounts to typing in the 
procedure name and its arguments.  It really cannot be any simpler than 
that.  On the other hand QMP calls for a hand written emitter and parser 
for each command.

> For RPCs that are not self-described, you need an IDL compiler to make 
> any sense out of the packets.  IDL compilers are a huge problem 
> because you need to build the compiler and run it on the same platform 
> unless you choose from the handful of common compilers.  Those 
> compilers/RPCs tend to produce absolutely terrible C code so you need 
> to do all sorts of tricks to avoid excessive warnings.

Writing a lot of code is worse than crap code someone else writes.

> For self-describing RPCs (like XML-RPC), you can avoid IDL which is 
> good.  You still end up with pretty funky C code though and the type 
> system almost always comes back to haunt you.

Don't read that C code.

I doubt we'll have funky type problems.

>> Anything written in a high level language (say, Python) will be 
>> trivial to port to an RPC.  RPC clients in Python can be typed in 
>> with one finger in a couple of minutes.
>
> Reality is less rosy.  Take a look at Xend's XML-RPC server/client.  
> Python doesn't provide anything but the most dumb transports.  
> Implementing custom transports requires poking internal bits of the 
> class library.  It gets ugly really quickly.
>

I've seen this, and while it fell below by expectations, it's something 
done once rather than per procedure.

> Also, typing really hurts with dynamic languages because the type 
> semantics never match up well which causes unexpected failures.  A 
> classic example with XML-RPC is None.  There is no standard way to 
> transport None in XML-RPC although there are common extensions to 
> support it.  Of course, not all clients support it and there isn't 
> really a standard way to negotiate it.  It's really a mess.

With QMP we don't have a standard way to transport anything (the None 
problem can be avoided by not allowing it at all).

> At this point, we have a close-to-mergable set of patches that provide 
> real value.  I don't see a reason not to merge them once they've been 
> resubmitted.

Merging is a meaningless milestone since the protocol will not be 
enabled.  Since we'll miss 0.11 in reality we have 7-8 months before the 
protocol can be used in production.  IMO you're optimizing for the short 
term, disregarding long-term maintenance, and disregarding ease of 
implementation for users of the qemu monitor.

> I don't believe an RPC is going to be a dramatic simplification.  It 
> could be if there were a high quality RPC out there that had well 
> maintained client/server libraries.  I don't believe such a thing 
> exists though.  If you want to keep arguing this, let's focus on 
> actual RPC implementations.  I think we've agreed that JSON-RPC isn't 
> going to work because of the lack of client/server library 
> availability.  XML-RPC falls into the same boat for C as far as I'm 
> concerned.  If there are any other ones you want to consider, let's 
> look at them so that we can make progress in this discussion.

We are not in agreement on jsonrpc (we could easily use a json library 
and implement the rpc ourselves).  Even though I dislike xml-rpc, I'd 
prefer it to a custom protocol.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/26/2009 12:10 PM, Daniel P. Berrange wrote:
>
>> Let's turn this around.  IIRC libvirt uses an RPC interface for its
>> clients.  How would you estimate the effort to port this interface to
>> QMP, and adapt all its clients?
>>      
>
> The libvirt RPC interface is a binary message based protocol, using XDR
> to encode arguments/return values, and emit signals. The libvirt for
> handling this is quite complex because it has to deal with TLS, and
> SASL for data encryption, as well as alllowing overlapping RPC requests
> on a single TCP connection ( to allow multi-threaded clients to have
> parallelized method calls without opening multiple connections). So
> it would be a fairly significant amount of work, in comparison to the
> current QMP proposal.
>    

You're describing an RPC with asynchronous commands and notifications, 
with support for authentication and authorization.  While qemu doesn't 
need auth*, it does need the other features, so RPC looks like a good fit.

> The existing libvirt code to talk to QEMU's monitor is actually rather
> simple by comparison to our native RPC code. The problems we have
> historically had with the QEMU monitor were, ill defined data printed
> when an error occurs,  the changing of monitor commands between QEMU
> releases in incompatible ways, the lack of quoting for arguments and
> no support for async events.
>    

No doubt RPC is complicated if you implement it yourself, but surely you 
used a library?  Actually using the RPC is simpler than a line protocol.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Avi Kivity]

On 06/25/2009 10:11 PM, Luiz Capitulino wrote:
>
>   Yes, having a library was suggested by Amit some months ago. The
> problem is that it has various issues wrt maintainability.
>
>   For example, libvirt is able to run two instances of different
> versions of qemu at the same time. How to handle this if you
> update libmonitor.so?
>    

You could use dlopen().  The problem with a library is that it assumes 
everyone is still trapped in C, which is not the case.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Daniel P. Berrange]

On Fri, Jun 26, 2009 at 12:21:01PM +0300, Avi Kivity wrote:
> On 06/25/2009 10:11 PM, Luiz Capitulino wrote:
> >
> >  Yes, having a library was suggested by Amit some months ago. The
> >problem is that it has various issues wrt maintainability.
> >
> >  For example, libvirt is able to run two instances of different
> >versions of qemu at the same time. How to handle this if you
> >update libmonitor.so?
> >   

The sane way is to *NOT* break ABI of libmonitor.so, and not change
the wire protocol in a non backwards compatible way. This is entirely
doable, it just the maintainer of libmonitor/qemu to decide that  ABI
stability is important. So if you find an existing API  / command
needs to gain an extra argument, you don't change the existing API,
you add a new one. Or ideally design the API upfront so that it 
can be extended without breaking back compatability.

> You could use dlopen().  The problem with a library is that it assumes 
> everyone is still trapped in C, which is not the case.

dlopen() of a different libmonitor.so for every version of QEMU is just
crazy, whether C or not. Maintain ABI & protocol back compatability 
with new releases and avoid the issue in the first place.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Daniel P. Berrange]

On Fri, Jun 26, 2009 at 12:16:38PM +0300, Avi Kivity wrote:
> On 06/26/2009 12:10 PM, Daniel P. Berrange wrote:
> >
> >>Let's turn this around.  IIRC libvirt uses an RPC interface for its
> >>clients.  How would you estimate the effort to port this interface to
> >>QMP, and adapt all its clients?
> >>     
> >
> >The libvirt RPC interface is a binary message based protocol, using XDR
> >to encode arguments/return values, and emit signals. The libvirt for
> >handling this is quite complex because it has to deal with TLS, and
> >SASL for data encryption, as well as alllowing overlapping RPC requests
> >on a single TCP connection ( to allow multi-threaded clients to have
> >parallelized method calls without opening multiple connections). So
> >it would be a fairly significant amount of work, in comparison to the
> >current QMP proposal.
> >   
> 
> You're describing an RPC with asynchronous commands and notifications, 
> with support for authentication and authorization.  While qemu doesn't 
> need auth*, it does need the other features, so RPC looks like a good fit.
> 
> >The existing libvirt code to talk to QEMU's monitor is actually rather
> >simple by comparison to our native RPC code. The problems we have
> >historically had with the QEMU monitor were, ill defined data printed
> >when an error occurs,  the changing of monitor commands between QEMU
> >releases in incompatible ways, the lack of quoting for arguments and
> >no support for async events.
> >   
> 
> No doubt RPC is complicated if you implement it yourself, but surely you 
> used a library?  Actually using the RPC is simpler than a line protocol.

We only used a library for the data serialization/de-serialization. It
would have been nice to use libdbus.so in peer-to-peer (non-bus) mode,
but sadly it still lacks any useful form of encryption, or authetnication
for TCP, and the windows port is still an incomplete hacked up fork of
the code.


Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Avi Kivity]

On 06/26/2009 12:42 PM, Daniel P. Berrange wrote:
> On Fri, Jun 26, 2009 at 12:21:01PM +0300, Avi Kivity wrote:
>    
>> On 06/25/2009 10:11 PM, Luiz Capitulino wrote:
>>      
>>>   Yes, having a library was suggested by Amit some months ago. The
>>> problem is that it has various issues wrt maintainability.
>>>
>>>   For example, libvirt is able to run two instances of different
>>> versions of qemu at the same time. How to handle this if you
>>> update libmonitor.so?
>>>
>>>        
>
> The sane way is to *NOT* break ABI of libmonitor.so, and not change
> the wire protocol in a non backwards compatible way. This is entirely
> doable, it just the maintainer of libmonitor/qemu to decide that  ABI
> stability is important. So if you find an existing API  / command
> needs to gain an extra argument, you don't change the existing API,
> you add a new one. Or ideally design the API upfront so that it
> can be extended without breaking back compatability.
>
>    

Yeah.

>> You could use dlopen().  The problem with a library is that it assumes
>> everyone is still trapped in C, which is not the case.
>>      
>
> dlopen() of a different libmonitor.so for every version of QEMU is just
> crazy, whether C or not. Maintain ABI&  protocol back compatability
> with new releases and avoid the issue in the first place.
>    

Right, I just like dlopen().

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Thu, Jun 25, 2009 at 02:54:40PM -0500, Anthony Liguori wrote:
> Avi Kivity wrote:
>> Adopting an RPC should be easier than adopting QMP, since all you have  
>> to do is compile the IDL.  Again I point to the incremental cost of  
>> adding a command.
>
> Having written clients for XML-RPC, SExpr/HTTP, and the QEMU monitor, in  
> both C and in Python, along with having spent way too much time with  
> various IDL compilers, I really don't agree with you here.

did you write any ad-hoc text base protocol then ?

> It is relatively easy to parse a line based protocol.

it's not that hard indeed, but now what's the cost of having to replicate this
ad-hoc protocol in the management layer that use different languages ?
what about the cost of extensibility in the future (hello IMAP) ?

> Certainly easier  
> than most poorly designed RPCs.  For RPCs that are not self-described,  
> you need an IDL compiler to make any sense out of the packets.  IDL  
> compilers are a huge problem because you need to build the compiler and  
> run it on the same platform unless you choose from the handful of common  
> compilers.  Those compilers/RPCs tend to produce absolutely terrible C  
> code so you need to do all sorts of tricks to avoid excessive warnings.

> For self-describing RPCs (like XML-RPC), you can avoid IDL which is  
> good.  You still end up with pretty funky C code though and the type  
> system almost always comes back to haunt you.

you can avoid IDL in all case, if you choose. it requires more boilerplate
in C obviously, but looking at QMP .. it's adding exactly the same boilerplate.

> Reality is less rosy.  Take a look at Xend's XML-RPC server/client.   
> Python doesn't provide anything but the most dumb transports.   
> Implementing custom transports requires poking internal bits of the  
> class library.  It gets ugly really quickly.

but it's not the fault of xmlrpc, it's the implementation fault.
there's other language out there that got a perfectly fine xmlrpc library,
that doesn't assume anything about the transport.

worst come to worst, reimplementing XMLRPC is *easy* as long as the marshaling
protocol library (XML) is available in a good quality, which is certainly the
case in python.

The same hold true of JSonRPC, absolutely trivial to implement as long as you
got the underlying JSon library.

> Also, typing really hurts with dynamic languages because the type  
> semantics never match up well which causes unexpected failures.  A  
> classic example with XML-RPC is None.  There is no standard way to  
> transport None in XML-RPC although there are common extensions to  
> support it.  Of course, not all clients support it and there isn't  
> really a standard way to negotiate it.  It's really a mess.

how does QMP handle None then ?

> At this point, we have a close-to-mergable set of patches that provide  
> real value.  I don't see a reason not to merge them once they've been  
> resubmitted.

it provides no real value at this point. it start providing real value when
everything out (libvirt, xend, ..) there is using it.

script people won't be able to use QMP anyway, and probably still going to 
use the monitor as it is (you don't expect them to parse a line protocol with
quoted string in bash do you ?)

> I don't believe an RPC is going to be a dramatic simplification.

it's not really a simplification, but QMP while it look simple now
(just looking at the C side, without factoring the cost of reimplementing this
text protocol in language that want to use it), it's becoming not simple.

already the protocol is growing quoted string, utf8 support, ...

What's next ? multi line protocol to extends it ?
IMAP grew probably the same way, as a simple protocol in the beggining,
but look at the very useful description that Felip made (Thanks Felip btw)

it's like printing XML yourself with printf instead of using
the outputing code of the library because it's "harder". you end up
with some not correctly escaped XML, etc

> It  
> could be if there were a high quality RPC out there that had well  
> maintained client/server libraries.  I don't believe such a thing exists  
> though.  If you want to keep arguing this, let's focus on actual RPC  
> implementations.  I think we've agreed that JSON-RPC isn't going to work  
> because of the lack of client/server library availability.

So at this point your dismissing JSON-RPC just on the fact that there's no
server/client implementation available right away, but you don't dismiss it on
any other ground related to the protocol itself ? is that right ?

> XML-RPC  
> falls into the same boat for C as far as I'm concerned.  If there are  
> any other ones you want to consider, let's look at them so that we can  
> make progress in this discussion.

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Vincent Hanquez]

On Fri, Jun 26, 2009 at 10:44:40AM +0100, Daniel P. Berrange wrote:
> We only used a library for the data serialization/de-serialization.

that's what has been proposed here so far, xmlrpc and jsonrpc are just
serialization/de-serialization libraries of a "query". dbus is a bit more
hence why it got some shortcoming related to qemu related to tcp auth and stuff.

the question is why did you not do your own data seraliazation/deserialization
library then ? and then why do your previous answer doesn't apply to qemu too ?

-- 
Vincent

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> Merging is a meaningless milestone since the protocol will not be 
> enabled.  Since we'll miss 0.11 in reality we have 7-8 months before 
> the protocol can be used in production.  IMO you're optimizing for the 
> short term, disregarding long-term maintenance, and disregarding ease 
> of implementation for users of the qemu monitor.

QMP is:

__skip__: [ \t]+

symbol: [A-Za-z_\-]+
decimalinteger: [1-9][0-9]+
hexinteger: 0x[0-9]+
float: [0-9]+\.[0-9]+
string: \"([^\"\\]|(\\[0-9][0-9][0-9]))*\" | symbol

number: float | decimalinteger | hexinteger
value: string | number

arg_list: value arg_list?
command: symbol arg_list? '\n'

response_status: ('+' | '-') number arg_list? '\n'
response_data: '=' arg_list '\n'
async_msg: '*' arg_list '\n'

response: async_msg | response_data* response_status

Client sends 'command's, server responses with 'response'.  Responses of 
response type '*' are server initiated messages.  Otherwise, a server 
always responses to a command with a response type of '+' or '-'.  A 
client sits in a 'response' parsing loop.  Each sent 'command' is queued 
in a FIFO and responses are associated with the commands based on the 
FIFO order.  'async_msg's do not have associated commands.  All strings 
are UTF-8.  Human readable messages depend on the locale used by QEMU.

N.B. there are some subtle changes to Luiz's original proposal to make 
QMP more well-formed.  In particular, more things have to be encoded as 
strings than previously.

This is a very well formed RPC now.  It can be trivially implemented in 
something like Python.  We can add higher level types.  For instance, we 
could:

-value: string | number
+value: (string | number | list | dictionary)
+comma_arg_list: value (',' comma_arg_list)?
+list: '[' comma_arg_list? ']'
+
+dict_list: string ':' value (',' dict_list)?
+dictionary: '{' dict_list? '}'

And then we basically have JSON.  The changes on top of Luiz's current 
patchset would be introducing an error code and adding more structure to 
monitor_printf() which I asked for anyway.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

Anthony Liguori wrote:
> Avi Kivity wrote:
>> Merging is a meaningless milestone since the protocol will not be 
>> enabled.  Since we'll miss 0.11 in reality we have 7-8 months before 
>> the protocol can be used in production.  IMO you're optimizing for 
>> the short term, disregarding long-term maintenance, and disregarding 
>> ease of implementation for users of the qemu monitor.
>
Here's an updated grammar.

One benefit of using our own grammar over just JSON-RPC that is line 
based is that lines form discrete PDUs.  If you wanted to implement a 
JSON-RPC parser, you would have to be able to either invent your own PDU 
transmission mechanism or rely on the JSON parser to determine message 
boundaries.  Since the later is likely to be a recursive decent parser, 
this is difficult to work into an asynchronous client library.  The same 
is true for XML-RPC.   With XML-RPC, you rely on HTTP's Content-Length 
header to determine boundaries.

By having a well defined PDU, you can hand lines to the parser and be 
sure that the recursion will complete without requiring more data.

Regards,

Anthony Liguori

[-- Attachment #2: qmp-v2.g --]
[-- Type: text/plain, Size: 823 bytes --]

# QMP 1.0

# v1 -> v2
#  o allow integer in symbol names (C-style)
#  o allow negative integer/floats
#  o no newlines in strings
#  o one value per response_data line

__skip__: [ \t]+

symbol: [A-Za-z_\-][A-Za-z_\-0-9]*

decimalinteger: (-)?[1-9][0-9]+
hexinteger: 0x[0-9]+
float: (-)?[0-9]+\.[0-9]+

string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol

number: float | decimalinteger | hexinteger

comma_arg_list: value (',' comma_arg_list)?
list: '[' comma_arg_list? ']'

dict_list: string ':' value (',' dict_list)?
dictionary: '{' dict_list? '}' 

value: (string | number | list | dictionary)

arg_list: value arg_list?

command: symbol arg_list? '\n'

response_status: ('+' | '-') number string? '\n'
response_data: '=' value '\n'
async_msg: '*' arg_list '\n'

response: async_msg | response_data* response_status

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Fri, Jun 26, 2009 at 5:02 PM, Anthony Liguori<anthony@codemonkey.ws> wrote:
> Anthony Liguori wrote:
>>
>> Avi Kivity wrote:
>>>
>>> Merging is a meaningless milestone since the protocol will not be
>>> enabled.  Since we'll miss 0.11 in reality we have 7-8 months before the
>>> protocol can be used in production.  IMO you're optimizing for the short
>>> term, disregarding long-term maintenance, and disregarding ease of
>>> implementation for users of the qemu monitor.
>>
> Here's an updated grammar.
>
> One benefit of using our own grammar over just JSON-RPC that is line based
> is that lines form discrete PDUs.  If you wanted to implement a JSON-RPC
> parser, you would have to be able to either invent your own PDU transmission
> mechanism or rely on the JSON parser to determine message boundaries.  Since
> the later is likely to be a recursive decent parser, this is difficult to
> work into an asynchronous client library.  The same is true for XML-RPC.
> With XML-RPC, you rely on HTTP's Content-Length header to determine
> boundaries.
>
> By having a well defined PDU, you can hand lines to the parser and be sure
> that the recursion will complete without requiring more data.
>
> Regards,
>
> Anthony Liguori
>
> # QMP 1.0
>
> # v1 -> v2
> #  o allow integer in symbol names (C-style)
> #  o allow negative integer/floats
> #  o no newlines in strings
> #  o one value per response_data line
>
> __skip__: [ \t]+
>
> symbol: [A-Za-z_\-][A-Za-z_\-0-9]*
>
> decimalinteger: (-)?[1-9][0-9]+
> hexinteger: 0x[0-9]+
> float: (-)?[0-9]+\.[0-9]+
>
> string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol
>
> number: float | decimalinteger | hexinteger
>
> comma_arg_list: value (',' comma_arg_list)?
> list: '[' comma_arg_list? ']'
>
> dict_list: string ':' value (',' dict_list)?
> dictionary: '{' dict_list? '}'
>
> value: (string | number | list | dictionary)
>
> arg_list: value arg_list?
>
> command: symbol arg_list? '\n'
>
> response_status: ('+' | '-') number string? '\n'
> response_data: '=' value '\n'
> async_msg: '*' arg_list '\n'
>
> response: async_msg | response_data* response_status
>
>

Thanks for writing this down, I really appreciate it.

It answers some of the issues I had with the original protocol
proposal (interruption of multi-line responses by asynchronous
events). What bothers me though is that it's not LL(1) grammar (not a
big problem, but writing LL parsers by hand is much easier) and not
all floats can be represented (eg. NaN). String is underdefined - does
the syntax represent bytes or characters? If characters then the \nnn
syntax is insufficient. There's no syntax for binary data which may
eventually be required (FDT machine description), but that can be
relatively easily added with something like

blob := 'B' base64-string
base64-string := \" ([A-Za-z0-9+/]*) \"

and adjusting "value".

Note that IMAP uses special literal symbols to send binary data and
the data transfer must be negotiated (unless LITERAL+ extension is
used). It looks like this

S: ... RFC822.BODY {<size>}
C: + Ok, send the data
S: <binary data of the announced size>

It makes the protocol much harder to implement and almost nobody does
it right (including major servers), so I would recommend against
anything like that. In the ideal case QEMU will never need large
enough blobs to warrant it.

I'm still not convinced that reinventing a text-based protocol is way
to go, but I welcome the effort for formal specification of what you
had in mind.

Best regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Filip Navara wrote:
> On Fri, Jun 26, 2009 at 5:02 PM, Anthony Liguori<anthony@codemonkey.ws> wrote:
>   


> It answers some of the issues I had with the original protocol
> proposal (interruption of multi-line responses by asynchronous
> events). What bothers me though is that it's not LL(1) grammar (not a
> big problem, but writing LL parsers by hand is much easier)

Perhaps you mean LR(1) (i.e. something that can be understood by yacc).  
When writing parsers by hand, I tend to prefer recursive decent which 
easily handles LL(k).

Fortunately, this grammar is very simple and can be reasonable converted 
to LR(1).

>  and not
> all floats can be represented (eg. NaN).

I'm not sure that's so important.  JSON doesn't support NaN.

>  String is underdefined - does
> the syntax represent bytes or characters? If characters then the \nnn
> syntax is insufficient.

Strings are UTF-8.  Based on the definition of string, the only thing 
needing escaping is ", \\, and \n.  Anything else is just cosmetic.  My 
intention was for \nnn to be a single octal byte as an escape.  You 
could certainly use multiple escape sequences to encoding a large UTF 
character.

>  There's no syntax for binary data which may
> eventually be required (FDT machine description), but that can be
> relatively easily added with something like
>   

I don't think binary data is a requirement.  An FDT should be 
transmitted as a tree, not as a binary blob.  You could also transmit 
binary as a list of bytes though.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Fri, Jun 26, 2009 at 9:36 PM, Anthony Liguori<anthony@codemonkey.ws> wrote:
> Filip Navara wrote:
>>
>> On Fri, Jun 26, 2009 at 5:02 PM, Anthony Liguori<anthony@codemonkey.ws>
>> wrote:
>>
>
>
>> It answers some of the issues I had with the original protocol
>> proposal (interruption of multi-line responses by asynchronous
>> events). What bothers me though is that it's not LL(1) grammar (not a
>> big problem, but writing LL parsers by hand is much easier)
>
> Perhaps you mean LR(1) (i.e. something that can be understood by yacc).
>  When writing parsers by hand, I tend to prefer recursive decent which
> easily handles LL(k).

I meant LL(1). Nevertheless I realized that it's easy to write a lexer
and LL(1) parser given this grammar, so ignore this objection. I
prefer to write my parsers by hand instead of using yacc or other
tools as long as the grammar is simple enough.

> Fortunately, this grammar is very simple and can be reasonable converted to
> LR(1).
>
>>  and not
>> all floats can be represented (eg. NaN).
>
> I'm not sure that's so important.  JSON doesn't support NaN.

It may not be important now, but I consider it important to get the
syntax right the first time. Adding and extending commands to existing
implementation is easy (or easier at least) and can be done in
backward-compatible way if the syntax of the protocol doesn't change
between versions. What JSON does or doesn't support is irrelevant
here, most platforms QEMU runs on use IEEE 754 floats and if you want
to express them in the syntax then express all of their features -
Inf, NaN and exponents (maybe I forgot something). The benefit is that
you can use printf functions to output them if the syntax matches what
C uses.

>>  String is underdefined - does
>> the syntax represent bytes or characters? If characters then the \nnn
>> syntax is insufficient.
>
> Strings are UTF-8.  Based on the definition of string, the only thing
> needing escaping is ", \\, and \n.  Anything else is just cosmetic.  My
> intention was for \nnn to be a single octal byte as an escape.  You could
> certainly use multiple escape sequences to encoding a large UTF character.

I guess you mean that the "string" in grammar is sequence of bytes.
These bytes should be interpreted as UTF-8 characters then.

Note that this is very important to specify. For example the GREEK
CAPITAL LETTER PI is unicode code point 928. If the "string" syntax
was meant to represent characters I would expect "\928" to refer to
single character string with PI being the character, on the other hand
in byte-based UTF-8 string you'd write "\206\160".

>
>>  There's no syntax for binary data which may
>> eventually be required (FDT machine description), but that can be
>> relatively easily added with something like
>>
>
> I don't think binary data is a requirement.  An FDT should be transmitted as
> a tree, not as a binary blob.  You could also transmit binary as a list of
> bytes though.

You can transmit it as list of bytes and it's woefully inefficient. I
gave the FDT as an example, but I believe that binary data may be
needed in future for one purpose or another. The monitor already
provides a way to dump guest memory and I see no reason to not
transfer it using some "binary" encoding.

>
> Regards,
>
> Anthony Liguori
>

Regards,
Filip Navara

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Fri, Jun 26, 2009 at 9:36 PM, Anthony Liguori<anthony@codemonkey.ws> wrote:
> Filip Navara wrote:
>>  and not
>> all floats can be represented (eg. NaN).
>
> I'm not sure that's so important.  JSON doesn't support NaN.

BTW, JSON doesn't even pretend to support floats. What it supports is
a notation to write decimal fractions, in many languages that maps to
the DECIMAL type (number with sign, mantissa and exponent in base-10)
and not float. Calling it float is a misnomer.

Best regards,
Filip Navara

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Luiz Capitulino]

On Fri, 26 Jun 2009 10:42:24 +0100
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Fri, Jun 26, 2009 at 12:21:01PM +0300, Avi Kivity wrote:
> > On 06/25/2009 10:11 PM, Luiz Capitulino wrote:
> > >
> > >  Yes, having a library was suggested by Amit some months ago. The
> > >problem is that it has various issues wrt maintainability.
> > >
> > >  For example, libvirt is able to run two instances of different
> > >versions of qemu at the same time. How to handle this if you
> > >update libmonitor.so?
> > >   
> 
> The sane way is to *NOT* break ABI of libmonitor.so, and not change
> the wire protocol in a non backwards compatible way. This is entirely
> doable, it just the maintainer of libmonitor/qemu to decide that  ABI
> stability is important. So if you find an existing API  / command
> needs to gain an extra argument, you don't change the existing API,
> you add a new one. Or ideally design the API upfront so that it 
> can be extended without breaking back compatability.

 [ Looks like my (evil) twin brother has taken the keyboard while I
was away and has started saying libmonitor.so non-sense. ]

 I think I have mixed up problems here, but anyway, working on a C library
seems a step back to me. I believe that defining the API for it will
generate more discussion than the protocol, will probably be harder
to write and to document too.

 In a general way the protocol idea is very simple: we just have to
make monitor's output parsable. As Anthony has put it, the patchset
is not far from a mergeable state. I can't even image how long it would
take to have the C API in the same shape.

 That said, I think the protocol discussion went way too far. Of
course that I want to do the right thing and we should always be
concerned with the future, but the priority here is not to design/choose
the next-ultra-super-hiper-mathematically-proven-and-loved protocol.

 So, IMHO both solutions (QMP and JSON) solves the problem and I
would work on either one. I just would like that Anthony and Avi
get in agreement, because the project will fail if it becomes
one more difference between qemu and qemu-kvm.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/26/2009 06:02 PM, Anthony Liguori wrote:
> Anthony Liguori wrote:
>> Avi Kivity wrote:
>>> Merging is a meaningless milestone since the protocol will not be 
>>> enabled.  Since we'll miss 0.11 in reality we have 7-8 months before 
>>> the protocol can be used in production.  IMO you're optimizing for 
>>> the short term, disregarding long-term maintenance, and disregarding 
>>> ease of implementation for users of the qemu monitor.
>>
> Here's an updated grammar.
>
> One benefit of using our own grammar over just JSON-RPC that is line 
> based is that lines form discrete PDUs.  If you wanted to implement a 
> JSON-RPC parser, you would have to be able to either invent your own 
> PDU transmission mechanism or rely on the JSON parser to determine 
> message boundaries.  Since the later is likely to be a recursive 
> decent parser, this is difficult to work into an asynchronous client 
> library.  The same is true for XML-RPC.   With XML-RPC, you rely on 
> HTTP's Content-Length header to determine boundaries.

You could also abort the parse if you reach EOF, and start it from the 
beginning when you get more data.  It's reasonably efficient in a local 
protocol, and when you expect small messages.

I'm fine with a newline delimiter though.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/26/2009 06:02 PM, Anthony Liguori wrote:
> # v1 ->  v2
> #  o allow integer in symbol names (C-style)
> #  o allow negative integer/floats
> #  o no newlines in strings
> #  o one value per response_data line
>
> __skip__: [ \t]+
>
> symbol: [A-Za-z_\-][A-Za-z_\-0-9]*
>
> decimalinteger: (-)?[1-9][0-9]+
>    
>

Traditionally, 0 is recognized as an integer.

> hexinteger: 0x[0-9]+
>    

[a-fA-f]

> float: (-)?[0-9]+\.[0-9]+
>    

5, 3., .10, 1e9 are all floats.

> string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol
>    

I think json recognises 'blah' but not symbol.  Also it may recognize 
other escapes.

It's important to allow clients to use json emitters so they can 
construct a dict and throw it at qemu, similarly to parse qemu results.

> arg_list: value arg_list?
>
> command: symbol arg_list? '\n'
>    

Better to use a dictionary for the argument list (more self-documenting, 
easier expansion).

I'd also adopt jsonrpc's request and id attributes, they allow async 
commands and remove the need for any specialized parsing.

> response_status: ('+' | '-') number string? '\n'
> response_data: '=' value '\n'
> async_msg: '*' arg_list '\n'
>
>    

Similarly, return a value that contains message type, id, and return value.

I think it's fine to drop jsonrpc as the standard (but retain its 
features) and just adopt json for encoding.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/26/2009 11:25 PM, Filip Navara wrote:
>> I don't think binary data is a requirement.  An FDT should be transmitted as
>> a tree, not as a binary blob.  You could also transmit binary as a list of
>> bytes though.
>>      
>
> You can transmit it as list of bytes and it's woefully inefficient. I
> gave the FDT as an example, but I believe that binary data may be
> needed in future for one purpose or another. The monitor already
> provides a way to dump guest memory and I see no reason to not
> transfer it using some "binary" encoding.
>    

Another candidate is screendump.  There's no reason to go through a 
file.  We could pass a pipe as an fd and request a screendump to that 
fd, but that's rather roundabout.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Avi Kivity]

On 06/27/2009 06:58 PM, Luiz Capitulino wrote:
>   So, IMHO both solutions (QMP and JSON) solves the problem and I
> would work on either one. I just would like that Anthony and Avi
> get in agreement, because the project will fail if it becomes
> one more difference between qemu and qemu-kvm.
>    

There's no danger of a diverging implementation in this case since no 
one is proposing to have different monitor protocols.  We just need to 
find the best protocol.  Anthony's looking for minimal churn for the 
existing monitor command set and for libvirt, while I am considering the 
additional effort for new commands and for new clients.  It really isn't 
very complicated, and the thread only got so long because the topic is 
relatively simple.  Post an RFC and a mile-long patchset about changing 
TCG to SSA form, and see how you get no replies.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Blue Swirl]

On 6/28/09, Avi Kivity <avi@redhat.com> wrote:
> On 06/27/2009 06:58 PM, Luiz Capitulino wrote:
>
> >  So, IMHO both solutions (QMP and JSON) solves the problem and I
> > would work on either one. I just would like that Anthony and Avi
> > get in agreement, because the project will fail if it becomes
> > one more difference between qemu and qemu-kvm.
> >
> >
>
>  There's no danger of a diverging implementation in this case since no one
> is proposing to have different monitor protocols.  We just need to find the
> best protocol.  Anthony's looking for minimal churn for the existing monitor
> command set and for libvirt, while I am considering the additional effort
> for new commands and for new clients.  It really isn't very complicated, and
> the thread only got so long because the topic is relatively simple.  Post an
> RFC and a mile-long patchset about changing TCG to SSA form, and see how you
> get no replies.

No, I think the trick is to have the mile-long patchset that twiddles
something in the block layer. It can potentially cause deadly bugs for
every guest and the benefit may not be obvious.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> On 06/26/2009 06:02 PM, Anthony Liguori wrote:
>> # v1 ->  v2
>> #  o allow integer in symbol names (C-style)
>> #  o allow negative integer/floats
>> #  o no newlines in strings
>> #  o one value per response_data line
>>
>> __skip__: [ \t]+
>>
>> symbol: [A-Za-z_\-][A-Za-z_\-0-9]*
>>
>> decimalinteger: (-)?[1-9][0-9]+
>>   
>
> Traditionally, 0 is recognized as an integer.
>> hexinteger: 0x[0-9]+
>>    
>
> [a-fA-f]

I have an updated grammer that has both of these fixes.

>> float: (-)?[0-9]+\.[0-9]+
>>    
>
> 5, 3., .10, 1e9 are all floats.

I was following JSON style floats as opposed to C float.  C floats are 
irregular and difficult to parse with an automatic parser.  If we care, 
it's worth doing correctly though.

>> string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol
>>    
>
> I think json recognises 'blah' but not symbol.  Also it may recognize 
> other escapes.

We can drop symbol.  The original intention was to output format similar 
to what we have today but we're pretty far from that now.

> It's important to allow clients to use json emitters so they can 
> construct a dict and throw it at qemu, similarly to parse qemu results.
>
>> arg_list: value arg_list?
>>
>> command: symbol arg_list? '\n'
>>    
>
> Better to use a dictionary for the argument list (more 
> self-documenting, easier expansion).

Verses a list?  I'm not sure a dict fits the current QEMU model very 
well.  How would you propose fitting it into the current monitor 
infrastructure?

> I'd also adopt jsonrpc's request and id attributes, they allow async 
> commands and remove the need for any specialized parsing.

I think command tagging is a very good thing to do.

>> response_status: ('+' | '-') number string? '\n'
>> response_data: '=' value '\n'
>> async_msg: '*' arg_list '\n'
>>
>>    
>
> Similarly, return a value that contains message type, id, and return 
> value.

I think command outputs are basically return values.  What's nice about 
the current QMP proposal is that we're basically returning tuples.

> I think it's fine to drop jsonrpc as the standard (but retain its 
> features) and just adopt json for encoding.

Yeah, I think if we focus on defining a grammar and semantics, if it 
happens to look like JSON, I don't really care.  I just don't want to 
adopt the baggage of json and particularly json-rpc.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Filip Navara]

On Sun, Jun 28, 2009 at 5:52 PM, Avi Kivity<avi@redhat.com> wrote:
> On 06/27/2009 06:58 PM, Luiz Capitulino wrote:
>>
>>  So, IMHO both solutions (QMP and JSON) solves the problem and I
>> would work on either one. I just would like that Anthony and Avi
>> get in agreement, because the project will fail if it becomes
>> one more difference between qemu and qemu-kvm.
>>
>
> There's no danger of a diverging implementation in this case since no one is
> proposing to have different monitor protocols.  We just need to find the
> best protocol.  Anthony's looking for minimal churn for the existing monitor
> command set and for libvirt, while I am considering the additional effort
> for new commands and for new clients.

I'm with Avi on this issue, but I will be happy as long as the
protocol is precisely described and extensible for future. Moreover I
believe that converting the current code to use a new function like
monitor_print_data could be done now even without knowing the exact
details of the on-wire protocol. The monitor_print_data function could
be then adjusted to understand the protocol specifics and emit the
data accordingly.

> It really isn't very complicated, and
> the thread only got so long because the topic is relatively simple.  Post an
> RFC and a mile-long patchset about changing TCG to SSA form, and see how you
> get no replies.

I wouldn't even dare to push the SSA patch... Mile-long doesn't
describe it precisely enough. Imagine it was applied to all the
targets.

F.

 target-arm/translate.c | 3846 +++++++++++++++++++++++++++---------------------
 tcg/tcg.c              |  327 +++-
 tcg/tcg.h              |    2 +-
 tcg/tcg-op.h           | 1614 ++++++++++++-------

 4 files changed, 3483 insertions(+), 2306 deletions(-)

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> On 06/26/2009 11:25 PM, Filip Navara wrote:
>>> I don't think binary data is a requirement.  An FDT should be 
>>> transmitted as
>>> a tree, not as a binary blob.  You could also transmit binary as a 
>>> list of
>>> bytes though.
>>>      
>>
>> You can transmit it as list of bytes and it's woefully inefficient. I
>> gave the FDT as an example, but I believe that binary data may be
>> needed in future for one purpose or another. The monitor already
>> provides a way to dump guest memory and I see no reason to not
>> transfer it using some "binary" encoding.
>>    
>
> Another candidate is screendump.  There's no reason to go through a 
> file.  We could pass a pipe as an fd and request a screendump to that 
> fd, but that's rather roundabout.

Instead of inventing binary syntaxes, maybe we could use dictionary 
syntax to denote a mime encoded value?

So something like:

{"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", 
"Content": "AB234SDFSDf=="}

A special brace format could be used if we wanted to make sure it was 
independently parsable:

<"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", 
"Content": "AB234SDFSDf==">

But I suspect dicts are good enough.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/28/2009 08:23 PM, Anthony Liguori wrote:
>> It's important to allow clients to use json emitters so they can 
>> construct a dict and throw it at qemu, similarly to parse qemu results.
>>
>>> arg_list: value arg_list?
>>>
>>> command: symbol arg_list? '\n'
>>
>> Better to use a dictionary for the argument list (more 
>> self-documenting, easier expansion).
>
> Verses a list?  I'm not sure a dict fits the current QEMU model very 
> well.  How would you propose fitting it into the current monitor 
> infrastructure?

First, I would like to consider the protocol mostly from the point of 
view of clients, not qemu, and certainly not current qemu code.  There 
is only one qemu, but many clients.  Code problems can be solved 
tomorrow, but interface problems remain forever.

Second, I think it fits quite nicely.  Positional arguments are nasty 
when you have many of them, especially when some are optional.  And we 
already support dictionaries: -drive file=blah,cache=none.  We make a 
dictionary interface, and have both the command line parser and the 
monitor json thing implement it.

(note: it would be nice to have this interface detect unused arguments 
so we can flag typos in keywords).

>> Similarly, return a value that contains message type, id, and return 
>> value.
>
> I think command outputs are basically return values.  What's nice 
> about the current QMP proposal is that we're basically returning tuples.

Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }.  
'type' is the new =, and 'id' is the tag.  If multiple return values are 
needed, value can be a list or dictionary.

>> I think it's fine to drop jsonrpc as the standard (but retain its 
>> features) and just adopt json for encoding.
>
> Yeah, I think if we focus on defining a grammar and semantics, if it 
> happens to look like JSON, I don't really care.  I just don't want to 
> adopt the baggage of json and particularly json-rpc.

If 'looks like' == 'compatible with', I agree.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/28/2009 08:30 PM, Anthony Liguori wrote:
>> Another candidate is screendump.  There's no reason to go through a 
>> file.  We could pass a pipe as an fd and request a screendump to that 
>> fd, but that's rather roundabout.
>
>
> Instead of inventing binary syntaxes, maybe we could use dictionary 
> syntax to denote a mime encoded value?
>
> So something like:
>
> {"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", 
> "Content": "AB234SDFSDf=="}
>
> A special brace format could be used if we wanted to make sure it was 
> independently parsable:
>
> <"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", 
> "Content": "AB234SDFSDf==">
>
> But I suspect dicts are good enough.

Yes.

(I see a future thread about how we negotiate content types etc.)

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Filip Navara]

On Sun, Jun 28, 2009 at 7:23 PM, Anthony Liguori<anthony@codemonkey.ws> wrote:
> Avi Kivity wrote:
[snip]
>> 5, 3., .10, 1e9 are all floats.
>
> I was following JSON style floats as opposed to C float.  C floats are
> irregular and difficult to parse with an automatic parser.  If we care, it's
> worth doing correctly though.

I'd very much prefer to do it correctly and express all C floats in the syntax.

Going back to my analogy with IMAP, the IMAP syntax doesn't allow for
multipart messages with zero parts. While this doesn't sound like
anything useful it's a discrepancy with the MIME standard of multipart
messages which allows it. And since the MIME standard allows it,
applications started to use it (some MMS operator gateways iirc). The
result is that instead of expressing the MIME structure directly in
IMAP (as was the original intent) one has to first convert it into
something that can be expressed in the IMAP syntax (by adding dummy
parts to the message).

Best regards,
Filip Navara

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Avi Kivity]

On 06/28/2009 08:23 PM, Filip Navara wrote:
> On Sun, Jun 28, 2009 at 5:52 PM, Avi Kivity<avi@redhat.com>  wrote:
>    
>> On 06/27/2009 06:58 PM, Luiz Capitulino wrote:
>>      
>>>   So, IMHO both solutions (QMP and JSON) solves the problem and I
>>> would work on either one. I just would like that Anthony and Avi
>>> get in agreement, because the project will fail if it becomes
>>> one more difference between qemu and qemu-kvm.
>>>
>>>        
>> There's no danger of a diverging implementation in this case since no one is
>> proposing to have different monitor protocols.  We just need to find the
>> best protocol.  Anthony's looking for minimal churn for the existing monitor
>> command set and for libvirt, while I am considering the additional effort
>> for new commands and for new clients.
>>      
>
> I'm with Avi on this issue, but I will be happy as long as the
> protocol is precisely described and extensible for future. Moreover I
> believe that converting the current code to use a new function like
> monitor_print_data could be done now even without knowing the exact
> details of the on-wire protocol. The monitor_print_data function could
> be then adjusted to understand the protocol specifics and emit the
> data accordingly.
>    

In terms of implementation, I think we could structure the human and 
machine monitor as follows:

- the human monitor parses a command (with the current parser) and 
generates a dict
- the machine monitor (using a json-like parser) parses its input which 
contains a parameter dict
- the dict is handed off to common code
- common code does something, returns an error code and a return value
- the human monitor prints the return value (with the current printfs)
- the machine monitor packs the return value according to the protocol 
and lets the json-like emitter emit it

So we don't have to duplicate parsers an emitters; instead we have to 
write some accessor functions for generalized values (which can come 
from the human monitor, command line, machine monitor, and maybe config 
file).

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Blue Swirl]

On 6/28/09, Filip Navara <filip.navara@gmail.com> wrote:
> On Sun, Jun 28, 2009 at 5:52 PM, Avi Kivity<avi@redhat.com> wrote:
>  > It really isn't very complicated, and
>  > the thread only got so long because the topic is relatively simple.  Post an
>  > RFC and a mile-long patchset about changing TCG to SSA form, and see how you
>  > get no replies.
>
>
> I wouldn't even dare to push the SSA patch... Mile-long doesn't
>  describe it precisely enough. Imagine it was applied to all the
>  targets.

Is there a speed benefit from using the SSA form?

In theory, a new code generator could be added alongside the current
one and when (if?) everyone is happy with it, switch to the new one.

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Avi Kivity]

On 06/28/2009 08:51 PM, Blue Swirl wrote:
> On 6/28/09, Filip Navara<filip.navara@gmail.com>  wrote:
>    
>> On Sun, Jun 28, 2009 at 5:52 PM, Avi Kivity<avi@redhat.com>  wrote:
>>   >  It really isn't very complicated, and
>>   >  the thread only got so long because the topic is relatively simple.  Post an
>>   >  RFC and a mile-long patchset about changing TCG to SSA form, and see how you
>>   >  get no replies.
>>
>>
>> I wouldn't even dare to push the SSA patch... Mile-long doesn't
>>   describe it precisely enough. Imagine it was applied to all the
>>   targets.
>>      
>
> Is there a speed benefit from using the SSA form?
>    

I'm no compiler expert.  My understanding is that SSA enables 
optimizations, so translation speed would decrease, but generated code 
speed would increase.

Other jits increase optimization effort on long running loops, so if a 
code is run heavily it receives additional attention from the optimizer.


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command

[Jamie Lokier]

Blue Swirl wrote:
> On 6/28/09, Filip Navara <filip.navara@gmail.com> wrote:
> > On Sun, Jun 28, 2009 at 5:52 PM, Avi Kivity<avi@redhat.com> wrote:
> >  > It really isn't very complicated, and
> >  > the thread only got so long because the topic is relatively simple.  Post an
> >  > RFC and a mile-long patchset about changing TCG to SSA form, and see how you
> >  > get no replies.
> >
> >
> > I wouldn't even dare to push the SSA patch... Mile-long doesn't
> >  describe it precisely enough. Imagine it was applied to all the
> >  targets.
> 
> Is there a speed benefit from using the SSA form?

You can do some nice optimisations on SSA form, but you can do them
without SSA too.  It's just that SSA makes it easier to think about
some things.

SSA is only useful for interesting control flow graphs with many basic
blocks, though.  As far as I know, QEMU only translates code in linear
instruction traces - so most SSA optimisations don't apply, and those
which remain are easy to think about in any representation.

-- Jamie

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> First, I would like to consider the protocol mostly from the point of 
> view of clients, not qemu, and certainly not current qemu code.  There 
> is only one qemu, but many clients.  Code problems can be solved 
> tomorrow, but interface problems remain forever.

I'm trying to consider both.  We need something that we can get into a 
robust state in 6 months.  I don't want to undertake a massive effort 
that will require major changes because history shows that we'll end up 
with something that doesn't really do the job.

> Second, I think it fits quite nicely.  Positional arguments are nasty 
> when you have many of them, especially when some are optional.  And we 
> already support dictionaries: -drive file=blah,cache=none.  We make a 
> dictionary interface, and have both the command line parser and the 
> monitor json thing implement it.

It a fair bit of complexity but I don't think it's too problematic.  The 
current monitor abstraction doesn't have a mechanism to define names of 
arguments but I think it could be added relatively easily although it 
wouldn't be so elegant (another parameter with comma deliminated names 
would do the trick).

>
> Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }.  
> 'type' is the new =, and 'id' is the tag.  If multiple return values 
> are needed, value can be a list or dictionary.

I really don't like the syntax you proposed but that's an aesthetic 
problem.  This is an interface consumed by non humans so I don't think 
it's worth arguing about something that is just as easily parsed by a 
computer even if it's hideously ugly :-)

So I'll defer to whatever Luiz (or whoever sends patches) decides to do 
as long as it meets the same requirements.

The main thing missing from the first patchset that this thread has 
revealed is a formal grammar for the data.  This is an important 
requirement to ensure that it's reasonably parse-able by both ends.

>>> I think it's fine to drop jsonrpc as the standard (but retain its 
>>> features) and just adopt json for encoding.
>>
>> Yeah, I think if we focus on defining a grammar and semantics, if it 
>> happens to look like JSON, I don't really care.  I just don't want to 
>> adopt the baggage of json and particularly json-rpc.
>
> If 'looks like' == 'compatible with', I agree.

You lost me here.  JSON interoperability cannot be a requirement.  
Otherwise, we're going to get stuck dealing with all the warts of JSON.  
I don't want people to assume they can just use a JSON parser.

You seem to have conflicting requirements.   You want JSON compatibility 
but then you want C-style floats which JSON doesn't support.

I don't mind if this ends up looking like JSON, but I don't want to have 
to enforce that it's remains JSON compatible in the long term.  We 
should have our own grammar that we can version and that people can use 
as the basis of a client.

If being pure JSON is important to someone (and I don't think it will 
be), then someone can write a JSON proxy.  You don't need any RPC 
knowledge to do that so it can be relatively future-proof.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/29/2009 03:42 AM, Anthony Liguori wrote:
> Avi Kivity wrote:
>> First, I would like to consider the protocol mostly from the point of 
>> view of clients, not qemu, and certainly not current qemu code.  
>> There is only one qemu, but many clients.  Code problems can be 
>> solved tomorrow, but interface problems remain forever.
>
> I'm trying to consider both.  We need something that we can get into a 
> robust state in 6 months.  I don't want to undertake a massive effort 
> that will require major changes because history shows that we'll end 
> up with something that doesn't really do the job.

History also shows that band aids rarely work, that trying to reinvent 
wheels leads to unusual shapes, and that we're pretty bad at predicting 
future needs.

>> Second, I think it fits quite nicely.  Positional arguments are nasty 
>> when you have many of them, especially when some are optional.  And 
>> we already support dictionaries: -drive file=blah,cache=none.  We 
>> make a dictionary interface, and have both the command line parser 
>> and the monitor json thing implement it.
>
> It a fair bit of complexity but I don't think it's too problematic.  
> The current monitor abstraction doesn't have a mechanism to define 
> names of arguments but I think it could be added relatively easily 
> although it wouldn't be so elegant (another parameter with comma 
> deliminated names would do the trick).

Instead of 7 callback types you get one callback type with one parameter 
(an object).  We can retain the table driven parsing for the human 
monitor by, for example, changing

     { "memsave", "lis", do_memory_save,
       "addr size file", "save to disk virtual memory dump starting at 
'addr' of size 'size'", },

to
     { "memsave", "address:l,size:i,file:s", do_memory_save,
       "save to disk virtual memory dump starting at 'address' of size 
'size'", },

>>
>> Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }.  
>> 'type' is the new =, and 'id' is the tag.  If multiple return values 
>> are needed, value can be a list or dictionary.
>
> I really don't like the syntax you proposed but that's an aesthetic 
> problem.  This is an interface consumed by non humans so I don't think 
> it's worth arguing about something that is just as easily parsed by a 
> computer even if it's hideously ugly :-)

I too prefer looking at the original QMP with hand crafted syntax.  But 
even more I prefer not looking at it, and the more we make it machine 
consumable, the less we have to dig in.

> The main thing missing from the first patchset that this thread has 
> revealed is a formal grammar for the data.  This is an important 
> requirement to ensure that it's reasonably parse-able by both ends.

That's something else gained from json -- someone else has written the 
grammar and made sure it works in real life.


>>>> I think it's fine to drop jsonrpc as the standard (but retain its 
>>>> features) and just adopt json for encoding.
>>>
>>> Yeah, I think if we focus on defining a grammar and semantics, if it 
>>> happens to look like JSON, I don't really care.  I just don't want 
>>> to adopt the baggage of json and particularly json-rpc.
>>
>> If 'looks like' == 'compatible with', I agree.
>
> You lost me here.  JSON interoperability cannot be a requirement.  
> Otherwise, we're going to get stuck dealing with all the warts of JSON. 

Look at the top half of http://www.json.org/.  It's remarkably wart free.

> I don't want people to assume they can just use a JSON parser.

Look at the bottom half of http://www.json.org/.  It's remarkably parser 
full.  It means that anyone can get up and running writing a cloud in 
haXe or Fan without studying and implementing the qemu data format.

>
> You seem to have conflicting requirements.   You want JSON 
> compatibility but then you want C-style floats which JSON doesn't 
> support.

According to the json grammar, it supports floats (including exponents), 
except for NaN and Inf.  Personally,  I'm happy to limit qemu to finite 
numbers.

> I don't mind if this ends up looking like JSON, but I don't want to 
> have to enforce that it's remains JSON compatible in the long term.  
> We should have our own grammar that we can version and that people can 
> use as the basis of a client.

That really reduces the attractiveness of the whole thing.  Writing 
parsers and emitters should be an optional part of writing a qemu 
control program, not a required part.


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

On Sun, 28 Jun 2009, Anthony Liguori wrote:
> Instead of inventing binary syntaxes, maybe we could use dictionary 
> syntax to denote a mime encoded value?
> 
> So something like:
> 
> {"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", 
> "Content": "AB234SDFSDf=="}
> 
> A special brace format could be used if we wanted to make sure it was 
> independently parsable:
> 
> <"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", 
> "Content": "AB234SDFSDf==">
> 
> But I suspect dicts are good enough.
> 

I think it would be a good idea to allow binary as
Content-Transfer-Encoding, since base64 is very inefficient for
transferring large amount of data.
For example we may want to save the state of the guest, including the
whole memory, and transfer it over the network without saving it to file
first.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

On Mon, 29 Jun 2009, Avi Kivity wrote:
> > I don't mind if this ends up looking like JSON, but I don't want to 
> > have to enforce that it's remains JSON compatible in the long term.  
> > We should have our own grammar that we can version and that people can 
> > use as the basis of a client.
> 
> That really reduces the attractiveness of the whole thing.  Writing 
> parsers and emitters should be an optional part of writing a qemu 
> control program, not a required part.
> 

I think that using JSON syntax, hence re-using JSON parsers and
emitters, it is a good compromise between supporting a full-blown RPC
and implementing our own IMAP like protocol.
This way both parties should be satisfied.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/29/2009 12:44 PM, Stefano Stabellini wrote:
> I think it would be a good idea to allow binary as
> Content-Transfer-Encoding, since base64 is very inefficient for
> transferring large amount of data.
> For example we may want to save the state of the guest, including the
> whole memory, and transfer it over the network without saving it to file
> first.
>    

The migrate command does that.  Give it an fd and migrate away.

Mixing binary in the data format seems a bad idea to me, especially if 
we stick to json as our encoding.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

On Mon, 29 Jun 2009, Avi Kivity wrote:
> On 06/29/2009 12:44 PM, Stefano Stabellini wrote:
> > I think it would be a good idea to allow binary as
> > Content-Transfer-Encoding, since base64 is very inefficient for
> > transferring large amount of data.
> > For example we may want to save the state of the guest, including the
> > whole memory, and transfer it over the network without saving it to file
> > first.
> >    
> 
> The migrate command does that.  Give it an fd and migrate away.
> 
> Mixing binary in the data format seems a bad idea to me, especially if 
> we stick to json as our encoding.
> 

JSON compatibility is more important to me than binary data, but as a
reply to the discussion about mime encodings, I was just pointing out
that mime does support binary encodings:

http://tools.ietf.org/html/rfc1652
http://tools.ietf.org/html/rfc3030

and if it doesn't prove to be too hard we could start thinking about
supporting them.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

On Mon, 29 Jun 2009, Anthony Liguori wrote:
> >
> > Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }.  
> > 'type' is the new =, and 'id' is the tag.  If multiple return values 
> > are needed, value can be a list or dictionary.
> 
> I really don't like the syntax you proposed but that's an aesthetic 
> problem.  This is an interface consumed by non humans so I don't think 
> it's worth arguing about something that is just as easily parsed by a 
> computer even if it's hideously ugly :-)
> 
> So I'll defer to whatever Luiz (or whoever sends patches) decides to do 
> as long as it meets the same requirements.
> 
> The main thing missing from the first patchset that this thread has 
> revealed is a formal grammar for the data.  This is an important 
> requirement to ensure that it's reasonably parse-able by both ends.
> 

Even though it is not a fun thing to do I think the interface should be
written in a more formal specification language than a spec.txt or a C
header file.
We can even come up with our own language to do it as long as it forces
us to maintain a robust interface and allow us to write simple scripts
to validate the methods we use to implement the interface in our own
language of choice.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/29/2009 05:41 PM, Stefano Stabellini wrote:
> Even though it is not a fun thing to do I think the interface should be
> written in a more formal specification language than a spec.txt or a C
> header file.
> We can even come up with our own language to do it as long as it forces
> us to maintain a robust interface and allow us to write simple scripts
> to validate the methods we use to implement the interface in our own
> language of choice.
>    

I agree.  It's really easy to screw up interfaces.  It can be a simple 
validator that runs on requests and responses, or a full fledged IDL 
compiler, but we need something.  I don't think this should block 
merging so long as it's added before 0.12.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
>> I don't mind if this ends up looking like JSON, but I don't want to 
>> have to enforce that it's remains JSON compatible in the long term.  
>> We should have our own grammar that we can version and that people 
>> can use as the basis of a client.
>
> That really reduces the attractiveness of the whole thing.  Writing 
> parsers and emitters should be an optional part of writing a qemu 
> control program, not a required part.
I really disagree but this thread is also killing my soul :-)

Whatever we do for QMP, it will not be enabled for 0.11 so we have 6 
months to get it right.  In the interest of moving forward and writing 
patches instead of emails, here's what I'm thinking:

1) Update the QMP patches to support return values for monitor commands
2) Update patches to support structured command output
3) Update the patches to support higher level data types (like lists and 
dictionaries)
4) For whatever the emission format is, provide a regular grammar to 
parse that output

I will commit a patch series that meets these goals.

Given a regular grammar, it's extremely easy to convert to outputting 
JSON, XML, or whatever.  We can keep arguing about whether JSON is the 
right format long term.  However, I don't want the useful work of 
conversing monitor commands to satisify 1-3 to be held up by arguments 
about the emission format which is largely unrelated to the command 
conversions.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/29/2009 11:23 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>>> I don't mind if this ends up looking like JSON, but I don't want to 
>>> have to enforce that it's remains JSON compatible in the long term.  
>>> We should have our own grammar that we can version and that people 
>>> can use as the basis of a client.
>>
>> That really reduces the attractiveness of the whole thing.  Writing 
>> parsers and emitters should be an optional part of writing a qemu 
>> control program, not a required part.
> I really disagree

Writing parsers and emitters should be a required part of writing a qemu 
control program?

>
> Whatever we do for QMP, it will not be enabled for 0.11 so we have 6 
> months to get it right.  In the interest of moving forward and writing 
> patches instead of emails, here's what I'm thinking:
>
> 1) Update the QMP patches to support return values for monitor commands
> 2) Update patches to support structured command output
> 3) Update the patches to support higher level data types (like lists 
> and dictionaries)
> 4) For whatever the emission format is, provide a regular grammar to 
> parse that output
>
> I will commit a patch series that meets these goals.

We have six months so you'll commit some unreviewed patches now?

This is deeply dissatisfying.

> Given a regular grammar, it's extremely easy to convert to outputting 
> JSON, XML, or whatever.  We can keep arguing about whether JSON is the 
> right format long term.  However, I don't want the useful work of 
> conversing monitor commands to satisify 1-3 to be held up by arguments 
> about the emission format which is largely unrelated to the command 
> conversions.

We can certainly get consensus on some things earlier.  I don't see the 
need to commit without review though.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> Writing parsers and emitters should be a required part of writing a 
> qemu control program?

That's what libraries are for.  The thing that you still haven't 
addressed with JSON, is that there isn't a huge community of JSON 
parsers out there.  The only json library available in Fedora 10 for C 
appears to be json-glib which carries a GObject dependency.  Practically 
speaking, if we did use JSON, we would have to provide our own parser 
library.

>> I will commit a patch series that meets these goals.
>
> We have six months so you'll commit some unreviewed patches now?

Where does unreviewed come from?  Why does the emission format have to 
be perfect in order to start refactoring the monitor interfaces?  There 
is a large bit of this work effort that has nothing to do with output 
format.  I don't want to block that work effort.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Stefano Stabellini]

On Tue, 30 Jun 2009, Anthony Liguori wrote:
> Avi Kivity wrote:
> > Writing parsers and emitters should be a required part of writing a 
> > qemu control program?
> 
> That's what libraries are for.  The thing that you still haven't 
> addressed with JSON, is that there isn't a huge community of JSON 
> parsers out there.  The only json library available in Fedora 10 for C 
> appears to be json-glib which carries a GObject dependency.  Practically 
> speaking, if we did use JSON, we would have to provide our own parser 
> library.
> 


I am just re-posting this portion of a week old email to qemu-devel from
Vincent, about C libraries for JSON, because this week he is on vacation:


> it might be missing for C, but lots of language got a maintained and used
> library nowadays. We're using it in ocaml, and i have to say we haven't 
> been disappointed using it, compared to the number of disappointments with
> xmlrpc (and mostly xml in general).
> 
> That said, I'll make a C library even if qemu doesn't end up using it.
> 

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/30/2009 04:30 PM, Anthony Liguori wrote:
> Avi Kivity wrote:
>> Writing parsers and emitters should be a required part of writing a 
>> qemu control program?
>
> That's what libraries are for.  The thing that you still haven't 
> addressed with JSON, is that there isn't a huge community of JSON 
> parsers out there.  The only json library available in Fedora 10 for C 
> appears to be json-glib which carries a GObject dependency.  
> Practically speaking, if we did use JSON, we would have to provide our 
> own parser library.

There are around 100 parsers listed on json.org.  We can make the 
machine protocol optional and dependant on the existence of a json 
parser of our choice, and Fedora will have to package that dependency if 
they want to build the machine protocol.

Clients are free to use a library or write their own parser.  If we 
provided a parser library it would be useless for all but one client.  
If we support a common format, clients can reuse an existing library.

>>> I will commit a patch series that meets these goals.
>>
>> We have six months so you'll commit some unreviewed patches now?
>
> Where does unreviewed come from? 

Sorry, I understood "I will commit" as "I will personally write and 
commit", not as "I will commit Luiz's suitably modified patchset".

> Why does the emission format have to be perfect in order to start 
> refactoring the monitor interfaces?  There is a large bit of this work 
> effort that has nothing to do with output format.  I don't want to 
> block that work effort.

It does not, and modifying the internals to support a serializable 
object model can be done in parallel to the json flamewar.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
>> Why does the emission format have to be perfect in order to start 
>> refactoring the monitor interfaces?  There is a large bit of this 
>> work effort that has nothing to do with output format.  I don't want 
>> to block that work effort.
>
> It does not, and modifying the internals to support a serializable 
> object model can be done in parallel to the json flamewar.

Great.  So Luiz, do you understand and agree with the proposed changes 
to your series?

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Luiz Capitulino]

On Tue, 30 Jun 2009 08:56:26 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> Avi Kivity wrote:
> >> Why does the emission format have to be perfect in order to start 
> >> refactoring the monitor interfaces?  There is a large bit of this 
> >> work effort that has nothing to do with output format.  I don't want 
> >> to block that work effort.
> >
> > It does not, and modifying the internals to support a serializable 
> > object model can be done in parallel to the json flamewar.
> 
> Great.  So Luiz, do you understand and agree with the proposed changes 
> to your series?

 Yes, only the lists and dictionaries types are not very clear
to me.

 Should command handlers return data in those types to the
monitor and thus get printed by it?

 Detailed is welcome, anyway.

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/30/2009 05:03 PM, Luiz Capitulino wrote:
>> Great.  So Luiz, do you understand and agree with the proposed changes
>> to your series?
>>      
>
>   Yes, only the lists and dictionaries types are not very clear
> to me.
>
>   Should command handlers return data in those types to the
> monitor and thus get printed by it?
>
>   Detailed is welcome, anyway.
>    

I would recommend proceeding as follows:

1. Add QValue, QArray, QDictionary

QValue is a polymorphic object that can be a number, a string, an array, 
or a dictionary.  It needs a bunch of accessors like qvalue_to_qarray() 
to check if a value is an array, and things like qarray_get() or 
qdict_put() to manipulate them.  For the simpler data types, 
qvalue_to_int64() and qvalue_from_int64() should suffice.

2. Split the human monitor command implementations into protocol 
adapters and implementation.

The protocol adapter reads the command, parses it according to 
qemu-monitor.hx, and constructs a QArray of the parameter list and 
passes it down to the command implementation.  The command 
implementation returns a QValue, which the protocol adapter formats to 
the human monitor response format.

3. Refine the object model

Instead of passing a QArray for the argument list, pass a QDictionary.  
For arguments that are themselves dictionaries 
(file=blah.img,cache=none), pass them as dictionaries instead of strings 
({file: blah.img, cache: none}).

When the dust settles down on the encoding selection, we simply make the 
machine protocol parser generate QValues and pass these to the 
newly-splitted command implementations.  We pass the return value to an 
emitter of our choice which can convert an arbitrary QValue into a 
string representation.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Anthony Liguori]

Avi Kivity wrote:
> On 06/30/2009 05:03 PM, Luiz Capitulino wrote:
>>> Great.  So Luiz, do you understand and agree with the proposed changes
>>> to your series?
>>>      
>>
>>   Yes, only the lists and dictionaries types are not very clear
>> to me.
>>
>>   Should command handlers return data in those types to the
>> monitor and thus get printed by it?
>>
>>   Detailed is welcome, anyway.
>>    
>
> I would recommend proceeding as follows:
>
> 1. Add QValue, QArray, QDictionary
>
> QValue is a polymorphic object that can be a number, a string, an 
> array, or a dictionary.  It needs a bunch of accessors like 
> qvalue_to_qarray() to check if a value is an array, and things like 
> qarray_get() or qdict_put() to manipulate them.  For the simpler data 
> types, qvalue_to_int64() and qvalue_from_int64() should suffice.

And add a return value to the monitor functions.  I think this is a good 
first step.

Passing a QArray to each monitor function is one way to go.  Another way 
to go is to use something like libffi to do proper dynamic dispatch.  
The later approach has the benefit of requiring less code churn and 
reusing the static type checking.

> 2. Split the human monitor command implementations into protocol 
> adapters and implementation.
>
> The protocol adapter reads the command, parses it according to 
> qemu-monitor.hx, and constructs a QArray of the parameter list and 
> passes it down to the command implementation.  The command 
> implementation returns a QValue, which the protocol adapter formats to 
> the human monitor response format.

One place to start would be to convert something like:

monitor_printf(mon, "file: f=%s,b=%s,c=%d", ...)

to:

value = qlist_create(qstring_create("file"), qdict_create("f", 
qstring_create(..), "b", qstring_create(..), "c", qstring_create(..), 
NULL));

But since this is a very regular pattern, you could also do:

value = qmonlist_create("file",
                                      "f", QVALUE_STRING, ...,
                                      "b", QVALUE_STRING, ...,
                                      "c", QVALUE_INT, ...,
                                      NULL);

You could potentially introduce some serious ninja action by doing 
something like:

value = qvalue_create("[%s, {'f': %s, 'b': %s', 'c': %d}]", ...);

What's cool about this is that you can mark qvalue_create() as having 
printf formatting so GCC will help you with type checking.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file

[Avi Kivity]

On 06/30/2009 09:21 PM, Anthony Liguori wrote:
>> 1. Add QValue, QArray, QDictionary
>>
>> QValue is a polymorphic object that can be a number, a string, an 
>> array, or a dictionary.  It needs a bunch of accessors like 
>> qvalue_to_qarray() to check if a value is an array, and things like 
>> qarray_get() or qdict_put() to manipulate them.  For the simpler data 
>> types, qvalue_to_int64() and qvalue_from_int64() should suffice.
>
>
> And add a return value to the monitor functions.  I think this is a 
> good first step.
>
> Passing a QArray to each monitor function is one way to go.  Another 
> way to go is to use something like libffi to do proper dynamic 
> dispatch.  The later approach has the benefit of requiring less code 
> churn and reusing the static type checking.

If/when we switch to passing dictionaries instead of arrays, this won't 
work.  I also think libffi is a little to exotic for something like the 
monitor.
You could potentially introduce some serious ninja action by doing 
something like:

> value = qvalue_create("[%s, {'f': %s, 'b': %s', 'c': %d}]", ...);

Seriously nifty...

-- 
error compiling committee.c: too many arguments to function