Re: [Qemu-devel] [PATCH V12 02/10] colo-compare: introduce colo compare initialization

[Jason Wang <jasowang@redhat.com>]

On 2016年08月17日 16:10, Zhang Chen wrote:
> This a COLO net ascii figure:
>
>   Primary qemu                                                           Secondary qemu
> +--------------------------------------------------------------+       +----------------------------------------------------------------+
> | +----------------------------------------------------------+ |       |  +-----------------------------------------------------------+ |
> | |                                                          | |       |  |                                                           | |
> | |                        guest                             | |       |  |                        guest                              | |
> | |                                                          | |       |  |                                                           | |
> | +-------^--------------------------+-----------------------+ |       |  +---------------------+--------+----------------------------+ |
> |         |                          |                         |       |                        ^        |                              |
> |         |                          |                         |       |                        |        |                              |
> |         |  +------------------------------------------------------+  |                        |        |                              |
> |netfilter|  |                       |                         |    |  |   netfilter            |        |                              |
> | +----------+ +----------------------------+                  |    |  |  +-----------------------------------------------------------+ |
> | |       |  |                       |      |        out       |    |  |  |                     |        |  filter excute order       | |
> | |       |  |          +-----------------------------+        |    |  |  |                     |        | +------------------->      | |
> | |       |  |          |            |      |         |        |    |  |  |                     |        |   TCP                      | |
> | | +-----+--+-+  +-----v----+ +-----v----+ |pri +----+----+sec|    |  |  | +------------+  +---+----+---v+rewriter++  +------------+ | |
> | | |          |  |          | |          | |in  |         |in |    |  |  | |            |  |        |              |  |            | | |
> | | |  filter  |  |  filter  | |  filter  +------>  colo   <------+ +-------->  filter   +--> adjust |   adjust     +-->   filter   | | |
> | | |  mirror  |  |redirector| |redirector| |    | compare |   |  |    |  | | redirector |  | ack    |   seq        |  | redirector | | |
> | | |          |  |          | |          | |    |         |   |  |    |  | |            |  |        |              |  |            | | |
> | | +----^-----+  +----+-----+ +----------+ |    +---------+   |  |    |  | +------------+  +--------+--------------+  +---+--------+ | |
> | |      |   tx        |   rx           rx  |                  |  |    |  |            tx                        all       |  rx      | |
> | |      |             |                    |                  |  |    |  +-----------------------------------------------------------+ |
> | |      |             +--------------+     |                  |  |    |                                                   |            |
> | |      |   filter excute order      |     |                  |  |    |                                                   |            |
> | |      |  +---------------->        |     |                  |  +--------------------------------------------------------+            |
> | +-----------------------------------------+                  |       |                                                                |
> |        |                            |                        |       |                                                                |
> +--------------------------------------------------------------+       +----------------------------------------------------------------+
>           |guest receive               | guest send
>           |                            |
> +--------+----------------------------v------------------------+
> |                                                              |                          NOTE: filter direction is rx/tx/all
> |                         tap                                  |                          rx:receive packets sent to the netdev
> |                                                              |                          tx:receive packets sent by the netdev
> +--------------------------------------------------------------+

It's better to add a doc under docs to explain this configuration in 
detail on top of this series.

> In COLO-compare, we do packet comparing job.
> Packets coming from the primary char indev will be sent to outdev.
> Packets coming from the secondary char dev will be dropped after comparing.
> colo-comapre need two input chardev and one output chardev:
> primary_in=chardev1-id (source: primary send packet)
> secondary_in=chardev2-id (source: secondary send packet)
> outdev=chardev3-id
>
> usage:
>
> primary:
> -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
> -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
> -chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
> -chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
> -chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
> -chardev socket,id=compare0-0,host=3.3.3.3,port=9001
> -chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
> -chardev socket,id=compare_out0,host=3.3.3.3,port=9005
> -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
> -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
> -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
> -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0
>
> secondary:
> -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown
> -device e1000,netdev=hn0,mac=52:a4:00:12:78:66
> -chardev socket,id=red0,host=3.3.3.3,port=9003
> -chardev socket,id=red1,host=3.3.3.3,port=9004
> -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
> -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
>
> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---
>   net/Makefile.objs  |   1 +
>   net/colo-compare.c | 284 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>   qemu-options.hx    |  39 ++++++++
>   vl.c               |   3 +-
>   4 files changed, 326 insertions(+), 1 deletion(-)
>   create mode 100644 net/colo-compare.c
>
> diff --git a/net/Makefile.objs b/net/Makefile.objs
> index b7c22fd..ba92f73 100644
> --- a/net/Makefile.objs
> +++ b/net/Makefile.objs
> @@ -16,3 +16,4 @@ common-obj-$(CONFIG_NETMAP) += netmap.o
>   common-obj-y += filter.o
>   common-obj-y += filter-buffer.o
>   common-obj-y += filter-mirror.o
> +common-obj-y += colo-compare.o
> diff --git a/net/colo-compare.c b/net/colo-compare.c
> new file mode 100644
> index 0000000..cdc3e0e
> --- /dev/null
> +++ b/net/colo-compare.c
> @@ -0,0 +1,284 @@
> +/*
> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
> + * (a.k.a. Fault Tolerance or Continuous Replication)
> + *
> + * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
> + * Copyright (c) 2016 FUJITSU LIMITED
> + * Copyright (c) 2016 Intel Corporation
> + *
> + * Author: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or
> + * later.  See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/error-report.h"
> +#include "qemu-common.h"
> +#include "qapi/qmp/qerror.h"
> +#include "qapi/error.h"
> +#include "net/net.h"
> +#include "net/vhost_net.h"

Looks unnecessary.

> +#include "qom/object_interfaces.h"
> +#include "qemu/iov.h"
> +#include "qom/object.h"
> +#include "qemu/typedefs.h"
> +#include "net/queue.h"
> +#include "sysemu/char.h"
> +#include "qemu/sockets.h"
> +#include "qapi-visit.h"
> +
> +#define TYPE_COLO_COMPARE "colo-compare"
> +#define COLO_COMPARE(obj) \
> +    OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE)
> +
> +#define COMPARE_READ_LEN_MAX NET_BUFSIZE
> +
> +typedef struct CompareState {
> +    Object parent;
> +
> +    char *pri_indev;
> +    char *sec_indev;
> +    char *outdev;
> +    CharDriverState *chr_pri_in;
> +    CharDriverState *chr_sec_in;
> +    CharDriverState *chr_out;
> +    QTAILQ_ENTRY(CompareState) next;

This looks not used in this series but in commit "colo-compare and 
filter-rewriter work with colo-frame". We'd better delay the introducing 
to that patch.

> +    SocketReadState pri_rs;
> +    SocketReadState sec_rs;
> +} CompareState;
> +
> +typedef struct CompareClass {
> +    ObjectClass parent_class;
> +} CompareClass;
> +
> +typedef struct CompareChardevProps {
> +    bool is_socket;
> +    bool is_unix;
> +} CompareChardevProps;
> +
> +static char *compare_get_pri_indev(Object *obj, Error **errp)
> +{
> +    CompareState *s = COLO_COMPARE(obj);
> +
> +    return g_strdup(s->pri_indev);
> +}
> +
> +static void compare_set_pri_indev(Object *obj, const char *value, Error **errp)
> +{
> +    CompareState *s = COLO_COMPARE(obj);
> +
> +    g_free(s->pri_indev);
> +    s->pri_indev = g_strdup(value);
> +}
> +
> +static char *compare_get_sec_indev(Object *obj, Error **errp)
> +{
> +    CompareState *s = COLO_COMPARE(obj);
> +
> +    return g_strdup(s->sec_indev);
> +}
> +
> +static void compare_set_sec_indev(Object *obj, const char *value, Error **errp)
> +{
> +    CompareState *s = COLO_COMPARE(obj);
> +
> +    g_free(s->sec_indev);
> +    s->sec_indev = g_strdup(value);
> +}
> +
> +static char *compare_get_outdev(Object *obj, Error **errp)
> +{
> +    CompareState *s = COLO_COMPARE(obj);
> +
> +    return g_strdup(s->outdev);
> +}
> +
> +static void compare_set_outdev(Object *obj, const char *value, Error **errp)
> +{
> +    CompareState *s = COLO_COMPARE(obj);
> +
> +    g_free(s->outdev);
> +    s->outdev = g_strdup(value);
> +}
> +
> +static void compare_pri_rs_finalize(SocketReadState *pri_rs)
> +{
> +    /* if packet_enqueue pri pkt failed we will send unsupported packet */
> +}
> +
> +static void compare_sec_rs_finalize(SocketReadState *sec_rs)
> +{
> +    /* if packet_enqueue sec pkt failed we will notify trace */
> +}
> +
> +static int compare_chardev_opts(void *opaque,
> +                                const char *name, const char *value,
> +                                Error **errp)
> +{
> +    CompareChardevProps *props = opaque;
> +
> +    if (strcmp(name, "backend") == 0 && strcmp(value, "socket") == 0) {
> +        props->is_socket = true;
> +    } else if (strcmp(name, "host") == 0) {

Typo? net_vhost_chardev_opts() did:

     } else if (strcmp(name, "path") == 0) {
         props->is_unix = true;
     }



> +        props->is_unix = true;
> +    } else if (strcmp(name, "port") == 0) {
> +    } else if (strcmp(name, "server") == 0) {
> +    } else if (strcmp(name, "wait") == 0) {
> +    } else {
> +        error_setg(errp,
> +                   "COLO-compare does not support a chardev with option %s=%s",
> +                   name, value);
> +        return -1;
> +    }
> +    return 0;
> +}
> +
> +/*
> + * called from the main thread on the primary
> + * to setup colo-compare.
> + */
> +static void colo_compare_complete(UserCreatable *uc, Error **errp)
> +{
> +    CompareState *s = COLO_COMPARE(uc);
> +    CompareChardevProps props;
> +
> +    if (!s->pri_indev || !s->sec_indev || !s->outdev) {
> +        error_setg(errp, "colo compare needs 'primary_in' ,"
> +                   "'secondary_in','outdev' property set");
> +        return;
> +    } else if (!strcmp(s->pri_indev, s->outdev) ||
> +               !strcmp(s->sec_indev, s->outdev) ||
> +               !strcmp(s->pri_indev, s->sec_indev)) {
> +        error_setg(errp, "'indev' and 'outdev' could not be same "
> +                   "for compare module");
> +        return;
> +    }
> +
> +    s->chr_pri_in = qemu_chr_find(s->pri_indev);
> +    if (s->chr_pri_in == NULL) {
> +        error_setg(errp, "Primary IN Device '%s' not found",
> +                   s->pri_indev);
> +        return;
> +    }
> +
> +    /* inspect chardev opts */
> +    memset(&props, 0, sizeof(props));
> +    if (qemu_opt_foreach(s->chr_pri_in->opts, compare_chardev_opts, &props, errp)) {
> +        return;
> +    }
> +
> +    if (!props.is_socket || !props.is_unix) {
> +        error_setg(errp, "chardev \"%s\" is not a unix socket",
> +                   s->pri_indev);
> +        return;
> +    }
> +
> +    s->chr_sec_in = qemu_chr_find(s->sec_indev);
> +    if (s->chr_sec_in == NULL) {
> +        error_setg(errp, "Secondary IN Device '%s' not found",
> +                   s->sec_indev);
> +        return;
> +    }
> +
> +    memset(&props, 0, sizeof(props));
> +    if (qemu_opt_foreach(s->chr_sec_in->opts, compare_chardev_opts, &props, errp)) {
> +        return;
> +    }
> +
> +    if (!props.is_socket || !props.is_unix) {
> +        error_setg(errp, "chardev \"%s\" is not a unix socket",
> +                   s->sec_indev);

I believe tcp socket is also supported?

> +        return;
> +    }
> +
> +    s->chr_out = qemu_chr_find(s->outdev);
> +    if (s->chr_out == NULL) {
> +        error_setg(errp, "OUT Device '%s' not found", s->outdev);
> +        return;
> +    }
> +
> +    memset(&props, 0, sizeof(props));
> +    if (qemu_opt_foreach(s->chr_out->opts, compare_chardev_opts, &props, errp)) {
> +        return;
> +    }
> +
> +    if (!props.is_socket || !props.is_unix) {
> +        error_setg(errp, "chardev \"%s\" is not a unix socket",
> +                   s->outdev);

Ditto, and there's code duplication, please introduce a helper to do above.

> +        return;
> +    }
> +
> +    qemu_chr_fe_claim_no_fail(s->chr_pri_in);
> +
> +    qemu_chr_fe_claim_no_fail(s->chr_sec_in);
> +
> +    qemu_chr_fe_claim_no_fail(s->chr_out);
> +
> +    net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize);
> +    net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize);
> +
> +    return;
> +}
> +
> +static void colo_compare_class_init(ObjectClass *oc, void *data)
> +{
> +    UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc);
> +
> +    ucc->complete = colo_compare_complete;
> +}
> +
> +static void colo_compare_init(Object *obj)
> +{
> +    object_property_add_str(obj, "primary_in",
> +                            compare_get_pri_indev, compare_set_pri_indev,
> +                            NULL);
> +    object_property_add_str(obj, "secondary_in",
> +                            compare_get_sec_indev, compare_set_sec_indev,
> +                            NULL);
> +    object_property_add_str(obj, "outdev",
> +                            compare_get_outdev, compare_set_outdev,
> +                            NULL);
> +}
> +
> +static void colo_compare_finalize(Object *obj)
> +{
> +    CompareState *s = COLO_COMPARE(obj);
> +
> +    if (s->chr_pri_in) {
> +        qemu_chr_add_handlers(s->chr_pri_in, NULL, NULL, NULL, NULL);
> +        qemu_chr_fe_release(s->chr_pri_in);
> +    }
> +    if (s->chr_sec_in) {
> +        qemu_chr_add_handlers(s->chr_sec_in, NULL, NULL, NULL, NULL);
> +        qemu_chr_fe_release(s->chr_sec_in);
> +    }
> +    if (s->chr_out) {
> +        qemu_chr_fe_release(s->chr_out);
> +    }
> +
> +    g_free(s->pri_indev);
> +    g_free(s->sec_indev);
> +    g_free(s->outdev);
> +}
> +
> +static const TypeInfo colo_compare_info = {
> +    .name = TYPE_COLO_COMPARE,
> +    .parent = TYPE_OBJECT,
> +    .instance_size = sizeof(CompareState),
> +    .instance_init = colo_compare_init,
> +    .instance_finalize = colo_compare_finalize,
> +    .class_size = sizeof(CompareClass),
> +    .class_init = colo_compare_class_init,
> +    .interfaces = (InterfaceInfo[]) {
> +        { TYPE_USER_CREATABLE },
> +        { }
> +    }
> +};
> +
> +static void register_types(void)
> +{
> +    type_register_static(&colo_compare_info);
> +}
> +
> +type_init(register_types);
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 587de8f..33d5d0b 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -3866,6 +3866,45 @@ Dump the network traffic on netdev @var{dev} to the file specified by
>   The file format is libpcap, so it can be analyzed with tools such as tcpdump
>   or Wireshark.
>   
> +@item -object colo-compare,id=@var{id},primary_in=@var{chardevid},secondary_in=@var{chardevid},
> +outdev=@var{chardevid}
> +
> +Colo-compare gets packet from primary_in@var{chardevid} and secondary_in@var{chardevid}, than compare primary packet with
> +secondary packet. If the packet same, we will output primary

s/If the packet same/If the packets are same/.

> +packet to outdev@var{chardevid}, else we will notify colo-frame
> +do checkpoint and send primary packet to outdev@var{chardevid}.
> +
> +we can use it with the help of filter-mirror and filter-redirector.

s/we/We/ and looks like colo compare must be used with the help of 
mirror and redirector?

> +
> +@example
> +
> +primary:
> +-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
> +-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
> +-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
> +-chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
> +-chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
> +-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
> +-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
> +-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
> +-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
> +-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
> +-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
> +-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0
> +
> +secondary:
> +-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown
> +-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
> +-chardev socket,id=red0,host=3.3.3.3,port=9003
> +-chardev socket,id=red1,host=3.3.3.3,port=9004
> +-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
> +-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
> +
> +@end example
> +
> +If you want to know the detail of above command line, you can read
> +the colo-compare git log.
> +
>   @item -object secret,id=@var{id},data=@var{string},format=@var{raw|base64}[,keyid=@var{secretid},iv=@var{string}]
>   @item -object secret,id=@var{id},file=@var{filename},format=@var{raw|base64}[,keyid=@var{secretid},iv=@var{string}]
>   
> diff --git a/vl.c b/vl.c
> index cbe51ac..c6b9a6f 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -2865,7 +2865,8 @@ static bool object_create_initial(const char *type)
>       if (g_str_equal(type, "filter-buffer") ||
>           g_str_equal(type, "filter-dump") ||
>           g_str_equal(type, "filter-mirror") ||
> -        g_str_equal(type, "filter-redirector")) {
> +        g_str_equal(type, "filter-redirector") ||
> +        g_str_equal(type, "colo-compare")) {
>           return false;
>       }
>