[Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Baptiste Reynal]

A new memory backend, the shared memory backend, based on
the file memory backend.

This new backend allows a master QEMU instance to share a part of
his main memory whith a slave QEMU instance. It is then possible to load
a firmware on this memory and trigger the slave boot using a SDM
signal.

Such new backend enables, on a master side, to allocate the whole
memory as shareable (e.g. /dev/shm, or hugetlbfs).
On the slave side it enables the startup of QEMU without any main memory
allocated. Then the slave goes in a waiting state, the same used in the
case of an incoming migration, and the file descriptor of the shared
memory is sent by the master over a socket.
The waiting state ends when the master sends to the slave a signal
with size and offset to mmap and use as memory as payload.

You can test those patches using modules and test application available
on https://git.virtualopensystems.com/dev/qemu-het-tools.
- SDM Platform module on branch sdm_test_mod_v1
- Test application on branch sdm_test_app
- Slave files on branch demo_slaves_sdm

 QEMU code is avalaible here https://git.virtualopensystems.com/dev/qemu on branch rfc_v2.

Run QEMU master instance:
./qemu-system-arm -nographic \
        -kernel zImage \
        -M virt -m 1G \
        -initrd busybox.cpio \
        -object multi-socket-backend,id=sock,path=sock,listen \
        -object memory-backend-shared,id=mem,size=1G,mem-path=/dev/hugepages,socket=sock,master \
        -object sdm-signal-shboot,id=boot,shm=mem \
        -object sdm-communication-socket,id=scomm,socket=sock \
        -device virtio-sdm-device,comm=scomm,master,num-slaves=1,len-signals=1,signals[0]=boot \
        -append "mem=512M memmap=512M$0x60000000" \
        -numa node,memdev=mem -m 1G

Run QEMU slave instance:
./qemu-system-arm -nographic \
        -kernel zImage \
        -M virt -m 1G \
        -initrd busybox.cpio \
        -object multi-socket-backend,id=sock,path=sock \
        -object memory-backend-shared,id=mem,size=512M,socket=sock \
        -object sdm-signal-shboot,id=boot,shm=mem \
        -object sdm-communication-socket,id=scomm,socket=sock \
        -device sdm-platform,comm=scomm,len-signals=1,signals[0]="boot" \
        -incoming "shared:mem" \
        -numa node,memdev=mem -m 512M

On the master, load the SDM module (insmod sdm_test_mod_v1) and trigger the boot with
the following command:
./sdm-test -b 0 1 zImage slave.dtb

zImage and slave.dtb are available on branch demo_slaves_sdm.

This patch serie is a follow-up to "[RFC PATCH 0/8] Towards an Heterogeneous QEMU":
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg00171.html

This patch serie depends on:
- [RFC v2 0/6] SDM Interface
- [RFC v2 1/1] backend: multi-client-socket

This work has been sponsored by Huawei Technologies Duesseldorf GmbH.

Baptiste Reynal (3):
  backend: shared memory backend
  migration: add shared migration type
  hw/misc: sdm signal shboot

Christian Pinto (3):
  qemu: slave machine flag
  hw/arm: boot
  qemu: numa

 backends/Makefile.objs              |   2 +-
 backends/hostmem-shared.c           | 198 ++++++++++++++++++++++++++++++++++++
 hw/arm/boot.c                       |  13 +++
 hw/core/machine.c                   |  27 +++++
 hw/misc/Makefile.objs               |   1 +
 hw/misc/sdm-signal-shboot.c         |  62 +++++++++++
 include/hw/boards.h                 |   2 +
 include/hw/misc/sdm-signal-shboot.h |  38 +++++++
 include/migration/migration.h       |   2 +
 include/sysemu/hostmem-shared.h     |  66 ++++++++++++
 migration/Makefile.objs             |   2 +-
 migration/migration.c               |   2 +
 migration/shared.c                  |  33 ++++++
 numa.c                              |  17 +++-
 qemu-options.hx                     |   5 +-
 util/qemu-config.c                  |   5 +
 16 files changed, 471 insertions(+), 4 deletions(-)
 create mode 100644 backends/hostmem-shared.c
 create mode 100644 hw/misc/sdm-signal-shboot.c
 create mode 100644 include/hw/misc/sdm-signal-shboot.h
 create mode 100644 include/sysemu/hostmem-shared.h
 create mode 100644 migration/shared.c

-- 
2.7.3

[Qemu-devel] [RFC v2 1/6] backend: shared memory backend

[Baptiste Reynal]

This patch introduces a shared memory backend, allowing to share memory
between a master and many slaves.

The memory is implemented using hugetlbfs, and relies on the
multi-socket backend to share informations (size and offset for the
slaves).

Instantiation on the master:
-object memory-backend-shared,id=<id>,size=<memory_sizeK,M,G>,
	chardev=<multi-socket_id>,master=on

Instantiation on the slave:
-object memory-backend-shared,id=<id>,size=<slave_memory_sizeK,M,G>,
	chardev=<multi-socket_id>,master=off

Memory size on the slave can be smaller than on master. The master will
send to the slave the size and the offset of the memory segment it can
allocate in the total shared memory.

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 backends/Makefile.objs          |   2 +-
 backends/hostmem-shared.c       | 192 ++++++++++++++++++++++++++++++++++++++++
 include/sysemu/hostmem-shared.h |  66 ++++++++++++++
 3 files changed, 259 insertions(+), 1 deletion(-)
 create mode 100644 backends/hostmem-shared.c
 create mode 100644 include/sysemu/hostmem-shared.h

diff --git a/backends/Makefile.objs b/backends/Makefile.objs
index 689eac3..de76906 100644
--- a/backends/Makefile.objs
+++ b/backends/Makefile.objs
@@ -8,6 +8,6 @@ baum.o-cflags := $(SDL_CFLAGS)
 common-obj-$(CONFIG_TPM) += tpm.o
 
 common-obj-y += hostmem.o hostmem-ram.o
-common-obj-$(CONFIG_LINUX) += hostmem-file.o
+common-obj-$(CONFIG_LINUX) += hostmem-file.o hostmem-shared.o
 
 common-obj-y += multi-socket.o
diff --git a/backends/hostmem-shared.c b/backends/hostmem-shared.c
new file mode 100644
index 0000000..b45a87b
--- /dev/null
+++ b/backends/hostmem-shared.c
@@ -0,0 +1,192 @@
+/*
+ * QEMU Host Memory Backend for hugetlbfs
+ *
+ * Copyright (C) 2015 - Virtual Open Systems
+ *
+ * Author: Baptiste Reynal <b.reynal@virtualopensystems.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#include "sysemu/hostmem-shared.h"
+
+static void shm_map(HostMemoryBackendShared *shm, size_t size, off_t offset)
+{
+    void *shared_ram;
+    HostMemoryBackend *backend = MEMORY_BACKEND(shm);
+
+    shared_ram = mmap(0, size, PROT_READ | PROT_WRITE,
+            MAP_SHARED, shm->fd, offset);
+    close(shm->fd);
+
+    if (shared_ram == MAP_FAILED) {
+        perror("Map failed");
+    }
+
+    memory_region_init_ram_ptr(&shm->shared_region, OBJECT(backend),
+            "shared_mem", size, shared_ram);
+
+    memory_region_add_subregion(&backend->mr,
+            0, &shm->shared_region);
+
+    vmstate_register_ram_global(&shm->shared_region);
+
+    event_notifier_set(shm->levent);
+}
+
+/* Callback function if a fd is received over the socket */
+static void set_shared_memory(MSClient *c, const char *message, void *opaque)
+{
+    HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(opaque);
+
+    int fd = 0;
+    multi_socket_get_fds_from(c, &fd);
+
+    if (fd <= 0) {
+        printf("Error receiving fd: %d", fd);
+        exit(-1);
+    }
+
+    shm->fd = fd;
+}
+
+static void shm_send_fd(MSClient *client, void *opaque)
+{
+    int fd;
+    const char *message = "send_fd";
+    HostMemoryBackend *hm = MEMORY_BACKEND(opaque);
+
+    fd = memory_region_get_fd(&hm->mr);
+
+    multi_socket_send_fds_to(client, &fd, 1, message, strlen(message) + 1);
+}
+
+static void
+shared_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
+{
+    HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(backend);
+
+    if (!backend->size) {
+        error_setg(errp, "can't create backend with size 0");
+        return;
+    }
+
+    if (!memory_region_size(&backend->mr)) {
+        if (shm->master) {
+            backend->force_prealloc = mem_prealloc;
+            memory_region_init_ram_from_file(&backend->mr, OBJECT(backend),
+                    object_get_canonical_path(OBJECT(backend)),
+                    backend->size, true,
+                    shm->mem_path, errp);
+            multi_socket_add_reg_handler(shm->ms, shm_send_fd, shm);
+        } else {
+            backend->force_prealloc = mem_prealloc;
+
+            /*
+             * Initialize only the main fields
+             * the rest is initialized when the fd is received
+            */
+            memory_region_init(&backend->mr, OBJECT(backend),
+                    object_get_canonical_path(OBJECT(backend)),
+                    backend->size);
+
+            multi_socket_add_handler(shm->ms, "send_fd",
+                    set_shared_memory, shm);
+        }
+    }
+}
+
+static void
+shared_memory_backend_complete(UserCreatable *uc, Error **errp)
+{
+    HostMemoryBackend *hm = MEMORY_BACKEND(uc);
+    HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(uc);
+    HostMemoryBackendClass *bc = MEMORY_BACKEND_GET_CLASS(uc);
+    HostMemoryBackendSharedClass *bsc = MEMORY_BACKEND_SHARED_GET_CLASS(uc);
+
+    if (shm->master) {
+        bsc->parent_complete(uc, errp);
+    } else {
+        bc->alloc(hm, errp);
+    }
+}
+
+static void shared_backend_class_init(ObjectClass *oc, void *data)
+{
+    UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc);
+    HostMemoryBackendClass *bc = MEMORY_BACKEND_CLASS(oc);
+    HostMemoryBackendSharedClass *bsc = MEMORY_BACKEND_SHARED_CLASS(oc);
+
+    bc->alloc = shared_backend_memory_alloc;
+    bsc->parent_complete = ucc->complete;
+    bsc->map = shm_map;
+    ucc->complete = shared_memory_backend_complete;
+}
+
+static char *get_mem_path(Object *o, Error **errp)
+{
+    HostMemoryBackendShared *backend = MEMORY_BACKEND_SHARED(o);
+
+    return g_strdup(backend->mem_path);
+}
+
+static void set_mem_path(Object *o, const char *str, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+    HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(o);
+
+    if (memory_region_size(&backend->mr)) {
+        error_setg(errp, "cannot change property value");
+        return;
+    }
+    g_free(shm->mem_path);
+
+    shm->mem_path = g_strdup(str);
+}
+
+static bool get_master(Object *o, Error **errp)
+{
+    HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(o);
+
+    return shm->master;
+}
+
+static void set_master(Object *o, bool value, Error **errp)
+{
+    HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(o);
+
+    shm->master = value;
+}
+
+static void shared_backend_instance_init(Object *o)
+{
+    HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(o);
+
+    object_property_add_bool(o, "master", get_master,
+                        set_master, NULL);
+    object_property_add_str(o, "mem-path", get_mem_path,
+                            set_mem_path, NULL);
+    object_property_add_link(o, "socket", TYPE_MULTI_SOCKET_BACKEND,
+            (Object **)&shm->ms,
+            object_property_allow_set_link,
+            OBJ_PROP_LINK_UNREF_ON_RELEASE,
+            &error_abort);
+}
+
+static const TypeInfo shared_backend_info = {
+    .name = TYPE_MEMORY_BACKEND_SHARED,
+    .parent = TYPE_MEMORY_BACKEND,
+    .class_init = shared_backend_class_init,
+    .class_size = sizeof(HostMemoryBackendSharedClass),
+    .instance_init = shared_backend_instance_init,
+    .instance_size = sizeof(HostMemoryBackendShared),
+};
+
+static void register_types(void)
+{
+    type_register_static(&shared_backend_info);
+}
+
+type_init(register_types);
diff --git a/include/sysemu/hostmem-shared.h b/include/sysemu/hostmem-shared.h
new file mode 100644
index 0000000..7db3b37
--- /dev/null
+++ b/include/sysemu/hostmem-shared.h
@@ -0,0 +1,66 @@
+/*
+ * QEMU Host Memory Backend for hugetlbfs
+ *
+ * Copyright (C) 2015 - Virtual Open Systems
+ *
+ * Author: Baptiste Reynal <b.reynal@virtualopensystems.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_HM_H
+#define QEMU_HM_H
+
+#include "qemu-common.h"
+#include "sysemu/hostmem.h"
+#include "sysemu/sysemu.h"
+#include "qemu/multi-socket.h"
+#include "qom/object_interfaces.h"
+#include "qapi-visit.h"
+
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/statfs.h>
+#include <fcntl.h>
+
+typedef struct HostMemoryBackendShared HostMemoryBackendShared;
+typedef struct HostMemoryBackendSharedClass HostMemoryBackendSharedClass;
+
+struct HostMemoryBackendShared {
+    HostMemoryBackend parent_obj;
+
+    bool master;
+
+    char *mem_path;
+
+    int event;
+    int fd;
+
+    EventNotifier *levent;
+
+    MSBackend *ms;
+    MemoryRegion shared_region;
+};
+
+struct HostMemoryBackendSharedClass {
+    HostMemoryBackendClass parent_class;
+
+    void (*parent_complete)(UserCreatable *uc, Error **errp);
+    void (*map)(HostMemoryBackendShared *shm, size_t size, off_t offset);
+};
+
+#define TYPE_MEMORY_BACKEND_SHARED "memory-backend-shared"
+
+#define MEMORY_BACKEND_SHARED(obj) \
+    OBJECT_CHECK(HostMemoryBackendShared, (obj), TYPE_MEMORY_BACKEND_SHARED)
+#define MEMORY_BACKEND_SHARED_GET_CLASS(obj) \
+    OBJECT_GET_CLASS(HostMemoryBackendSharedClass, (obj), \
+            TYPE_MEMORY_BACKEND_SHARED)
+#define MEMORY_BACKEND_SHARED_CLASS(klass) \
+    OBJECT_CLASS_CHECK(HostMemoryBackendSharedClass, (klass), \
+            TYPE_MEMORY_BACKEND_SHARED)
+#define IS_MEMORY_BACKEND_SHARED(obj) \
+    object_dynamic_cast(OBJECT(obj), TYPE_MEMORY_BACKEND_SHARED)
+
+#endif
-- 
2.7.3

[Qemu-devel] [RFC v2 2/6] migration: add shared migration type

[Baptiste Reynal]

A QEMU instance can now wait for the instantiation of the memory by the
master while shared-memory backend is used.

Use:
-incoming "shared:<shared-memory_id>"

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 backends/hostmem-shared.c     |  6 ++++++
 include/migration/migration.h |  2 ++
 migration/Makefile.objs       |  2 +-
 migration/migration.c         |  2 ++
 migration/shared.c            | 33 +++++++++++++++++++++++++++++++++
 5 files changed, 44 insertions(+), 1 deletion(-)
 create mode 100644 migration/shared.c

diff --git a/backends/hostmem-shared.c b/backends/hostmem-shared.c
index b45a87b..e703c30 100644
--- a/backends/hostmem-shared.c
+++ b/backends/hostmem-shared.c
@@ -11,6 +11,7 @@
  */
 
 #include "sysemu/hostmem-shared.h"
+#include "migration/vmstate.h"
 
 static void shm_map(HostMemoryBackendShared *shm, size_t size, off_t offset)
 {
@@ -96,6 +97,11 @@ shared_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
                     set_shared_memory, shm);
         }
     }
+
+    shm->levent = g_new(EventNotifier, 1);
+    event_notifier_init(shm->levent, 0);
+
+    shm->event = event_notifier_get_fd(shm->levent);
 }
 
 static void
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 8334621..0d4efa5 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -114,6 +114,8 @@ void migrate_fd_connect(MigrationState *s);
 
 int migrate_fd_close(MigrationState *s);
 
+void shared_start_incoming_migration(const char *name, Error **errp);
+
 void add_migration_state_change_notifier(Notifier *notify);
 void remove_migration_state_change_notifier(Notifier *notify);
 bool migration_in_setup(MigrationState *);
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index d929e96..08c96f7 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -7,4 +7,4 @@ common-obj-$(CONFIG_RDMA) += rdma.o
 common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
 
 common-obj-y += block.o
-
+common-obj-y += shared.o
diff --git a/migration/migration.c b/migration/migration.c
index 662e77e..798801c 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -249,6 +249,8 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
         deferred_incoming_migration(errp);
     } else if (strstart(uri, "tcp:", &p)) {
         tcp_start_incoming_migration(p, errp);
+    } else if (strstart(uri, "shared:", &p)) {
+        shared_start_incoming_migration(p, errp);
 #ifdef CONFIG_RDMA
     } else if (strstart(uri, "rdma:", &p)) {
         rdma_start_incoming_migration(p, errp);
diff --git a/migration/shared.c b/migration/shared.c
new file mode 100644
index 0000000..1371a71
--- /dev/null
+++ b/migration/shared.c
@@ -0,0 +1,33 @@
+#include "qemu-common.h"
+#include "qemu/main-loop.h"
+#include "qemu/sockets.h"
+#include "migration/migration.h"
+#include "monitor/monitor.h"
+#include "migration/qemu-file.h"
+#include "block/block.h"
+#include "sysemu/hostmem-shared.h"
+
+static void shared_accept_incoming_migration(void *opaque)
+{
+    QEMUFile *f = opaque;
+    printf("Start !\n");
+
+    qemu_set_fd_handler(qemu_get_fd(f), NULL, NULL, NULL);
+    vm_start();
+}
+
+void shared_start_incoming_migration(const char *id, Error **errp)
+{
+    HostMemoryBackendShared *shm = (HostMemoryBackendShared *)
+        object_resolve_path_type(id, TYPE_MEMORY_BACKEND_SHARED, NULL);
+    QEMUFile *f;
+
+    if (shm == NULL) {
+        printf("Error: Cannot find shared memory %s\n", id);
+        exit(-1);
+    }
+
+    f = qemu_fdopen(shm->event, "rb");
+
+    qemu_set_fd_handler(shm->event, shared_accept_incoming_migration, NULL, f);
+}
-- 
2.7.3

[Qemu-devel] [RFC v2 3/6] hw/misc: sdm signal shboot

[Baptiste Reynal]

This patch introduces a new signal for SDM device, which triggers the
boot of a machine using a shared memory.

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 hw/misc/Makefile.objs               |  1 +
 hw/misc/sdm-signal-shboot.c         | 62 +++++++++++++++++++++++++++++++++++++
 include/hw/misc/sdm-signal-shboot.h | 38 +++++++++++++++++++++++
 3 files changed, 101 insertions(+)
 create mode 100644 hw/misc/sdm-signal-shboot.c
 create mode 100644 include/hw/misc/sdm-signal-shboot.h

diff --git a/hw/misc/Makefile.objs b/hw/misc/Makefile.objs
index b9f75db..20a7d82 100644
--- a/hw/misc/Makefile.objs
+++ b/hw/misc/Makefile.objs
@@ -27,6 +27,7 @@ obj-$(CONFIG_SDM) += sdm-signal.o
 obj-$(CONFIG_SDM) += sdm-platform.o
 obj-$(CONFIG_SDM) += sdm-communication-local.o
 obj-$(CONFIG_SDM) += sdm-communication-socket.o
+obj-$(CONFIG_SDM) += sdm-signal-shboot.o
 
 obj-$(CONFIG_REALVIEW) += arm_sysctl.o
 obj-$(CONFIG_NSERIES) += cbus.o
diff --git a/hw/misc/sdm-signal-shboot.c b/hw/misc/sdm-signal-shboot.c
new file mode 100644
index 0000000..4664e31
--- /dev/null
+++ b/hw/misc/sdm-signal-shboot.c
@@ -0,0 +1,62 @@
+/*
+ * SDM Signal Shared Boot
+ *
+ * Copyright (C) 2016 - Virtual Open Systems
+ *
+ * Author: Baptiste Reynal <b.reynalb@virtualopensystems.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+#include "hw/misc/sdm-signal-shboot.h"
+
+static int sdm_signal_shboot_hw_ops(SDMSignal *signal, SDMSignalData *data)
+{
+    SDMSignalShBoot *shb = SDM_SIGNAL_SHBOOT(signal);
+    HostMemoryBackendSharedClass *shmc =
+        MEMORY_BACKEND_SHARED_GET_CLASS(shb->shm);
+
+    shmc->map(shb->shm, data->payload[0], data->payload[1]);
+
+    return 0;
+}
+
+static bool sdm_signal_shboot_hw_only(SDMSignal *signal)
+{
+    return true;
+}
+
+static void sdm_signal_shboot_init(Object *obj)
+{
+    SDMSignalShBoot *signal = SDM_SIGNAL_SHBOOT(obj);
+
+    object_property_add_link(obj, "shm",
+            TYPE_MEMORY_BACKEND_SHARED,
+            (Object **)&signal->shm,
+            object_property_allow_set_link,
+            OBJ_PROP_LINK_UNREF_ON_RELEASE,
+            &error_abort);
+}
+
+static void sdm_signal_shboot_class_init(ObjectClass *oc, void *data)
+{
+    SDMSignalClass *signalc = SDM_SIGNAL_CLASS(oc);
+
+    signalc->hw_ops = sdm_signal_shboot_hw_ops;
+    signalc->hw_only = sdm_signal_shboot_hw_only;
+}
+
+static const TypeInfo sdm_signal_shboot_info = {
+    .name = TYPE_SDM_SIGNAL_SHBOOT,
+    .parent = TYPE_SDM_SIGNAL,
+    .class_init = sdm_signal_shboot_class_init,
+    .instance_size = sizeof(SDMSignalShBoot),
+    .instance_init = sdm_signal_shboot_init,
+};
+
+static void register_types(void)
+{
+    type_register_static(&sdm_signal_shboot_info);
+}
+
+type_init(register_types);
diff --git a/include/hw/misc/sdm-signal-shboot.h b/include/hw/misc/sdm-signal-shboot.h
new file mode 100644
index 0000000..37fd58b
--- /dev/null
+++ b/include/hw/misc/sdm-signal-shboot.h
@@ -0,0 +1,38 @@
+/*
+ * SDM Signal Shared Boot
+ *
+ * Copyright (C) 2016 - Virtual Open Systems
+ *
+ * Author: Baptiste Reynal <b.reynal@virtualopensystems.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ *
+ * This signal triggers the boot of a shared memory backend. It is intended to
+ * find the size in PAYLOAD_REG0 and the offset in PAYLOAD_REG1.
+ */
+#ifndef HW_SDM_SIGNAL_SHBOOT_H
+#define HW_SDM_SIGNAL_SHBOOT_H
+
+#include "hw/misc/sdm-signal.h"
+#include "sysemu/hostmem-shared.h"
+
+#define TYPE_SDM_SIGNAL_SHBOOT "sdm-signal-shboot"
+#define SDM_SIGNAL_SHBOOT(obj) \
+    OBJECT_CHECK(SDMSignalShBoot, (obj), TYPE_SDM_SIGNAL)
+
+typedef struct SDMSignalShBoot SDMSignalShBoot;
+
+/**
+ *
+ * @SDMSignalShBoot
+ *
+ * @parent: opaque parent object container
+ */
+struct SDMSignalShBoot {
+    /* private */
+    SDMSignal parent;
+
+    HostMemoryBackendShared *shm;
+};
+#endif
-- 
2.7.3

[Qemu-devel] [RFC v2 4/6] qemu: slave machine flag

[Baptiste Reynal]

From: Christian Pinto <c.pinto@virtualopensystems.com>

This patch adds a new machine flag, to configure qemu as a slave instance.

Usage

-machine -slave=[on|off] (default=off)

Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
---
 hw/core/machine.c   | 27 +++++++++++++++++++++++++++
 include/hw/boards.h |  2 ++
 qemu-options.hx     |  5 ++++-
 util/qemu-config.c  |  5 +++++
 4 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/hw/core/machine.c b/hw/core/machine.c
index f4db340..b3e1e28 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -283,6 +283,20 @@ static bool machine_get_suppress_vmdesc(Object *obj, Error **errp)
     return ms->suppress_vmdesc;
 }
 
+static bool machine_get_slave(Object *obj, Error **errp)
+{
+    MachineState *ms = MACHINE(obj);
+
+    return ms->slave;
+}
+
+static void machine_set_slave(Object *obj, bool value, Error **errp)
+{
+    MachineState *ms = MACHINE(obj);
+
+    ms->slave = value;
+}
+
 static int error_on_sysbus_device(SysBusDevice *sbdev, void *opaque)
 {
     error_report("Option '-device %s' cannot be handled by this machine",
@@ -335,6 +349,7 @@ static void machine_initfn(Object *obj)
     ms->kvm_shadow_mem = -1;
     ms->dump_guest_core = true;
     ms->mem_merge = true;
+    ms->slave = false;
 
     object_property_add_str(obj, "accel",
                             machine_get_accel, machine_set_accel, NULL);
@@ -437,6 +452,13 @@ static void machine_initfn(Object *obj)
     object_property_set_description(obj, "suppress-vmdesc",
                                     "Set on to disable self-describing migration",
                                     NULL);
+    object_property_add_bool(obj, "slave",
+                             machine_get_slave,
+                             machine_set_slave,
+                             NULL);
+    object_property_set_description(obj, "slave",
+                                    "Enables a slave (remote) machine instance",
+                                    NULL);
 
     /* Register notifier when init is done for sysbus sanity checks */
     ms->sysbus_notifier.notify = machine_init_notify;
@@ -497,6 +519,11 @@ bool machine_mem_merge(MachineState *machine)
     return machine->mem_merge;
 }
 
+bool machine_slave(MachineState *machine)
+{
+    return machine->slave;
+}
+
 static const TypeInfo machine_info = {
     .name = TYPE_MACHINE,
     .parent = TYPE_OBJECT,
diff --git a/include/hw/boards.h b/include/hw/boards.h
index 3e9a92c..523cfc2 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -40,6 +40,7 @@ int machine_kvm_shadow_mem(MachineState *machine);
 int machine_phandle_start(MachineState *machine);
 bool machine_dump_guest_core(MachineState *machine);
 bool machine_mem_merge(MachineState *machine);
+bool machine_slave(MachineState *machine);
 
 /**
  * MachineClass:
@@ -120,6 +121,7 @@ struct MachineState {
     char *firmware;
     bool iommu;
     bool suppress_vmdesc;
+    bool slave;
 
     ram_addr_t ram_size;
     ram_addr_t maxram_size;
diff --git a/qemu-options.hx b/qemu-options.hx
index 7e147b8..96300ba 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -41,7 +41,8 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
     "                igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n"
     "                aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n"
     "                dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n"
-    "                suppress-vmdesc=on|off disables self-describing migration (default=off)\n",
+    "                suppress-vmdesc=on|off disables self-describing migration (default=off)\n"
+    "                slave=on|off enables a slave (remote) machine instance (default=off)",
     QEMU_ARCH_ALL)
 STEXI
 @item -machine [type=]@var{name}[,prop=@var{value}[,...]]
@@ -80,6 +81,8 @@ execution of AES cryptographic functions.  The default is on.
 Enables or disables DEA key wrapping support on s390-ccw hosts. This feature
 controls whether DEA wrapping keys will be created to allow
 execution of DEA cryptographic functions.  The default is on.
+@item slave=on|off
+Enables a slave (remote) machine instance
 @end table
 ETEXI
 
diff --git a/util/qemu-config.c b/util/qemu-config.c
index 5fcfd0e..696408d 100644
--- a/util/qemu-config.c
+++ b/util/qemu-config.c
@@ -219,6 +219,11 @@ static QemuOptsList machine_opts = {
             .name = "suppress-vmdesc",
             .type = QEMU_OPT_BOOL,
             .help = "Set on to disable self-describing migration",
+        },{
+            .name = "slave",
+            .type = QEMU_OPT_BOOL,
+            .help = "Indicates the machine is a slave instance "
+                    "(e.g. remoteproc)",
         },
         { /* End of list */ }
     }
-- 
2.7.3

[Qemu-devel] [RFC v2 5/6] hw/arm: boot

[Baptiste Reynal]

From: Christian Pinto <c.pinto@virtualopensystems.com>

This patch modifies the boot process of an ARM machine in otrder to check
whether if it is a slave, by checking the slave machine flag.

When the slave flag is on, no kernel, dtb or initrd are loaded into memory.
The boot address of each core is set to the start address of the RAM,
that depends on the machine model executed.

Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
---
 hw/arm/boot.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/hw/arm/boot.c b/hw/arm/boot.c
index bef451b..ee0c4a1 100644
--- a/hw/arm/boot.c
+++ b/hw/arm/boot.c
@@ -590,6 +590,19 @@ static void arm_load_kernel_notify(Notifier *notifier, void *data)
     /* Load the kernel.  */
     if (!info->kernel_filename || info->firmware_loaded) {
 
+        if (!info->kernel_filename && machine_slave(current_machine)) {
+            /* If a machine is booted as a slave instance there is no need to
+             * provide the DTB blob or kernel image, that will instead
+             * be copied into memory later by a master instance.
+             * The boot address is set to be at the beginning of the RAM.
+             */
+            info->entry = info->loader_start;
+            CPU_FOREACH(cs) {
+                ARM_CPU(cs)->env.boot_info = info;
+            }
+            return;
+        }
+
         if (have_dtb(info)) {
             /* If we have a device tree blob, but no kernel to supply it to (or
              * the kernel is supposed to be loaded by the bootloader), copy the
-- 
2.7.3

[Qemu-devel] [RFC v2 6/6] qemu: numa

[Baptiste Reynal]

From: Christian Pinto <c.pinto@virtualopensystems.com>

This patch modifies the behavior of
memory_region_allocate_system_memory, when the new shared memory backend
is used from a slave qemu instance.
In such case there is not yet a valid pointer for the memory region pointed
by the backend (it will be innitilized later when received from the master.)
and vmstate_register would fail.

The patch skips the call to vmstate_register in case of slave shared memory
backend, that will be performed later after the actual memory pointer is
available.

Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
---
 numa.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/numa.c b/numa.c
index e9b18f5..b39a892 100644
--- a/numa.c
+++ b/numa.c
@@ -33,6 +33,7 @@
 #include "qapi/dealloc-visitor.h"
 #include "hw/boards.h"
 #include "sysemu/hostmem.h"
+#include "sysemu/hostmem-shared.h"
 #include "qmp-commands.h"
 #include "hw/mem/pc-dimm.h"
 #include "qemu/option.h"
@@ -442,6 +443,7 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner,
 {
     uint64_t addr = 0;
     int i;
+    bool vmstate_register = true;
 
     if (nb_numa_nodes == 0 || !have_memdevs) {
         allocate_system_memory_nonnuma(mr, owner, name, ram_size);
@@ -453,9 +455,18 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner,
         Error *local_err = NULL;
         uint64_t size = numa_info[i].node_mem;
         HostMemoryBackend *backend = numa_info[i].node_memdev;
+
         if (!backend) {
             continue;
         }
+
+        if (IS_MEMORY_BACKEND_SHARED(backend)) {
+            HostMemoryBackendShared *shm = MEMORY_BACKEND_SHARED(backend);
+            if (!shm->master) {
+                vmstate_register = false;
+            }
+        }
+
         MemoryRegion *seg = host_memory_backend_get_memory(backend, &local_err);
         if (local_err) {
             error_report_err(local_err);
@@ -471,7 +482,11 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner,
         }
 
         memory_region_add_subregion(mr, addr, seg);
-        vmstate_register_ram_global(seg);
+
+        if (vmstate_register) {
+            vmstate_register_ram_global(seg);
+        }
+
         addr += size;
     }
 }
-- 
2.7.3

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Markus Armbruster]

Copying Paolo, author of memory-backend-file.

Baptiste Reynal <b.reynal@virtualopensystems.com> writes:

> A new memory backend, the shared memory backend, based on
> the file memory backend.
>
> This new backend allows a master QEMU instance to share a part of
> his main memory whith a slave QEMU instance. It is then possible to load
> a firmware on this memory and trigger the slave boot using a SDM
> signal.

Can you explain why that's useful in a bit more detail?

> Such new backend enables, on a master side, to allocate the whole
> memory as shareable (e.g. /dev/shm, or hugetlbfs).

memory-backend-file can do that already.  What exactly does
memory-backend-shared add to it?

> On the slave side it enables the startup of QEMU without any main memory
> allocated. Then the slave goes in a waiting state, the same used in the
> case of an incoming migration, and the file descriptor of the shared
> memory is sent by the master over a socket.
> The waiting state ends when the master sends to the slave a signal
> with size and offset to mmap and use as memory as payload.
>
> You can test those patches using modules and test application available
> on https://git.virtualopensystems.com/dev/qemu-het-tools.
> - SDM Platform module on branch sdm_test_mod_v1
> - Test application on branch sdm_test_app
> - Slave files on branch demo_slaves_sdm
>
>  QEMU code is avalaible here https://git.virtualopensystems.com/dev/qemu on branch rfc_v2.
>
> Run QEMU master instance:
> ./qemu-system-arm -nographic \
>         -kernel zImage \
>         -M virt -m 1G \
>         -initrd busybox.cpio \
>         -object multi-socket-backend,id=sock,path=sock,listen \
>         -object memory-backend-shared,id=mem,size=1G,mem-path=/dev/hugepages,socket=sock,master \
>         -object sdm-signal-shboot,id=boot,shm=mem \
>         -object sdm-communication-socket,id=scomm,socket=sock \
>         -device virtio-sdm-device,comm=scomm,master,num-slaves=1,len-signals=1,signals[0]=boot \
>         -append "mem=512M memmap=512M$0x60000000" \
>         -numa node,memdev=mem -m 1G
>
> Run QEMU slave instance:
> ./qemu-system-arm -nographic \
>         -kernel zImage \
>         -M virt -m 1G \
>         -initrd busybox.cpio \
>         -object multi-socket-backend,id=sock,path=sock \
>         -object memory-backend-shared,id=mem,size=512M,socket=sock \
>         -object sdm-signal-shboot,id=boot,shm=mem \
>         -object sdm-communication-socket,id=scomm,socket=sock \
>         -device sdm-platform,comm=scomm,len-signals=1,signals[0]="boot" \
>         -incoming "shared:mem" \
>         -numa node,memdev=mem -m 512M
>
> On the master, load the SDM module (insmod sdm_test_mod_v1) and trigger the boot with
> the following command:
> ./sdm-test -b 0 1 zImage slave.dtb
>
> zImage and slave.dtb are available on branch demo_slaves_sdm.
>
> This patch serie is a follow-up to "[RFC PATCH 0/8] Towards an Heterogeneous QEMU":
> https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg00171.html
>
> This patch serie depends on:
> - [RFC v2 0/6] SDM Interface
> - [RFC v2 1/1] backend: multi-client-socket
>
> This work has been sponsored by Huawei Technologies Duesseldorf GmbH.
>
> Baptiste Reynal (3):
>   backend: shared memory backend
>   migration: add shared migration type
>   hw/misc: sdm signal shboot
>
> Christian Pinto (3):
>   qemu: slave machine flag
>   hw/arm: boot
>   qemu: numa
>
>  backends/Makefile.objs              |   2 +-
>  backends/hostmem-shared.c           | 198 ++++++++++++++++++++++++++++++++++++
>  hw/arm/boot.c                       |  13 +++
>  hw/core/machine.c                   |  27 +++++
>  hw/misc/Makefile.objs               |   1 +
>  hw/misc/sdm-signal-shboot.c         |  62 +++++++++++
>  include/hw/boards.h                 |   2 +
>  include/hw/misc/sdm-signal-shboot.h |  38 +++++++
>  include/migration/migration.h       |   2 +
>  include/sysemu/hostmem-shared.h     |  66 ++++++++++++
>  migration/Makefile.objs             |   2 +-
>  migration/migration.c               |   2 +
>  migration/shared.c                  |  33 ++++++
>  numa.c                              |  17 +++-
>  qemu-options.hx                     |   5 +-
>  util/qemu-config.c                  |   5 +
>  16 files changed, 471 insertions(+), 4 deletions(-)
>  create mode 100644 backends/hostmem-shared.c
>  create mode 100644 hw/misc/sdm-signal-shboot.c
>  create mode 100644 include/hw/misc/sdm-signal-shboot.h
>  create mode 100644 include/sysemu/hostmem-shared.h
>  create mode 100644 migration/shared.c

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Baptiste Reynal]

On Tue, Mar 22, 2016 at 3:14 PM, Markus Armbruster <armbru@redhat.com> wrote:
>
> Copying Paolo, author of memory-backend-file.
>
> Baptiste Reynal <b.reynal@virtualopensystems.com> writes:
>
> > A new memory backend, the shared memory backend, based on
> > the file memory backend.
> >
> > This new backend allows a master QEMU instance to share a part of
> > his main memory whith a slave QEMU instance. It is then possible to load
> > a firmware on this memory and trigger the slave boot using a SDM
> > signal.
>
> Can you explain why that's useful in a bit more detail?
>

The purpose of such memory backend is to enable the modeling
of asymmetric multiprocessing (AMP) configurations. An example is
a multi-core ARM CPU working alongside with two Cortex-M
micro controllers.

On such configurations many CPU (modeled by many QEMU
instances) share the same memory bus. This patch series allows multiple
QEMU instances to share the same file-backed memory bus.

>
> > Such new backend enables, on a master side, to allocate the whole
> > memory as shareable (e.g. /dev/shm, or hugetlbfs).
>
> memory-backend-file can do that already.  What exactly does
> memory-backend-shared add to it?

This memory-backend-shared allows multiple QEMU instances to share the
same memory backend. A slave instance will have access only to a subset of
the whole AMP system memory, by mapping just one part of the file as its
memory. This can be considered as an MMU configured by the master processor
to restrict slave memory accesses to only a subset of the whole system memory.

For the purpose of AMP modeling, a QEMU instance can be put in waiting mode
(by using QEMU migration) until a master QEMU triggers its boot. It allows the
master to fill the slave memory with its firmware before boot, in order to be
compliant with the remoteproc framework (power on, load firmware, power off).

>
>
> > On the slave side it enables the startup of QEMU without any main memory
> > allocated. Then the slave goes in a waiting state, the same used in the
> > case of an incoming migration, and the file descriptor of the shared
> > memory is sent by the master over a socket.
> > The waiting state ends when the master sends to the slave a signal
> > with size and offset to mmap and use as memory as payload.
> >
> > You can test those patches using modules and test application available
> > on https://git.virtualopensystems.com/dev/qemu-het-tools.
> > - SDM Platform module on branch sdm_test_mod_v1
> > - Test application on branch sdm_test_app
> > - Slave files on branch demo_slaves_sdm
> >
> >  QEMU code is avalaible here https://git.virtualopensystems.com/dev/qemu on branch rfc_v2.
> >
> > Run QEMU master instance:
> > ./qemu-system-arm -nographic \
> >         -kernel zImage \
> >         -M virt -m 1G \
> >         -initrd busybox.cpio \
> >         -object multi-socket-backend,id=sock,path=sock,listen \
> >         -object memory-backend-shared,id=mem,size=1G,mem-path=/dev/hugepages,socket=sock,master \
> >         -object sdm-signal-shboot,id=boot,shm=mem \
> >         -object sdm-communication-socket,id=scomm,socket=sock \
> >         -device virtio-sdm-device,comm=scomm,master,num-slaves=1,len-signals=1,signals[0]=boot \
> >         -append "mem=512M memmap=512M$0x60000000" \
> >         -numa node,memdev=mem -m 1G
> >
> > Run QEMU slave instance:
> > ./qemu-system-arm -nographic \
> >         -kernel zImage \
> >         -M virt -m 1G \
> >         -initrd busybox.cpio \
> >         -object multi-socket-backend,id=sock,path=sock \
> >         -object memory-backend-shared,id=mem,size=512M,socket=sock \
> >         -object sdm-signal-shboot,id=boot,shm=mem \
> >         -object sdm-communication-socket,id=scomm,socket=sock \
> >         -device sdm-platform,comm=scomm,len-signals=1,signals[0]="boot" \
> >         -incoming "shared:mem" \
> >         -numa node,memdev=mem -m 512M
> >
> > On the master, load the SDM module (insmod sdm_test_mod_v1) and trigger the boot with
> > the following command:
> > ./sdm-test -b 0 1 zImage slave.dtb
> >
> > zImage and slave.dtb are available on branch demo_slaves_sdm.
> >
> > This patch serie is a follow-up to "[RFC PATCH 0/8] Towards an Heterogeneous QEMU":
> > https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg00171.html
> >
> > This patch serie depends on:
> > - [RFC v2 0/6] SDM Interface
> > - [RFC v2 1/1] backend: multi-client-socket
> >
> > This work has been sponsored by Huawei Technologies Duesseldorf GmbH.
> >
> > Baptiste Reynal (3):
> >   backend: shared memory backend
> >   migration: add shared migration type
> >   hw/misc: sdm signal shboot
> >
> > Christian Pinto (3):
> >   qemu: slave machine flag
> >   hw/arm: boot
> >   qemu: numa
> >
> >  backends/Makefile.objs              |   2 +-
> >  backends/hostmem-shared.c           | 198 ++++++++++++++++++++++++++++++++++++
> >  hw/arm/boot.c                       |  13 +++
> >  hw/core/machine.c                   |  27 +++++
> >  hw/misc/Makefile.objs               |   1 +
> >  hw/misc/sdm-signal-shboot.c         |  62 +++++++++++
> >  include/hw/boards.h                 |   2 +
> >  include/hw/misc/sdm-signal-shboot.h |  38 +++++++
> >  include/migration/migration.h       |   2 +
> >  include/sysemu/hostmem-shared.h     |  66 ++++++++++++
> >  migration/Makefile.objs             |   2 +-
> >  migration/migration.c               |   2 +
> >  migration/shared.c                  |  33 ++++++
> >  numa.c                              |  17 +++-
> >  qemu-options.hx                     |   5 +-
> >  util/qemu-config.c                  |   5 +
> >  16 files changed, 471 insertions(+), 4 deletions(-)
> >  create mode 100644 backends/hostmem-shared.c
> >  create mode 100644 hw/misc/sdm-signal-shboot.c
> >  create mode 100644 include/hw/misc/sdm-signal-shboot.h
> >  create mode 100644 include/sysemu/hostmem-shared.h
> >  create mode 100644 migration/shared.c

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 589 bytes --]

On Fri, Mar 18, 2016 at 10:13:52AM +0100, Baptiste Reynal wrote:
> A new memory backend, the shared memory backend, based on
> the file memory backend.
> 
> This new backend allows a master QEMU instance to share a part of
> his main memory whith a slave QEMU instance. It is then possible to load
> a firmware on this memory and trigger the slave boot using a SDM
> signal.
> 
> Such new backend enables, on a master side, to allocate the whole
> memory as shareable (e.g. /dev/shm, or hugetlbfs).

How is this different from qemu -mem-path which can be used for
hugetlbfs?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Baptiste Reynal]

On Thu, Mar 31, 2016 at 11:14 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Fri, Mar 18, 2016 at 10:13:52AM +0100, Baptiste Reynal wrote:
> > A new memory backend, the shared memory backend, based on
> > the file memory backend.
> >
> > This new backend allows a master QEMU instance to share a part of
> > his main memory whith a slave QEMU instance. It is then possible to load
> > a firmware on this memory and trigger the slave boot using a SDM
> > signal.
> >
> > Such new backend enables, on a master side, to allocate the whole
> > memory as shareable (e.g. /dev/shm, or hugetlbfs).
>
> How is this different from qemu -mem-path which can be used for
> hugetlbfs?

A new mechanism is integrated here. This shared memory allows the
slave to map dynamically only a subregion of the master memory, when
with mempath the entire memory is shared.
A use-case can be the use of remoteproc framework on Linux. Remoteproc
carveout a memory region and load the slave firmware to this region,
then send the size and offset to the slave before boot. The slave
memory region is unknown before master execution, it cannot be set in
QEMU command line.

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

On Tue, Apr 05, 2016 at 02:00:40PM +0200, Baptiste Reynal wrote:
> On Thu, Mar 31, 2016 at 11:14 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > On Fri, Mar 18, 2016 at 10:13:52AM +0100, Baptiste Reynal wrote:
> > > A new memory backend, the shared memory backend, based on
> > > the file memory backend.
> > >
> > > This new backend allows a master QEMU instance to share a part of
> > > his main memory whith a slave QEMU instance. It is then possible to load
> > > a firmware on this memory and trigger the slave boot using a SDM
> > > signal.
> > >
> > > Such new backend enables, on a master side, to allocate the whole
> > > memory as shareable (e.g. /dev/shm, or hugetlbfs).
> >
> > How is this different from qemu -mem-path which can be used for
> > hugetlbfs?
> 
> A new mechanism is integrated here. This shared memory allows the
> slave to map dynamically only a subregion of the master memory, when
> with mempath the entire memory is shared.
> A use-case can be the use of remoteproc framework on Linux. Remoteproc
> carveout a memory region and load the slave firmware to this region,
> then send the size and offset to the slave before boot. The slave
> memory region is unknown before master execution, it cannot be set in
> QEMU command line.

Igor: Do you know what level of control QEMU already has for individual
DIMMs backed by hugetlbfs?  Is it possible to have just a specific
physical memory range on hugetlbfs by defining individual DIMMs on the
command-line?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Igor Mammedov]

On Wed, 6 Apr 2016 09:57:57 +0100
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Tue, Apr 05, 2016 at 02:00:40PM +0200, Baptiste Reynal wrote:
> > On Thu, Mar 31, 2016 at 11:14 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:  
> > >
> > > On Fri, Mar 18, 2016 at 10:13:52AM +0100, Baptiste Reynal wrote:  
> > > > A new memory backend, the shared memory backend, based on
> > > > the file memory backend.
> > > >
> > > > This new backend allows a master QEMU instance to share a part of
> > > > his main memory whith a slave QEMU instance. It is then possible to load
> > > > a firmware on this memory and trigger the slave boot using a SDM
> > > > signal.
> > > >
> > > > Such new backend enables, on a master side, to allocate the whole
> > > > memory as shareable (e.g. /dev/shm, or hugetlbfs).  
> > >
> > > How is this different from qemu -mem-path which can be used for
> > > hugetlbfs?  
> > 
> > A new mechanism is integrated here. This shared memory allows the
> > slave to map dynamically only a subregion of the master memory, when
> > with mempath the entire memory is shared.
> > A use-case can be the use of remoteproc framework on Linux. Remoteproc
> > carveout a memory region and load the slave firmware to this region,
> > then send the size and offset to the slave before boot. The slave
> > memory region is unknown before master execution, it cannot be set in
> > QEMU command line.  
> 
> Igor: Do you know what level of control QEMU already has for individual
> DIMMs backed by hugetlbfs?  Is it possible to have just a specific
> physical memory range on hugetlbfs by defining individual DIMMs on the
> command-line?
it should be possible by using memory-backend-file object with "mem-path"
property pointing to hugetlbfs and property "share" should help to make
it shared.

Also one could use backend for initial memory as well via
 -numa xxx,memdev=foo
option

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Markus Armbruster]

Baptiste Reynal <b.reynal@virtualopensystems.com> writes:

> On Tue, Mar 22, 2016 at 3:14 PM, Markus Armbruster <armbru@redhat.com> wrote:
>>
>> Copying Paolo, author of memory-backend-file.
>>
>> Baptiste Reynal <b.reynal@virtualopensystems.com> writes:
>>
>> > A new memory backend, the shared memory backend, based on
>> > the file memory backend.
>> >
>> > This new backend allows a master QEMU instance to share a part of
>> > his main memory whith a slave QEMU instance. It is then possible to load
>> > a firmware on this memory and trigger the slave boot using a SDM
>> > signal.
>>
>> Can you explain why that's useful in a bit more detail?
>>
>
> The purpose of such memory backend is to enable the modeling
> of asymmetric multiprocessing (AMP) configurations. An example is
> a multi-core ARM CPU working alongside with two Cortex-M
> micro controllers.
>
> On such configurations many CPU (modeled by many QEMU
> instances) share the same memory bus. This patch series allows multiple
> QEMU instances to share the same file-backed memory bus.

All right, this is for implementing (heterogenous) multi-core via
multiple QEMU instances.  The alternative would be making one QEMU
instance support more different CPUs.  Which one is the more promising
approach I can't say.

>> > Such new backend enables, on a master side, to allocate the whole
>> > memory as shareable (e.g. /dev/shm, or hugetlbfs).
>>
>> memory-backend-file can do that already.  What exactly does
>> memory-backend-shared add to it?
>
> This memory-backend-shared allows multiple QEMU instances to share the
> same memory backend. A slave instance will have access only to a subset of
> the whole AMP system memory, by mapping just one part of the file as its
> memory. This can be considered as an MMU configured by the master processor
> to restrict slave memory accesses to only a subset of the whole system memory.
>
> For the purpose of AMP modeling, a QEMU instance can be put in waiting mode
> (by using QEMU migration) until a master QEMU triggers its boot. It allows the
> master to fill the slave memory with its firmware before boot, in order to be
> compliant with the remoteproc framework (power on, load firmware, power off).

If I understand you correctly, you need two things: multiple QEMU
instances sharing (some) memory, and a signalling mechanism between
them.

memory-backend-file provides the former, but by itself can't provide the
latter.

Is this roughly correct?

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Baptiste Reynal]

On Wed, Apr 6, 2016 at 3:47 PM, Igor Mammedov <imammedo@redhat.com> wrote:
> On Wed, 6 Apr 2016 09:57:57 +0100
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
>> On Tue, Apr 05, 2016 at 02:00:40PM +0200, Baptiste Reynal wrote:
>> > On Thu, Mar 31, 2016 at 11:14 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> > >
>> > > On Fri, Mar 18, 2016 at 10:13:52AM +0100, Baptiste Reynal wrote:
>> > > > A new memory backend, the shared memory backend, based on
>> > > > the file memory backend.
>> > > >
>> > > > This new backend allows a master QEMU instance to share a part of
>> > > > his main memory whith a slave QEMU instance. It is then possible to load
>> > > > a firmware on this memory and trigger the slave boot using a SDM
>> > > > signal.
>> > > >
>> > > > Such new backend enables, on a master side, to allocate the whole
>> > > > memory as shareable (e.g. /dev/shm, or hugetlbfs).
>> > >
>> > > How is this different from qemu -mem-path which can be used for
>> > > hugetlbfs?
>> >
>> > A new mechanism is integrated here. This shared memory allows the
>> > slave to map dynamically only a subregion of the master memory, when
>> > with mempath the entire memory is shared.
>> > A use-case can be the use of remoteproc framework on Linux. Remoteproc
>> > carveout a memory region and load the slave firmware to this region,
>> > then send the size and offset to the slave before boot. The slave
>> > memory region is unknown before master execution, it cannot be set in
>> > QEMU command line.
>>
>> Igor: Do you know what level of control QEMU already has for individual
>> DIMMs backed by hugetlbfs?  Is it possible to have just a specific
>> physical memory range on hugetlbfs by defining individual DIMMs on the
>> command-line?
> it should be possible by using memory-backend-file object with "mem-path"
> property pointing to hugetlbfs and property "share" should help to make
> it shared.
>
> Also one could use backend for initial memory as well via
>  -numa xxx,memdev=foo
> option

Thanks for the help.
>From my understanding the solution will be each slave backed by a
different hugetlbfs file shared with the master. This solution seems
acceptable. The dynamic aspect is lost but it is not mandatory. In
some AMP implementation (i.e OMAP remoteproc) the entire memory is
shared and the memory protection is assured by an IOMMU, the lack of
IOMMU in this project forces us to have some static configuration at
some point.
The next version will then focus on the incoming state with a boot signal.

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Baptiste Reynal]

On Thu, Apr 7, 2016 at 6:27 PM, Markus Armbruster <armbru@redhat.com> wrote:
> Baptiste Reynal <b.reynal@virtualopensystems.com> writes:
>
>> On Tue, Mar 22, 2016 at 3:14 PM, Markus Armbruster <armbru@redhat.com> wrote:
>>>
>>> Copying Paolo, author of memory-backend-file.
>>>
>>> Baptiste Reynal <b.reynal@virtualopensystems.com> writes:
>>>
>>> > A new memory backend, the shared memory backend, based on
>>> > the file memory backend.
>>> >
>>> > This new backend allows a master QEMU instance to share a part of
>>> > his main memory whith a slave QEMU instance. It is then possible to load
>>> > a firmware on this memory and trigger the slave boot using a SDM
>>> > signal.
>>>
>>> Can you explain why that's useful in a bit more detail?
>>>
>>
>> The purpose of such memory backend is to enable the modeling
>> of asymmetric multiprocessing (AMP) configurations. An example is
>> a multi-core ARM CPU working alongside with two Cortex-M
>> micro controllers.
>>
>> On such configurations many CPU (modeled by many QEMU
>> instances) share the same memory bus. This patch series allows multiple
>> QEMU instances to share the same file-backed memory bus.
>
> All right, this is for implementing (heterogenous) multi-core via
> multiple QEMU instances.  The alternative would be making one QEMU
> instance support more different CPUs.  Which one is the more promising
> approach I can't say.

We decided to implement it using multiple QEMU instances in order to
reduce the impact on QEMU code.

>
>>> > Such new backend enables, on a master side, to allocate the whole
>>> > memory as shareable (e.g. /dev/shm, or hugetlbfs).
>>>
>>> memory-backend-file can do that already.  What exactly does
>>> memory-backend-shared add to it?
>>
>> This memory-backend-shared allows multiple QEMU instances to share the
>> same memory backend. A slave instance will have access only to a subset of
>> the whole AMP system memory, by mapping just one part of the file as its
>> memory. This can be considered as an MMU configured by the master processor
>> to restrict slave memory accesses to only a subset of the whole system memory.
>>
>> For the purpose of AMP modeling, a QEMU instance can be put in waiting mode
>> (by using QEMU migration) until a master QEMU triggers its boot. It allows the
>> master to fill the slave memory with its firmware before boot, in order to be
>> compliant with the remoteproc framework (power on, load firmware, power off).
>
> If I understand you correctly, you need two things: multiple QEMU
> instances sharing (some) memory, and a signalling mechanism between
> them.
>
> memory-backend-file provides the former, but by itself can't provide the
> latter.
>
> Is this roughly correct?

Exactly. The signaling mechanism is handled by the SDM device,
presented here:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg361316.html

Re: [Qemu-devel] [RFC v2 0/6] QEMU shared-memory backend

[Markus Armbruster]

Baptiste Reynal <b.reynal@virtualopensystems.com> writes:

> On Thu, Apr 7, 2016 at 6:27 PM, Markus Armbruster <armbru@redhat.com> wrote:
>> Baptiste Reynal <b.reynal@virtualopensystems.com> writes:
>>
>>> On Tue, Mar 22, 2016 at 3:14 PM, Markus Armbruster <armbru@redhat.com> wrote:
>>>>
>>>> Copying Paolo, author of memory-backend-file.
>>>>
>>>> Baptiste Reynal <b.reynal@virtualopensystems.com> writes:
>>>>
>>>> > A new memory backend, the shared memory backend, based on
>>>> > the file memory backend.
>>>> >
>>>> > This new backend allows a master QEMU instance to share a part of
>>>> > his main memory whith a slave QEMU instance. It is then possible to load
>>>> > a firmware on this memory and trigger the slave boot using a SDM
>>>> > signal.
>>>>
>>>> Can you explain why that's useful in a bit more detail?
>>>>
>>>
>>> The purpose of such memory backend is to enable the modeling
>>> of asymmetric multiprocessing (AMP) configurations. An example is
>>> a multi-core ARM CPU working alongside with two Cortex-M
>>> micro controllers.
>>>
>>> On such configurations many CPU (modeled by many QEMU
>>> instances) share the same memory bus. This patch series allows multiple
>>> QEMU instances to share the same file-backed memory bus.
>>
>> All right, this is for implementing (heterogenous) multi-core via
>> multiple QEMU instances.  The alternative would be making one QEMU
>> instance support more different CPUs.  Which one is the more promising
>> approach I can't say.
>
> We decided to implement it using multiple QEMU instances in order to
> reduce the impact on QEMU code.

Understood.

>>>> > Such new backend enables, on a master side, to allocate the whole
>>>> > memory as shareable (e.g. /dev/shm, or hugetlbfs).
>>>>
>>>> memory-backend-file can do that already.  What exactly does
>>>> memory-backend-shared add to it?
>>>
>>> This memory-backend-shared allows multiple QEMU instances to share the
>>> same memory backend. A slave instance will have access only to a subset of
>>> the whole AMP system memory, by mapping just one part of the file as its
>>> memory. This can be considered as an MMU configured by the master processor
>>> to restrict slave memory accesses to only a subset of the whole system memory.
>>>
>>> For the purpose of AMP modeling, a QEMU instance can be put in waiting mode
>>> (by using QEMU migration) until a master QEMU triggers its boot. It allows the
>>> master to fill the slave memory with its firmware before boot, in order to be
>>> compliant with the remoteproc framework (power on, load firmware, power off).
>>
>> If I understand you correctly, you need two things: multiple QEMU
>> instances sharing (some) memory, and a signalling mechanism between
>> them.
>>
>> memory-backend-file provides the former, but by itself can't provide the
>> latter.
>>
>> Is this roughly correct?
>
> Exactly. The signaling mechanism is handled by the SDM device,
> presented here:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg361316.html

Got it, thanks.

I think your commit message needs to explain why
memory-backend-file,share doesn't suffice for your use case (assuming it
doesn't).  You elaborated on your use case in reply to my question
(quoted above), but 1. you didn't quite answer my question why
memory-backend-file won't do, and 2. it should really be in git.