Re: [RFC v4 PATCH 00/49] Initial support of multi-process qemu - status update

[Elena Ufimtseva <elena.ufimtseva@oracle.com>]

Hi

We would like to give a short update to the community about the multi-process project.

Firstly, we appreciate the feedback and all productive discussions we had
at KVM 2019 forum.
As an outcome of the conference, we have switched gears and are investigating
the ways of using the muser framework in our project.

At this moment we are working on the evaluation and a first prototype
of qemu-multiprocess based on muser framework.
We first heard about it at the conference from the presentation given by
Thanos Makatos and Swapnil Ingle from Nutanix.
Their presentation is available https://static.sched.com/hosted_files/kvmforum2019/3b/muser.pdf
 along with github link to the source repo.
After the conversation we had with a group of people including Felipe Franciosi,
Stefan Hajnoczi, Daniel Berrangé, Konrad Wilk, Peter Maydell, John Jonson and few others
(apologies if some names are missing), we have gathered important answers on how to move
forward with qemu-multiprocess.

At this moment we are working on the first stage of the project with help of
the Nutanix developers.
The questions we have gathered so far will be addressed with muser
and Qemu developers after we finish the first stage and make sure we understand
what it will take for us to move onto the next stage.

We will also incorporate relevant review from Stefan that he provided
on the series 4 of the patchset. Thank you Stefan.

If anyone have any further suggestions or questions about the status,
please reply to this email.

Thank you

JJ, Jag & Elena

On Thu, Oct 24, 2019 at 05:08:41AM -0400, Jagannathan Raman wrote:
> Started with the presentation in October 2017 made by Marc-Andre (Red Hat)
> and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018,
> the multi-process project is now a prototype and presented in this patchset.
> John & Elena will present the status of this project in KVM Forum 2019.
> 
> This first series enables the emulation of lsi53c895a in a separate process.
> 
> We posted the Proof Of Concept patches [2] before the BoF session in 2018.
> Subsequently, we posted RFC v1 [3], RFC v2 [4] and RFC v3 [5] of this series. 
> 
> We want to present version 4 of this series, which incorporates the feedback
> we received for v3 & adds support for live migrating the remote process.
> 
> Following people contributed to this patchset:
> 
> John G Johnson <john.g.johnson@oracle.com>
> Jagannathan Raman <jag.raman@oracle.com>
> Elena Ufimtseva <elena.ufimtseva@oracle.com>
> Kanth Ghatraju <kanth.ghatraju@oracle.com>
> 
> For full concept writeup about QEMU disaggregation refer to
> docs/devel/qemu-multiprocess.rst. Please refer to 
> docs/qemu-multiprocess.txt for usage information.
> 
> We are planning on making the following improvements in the future:
>  - Performance improvements
>  - Libvirt support
>  - Enforcement of security policies
>  - blockdev support
> 
> We welcome all your ideas, concerns, and questions for this patchset.
> 
> Thank you!
> 
> [1]: http://events17.linuxfoundation.org/sites/events/files/slides/KVM%20FORUM%20multi-process.pdf
> [1]: https://www.youtube.com/watch?v=Kq1-coHh7lg
> [2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
> [3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
> [4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
> [5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
> 
> Elena Ufimtseva (22):
>   multi-process: add a command line option for debug file
>   multi-process: introduce proxy object
>   mutli-process: build remote command line args
>   multi-process: configure remote side devices
>   multi-process: add qdev_proxy_add to create proxy devices
>   multi-process: remote: add setup_devices and setup_drive msg
>     processing
>   multi-process: remote: use fd for socket from parent process
>   multi-process: remote: add create_done condition
>   multi-process: add processing of remote drive and device command line
>   multi-process: refractor vl.c code to re-use in remote
>   multi-process: add remote option
>   multi-process: add remote options parser
>   multi-process: add parse_cmdline in remote process
>   multi-process: send heartbeat messages to remote
>   multi-process: handle heartbeat messages in remote process
>   multi-process/mon: choose HMP commands based on target
>   multi-process/mig: Load VMSD in the proxy object
>   multi-process/mig: refactor runstate_check into common file
>   multi-process/mig: Synchronize runstate of remote process
>   multi-process/mig: Restore the VMSD in remote process
>   multi-process: Enable support for multiple devices in remote
>   multi-process: add configure and usage information
> 
> Jagannathan Raman (26):
>   multi-process: memory: alloc RAM from file at offset
>   multi-process: util: Add qemu_thread_cancel() to cancel running thread
>   multi-process: Add stub functions to facilate build of multi-process
>   multi-process: Add config option for multi-process QEMU
>   multi-process: build system for remote device process
>   multi-process: define mpqemu-link object
>   multi-process: add functions to synchronize proxy and remote endpoints
>   multi-process: setup PCI host bridge for remote device
>   multi-process: setup a machine object for remote device process
>   multi-process: setup memory manager for remote device
>   multi-process: remote process initialization
>   multi-process: PCI BAR read/write handling for proxy & remote
>     endpoints
>   multi-process: Add LSI device proxy object
>   multi-process: Synchronize remote memory
>   multi-process: create IOHUB object to handle irq
>   multi-process: Introduce build flags to separate remote process code
>   multi-process: Use separate MMIO communication channel
>   multi-process: perform device reset in the remote process
>   multi-process/mon: stub functions to enable QMP module for remote
>     process
>   multi-process/mon: enable QMP module support in the remote process
>   multi-process/mon: Refactor monitor/chardev functions out of vl.c
>   multi-process/mon: Initialize QMP module for remote processes
>   multi-process: prevent duplicate memory initialization in remote
>   multi-process/mig: build migration module in the remote process
>   multi-process/mig: Enable VMSD save in the Proxy object
>   multi-process/mig: Send VMSD of remote to the Proxy object
> 
> John G Johnson (1):
>   multi-process: add the concept description to
>     docs/devel/qemu-multiprocess
> 
>  Makefile                            |    2 +
>  Makefile.objs                       |   39 ++
>  Makefile.target                     |   94 ++-
>  accel/stubs/kvm-stub.c              |    5 +
>  accel/stubs/tcg-stub.c              |  106 ++++
>  backends/Makefile.objs              |    2 +
>  block/Makefile.objs                 |    2 +
>  chardev/char.c                      |   14 +
>  configure                           |   15 +
>  docs/devel/index.rst                |    1 +
>  docs/devel/qemu-multiprocess.rst    | 1102 +++++++++++++++++++++++++++++++++++
>  docs/qemu-multiprocess.txt          |   86 +++
>  exec.c                              |   14 +-
>  hmp-commands-info.hx                |   10 +
>  hmp-commands.hx                     |   25 +-
>  hw/Makefile.objs                    |    9 +
>  hw/block/Makefile.objs              |    2 +
>  hw/core/Makefile.objs               |   17 +
>  hw/nvram/Makefile.objs              |    2 +
>  hw/pci/Makefile.objs                |    4 +
>  hw/proxy/Makefile.objs              |    1 +
>  hw/proxy/memory-sync.c              |  226 +++++++
>  hw/proxy/proxy-lsi53c895a.c         |   97 +++
>  hw/proxy/qemu-proxy.c               |  807 +++++++++++++++++++++++++
>  hw/scsi/Makefile.objs               |    2 +
>  include/chardev/char.h              |    1 +
>  include/exec/address-spaces.h       |    2 +
>  include/exec/ram_addr.h             |    2 +-
>  include/hw/pci/pci_ids.h            |    3 +
>  include/hw/proxy/memory-sync.h      |   51 ++
>  include/hw/proxy/proxy-lsi53c895a.h |   42 ++
>  include/hw/proxy/qemu-proxy.h       |  125 ++++
>  include/hw/qdev-core.h              |    2 +
>  include/io/mpqemu-link.h            |  214 +++++++
>  include/monitor/monitor.h           |    2 +
>  include/monitor/qdev.h              |   25 +
>  include/qemu-common.h               |    8 +
>  include/qemu/log.h                  |    1 +
>  include/qemu/mmap-alloc.h           |    3 +-
>  include/qemu/thread.h               |    1 +
>  include/remote/iohub.h              |   63 ++
>  include/remote/machine.h            |   48 ++
>  include/remote/memory.h             |   34 ++
>  include/remote/pcihost.h            |   59 ++
>  include/sysemu/runstate.h           |    3 +
>  io/Makefile.objs                    |    2 +
>  io/mpqemu-link.c                    |  351 +++++++++++
>  memory.c                            |    2 +-
>  migration/Makefile.objs             |   12 +
>  migration/savevm.c                  |   63 ++
>  migration/savevm.h                  |    3 +
>  monitor/Makefile.objs               |    3 +
>  monitor/misc.c                      |   84 +--
>  monitor/monitor-internal.h          |   38 ++
>  monitor/monitor.c                   |   83 ++-
>  net/Makefile.objs                   |    2 +
>  qapi/Makefile.objs                  |    2 +
>  qdev-monitor.c                      |  270 ++++++++-
>  qemu-options.hx                     |   21 +
>  qom/Makefile.objs                   |    4 +
>  remote/Makefile.objs                |    6 +
>  remote/iohub.c                      |  159 +++++
>  remote/machine.c                    |  133 +++++
>  remote/memory.c                     |   99 ++++
>  remote/pcihost.c                    |   85 +++
>  remote/remote-main.c                |  633 ++++++++++++++++++++
>  remote/remote-opts.c                |  131 +++++
>  remote/remote-opts.h                |   31 +
>  replay/Makefile.objs                |    2 +-
>  rules.mak                           |    2 +-
>  runstate.c                          |   41 ++
>  scripts/hxtool                      |   44 +-
>  stubs/audio.c                       |   12 +
>  stubs/gdbstub.c                     |   21 +
>  stubs/machine-init-done.c           |    4 +
>  stubs/migration.c                   |  211 +++++++
>  stubs/monitor.c                     |   72 +++
>  stubs/net-stub.c                    |  121 ++++
>  stubs/qapi-misc.c                   |   43 ++
>  stubs/qapi-target.c                 |   49 ++
>  stubs/replay.c                      |   26 +
>  stubs/runstate-check.c              |    3 +
>  stubs/ui-stub.c                     |  130 +++++
>  stubs/vl-stub.c                     |  193 ++++++
>  stubs/vmstate.c                     |   20 +
>  stubs/xen-mapcache.c                |   22 +
>  ui/Makefile.objs                    |    2 +
>  util/log.c                          |    2 +
>  util/mmap-alloc.c                   |    7 +-
>  util/oslib-posix.c                  |    2 +-
>  util/qemu-thread-posix.c            |   10 +
>  vl-parse.c                          |  161 +++++
>  vl.c                                |  310 ++++------
>  vl.h                                |   54 ++
>  94 files changed, 6908 insertions(+), 246 deletions(-)
>  create mode 100644 docs/devel/qemu-multiprocess.rst
>  create mode 100644 docs/qemu-multiprocess.txt
>  create mode 100644 hw/proxy/Makefile.objs
>  create mode 100644 hw/proxy/memory-sync.c
>  create mode 100644 hw/proxy/proxy-lsi53c895a.c
>  create mode 100644 hw/proxy/qemu-proxy.c
>  create mode 100644 include/hw/proxy/memory-sync.h
>  create mode 100644 include/hw/proxy/proxy-lsi53c895a.h
>  create mode 100644 include/hw/proxy/qemu-proxy.h
>  create mode 100644 include/io/mpqemu-link.h
>  create mode 100644 include/remote/iohub.h
>  create mode 100644 include/remote/machine.h
>  create mode 100644 include/remote/memory.h
>  create mode 100644 include/remote/pcihost.h
>  create mode 100644 io/mpqemu-link.c
>  create mode 100644 remote/Makefile.objs
>  create mode 100644 remote/iohub.c
>  create mode 100644 remote/machine.c
>  create mode 100644 remote/memory.c
>  create mode 100644 remote/pcihost.c
>  create mode 100644 remote/remote-main.c
>  create mode 100644 remote/remote-opts.c
>  create mode 100644 remote/remote-opts.h
>  create mode 100644 runstate.c
>  mode change 100644 => 100755 scripts/hxtool
>  create mode 100644 stubs/audio.c
>  create mode 100644 stubs/migration.c
>  create mode 100644 stubs/net-stub.c
>  create mode 100644 stubs/qapi-misc.c
>  create mode 100644 stubs/qapi-target.c
>  create mode 100644 stubs/ui-stub.c
>  create mode 100644 stubs/vl-stub.c
>  create mode 100644 stubs/xen-mapcache.c
>  create mode 100644 vl-parse.c
>  create mode 100644 vl.h
> 
> -- 
> 1.8.3.1
>