[PATCH v12 00/19] Initial support for multi-process Qemu

[Jagannathan Raman <jag.raman@oracle.com>]

Hello,

This is the v12 of the patchset. Thank you very much for the
review of the v11 of the series.

We made changes to the following patches in this version:
  - Moved patches 18 & 19 in v11 to the front of the series based
    on feedback from Phil
  - [PATCH v12 02/19 ] multi-process: add configure and usage information
  - [PATCH v12 04/19 ] multi-process: Add config option for multi-process QEMU
  - [PATCH v12 08/19] multi-process: define MPQemuMsg format and transmission
    functions.

In summary, we replaced "scripts/mpqemu-launcher.py" with
"tests/multiprocess/multiprocess-lsi53c895a.py". We tested this test on
x86_64 and aarch64 architectures, which we have access to.

We changed the name of the config variable called CONFIG_MPQEMU
to CONFIG_MULTIPROCESS. We also moved the config variable
definition out of the "configure" script, and into the Kconfig
system. Previously, the user specified if multiprocess was enabled
using the "--enable-mpqemu" argument to the configure script.
In this version, we changed that. The multiprocess support is enabled
automatically if Kconfig system detects KVM support. This is needed
to run acceptance tests in the future.

We are working on acceptance tests (tests/acceptance/) for this
project. However, we have hit a roadblock and are working with the
avocado-devel community to resolve the issue.

We noticed that checkpatch.pl script flagged a warning for Patch 4 for
this series, but we don't believe that's a valid concern. We generated
the patches using QEMU's git orderfile
(git format-patch -O scripts/git.orderfile ...).

To touch upon the history of this project, we posted the Proof Of
Concept patches before the BoF session in 2018. Subsequently, we have
posted 11 versions on the qemu-devel mailing list. You can find them
by following the links below ([1] - [11]).Following people contributed
to the design and implementation of this project:
Jagannathan Raman <jag.raman@oracle.com>
Elena Ufimtseva <elena.ufimtseva@oracle.com>
John G Johnson <john.g.johnson@oracle.com>
Stefan Hajnoczi <stefanha@redhat.com>
Konrad Wilk <konrad.wilk@oracle.com>
Kanth Ghatraju <kanth.ghatraju@oracle.com>

We would like to thank QEMU community for your feedback in the
design and implementation of this project.Qemu wiki page:
https://wiki.qemu.org/Features/MultiProcessQEMU

For the full concept writeup about QEMU multi-process, please
refer to docs/devel/qemu-multiprocess.rst. Also see
docs/qemu-multiprocess.txt for usage information. We welcome
all your ideas, concerns, and questions for this patchset.

Thank you!

[POC]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
[1]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
[2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
[3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
[4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg655118.html
[5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg682429.html
[6]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg697484.html
[7]: https://patchew.org/QEMU/cover.1593273671.git.elena.ufimtseva@oracle.com/
[8]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg727007.html
[9]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg734275.html
[10]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg747638.html
[11]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg750972.htmlThank you!


Elena Ufimtseva (7):
  multi-process: add configure and usage information
  multi-process: add qio channel function to transmit data and fds
  multi-process: define MPQemuMsg format and transmission functions
  multi-process: introduce proxy object
  multi-process: add proxy communication functions
  multi-process: Forward PCI config space acceses to the remote process
  multi-process: perform device reset in the remote process

Jagannathan Raman (11):
  memory: alloc RAM from file at offset
  multi-process: Add config option for multi-process QEMU
  multi-process: setup PCI host bridge for remote device
  multi-process: setup a machine object for remote device process
  multi-process: Initialize message handler in remote device
  multi-process: Associate fd of a PCIDevice with its object
  multi-process: setup memory manager for remote device
  multi-process: PCI BAR read/write handling for proxy & remote
    endpoints
  multi-process: Synchronize remote memory
  multi-process: create IOHUB object to handle irq
  multi-process: Retrieve PCI info from remote process

John G Johnson (1):
  multi-process: add the concept description to
    docs/devel/qemu-multiprocess

 docs/devel/index.rst                          |   1 +
 docs/devel/multi-process.rst                  | 966 ++++++++++++++++++++++++++
 docs/multi-process.rst                        |  66 ++
 include/exec/memory.h                         |   2 +
 include/exec/ram_addr.h                       |   2 +-
 include/hw/pci-host/remote.h                  |  31 +
 include/hw/pci/pci_ids.h                      |   3 +
 include/hw/remote/iohub.h                     |  42 ++
 include/hw/remote/machine.h                   |  40 ++
 include/hw/remote/memory-sync.h               |  27 +
 include/hw/remote/memory.h                    |  19 +
 include/hw/remote/mpqemu-link.h               |  98 +++
 include/hw/remote/proxy.h                     |  53 ++
 include/hw/remote/remote-obj.h                |  42 ++
 include/io/channel.h                          |  24 +
 include/qemu/mmap-alloc.h                     |   3 +-
 backends/hostmem-memfd.c                      |   2 +-
 hw/misc/ivshmem.c                             |   3 +-
 hw/pci-host/remote.c                          |  75 ++
 hw/remote/iohub.c                             | 123 ++++
 hw/remote/machine.c                           |  79 +++
 hw/remote/memory-sync.c                       | 210 ++++++
 hw/remote/memory.c                            |  58 ++
 hw/remote/message.c                           | 241 +++++++
 hw/remote/mpqemu-link.c                       | 308 ++++++++
 hw/remote/proxy.c                             | 378 ++++++++++
 hw/remote/remote-obj.c                        | 154 ++++
 io/channel.c                                  |  45 ++
 softmmu/memory.c                              |   3 +-
 softmmu/physmem.c                             |  11 +-
 util/mmap-alloc.c                             |   7 +-
 util/oslib-posix.c                            |   2 +-
 MAINTAINERS                                   |  25 +
 accel/Kconfig                                 |   1 +
 hw/Kconfig                                    |   1 +
 hw/meson.build                                |   1 +
 hw/pci-host/Kconfig                           |   3 +
 hw/pci-host/meson.build                       |   1 +
 hw/remote/Kconfig                             |   4 +
 hw/remote/meson.build                         |  13 +
 tests/multiprocess/multiprocess-lsi53c895a.py |  92 +++
 41 files changed, 3246 insertions(+), 13 deletions(-)
 create mode 100644 docs/devel/multi-process.rst
 create mode 100644 docs/multi-process.rst
 create mode 100644 include/hw/pci-host/remote.h
 create mode 100644 include/hw/remote/iohub.h
 create mode 100644 include/hw/remote/machine.h
 create mode 100644 include/hw/remote/memory-sync.h
 create mode 100644 include/hw/remote/memory.h
 create mode 100644 include/hw/remote/mpqemu-link.h
 create mode 100644 include/hw/remote/proxy.h
 create mode 100644 include/hw/remote/remote-obj.h
 create mode 100644 hw/pci-host/remote.c
 create mode 100644 hw/remote/iohub.c
 create mode 100644 hw/remote/machine.c
 create mode 100644 hw/remote/memory-sync.c
 create mode 100644 hw/remote/memory.c
 create mode 100644 hw/remote/message.c
 create mode 100644 hw/remote/mpqemu-link.c
 create mode 100644 hw/remote/proxy.c
 create mode 100644 hw/remote/remote-obj.c
 create mode 100644 hw/remote/Kconfig
 create mode 100644 hw/remote/meson.build
 create mode 100755 tests/multiprocess/multiprocess-lsi53c895a.py

-- 
1.8.3.1