[PATCH v1 0/2] some device emulation documentation cleanup

[PATCH v1 0/2] some device emulation documentation cleanup

[Alex Bennée]

Hi,

This series attempts to improve our documentation by:

  - grouping various device emulation documentation in one place
  - writing a preamble that explains concepts
  - writing a section on vhost-user

Alex Bennée (2):
  docs: collect the disparate device emulation docs into one section
  docs: add a section on the generalities of vhost-user

 docs/interop/vhost-user.rst               |  4 +-
 docs/system/device-emulation.rst          | 79 +++++++++++++++++++++++
 docs/system/{ => devices}/ivshmem.rst     |  0
 docs/system/{ => devices}/net.rst         |  0
 docs/system/{ => devices}/nvme.rst        |  0
 docs/system/{ => devices}/usb.rst         |  0
 docs/system/devices/vhost-user.rst        | 57 ++++++++++++++++
 docs/system/{ => devices}/virtio-pmem.rst |  0
 docs/system/index.rst                     |  6 +-
 9 files changed, 140 insertions(+), 6 deletions(-)
 create mode 100644 docs/system/device-emulation.rst
 rename docs/system/{ => devices}/ivshmem.rst (100%)
 rename docs/system/{ => devices}/net.rst (100%)
 rename docs/system/{ => devices}/nvme.rst (100%)
 rename docs/system/{ => devices}/usb.rst (100%)
 create mode 100644 docs/system/devices/vhost-user.rst
 rename docs/system/{ => devices}/virtio-pmem.rst (100%)

-- 
2.20.1

[PATCH v1 1/2] docs: collect the disparate device emulation docs into one section

[Alex Bennée]

While we are at it add a brief preamble that explains some of the
common concepts in QEMU's device emulation which will hopefully lead
to less confusing about our dizzying command line options.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 docs/system/device-emulation.rst          | 78 +++++++++++++++++++++++
 docs/system/{ => devices}/ivshmem.rst     |  0
 docs/system/{ => devices}/net.rst         |  0
 docs/system/{ => devices}/nvme.rst        |  0
 docs/system/{ => devices}/usb.rst         |  0
 docs/system/{ => devices}/virtio-pmem.rst |  0
 docs/system/index.rst                     |  6 +-
 7 files changed, 79 insertions(+), 5 deletions(-)
 create mode 100644 docs/system/device-emulation.rst
 rename docs/system/{ => devices}/ivshmem.rst (100%)
 rename docs/system/{ => devices}/net.rst (100%)
 rename docs/system/{ => devices}/nvme.rst (100%)
 rename docs/system/{ => devices}/usb.rst (100%)
 rename docs/system/{ => devices}/virtio-pmem.rst (100%)

diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
new file mode 100644
index 0000000000..a0f00bc340
--- /dev/null
+++ b/docs/system/device-emulation.rst
@@ -0,0 +1,78 @@
+.. _device-emulation:
+
+Device Emulation
+----------------
+
+QEMU supports the emulation of a large number of devices from
+peripherals such network cards and USB devices to integrated systems
+on a chip (SoCs). Configuration of these is often a source of
+confusion so it helps to have an understanding of some of the terms
+used to describes devices within QEMU.
+
+Common Terms
+~~~~~~~~~~~~
+
+Device Front End
+================
+
+A device front end is how a device is presented to the guest. The type
+of device presented should match the hardware that the guest operating
+system is expecting to see. All devices can be specified with the
+``--device`` command line option. Running QEMU with the command line
+options ``--device help`` will list all devices it is aware of. Using
+the command line ``--device foo,help`` will list the additional
+configuration options available for that device.
+
+A front end is often paired with a back end, which describes how the
+host's resources are used in the emulation.
+
+Device Buses
+============
+
+All devices exist on a BUS. Depending on the machine model you choose
+(``-M foo``) a number of buses will have been automatically created.
+In most cases the BUS a device is attached to can be inferred, for
+example PCI devices are generally automatically allocated to the next
+free slot of the PCI bus. However in complicated configurations you
+can explicitly specify what bus a device is attached to and its
+address. Some devices, for example a PCI SCSI host controller, will
+add an additional bus to the system that other devices can be attached
+to.
+
+Device Back End
+===============
+
+The back end describes how the data from the emulated device will be
+processed by QEMU. The configuration of the back end is usually
+specific to the class of device being emulated. For example serial
+devices will be backed by a ``--chardev`` which can redirect the data
+to a file or socket or some other system. Storage devices are handled
+by ``--blockdev`` which will specify how blocks are handled, for
+example being stored in a qcow2 file or accessing a raw host disk
+partition. Back ends can sometimes be stacked to implement features
+like snapshots.
+
+While the choice of back end is generally transparent to the guest
+there are cases where features will not be reported to the guest if
+the back end is unable to support it.
+
+Device Pass Through
+===================
+
+Device pass through is where the device is actually given access to
+the underlying hardware. This can be as simple as exposing a single
+USB device on the host system to the guest or dedicating a video card
+in a PCI slot to the exclusive use of the guest.
+
+
+Emulated Devices
+~~~~~~~~~~~~~~~~
+
+.. toctree::
+   :maxdepth: 1
+
+   devices/ivshmem.rst           
+   devices/net.rst
+   devices/nvme.rst
+   devices/usb.rst
+   devices/virtio-pmem.rst
diff --git a/docs/system/ivshmem.rst b/docs/system/devices/ivshmem.rst
similarity index 100%
rename from docs/system/ivshmem.rst
rename to docs/system/devices/ivshmem.rst
diff --git a/docs/system/net.rst b/docs/system/devices/net.rst
similarity index 100%
rename from docs/system/net.rst
rename to docs/system/devices/net.rst
diff --git a/docs/system/nvme.rst b/docs/system/devices/nvme.rst
similarity index 100%
rename from docs/system/nvme.rst
rename to docs/system/devices/nvme.rst
diff --git a/docs/system/usb.rst b/docs/system/devices/usb.rst
similarity index 100%
rename from docs/system/usb.rst
rename to docs/system/devices/usb.rst
diff --git a/docs/system/virtio-pmem.rst b/docs/system/devices/virtio-pmem.rst
similarity index 100%
rename from docs/system/virtio-pmem.rst
rename to docs/system/devices/virtio-pmem.rst
diff --git a/docs/system/index.rst b/docs/system/index.rst
index 6092eb2d91..641d243ba4 100644
--- a/docs/system/index.rst
+++ b/docs/system/index.rst
@@ -16,15 +16,12 @@ Contents:
 
    quickstart
    invocation
+   device-emulation
    keys
    mux-chardev
    monitor
    images
-   net
    virtio-net-failover
-   usb
-   nvme
-   ivshmem
    linuxboot
    generic-loader
    guest-loader
@@ -35,7 +32,6 @@ Contents:
    gdb
    managed-startup
    cpu-hotplug
-   virtio-pmem
    pr-manager
    targets
    security
-- 
2.20.1

[PATCH v1 2/2] docs: add a section on the generalities of vhost-user

[Alex Bennée]

While we do mention some of this stuff in the various daemons and
manuals the subtleties of the socket and memory sharing are sometimes
missed. This document attempts to give some background on vhost-user
daemons in general terms.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
---
 docs/interop/vhost-user.rst        |  4 ++-
 docs/system/device-emulation.rst   |  1 +
 docs/system/devices/vhost-user.rst | 57 ++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 1 deletion(-)
 create mode 100644 docs/system/devices/vhost-user.rst

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index d6085f7045..f46423b6f3 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -1,3 +1,5 @@
+.. _vhost_user_proto:
+
 ===================
 Vhost-user Protocol
 ===================
@@ -6,7 +8,7 @@ Vhost-user Protocol
 :Licence: This work is licensed under the terms of the GNU GPL,
           version 2 or later. See the COPYING file in the top-level
           directory.
-
+          
 .. contents:: Table of Contents
 
 Introduction
diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
index a0f00bc340..5aca6ed66b 100644
--- a/docs/system/device-emulation.rst
+++ b/docs/system/device-emulation.rst
@@ -75,4 +75,5 @@ Emulated Devices
    devices/net.rst
    devices/nvme.rst
    devices/usb.rst
+   devices/vhost-user.rst
    devices/virtio-pmem.rst
diff --git a/docs/system/devices/vhost-user.rst b/docs/system/devices/vhost-user.rst
new file mode 100644
index 0000000000..ecb1107d4c
--- /dev/null
+++ b/docs/system/devices/vhost-user.rst
@@ -0,0 +1,57 @@
+.. _vhost_user:
+
+vhost-user back ends
+--------------------
+
+vhost-user back ends are way to service the request of VirtIO devices
+outside of QEMU itself. To do this there are a number of things
+required.
+
+vhost-user device
+===================
+
+These are simple stub devices that ensure the VirtIO device is visible
+to the guest. The code is mostly boilerplate although each device has
+a ``chardev`` option which specifies the ID of the ``--chardev``
+device that connects via a socket to the vhost-user *daemon*.
+
+vhost-user daemon
+=================
+
+This is a separate process that is connected to by QEMU via a socket
+following the :ref:`vhost_user_proto`. There are a number of daemons
+that can be built when enabled by the project although any daemon that
+meets the specification for a given device can be used. The daemon
+will open a socket and *listen* for a connection from QEMU at which
+point the protocol will start its negotiation.
+
+Shared memory object
+====================
+
+In order for the daemon to access the VirtIO queues to process the
+requests it needs access to the guest's address space. This is
+achieved via the ``memory-backend-file`` object. A reference to a
+file-descriptor which can access this object will be passed via the
+socket as part of the protocol negotiation.
+
+Example
+=======
+
+First start you daemon.
+
+.. parsed-literal::
+
+  $ virtio-foo --socket-path=/var/run/foo.sock $OTHER_ARGS
+
+The you start your QEMU instance specifying the device, chardev and
+memory objects.
+
+.. parsed-literal::
+
+  $ |qemu_system| \\
+      -chardev socket,id=ba1,path=/var/run/foo.sock \\
+      -device vhost-user-foo,chardev=ba1,$OTHER_ARGS \\
+      -object memory-backend-memfd,id=mem,size=4G,share=on \\
+      -numa node,memdev=mem \\
+        ...
+
-- 
2.20.1

Re: [PATCH v1 2/2] docs: add a section on the generalities of vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4354 bytes --]

On Wed, Jul 07, 2021 at 07:45:49PM +0100, Alex Bennée wrote:
> While we do mention some of this stuff in the various daemons and
> manuals the subtleties of the socket and memory sharing are sometimes
> missed. This document attempts to give some background on vhost-user
> daemons in general terms.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  docs/interop/vhost-user.rst        |  4 ++-
>  docs/system/device-emulation.rst   |  1 +
>  docs/system/devices/vhost-user.rst | 57 ++++++++++++++++++++++++++++++
>  3 files changed, 61 insertions(+), 1 deletion(-)
>  create mode 100644 docs/system/devices/vhost-user.rst
> 
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index d6085f7045..f46423b6f3 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -1,3 +1,5 @@
> +.. _vhost_user_proto:
> +
>  ===================
>  Vhost-user Protocol
>  ===================
> @@ -6,7 +8,7 @@ Vhost-user Protocol
>  :Licence: This work is licensed under the terms of the GNU GPL,
>            version 2 or later. See the COPYING file in the top-level
>            directory.
> -
> +          
>  .. contents:: Table of Contents
>  
>  Introduction
> diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
> index a0f00bc340..5aca6ed66b 100644
> --- a/docs/system/device-emulation.rst
> +++ b/docs/system/device-emulation.rst
> @@ -75,4 +75,5 @@ Emulated Devices
>     devices/net.rst
>     devices/nvme.rst
>     devices/usb.rst
> +   devices/vhost-user.rst
>     devices/virtio-pmem.rst
> diff --git a/docs/system/devices/vhost-user.rst b/docs/system/devices/vhost-user.rst
> new file mode 100644
> index 0000000000..ecb1107d4c
> --- /dev/null
> +++ b/docs/system/devices/vhost-user.rst
> @@ -0,0 +1,57 @@
> +.. _vhost_user:
> +
> +vhost-user back ends
> +--------------------
> +
> +vhost-user back ends are way to service the request of VirtIO devices
> +outside of QEMU itself. To do this there are a number of things
> +required.
> +
> +vhost-user device
> +===================
> +
> +These are simple stub devices that ensure the VirtIO device is visible
> +to the guest. The code is mostly boilerplate although each device has
> +a ``chardev`` option which specifies the ID of the ``--chardev``
> +device that connects via a socket to the vhost-user *daemon*.
> +
> +vhost-user daemon
> +=================
> +
> +This is a separate process that is connected to by QEMU via a socket
> +following the :ref:`vhost_user_proto`. There are a number of daemons
> +that can be built when enabled by the project although any daemon that
> +meets the specification for a given device can be used. The daemon
> +will open a socket and *listen* for a connection from QEMU at which
> +point the protocol will start its negotiation.

I suggest dropping this line because it's more complicated and possibly
confusing at this point: vhost-user daemons can either listen or they
can connect (i.e. QEMU is the one that listens). I think both directions
were supported because they were convenient in different situations.

> +
> +Shared memory object
> +====================
> +
> +In order for the daemon to access the VirtIO queues to process the
> +requests it needs access to the guest's address space. This is
> +achieved via the ``memory-backend-file`` object. A reference to a

``memory-backend-file`` or ``memory-backend-memfd`` objects.

(memory-backend-ram doesn't have an fd so it cannot be passed over a
UNIX domain socket.)

> +file-descriptor which can access this object will be passed via the
> +socket as part of the protocol negotiation.
> +
> +Example
> +=======
> +
> +First start you daemon.
> +
> +.. parsed-literal::
> +
> +  $ virtio-foo --socket-path=/var/run/foo.sock $OTHER_ARGS
> +
> +The you start your QEMU instance specifying the device, chardev and
> +memory objects.
> +
> +.. parsed-literal::
> +
> +  $ |qemu_system| \\
> +      -chardev socket,id=ba1,path=/var/run/foo.sock \\
> +      -device vhost-user-foo,chardev=ba1,$OTHER_ARGS \\
> +      -object memory-backend-memfd,id=mem,size=4G,share=on \\
> +      -numa node,memdev=mem \\
> +        ...
> +
> -- 
> 2.20.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]