[PATCH] docs: driver-api: virtio: virtio on Linux

[PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
Minor fixes to existing virtio kerneldocs.

Signed-off-by: Ricardo Cañuelo <ricardo.canuelo@collabora.com>
---
 Documentation/driver-api/index.rst            |   1 +
 Documentation/driver-api/virtio/index.rst     |  11 +
 Documentation/driver-api/virtio/virtio.rst    | 274 ++++++++++++++++++
 .../virtio/writing_virtio_drivers.rst         | 190 ++++++++++++
 MAINTAINERS                                   |   1 +
 include/linux/virtio.h                        |   6 +-
 include/linux/virtio_config.h                 |   6 +-
 include/uapi/linux/virtio_ring.h              |  16 +-
 8 files changed, 494 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/driver-api/virtio/index.rst
 create mode 100644 Documentation/driver-api/virtio/virtio.rst
 create mode 100644 Documentation/driver-api/virtio/writing_virtio_drivers.rst

diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index d3a58f77328e..30a3de452b1d 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -106,6 +106,7 @@ available subsections can be seen below.
    vfio-mediated-device
    vfio
    vfio-pci-device-specific-driver-acceptance
+   virtio/index
    xilinx/index
    xillybus
    zorro
diff --git a/Documentation/driver-api/virtio/index.rst b/Documentation/driver-api/virtio/index.rst
new file mode 100644
index 000000000000..528b14b291e3
--- /dev/null
+++ b/Documentation/driver-api/virtio/index.rst
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======
+Virtio
+======
+
+.. toctree::
+   :maxdepth: 1
+
+   virtio
+   writing_virtio_drivers
diff --git a/Documentation/driver-api/virtio/virtio.rst b/Documentation/driver-api/virtio/virtio.rst
new file mode 100644
index 000000000000..049a8aefad92
--- /dev/null
+++ b/Documentation/driver-api/virtio/virtio.rst
@@ -0,0 +1,274 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _virtio:
+
+===============
+Virtio on Linux
+===============
+
+Introduction
+============
+
+Virtio is an open standard interface for virtual machines to access
+paravirtualized devices, ie. devices that aren't emulated by a
+hypervisor, but rather real host devices that are exposed by the
+hypervisor to the guest to achieve native performance. In other words,
+it provides a communication mechanism for a guest OS to use devices on
+the host machine.
+
+The concrete hardware details of the real host devices are abstracted in the
+hypervisor, which provides a set of simplified virtual devices that
+implement the Virtio protocol. These devices are defined in the Virtio
+spec [1] and they're the devices that the guest OS will ultimately
+handle. So, in that regard, the guest OS knows it's running in a virtual
+environment and that it needs to use the appropriate Virtio drivers to
+handle the devices instead of the regular device drivers it'd use in a
+native or purely virtual environment (with emulated devices).
+
+
+Device - Driver communication: virtqueues
+=========================================
+
+Although the Virtio devices are really an abstraction layer in the
+hypervisor, they're exposed to the guest as if they are physical devices
+either PCI or MMIO-based. We refer to that as the transport method and
+it's orthogonal to the device itself. The Virtio spec defines these two
+and other transport methods in detail, including device discovery,
+capabilities and interrupt handling.
+
+The communication between the driver in the guest OS and the device in
+the hypervisor is done through shared memory (that's what makes virtio
+devices so efficient) using specialized data structures called
+virtqueues. The use of these virtqueues for data transfers is referred
+to as the data plane, while the process of setting them up and
+coordinating the driver and the device during the setup stage is called
+the control plane.
+
+Virtqueues, which are used to communicate the driver and the device
+regardless of the transport method used, are actually ring buffers
+[#f1]_ of buffer descriptors similar to the ones used in a network
+device:
+
+.. kernel-doc:: include/uapi/linux/virtio_ring.h
+    :identifiers: struct vring_desc
+
+All the buffers the descriptors point to are allocated by the guest and
+used by the host either for reading or for writing but not for both.
+
+Refer to [2] for an illustrated overview of virtqueues and how the host
+device and the guest driver communicate.
+
+Each virtqueue defines three areas: the descriptor area (`desc`), which
+is an array of descriptors as described above, and the avail and used
+rings.
+
+The `avail` ring is where the driver puts the indexes of the descriptors
+that it has set up for the device to consume. The `used` ring is used by
+the device to return the consumed buffers (read or written) to the
+driver. For each used descriptor, the ring also contains the used buffer
+length in case it was written.
+
+These rings are laid out in the :c:type:`vring_virtqueue` struct
+together with other necessary management data, including a pointer to a
+transport-specific ``notify`` function that is used to let the host side
+know when new buffers have been put in the virtqueue for
+processing. Embedded in this struct is the :c:type:`virtqueue` struct,
+which is the data structure that's ultimately used by virtio drivers:
+
+.. kernel-doc:: include/linux/virtio.h
+    :identifiers: struct virtqueue
+
+The callback saved in this struct serves the same purpose as the
+``notify`` function in :c:type:`vring_virtqueue` but in the
+opposite direction. That is, the callback is triggered when the host has
+used the provided buffers. More specifically, the trigger will be an
+interrupt issued by the hypervisor (QEMU, for example). Interrupt
+request handlers are registered for a virtqueue during the virtqueue
+setup process::
+
+	static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs,
+			struct virtqueue *vqs[], vq_callback_t *callbacks[],
+			const char * const names[], const bool *ctx)
+	{
+		struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+		int i, err, queue_idx = 0;
+
+		vp_dev->vqs = kcalloc(nvqs, sizeof(*vp_dev->vqs), GFP_KERNEL);
+		if (!vp_dev->vqs)
+			return -ENOMEM;
+
+		err = request_irq(vp_dev->pci_dev->irq, vp_interrupt, IRQF_SHARED,
+				dev_name(&vdev->dev), vp_dev);
+		...
+
+In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
+called and it will ultimately lead to a call to
+:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
+function::
+
+	irqreturn_t vring_interrupt(int irq, void *_vq)
+	{
+		struct vring_virtqueue *vq = to_vvq(_vq);
+
+		...
+
+		pr_debug("virtqueue callback for %p (%p)\n", vq, vq->vq.callback);
+		if (vq->vq.callback)
+			vq->vq.callback(&vq->vq);
+
+		return IRQ_HANDLED;
+	}
+
+Virtqueues are allocated by the guest kernel and their memory addresses
+are then communicated to the hypervisor so it can access them. In the
+host side there could be some differences in this process depending on
+who is acting as the hypervisor, but from the guest point of view this
+communication depends on the transport method used. So for a PCI device
+in QEMU, for example, the QEMU setups the PCI BARs (which are memory
+regions defined in the virtual PCI device) and the guest kernel maps
+them to virtual memory, so when it writes to them it's actually writing
+to the host userspace memory, acting like some kind of guest-host IPC
+mechanism on top of PCI::
+
+	/*
+	 * vp_modern_queue_address - set the virtqueue address
+	 * @mdev: the modern virtio-pci device
+	 * @index: the queue index
+	 * @desc_addr: address of the descriptor area
+	 * @driver_addr: address of the driver area
+	 * @device_addr: address of the device area
+	 */
+	void vp_modern_queue_address(struct virtio_pci_modern_device *mdev,
+				     u16 index, u64 desc_addr, u64 driver_addr,
+				     u64 device_addr)
+	{
+		struct virtio_pci_common_cfg __iomem *cfg = mdev->common;
+
+		vp_iowrite16(index, &cfg->queue_select);
+
+		vp_iowrite64_twopart(desc_addr, &cfg->queue_desc_lo,
+				     &cfg->queue_desc_hi);
+		vp_iowrite64_twopart(driver_addr, &cfg->queue_avail_lo,
+				     &cfg->queue_avail_hi);
+		vp_iowrite64_twopart(device_addr, &cfg->queue_used_lo,
+				     &cfg->queue_used_hi);
+	}
+
+
+Device discovery and probing
+============================
+
+How a virtio device is found and configured by the kernel depends on how
+the hypervisor defines it. Taking the `QEMU virtio-console
+<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/char/virtio-console.c>`__
+device as an example, which uses PCI as a transport method, the device
+will present itself in the PCI bus with vendor 0x1af4 (RedHat, Inc.) and
+device id 0x1003 (Virtio console), as defined in the spec, so the kernel
+will detect it as it would do with any other PCI device.
+
+In more detail:
+
+The virtio core subsystem is composed of multiple modules/drivers, among
+them:
+
+- virtio.c: implements the virtio bus driver.
+- virtio_mmio.c: implements the MMIO transport, this is a platform
+  driver (virtio-mmio).
+- virtio_pci_common.c and virtio_pci_modern.c (and, optionally,
+  virtio_pci_legacy.c): implement the virtio-pci PCI driver
+
+Then there are individual virtio drivers for specific device types, such
+as the virtio-console driver. These are registered to the virtio bus
+driver.
+
+During the PCI enumeration process, if a device is found to match the
+virtio-pci driver (according to the virtio-pci device table, any PCI
+device with vendor id = 0x1af4)::
+
+	/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
+	static const struct pci_device_id virtio_pci_id_table[] = {
+		{ PCI_DEVICE(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_ANY_ID) },
+		{ 0 }
+	};
+
+then the virtio-pci driver is probed. This includes setting up the
+device ``virtio_config_ops`` functions and its virtqueue handling
+methods, such as :c:func:`setup_vq` to allocate and configure them, (but
+this method is not called yet)::
+
+	int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev)
+	{
+		struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
+		struct pci_dev *pci_dev = vp_dev->pci_dev;
+		int err;
+
+		mdev->pci_dev = pci_dev;
+
+		err = vp_modern_probe(mdev);
+		if (err)
+			return err;
+
+		if (mdev->device)
+			vp_dev->vdev.config = &virtio_pci_config_ops;
+		else
+			vp_dev->vdev.config = &virtio_pci_config_nodev_ops;
+
+		vp_dev->config_vector = vp_config_vector;
+		vp_dev->setup_vq = setup_vq;
+		vp_dev->del_vq = del_vq;
+		vp_dev->isr = mdev->isr;
+		vp_dev->vdev.id = mdev->id;
+
+		return 0;
+	}
+
+If the probing goes well, the device is registered to the virtio bus::
+
+	static int virtio_pci_probe(struct pci_dev *pci_dev,
+				    const struct pci_device_id *id)
+	{
+		...
+
+		if (force_legacy) {
+			rc = virtio_pci_legacy_probe(vp_dev);
+			/* Also try modern mode if we can't map BAR0 (no IO space). */
+			if (rc == -ENODEV || rc == -ENOMEM)
+				rc = virtio_pci_modern_probe(vp_dev);
+			if (rc)
+				goto err_probe;
+		} else {
+			rc = virtio_pci_modern_probe(vp_dev);
+			if (rc == -ENODEV)
+				rc = virtio_pci_legacy_probe(vp_dev);
+			if (rc)
+				goto err_probe;
+		}
+
+		...
+
+		rc = register_virtio_device(&vp_dev->vdev);
+
+When the device is registered to the virtio bus the kernel will look
+for a driver in the bus that can handle the device and call that
+driver's ``probe`` method.
+
+It's at this stage that the virtqueues will be setup by calling the
+appropriate `virtio_find` helper function, such as
+:c:func:`virtio_find_single_vq` or :c:func:`virtio_find_vqs`, which will
+end up calling the device ``find_vqs`` config op (transport-specific),
+which will allocate the virtqueues and configure them. In the case of
+virtio PCI devices, that's done by the ``setup_vq`` method seen above.
+
+
+References
+==========
+
+[1] Virtio Spec v1.1:
+https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
+
+[2] Virtqueues and virtio ring: How the data travels
+https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
+
+.. rubric:: Footnotes
+
+.. [#f1] that's why they may be also referred as virtrings.
diff --git a/Documentation/driver-api/virtio/writing_virtio_drivers.rst b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
new file mode 100644
index 000000000000..5cb088b817ae
--- /dev/null
+++ b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
@@ -0,0 +1,190 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _writing_virtio_drivers:
+
+======================
+Writing Virtio Drivers
+======================
+
+Introduction
+============
+
+The `Virtio spec
+<https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1930005>`__
+defines all the supported Virtio device types. Since these devices are,
+by definition, meant as abstractions for a wide variety of real
+hardware, the addition of new virtio drivers is not expected to be very
+frequent. Still, this document serves as a basic guideline for driver
+programmers that need to hack a new Virtio driver or understand the
+essentials of the existing ones. See :ref:`Virtio on Linux <virtio>` for
+a general overview of Virtio.
+
+
+Driver boilerplate
+==================
+
+As a bare minimum, a virtio driver should register in the virtio bus and
+configure the virtqueues for the device according to its spec, the
+configuration of the virtqueues in the driver side must match the
+virtqueue definitions in the device. A basic driver skeleton could look
+like this::
+
+	#include <linux/virtio.h>
+	#include <linux/virtio_ids.h>
+	#include <linux/virtio_config.h>
+	#include <linux/module.h>
+
+	/* device private data (one per device) */
+	struct virtio_dummy_dev {
+		struct virtqueue *vq;
+	};
+
+	static void virtio_dummy_recv_cb(struct virtqueue *vq)
+	{
+		struct virtio_dummy_dev *dev = vq->vdev->priv;
+		char *buf;
+		unsigned int len;
+
+		buf = virtqueue_get_buf(dev->vq, &len);
+		/* spurious callback? */
+		if (!buf)
+			return;
+
+		/* Process the received data */
+	}
+
+	static int virtio_dummy_probe(struct virtio_device *vdev)
+	{
+		struct virtio_dummy_dev *dev = NULL;
+
+		/* initialize device data */
+		dev = kzalloc(sizeof(struct virtio_dummy_dev), GFP_KERNEL);
+		if (!dev)
+			return -ENOMEM;
+
+		/* the device has a single virtqueue */
+		dev->vq = virtio_find_single_vq(vdev, virtio_dummy_recv_cb, "input");
+		if (IS_ERR(dev->vq)) {
+			kfree(dev);
+			return PTR_ERR(dev->vq);
+
+		}
+		vdev->priv = dev;
+
+		return 0;
+	}
+
+	static void virtio_dummy_remove(struct virtio_device *vdev)
+	{
+		struct virtio_dummy_dev *dev = vdev->priv;
+
+		/*
+		 * Disable vq interrupts: equivalent to
+		 * vdev->config->reset(vdev)
+		 */
+		virtio_reset_device(vdev);
+
+		/* remove virtqueues */
+		vdev->config->del_vqs(vdev);
+
+		kfree(dev);
+	}
+
+	static const struct virtio_device_id id_table[] = {
+		{ VIRTIO_ID_DUMMY, VIRTIO_DEV_ANY_ID },
+		{ 0 },
+	};
+
+	static struct virtio_driver virtio_dummy_driver = {
+		.driver.name =	KBUILD_MODNAME,
+		.driver.owner =	THIS_MODULE,
+		.id_table =	id_table,
+		.probe =	virtio_dummy_probe,
+		.remove =	virtio_dummy_remove,
+	};
+
+	module_virtio_driver(virtio_dummy_driver);
+	MODULE_DEVICE_TABLE(virtio, id_table);
+	MODULE_DESCRIPTION("Dummy virtio driver");
+	MODULE_LICENSE("GPL");
+
+This assumes the device is of a new virtio device type not defined
+before: ``VIRTIO_DEVICE_DUMMY``, which we should define in
+include/uapi/linux/virtio_ids.h. The device has only one virtqueue which
+is meant to be used to send data from the host to the guest.
+
+If your driver doesn't have to do anything special in its ``init`` and
+``exit`` methods, you can use the :c:func:`module_virtio_driver` helper to
+reduce the amount of boilerplate code.
+
+The ``probe`` method does the minimum driver setup in this case
+(memory allocation for the device data) and initializes the
+virtqueue. The virtqueues are automatically enabled after ``probe``
+returns, sending the appropriate "DRIVER_OK" status signal to the
+device. If the virtqueues need to be enabled before ``probe`` ends, they
+can be manually enabled by calling :c:func:`virtio_device_ready`:
+
+.. kernel-doc:: include/linux/virtio_config.h
+    :identifiers: virtio_device_ready
+
+
+Sending and receiving data
+==========================
+
+The :c:func:`virtio_dummy_recv_cb` callback in the code above will be
+triggered when the device notifies the driver after it finishes
+processing a descriptor or descriptor chain, either for reading or
+writing. However, that's only the second half of the Virtio
+device-driver communication process, as the communication is always
+started by the driver regardless of the direction of the data transfer.
+
+To configure a buffer transfer from the driver to the device, first you
+have to add the buffers -- packed as `scatterlists` -- to the
+appropriate virtqueue using any of the :c:func:`virtqueue_add_inbuf`,
+:c:func:`virtqueue_add_outbuf` or :c:func:`virtqueue_add_sgs`, depending
+on whether you need to add one input `scatterlist` (for the device to
+fill in), one output `scatterlist` (for the device to consume) or
+multiple `scatterlists`, respectively. Then, once the virtqueue is set
+up, a call to :c:func:`virtqueue_kick` sends a notification that will be
+serviced by the hypervisor that implements the device::
+
+	struct scatterlist sg[1];
+	sg_init_one(sg, buffer, BUFLEN);
+	virtqueue_add_inbuf(dev->vq, sg, 1, buffer, GFP_ATOMIC);
+	virtqueue_kick(dev->vq);
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_add_inbuf
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_add_outbuf
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_add_sgs
+
+Then, after the device has read or written the buffers prepared by the
+driver and notifies it back, the driver can call
+:c:func:`virtqueue_get_buf` to read the data produced by the device (if
+the virtqueue was set up with input buffers) or simply to reclaim the
+buffers if they were already consumed by the device:
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_get_buf_ctx
+
+The virtqueue callbacks can be disabled and re-enabled using the
+:c:func:`virtqueue_disable_cb` and the family of
+:c:func:`virtqueue_enable_cb` functions respectively. See
+drivers/virtio/virtio_ring.c for more details:
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_disable_cb
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_enable_cb
+
+
+References
+==========
+
+[1] Virtio Spec v1.1:
+https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
diff --git a/MAINTAINERS b/MAINTAINERS
index c0f958dfd289..1ea754e45e07 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -21465,6 +21465,7 @@ S:	Maintained
 F:	Documentation/ABI/testing/sysfs-bus-vdpa
 F:	Documentation/ABI/testing/sysfs-class-vduse
 F:	Documentation/devicetree/bindings/virtio/
+F:	Documentation/driver-api/virtio/
 F:	drivers/block/virtio_blk.c
 F:	drivers/crypto/virtio/
 F:	drivers/net/virtio_net.c
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index d8fdf170637c..fd8440e85933 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -11,7 +11,7 @@
 #include <linux/gfp.h>
 
 /**
- * virtqueue - a queue to register buffers for sending or receiving.
+ * struct virtqueue - a queue to register buffers for sending or receiving.
  * @list: the chain of virtqueues for this device
  * @callback: the function to call when buffers are consumed (can be NULL).
  * @name: the name of this virtqueue (mainly for debugging)
@@ -90,7 +90,7 @@ dma_addr_t virtqueue_get_avail_addr(struct virtqueue *vq);
 dma_addr_t virtqueue_get_used_addr(struct virtqueue *vq);
 
 /**
- * virtio_device - representation of a device using virtio
+ * struct virtio_device - representation of a device using virtio
  * @index: unique position on the virtio bus
  * @failed: saved value for VIRTIO_CONFIG_S_FAILED bit (for restore)
  * @config_enabled: configuration change reporting enabled
@@ -146,7 +146,7 @@ size_t virtio_max_dma_size(struct virtio_device *vdev);
 	list_for_each_entry(vq, &vdev->vqs, list)
 
 /**
- * virtio_driver - operations for a virtio I/O driver
+ * struct virtio_driver - operations for a virtio I/O driver
  * @driver: underlying device driver (populate name and owner).
  * @id_table: the ids serviced by this driver.
  * @feature_table: an array of feature numbers supported by this driver.
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index b47c2e7ed0ee..997801018ae4 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -225,7 +225,7 @@ int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs,
 
 /**
  * virtio_synchronize_cbs - synchronize with virtqueue callbacks
- * @vdev: the device
+ * @dev: the device
  */
 static inline
 void virtio_synchronize_cbs(struct virtio_device *dev)
@@ -244,7 +244,7 @@ void virtio_synchronize_cbs(struct virtio_device *dev)
 
 /**
  * virtio_device_ready - enable vq use in probe function
- * @vdev: the device
+ * @dev: the device
  *
  * Driver must call this to use vqs in the probe function.
  *
@@ -292,7 +292,7 @@ const char *virtio_bus_name(struct virtio_device *vdev)
 /**
  * virtqueue_set_affinity - setting affinity for a virtqueue
  * @vq: the virtqueue
- * @cpu: the cpu no.
+ * @cpu_mask: the cpu mask
  *
  * Pay attention the function are best-effort: the affinity hint may not be set
  * due to config support, irq type and sharing.
diff --git a/include/uapi/linux/virtio_ring.h b/include/uapi/linux/virtio_ring.h
index 476d3e5c0fe7..f8c20d3de8da 100644
--- a/include/uapi/linux/virtio_ring.h
+++ b/include/uapi/linux/virtio_ring.h
@@ -93,15 +93,21 @@
 #define VRING_USED_ALIGN_SIZE 4
 #define VRING_DESC_ALIGN_SIZE 16
 
-/* Virtio ring descriptors: 16 bytes.  These can chain together via "next". */
+/**
+ * struct vring_desc - Virtio ring descriptors,
+ * 16 bytes long. These can chain together via @next.
+ *
+ * @addr: buffer address (guest-physical)
+ * @len: buffer length
+ * @flags: descriptor flags
+ * @next: index of the next descriptor in the chain,
+ *        if the VRING_DESC_F_NEXT flag is set. We chain unused
+ *        descriptors via this, too.
+ */
 struct vring_desc {
-	/* Address (guest-physical). */
 	__virtio64 addr;
-	/* Length. */
 	__virtio32 len;
-	/* The flags as indicated above. */
 	__virtio16 flags;
-	/* We chain unused descriptors via this, too */
 	__virtio16 next;
 };
 
-- 
2.25.1

[PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
Minor fixes to existing virtio kerneldocs.

Signed-off-by: Ricardo Cañuelo <ricardo.canuelo@collabora.com>
---
 Documentation/driver-api/index.rst            |   1 +
 Documentation/driver-api/virtio/index.rst     |  11 +
 Documentation/driver-api/virtio/virtio.rst    | 274 ++++++++++++++++++
 .../virtio/writing_virtio_drivers.rst         | 190 ++++++++++++
 MAINTAINERS                                   |   1 +
 include/linux/virtio.h                        |   6 +-
 include/linux/virtio_config.h                 |   6 +-
 include/uapi/linux/virtio_ring.h              |  16 +-
 8 files changed, 494 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/driver-api/virtio/index.rst
 create mode 100644 Documentation/driver-api/virtio/virtio.rst
 create mode 100644 Documentation/driver-api/virtio/writing_virtio_drivers.rst

diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index d3a58f77328e..30a3de452b1d 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -106,6 +106,7 @@ available subsections can be seen below.
    vfio-mediated-device
    vfio
    vfio-pci-device-specific-driver-acceptance
+   virtio/index
    xilinx/index
    xillybus
    zorro
diff --git a/Documentation/driver-api/virtio/index.rst b/Documentation/driver-api/virtio/index.rst
new file mode 100644
index 000000000000..528b14b291e3
--- /dev/null
+++ b/Documentation/driver-api/virtio/index.rst
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======
+Virtio
+======
+
+.. toctree::
+   :maxdepth: 1
+
+   virtio
+   writing_virtio_drivers
diff --git a/Documentation/driver-api/virtio/virtio.rst b/Documentation/driver-api/virtio/virtio.rst
new file mode 100644
index 000000000000..049a8aefad92
--- /dev/null
+++ b/Documentation/driver-api/virtio/virtio.rst
@@ -0,0 +1,274 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _virtio:
+
+===============
+Virtio on Linux
+===============
+
+Introduction
+============
+
+Virtio is an open standard interface for virtual machines to access
+paravirtualized devices, ie. devices that aren't emulated by a
+hypervisor, but rather real host devices that are exposed by the
+hypervisor to the guest to achieve native performance. In other words,
+it provides a communication mechanism for a guest OS to use devices on
+the host machine.
+
+The concrete hardware details of the real host devices are abstracted in the
+hypervisor, which provides a set of simplified virtual devices that
+implement the Virtio protocol. These devices are defined in the Virtio
+spec [1] and they're the devices that the guest OS will ultimately
+handle. So, in that regard, the guest OS knows it's running in a virtual
+environment and that it needs to use the appropriate Virtio drivers to
+handle the devices instead of the regular device drivers it'd use in a
+native or purely virtual environment (with emulated devices).
+
+
+Device - Driver communication: virtqueues
+=========================================
+
+Although the Virtio devices are really an abstraction layer in the
+hypervisor, they're exposed to the guest as if they are physical devices
+either PCI or MMIO-based. We refer to that as the transport method and
+it's orthogonal to the device itself. The Virtio spec defines these two
+and other transport methods in detail, including device discovery,
+capabilities and interrupt handling.
+
+The communication between the driver in the guest OS and the device in
+the hypervisor is done through shared memory (that's what makes virtio
+devices so efficient) using specialized data structures called
+virtqueues. The use of these virtqueues for data transfers is referred
+to as the data plane, while the process of setting them up and
+coordinating the driver and the device during the setup stage is called
+the control plane.
+
+Virtqueues, which are used to communicate the driver and the device
+regardless of the transport method used, are actually ring buffers
+[#f1]_ of buffer descriptors similar to the ones used in a network
+device:
+
+.. kernel-doc:: include/uapi/linux/virtio_ring.h
+    :identifiers: struct vring_desc
+
+All the buffers the descriptors point to are allocated by the guest and
+used by the host either for reading or for writing but not for both.
+
+Refer to [2] for an illustrated overview of virtqueues and how the host
+device and the guest driver communicate.
+
+Each virtqueue defines three areas: the descriptor area (`desc`), which
+is an array of descriptors as described above, and the avail and used
+rings.
+
+The `avail` ring is where the driver puts the indexes of the descriptors
+that it has set up for the device to consume. The `used` ring is used by
+the device to return the consumed buffers (read or written) to the
+driver. For each used descriptor, the ring also contains the used buffer
+length in case it was written.
+
+These rings are laid out in the :c:type:`vring_virtqueue` struct
+together with other necessary management data, including a pointer to a
+transport-specific ``notify`` function that is used to let the host side
+know when new buffers have been put in the virtqueue for
+processing. Embedded in this struct is the :c:type:`virtqueue` struct,
+which is the data structure that's ultimately used by virtio drivers:
+
+.. kernel-doc:: include/linux/virtio.h
+    :identifiers: struct virtqueue
+
+The callback saved in this struct serves the same purpose as the
+``notify`` function in :c:type:`vring_virtqueue` but in the
+opposite direction. That is, the callback is triggered when the host has
+used the provided buffers. More specifically, the trigger will be an
+interrupt issued by the hypervisor (QEMU, for example). Interrupt
+request handlers are registered for a virtqueue during the virtqueue
+setup process::
+
+	static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs,
+			struct virtqueue *vqs[], vq_callback_t *callbacks[],
+			const char * const names[], const bool *ctx)
+	{
+		struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+		int i, err, queue_idx = 0;
+
+		vp_dev->vqs = kcalloc(nvqs, sizeof(*vp_dev->vqs), GFP_KERNEL);
+		if (!vp_dev->vqs)
+			return -ENOMEM;
+
+		err = request_irq(vp_dev->pci_dev->irq, vp_interrupt, IRQF_SHARED,
+				dev_name(&vdev->dev), vp_dev);
+		...
+
+In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
+called and it will ultimately lead to a call to
+:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
+function::
+
+	irqreturn_t vring_interrupt(int irq, void *_vq)
+	{
+		struct vring_virtqueue *vq = to_vvq(_vq);
+
+		...
+
+		pr_debug("virtqueue callback for %p (%p)\n", vq, vq->vq.callback);
+		if (vq->vq.callback)
+			vq->vq.callback(&vq->vq);
+
+		return IRQ_HANDLED;
+	}
+
+Virtqueues are allocated by the guest kernel and their memory addresses
+are then communicated to the hypervisor so it can access them. In the
+host side there could be some differences in this process depending on
+who is acting as the hypervisor, but from the guest point of view this
+communication depends on the transport method used. So for a PCI device
+in QEMU, for example, the QEMU setups the PCI BARs (which are memory
+regions defined in the virtual PCI device) and the guest kernel maps
+them to virtual memory, so when it writes to them it's actually writing
+to the host userspace memory, acting like some kind of guest-host IPC
+mechanism on top of PCI::
+
+	/*
+	 * vp_modern_queue_address - set the virtqueue address
+	 * @mdev: the modern virtio-pci device
+	 * @index: the queue index
+	 * @desc_addr: address of the descriptor area
+	 * @driver_addr: address of the driver area
+	 * @device_addr: address of the device area
+	 */
+	void vp_modern_queue_address(struct virtio_pci_modern_device *mdev,
+				     u16 index, u64 desc_addr, u64 driver_addr,
+				     u64 device_addr)
+	{
+		struct virtio_pci_common_cfg __iomem *cfg = mdev->common;
+
+		vp_iowrite16(index, &cfg->queue_select);
+
+		vp_iowrite64_twopart(desc_addr, &cfg->queue_desc_lo,
+				     &cfg->queue_desc_hi);
+		vp_iowrite64_twopart(driver_addr, &cfg->queue_avail_lo,
+				     &cfg->queue_avail_hi);
+		vp_iowrite64_twopart(device_addr, &cfg->queue_used_lo,
+				     &cfg->queue_used_hi);
+	}
+
+
+Device discovery and probing
+============================
+
+How a virtio device is found and configured by the kernel depends on how
+the hypervisor defines it. Taking the `QEMU virtio-console
+<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/char/virtio-console.c>`__
+device as an example, which uses PCI as a transport method, the device
+will present itself in the PCI bus with vendor 0x1af4 (RedHat, Inc.) and
+device id 0x1003 (Virtio console), as defined in the spec, so the kernel
+will detect it as it would do with any other PCI device.
+
+In more detail:
+
+The virtio core subsystem is composed of multiple modules/drivers, among
+them:
+
+- virtio.c: implements the virtio bus driver.
+- virtio_mmio.c: implements the MMIO transport, this is a platform
+  driver (virtio-mmio).
+- virtio_pci_common.c and virtio_pci_modern.c (and, optionally,
+  virtio_pci_legacy.c): implement the virtio-pci PCI driver
+
+Then there are individual virtio drivers for specific device types, such
+as the virtio-console driver. These are registered to the virtio bus
+driver.
+
+During the PCI enumeration process, if a device is found to match the
+virtio-pci driver (according to the virtio-pci device table, any PCI
+device with vendor id = 0x1af4)::
+
+	/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
+	static const struct pci_device_id virtio_pci_id_table[] = {
+		{ PCI_DEVICE(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_ANY_ID) },
+		{ 0 }
+	};
+
+then the virtio-pci driver is probed. This includes setting up the
+device ``virtio_config_ops`` functions and its virtqueue handling
+methods, such as :c:func:`setup_vq` to allocate and configure them, (but
+this method is not called yet)::
+
+	int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev)
+	{
+		struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
+		struct pci_dev *pci_dev = vp_dev->pci_dev;
+		int err;
+
+		mdev->pci_dev = pci_dev;
+
+		err = vp_modern_probe(mdev);
+		if (err)
+			return err;
+
+		if (mdev->device)
+			vp_dev->vdev.config = &virtio_pci_config_ops;
+		else
+			vp_dev->vdev.config = &virtio_pci_config_nodev_ops;
+
+		vp_dev->config_vector = vp_config_vector;
+		vp_dev->setup_vq = setup_vq;
+		vp_dev->del_vq = del_vq;
+		vp_dev->isr = mdev->isr;
+		vp_dev->vdev.id = mdev->id;
+
+		return 0;
+	}
+
+If the probing goes well, the device is registered to the virtio bus::
+
+	static int virtio_pci_probe(struct pci_dev *pci_dev,
+				    const struct pci_device_id *id)
+	{
+		...
+
+		if (force_legacy) {
+			rc = virtio_pci_legacy_probe(vp_dev);
+			/* Also try modern mode if we can't map BAR0 (no IO space). */
+			if (rc == -ENODEV || rc == -ENOMEM)
+				rc = virtio_pci_modern_probe(vp_dev);
+			if (rc)
+				goto err_probe;
+		} else {
+			rc = virtio_pci_modern_probe(vp_dev);
+			if (rc == -ENODEV)
+				rc = virtio_pci_legacy_probe(vp_dev);
+			if (rc)
+				goto err_probe;
+		}
+
+		...
+
+		rc = register_virtio_device(&vp_dev->vdev);
+
+When the device is registered to the virtio bus the kernel will look
+for a driver in the bus that can handle the device and call that
+driver's ``probe`` method.
+
+It's at this stage that the virtqueues will be setup by calling the
+appropriate `virtio_find` helper function, such as
+:c:func:`virtio_find_single_vq` or :c:func:`virtio_find_vqs`, which will
+end up calling the device ``find_vqs`` config op (transport-specific),
+which will allocate the virtqueues and configure them. In the case of
+virtio PCI devices, that's done by the ``setup_vq`` method seen above.
+
+
+References
+==========
+
+[1] Virtio Spec v1.1:
+https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
+
+[2] Virtqueues and virtio ring: How the data travels
+https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
+
+.. rubric:: Footnotes
+
+.. [#f1] that's why they may be also referred as virtrings.
diff --git a/Documentation/driver-api/virtio/writing_virtio_drivers.rst b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
new file mode 100644
index 000000000000..5cb088b817ae
--- /dev/null
+++ b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
@@ -0,0 +1,190 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _writing_virtio_drivers:
+
+======================
+Writing Virtio Drivers
+======================
+
+Introduction
+============
+
+The `Virtio spec
+<https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1930005>`__
+defines all the supported Virtio device types. Since these devices are,
+by definition, meant as abstractions for a wide variety of real
+hardware, the addition of new virtio drivers is not expected to be very
+frequent. Still, this document serves as a basic guideline for driver
+programmers that need to hack a new Virtio driver or understand the
+essentials of the existing ones. See :ref:`Virtio on Linux <virtio>` for
+a general overview of Virtio.
+
+
+Driver boilerplate
+==================
+
+As a bare minimum, a virtio driver should register in the virtio bus and
+configure the virtqueues for the device according to its spec, the
+configuration of the virtqueues in the driver side must match the
+virtqueue definitions in the device. A basic driver skeleton could look
+like this::
+
+	#include <linux/virtio.h>
+	#include <linux/virtio_ids.h>
+	#include <linux/virtio_config.h>
+	#include <linux/module.h>
+
+	/* device private data (one per device) */
+	struct virtio_dummy_dev {
+		struct virtqueue *vq;
+	};
+
+	static void virtio_dummy_recv_cb(struct virtqueue *vq)
+	{
+		struct virtio_dummy_dev *dev = vq->vdev->priv;
+		char *buf;
+		unsigned int len;
+
+		buf = virtqueue_get_buf(dev->vq, &len);
+		/* spurious callback? */
+		if (!buf)
+			return;
+
+		/* Process the received data */
+	}
+
+	static int virtio_dummy_probe(struct virtio_device *vdev)
+	{
+		struct virtio_dummy_dev *dev = NULL;
+
+		/* initialize device data */
+		dev = kzalloc(sizeof(struct virtio_dummy_dev), GFP_KERNEL);
+		if (!dev)
+			return -ENOMEM;
+
+		/* the device has a single virtqueue */
+		dev->vq = virtio_find_single_vq(vdev, virtio_dummy_recv_cb, "input");
+		if (IS_ERR(dev->vq)) {
+			kfree(dev);
+			return PTR_ERR(dev->vq);
+
+		}
+		vdev->priv = dev;
+
+		return 0;
+	}
+
+	static void virtio_dummy_remove(struct virtio_device *vdev)
+	{
+		struct virtio_dummy_dev *dev = vdev->priv;
+
+		/*
+		 * Disable vq interrupts: equivalent to
+		 * vdev->config->reset(vdev)
+		 */
+		virtio_reset_device(vdev);
+
+		/* remove virtqueues */
+		vdev->config->del_vqs(vdev);
+
+		kfree(dev);
+	}
+
+	static const struct virtio_device_id id_table[] = {
+		{ VIRTIO_ID_DUMMY, VIRTIO_DEV_ANY_ID },
+		{ 0 },
+	};
+
+	static struct virtio_driver virtio_dummy_driver = {
+		.driver.name =	KBUILD_MODNAME,
+		.driver.owner =	THIS_MODULE,
+		.id_table =	id_table,
+		.probe =	virtio_dummy_probe,
+		.remove =	virtio_dummy_remove,
+	};
+
+	module_virtio_driver(virtio_dummy_driver);
+	MODULE_DEVICE_TABLE(virtio, id_table);
+	MODULE_DESCRIPTION("Dummy virtio driver");
+	MODULE_LICENSE("GPL");
+
+This assumes the device is of a new virtio device type not defined
+before: ``VIRTIO_DEVICE_DUMMY``, which we should define in
+include/uapi/linux/virtio_ids.h. The device has only one virtqueue which
+is meant to be used to send data from the host to the guest.
+
+If your driver doesn't have to do anything special in its ``init`` and
+``exit`` methods, you can use the :c:func:`module_virtio_driver` helper to
+reduce the amount of boilerplate code.
+
+The ``probe`` method does the minimum driver setup in this case
+(memory allocation for the device data) and initializes the
+virtqueue. The virtqueues are automatically enabled after ``probe``
+returns, sending the appropriate "DRIVER_OK" status signal to the
+device. If the virtqueues need to be enabled before ``probe`` ends, they
+can be manually enabled by calling :c:func:`virtio_device_ready`:
+
+.. kernel-doc:: include/linux/virtio_config.h
+    :identifiers: virtio_device_ready
+
+
+Sending and receiving data
+==========================
+
+The :c:func:`virtio_dummy_recv_cb` callback in the code above will be
+triggered when the device notifies the driver after it finishes
+processing a descriptor or descriptor chain, either for reading or
+writing. However, that's only the second half of the Virtio
+device-driver communication process, as the communication is always
+started by the driver regardless of the direction of the data transfer.
+
+To configure a buffer transfer from the driver to the device, first you
+have to add the buffers -- packed as `scatterlists` -- to the
+appropriate virtqueue using any of the :c:func:`virtqueue_add_inbuf`,
+:c:func:`virtqueue_add_outbuf` or :c:func:`virtqueue_add_sgs`, depending
+on whether you need to add one input `scatterlist` (for the device to
+fill in), one output `scatterlist` (for the device to consume) or
+multiple `scatterlists`, respectively. Then, once the virtqueue is set
+up, a call to :c:func:`virtqueue_kick` sends a notification that will be
+serviced by the hypervisor that implements the device::
+
+	struct scatterlist sg[1];
+	sg_init_one(sg, buffer, BUFLEN);
+	virtqueue_add_inbuf(dev->vq, sg, 1, buffer, GFP_ATOMIC);
+	virtqueue_kick(dev->vq);
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_add_inbuf
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_add_outbuf
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_add_sgs
+
+Then, after the device has read or written the buffers prepared by the
+driver and notifies it back, the driver can call
+:c:func:`virtqueue_get_buf` to read the data produced by the device (if
+the virtqueue was set up with input buffers) or simply to reclaim the
+buffers if they were already consumed by the device:
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_get_buf_ctx
+
+The virtqueue callbacks can be disabled and re-enabled using the
+:c:func:`virtqueue_disable_cb` and the family of
+:c:func:`virtqueue_enable_cb` functions respectively. See
+drivers/virtio/virtio_ring.c for more details:
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_disable_cb
+
+.. kernel-doc:: drivers/virtio/virtio_ring.c
+    :identifiers: virtqueue_enable_cb
+
+
+References
+==========
+
+[1] Virtio Spec v1.1:
+https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
diff --git a/MAINTAINERS b/MAINTAINERS
index c0f958dfd289..1ea754e45e07 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -21465,6 +21465,7 @@ S:	Maintained
 F:	Documentation/ABI/testing/sysfs-bus-vdpa
 F:	Documentation/ABI/testing/sysfs-class-vduse
 F:	Documentation/devicetree/bindings/virtio/
+F:	Documentation/driver-api/virtio/
 F:	drivers/block/virtio_blk.c
 F:	drivers/crypto/virtio/
 F:	drivers/net/virtio_net.c
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index d8fdf170637c..fd8440e85933 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -11,7 +11,7 @@
 #include <linux/gfp.h>
 
 /**
- * virtqueue - a queue to register buffers for sending or receiving.
+ * struct virtqueue - a queue to register buffers for sending or receiving.
  * @list: the chain of virtqueues for this device
  * @callback: the function to call when buffers are consumed (can be NULL).
  * @name: the name of this virtqueue (mainly for debugging)
@@ -90,7 +90,7 @@ dma_addr_t virtqueue_get_avail_addr(struct virtqueue *vq);
 dma_addr_t virtqueue_get_used_addr(struct virtqueue *vq);
 
 /**
- * virtio_device - representation of a device using virtio
+ * struct virtio_device - representation of a device using virtio
  * @index: unique position on the virtio bus
  * @failed: saved value for VIRTIO_CONFIG_S_FAILED bit (for restore)
  * @config_enabled: configuration change reporting enabled
@@ -146,7 +146,7 @@ size_t virtio_max_dma_size(struct virtio_device *vdev);
 	list_for_each_entry(vq, &vdev->vqs, list)
 
 /**
- * virtio_driver - operations for a virtio I/O driver
+ * struct virtio_driver - operations for a virtio I/O driver
  * @driver: underlying device driver (populate name and owner).
  * @id_table: the ids serviced by this driver.
  * @feature_table: an array of feature numbers supported by this driver.
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index b47c2e7ed0ee..997801018ae4 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -225,7 +225,7 @@ int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs,
 
 /**
  * virtio_synchronize_cbs - synchronize with virtqueue callbacks
- * @vdev: the device
+ * @dev: the device
  */
 static inline
 void virtio_synchronize_cbs(struct virtio_device *dev)
@@ -244,7 +244,7 @@ void virtio_synchronize_cbs(struct virtio_device *dev)
 
 /**
  * virtio_device_ready - enable vq use in probe function
- * @vdev: the device
+ * @dev: the device
  *
  * Driver must call this to use vqs in the probe function.
  *
@@ -292,7 +292,7 @@ const char *virtio_bus_name(struct virtio_device *vdev)
 /**
  * virtqueue_set_affinity - setting affinity for a virtqueue
  * @vq: the virtqueue
- * @cpu: the cpu no.
+ * @cpu_mask: the cpu mask
  *
  * Pay attention the function are best-effort: the affinity hint may not be set
  * due to config support, irq type and sharing.
diff --git a/include/uapi/linux/virtio_ring.h b/include/uapi/linux/virtio_ring.h
index 476d3e5c0fe7..f8c20d3de8da 100644
--- a/include/uapi/linux/virtio_ring.h
+++ b/include/uapi/linux/virtio_ring.h
@@ -93,15 +93,21 @@
 #define VRING_USED_ALIGN_SIZE 4
 #define VRING_DESC_ALIGN_SIZE 16
 
-/* Virtio ring descriptors: 16 bytes.  These can chain together via "next". */
+/**
+ * struct vring_desc - Virtio ring descriptors,
+ * 16 bytes long. These can chain together via @next.
+ *
+ * @addr: buffer address (guest-physical)
+ * @len: buffer length
+ * @flags: descriptor flags
+ * @next: index of the next descriptor in the chain,
+ *        if the VRING_DESC_F_NEXT flag is set. We chain unused
+ *        descriptors via this, too.
+ */
 struct vring_desc {
-	/* Address (guest-physical). */
 	__virtio64 addr;
-	/* Length. */
 	__virtio32 len;
-	/* The flags as indicated above. */
 	__virtio16 flags;
-	/* We chain unused descriptors via this, too */
 	__virtio16 next;
 };
 
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Bagas Sanjaya]

On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
> Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
> Minor fixes to existing virtio kerneldocs.
> 

Hi,

I can't apply this patch on top of mainline. On what commit (and what
tree) was it based from?

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Cornelia Huck]

On Tue, Aug 02 2022, Ricardo Cañuelo <ricardo.canuelo@collabora.com> wrote:

> Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
> Minor fixes to existing virtio kerneldocs.

I think kerneldoc updates should be split out into a separate patch.

[No proper review, just some things I noticed below.]

>
> Signed-off-by: Ricardo Cañuelo <ricardo.canuelo@collabora.com>
> ---
>  Documentation/driver-api/index.rst            |   1 +
>  Documentation/driver-api/virtio/index.rst     |  11 +
>  Documentation/driver-api/virtio/virtio.rst    | 274 ++++++++++++++++++
>  .../virtio/writing_virtio_drivers.rst         | 190 ++++++++++++
>  MAINTAINERS                                   |   1 +
>  include/linux/virtio.h                        |   6 +-
>  include/linux/virtio_config.h                 |   6 +-
>  include/uapi/linux/virtio_ring.h              |  16 +-
>  8 files changed, 494 insertions(+), 11 deletions(-)
>  create mode 100644 Documentation/driver-api/virtio/index.rst
>  create mode 100644 Documentation/driver-api/virtio/virtio.rst
>  create mode 100644 Documentation/driver-api/virtio/writing_virtio_drivers.rst
>

(...)

> diff --git a/Documentation/driver-api/virtio/virtio.rst b/Documentation/driver-api/virtio/virtio.rst
> new file mode 100644
> index 000000000000..049a8aefad92
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/virtio.rst
> @@ -0,0 +1,274 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _virtio:
> +
> +===============
> +Virtio on Linux
> +===============
> +
> +Introduction
> +============
> +
> +Virtio is an open standard interface for virtual machines to access

There are devices that implement virtio in hardware.

> +paravirtualized devices, ie. devices that aren't emulated by a
> +hypervisor, but rather real host devices that are exposed by the
> +hypervisor to the guest to achieve native performance. In other words,
> +it provides a communication mechanism for a guest OS to use devices on
> +the host machine.
> +
> +The concrete hardware details of the real host devices are abstracted in the
> +hypervisor, which provides a set of simplified virtual devices that
> +implement the Virtio protocol. These devices are defined in the Virtio
> +spec [1] and they're the devices that the guest OS will ultimately
> +handle. So, in that regard, the guest OS knows it's running in a virtual
> +environment and that it needs to use the appropriate Virtio drivers to
> +handle the devices instead of the regular device drivers it'd use in a
> +native or purely virtual environment (with emulated devices).
> +
> +
> +Device - Driver communication: virtqueues
> +=========================================
> +
> +Although the Virtio devices are really an abstraction layer in the
> +hypervisor, they're exposed to the guest as if they are physical devices
> +either PCI or MMIO-based. We refer to that as the transport method and

Please also mention CCW (s390 channel devices), as Linux supports it as
well.

> +it's orthogonal to the device itself. The Virtio spec defines these two
> +and other transport methods in detail, including device discovery,
> +capabilities and interrupt handling.

(...)

> +Device discovery and probing
> +============================
> +
> +How a virtio device is found and configured by the kernel depends on how
> +the hypervisor defines it. Taking the `QEMU virtio-console
> +<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/char/virtio-console.c>`__
> +device as an example, which uses PCI as a transport method, the device

Should be the "virtio-console (...) device when using PCI...", I guess?

> +will present itself in the PCI bus with vendor 0x1af4 (RedHat, Inc.) and
> +device id 0x1003 (Virtio console), as defined in the spec, so the kernel
> +will detect it as it would do with any other PCI device.

(...)

> +References
> +==========
> +
> +[1] Virtio Spec v1.1:
> +https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html

v1.2 is out now :)

(I think it would be better to refer to the base spec directory?)

> +
> +[2] Virtqueues and virtio ring: How the data travels
> +https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
> +
> +.. rubric:: Footnotes
> +
> +.. [#f1] that's why they may be also referred as virtrings.
> diff --git a/Documentation/driver-api/virtio/writing_virtio_drivers.rst b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> new file mode 100644
> index 000000000000..5cb088b817ae
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> @@ -0,0 +1,190 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _writing_virtio_drivers:
> +
> +======================
> +Writing Virtio Drivers
> +======================
> +
> +Introduction
> +============
> +
> +The `Virtio spec
> +<https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1930005>`__

Chapter 5 ("Device Types") in the virtio specification [1]

(and add the same reference as above in this file)?

> +defines all the supported Virtio device types. Since these devices are,
> +by definition, meant as abstractions for a wide variety of real
> +hardware, the addition of new virtio drivers is not expected to be very
> +frequent. Still, this document serves as a basic guideline for driver
> +programmers that need to hack a new Virtio driver or understand the
> +essentials of the existing ones. See :ref:`Virtio on Linux <virtio>` for
> +a general overview of Virtio.

Very important point here: Don't add a device driver for a device that
has no reserved device id in the spec. Preferably, don't add a device
driver if no spec is available (it is useful to develop
devices/drivers/spec at the same time to avoid incompatibilities/fixups
later; and you can consider a device spec "final" if it has been
accepted into the spec, even if it isn't in a released version yet.)

(...)

> +This assumes the device is of a new virtio device type not defined
> +before: ``VIRTIO_DEVICE_DUMMY``, which we should define in
> +include/uapi/linux/virtio_ids.h. The device has only one virtqueue which

Don't do that :) You need to make clear that VIRTIO_DEVICE_DUMMY is only
a placeholder.

> +is meant to be used to send data from the host to the guest.

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Cornelia Huck]

On Tue, Aug 02 2022, Ricardo Cañuelo <ricardo.canuelo@collabora.com> wrote:

> Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
> Minor fixes to existing virtio kerneldocs.

I think kerneldoc updates should be split out into a separate patch.

[No proper review, just some things I noticed below.]

>
> Signed-off-by: Ricardo Cañuelo <ricardo.canuelo@collabora.com>
> ---
>  Documentation/driver-api/index.rst            |   1 +
>  Documentation/driver-api/virtio/index.rst     |  11 +
>  Documentation/driver-api/virtio/virtio.rst    | 274 ++++++++++++++++++
>  .../virtio/writing_virtio_drivers.rst         | 190 ++++++++++++
>  MAINTAINERS                                   |   1 +
>  include/linux/virtio.h                        |   6 +-
>  include/linux/virtio_config.h                 |   6 +-
>  include/uapi/linux/virtio_ring.h              |  16 +-
>  8 files changed, 494 insertions(+), 11 deletions(-)
>  create mode 100644 Documentation/driver-api/virtio/index.rst
>  create mode 100644 Documentation/driver-api/virtio/virtio.rst
>  create mode 100644 Documentation/driver-api/virtio/writing_virtio_drivers.rst
>

(...)

> diff --git a/Documentation/driver-api/virtio/virtio.rst b/Documentation/driver-api/virtio/virtio.rst
> new file mode 100644
> index 000000000000..049a8aefad92
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/virtio.rst
> @@ -0,0 +1,274 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _virtio:
> +
> +===============
> +Virtio on Linux
> +===============
> +
> +Introduction
> +============
> +
> +Virtio is an open standard interface for virtual machines to access

There are devices that implement virtio in hardware.

> +paravirtualized devices, ie. devices that aren't emulated by a
> +hypervisor, but rather real host devices that are exposed by the
> +hypervisor to the guest to achieve native performance. In other words,
> +it provides a communication mechanism for a guest OS to use devices on
> +the host machine.
> +
> +The concrete hardware details of the real host devices are abstracted in the
> +hypervisor, which provides a set of simplified virtual devices that
> +implement the Virtio protocol. These devices are defined in the Virtio
> +spec [1] and they're the devices that the guest OS will ultimately
> +handle. So, in that regard, the guest OS knows it's running in a virtual
> +environment and that it needs to use the appropriate Virtio drivers to
> +handle the devices instead of the regular device drivers it'd use in a
> +native or purely virtual environment (with emulated devices).
> +
> +
> +Device - Driver communication: virtqueues
> +=========================================
> +
> +Although the Virtio devices are really an abstraction layer in the
> +hypervisor, they're exposed to the guest as if they are physical devices
> +either PCI or MMIO-based. We refer to that as the transport method and

Please also mention CCW (s390 channel devices), as Linux supports it as
well.

> +it's orthogonal to the device itself. The Virtio spec defines these two
> +and other transport methods in detail, including device discovery,
> +capabilities and interrupt handling.

(...)

> +Device discovery and probing
> +============================
> +
> +How a virtio device is found and configured by the kernel depends on how
> +the hypervisor defines it. Taking the `QEMU virtio-console
> +<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/char/virtio-console.c>`__
> +device as an example, which uses PCI as a transport method, the device

Should be the "virtio-console (...) device when using PCI...", I guess?

> +will present itself in the PCI bus with vendor 0x1af4 (RedHat, Inc.) and
> +device id 0x1003 (Virtio console), as defined in the spec, so the kernel
> +will detect it as it would do with any other PCI device.

(...)

> +References
> +==========
> +
> +[1] Virtio Spec v1.1:
> +https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html

v1.2 is out now :)

(I think it would be better to refer to the base spec directory?)

> +
> +[2] Virtqueues and virtio ring: How the data travels
> +https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
> +
> +.. rubric:: Footnotes
> +
> +.. [#f1] that's why they may be also referred as virtrings.
> diff --git a/Documentation/driver-api/virtio/writing_virtio_drivers.rst b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> new file mode 100644
> index 000000000000..5cb088b817ae
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> @@ -0,0 +1,190 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _writing_virtio_drivers:
> +
> +======================
> +Writing Virtio Drivers
> +======================
> +
> +Introduction
> +============
> +
> +The `Virtio spec
> +<https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1930005>`__

Chapter 5 ("Device Types") in the virtio specification [1]

(and add the same reference as above in this file)?

> +defines all the supported Virtio device types. Since these devices are,
> +by definition, meant as abstractions for a wide variety of real
> +hardware, the addition of new virtio drivers is not expected to be very
> +frequent. Still, this document serves as a basic guideline for driver
> +programmers that need to hack a new Virtio driver or understand the
> +essentials of the existing ones. See :ref:`Virtio on Linux <virtio>` for
> +a general overview of Virtio.

Very important point here: Don't add a device driver for a device that
has no reserved device id in the spec. Preferably, don't add a device
driver if no spec is available (it is useful to develop
devices/drivers/spec at the same time to avoid incompatibilities/fixups
later; and you can consider a device spec "final" if it has been
accepted into the spec, even if it isn't in a released version yet.)

(...)

> +This assumes the device is of a new virtio device type not defined
> +before: ``VIRTIO_DEVICE_DUMMY``, which we should define in
> +include/uapi/linux/virtio_ids.h. The device has only one virtqueue which

Don't do that :) You need to make clear that VIRTIO_DEVICE_DUMMY is only
a placeholder.

> +is meant to be used to send data from the host to the guest.

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Matthew Wilcox]

On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
> +In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
> +called and it will ultimately lead to a call to
> +:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
> +function::

You don't need to use :c:func:`foo`.  You can just write foo() and the
tooling will convert it into :c:func:`foo` for you.

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Matthew Wilcox]

On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
> +In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
> +called and it will ultimately lead to a call to
> +:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
> +function::

You don't need to use :c:func:`foo`.  You can just write foo() and the
tooling will convert it into :c:func:`foo` for you.

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Bagas,

Thanks for reviewing

Bagas Sanjaya <bagasdotme@gmail.com> writes:
> I can't apply this patch on top of mainline. On what commit (and what
> tree) was it based from?

The patch applies cleanly on recent tags of linux-next (tested on
next-20220802). If this needs to be tested on a different tree/commit
let me know, as I'll have to submit a new version of it.

Cheers,
Ricardo

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Bagas,

Thanks for reviewing

Bagas Sanjaya <bagasdotme@gmail.com> writes:
> I can't apply this patch on top of mainline. On what commit (and what
> tree) was it based from?

The patch applies cleanly on recent tags of linux-next (tested on
next-20220802). If this needs to be tested on a different tree/commit
let me know, as I'll have to submit a new version of it.

Cheers,
Ricardo
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Michael S. Tsirkin]

On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
> Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
> Minor fixes to existing virtio kerneldocs.
> 
> Signed-off-by: Ricardo Cañuelo <ricardo.canuelo@collabora.com>
> ---
>  Documentation/driver-api/index.rst            |   1 +
>  Documentation/driver-api/virtio/index.rst     |  11 +
>  Documentation/driver-api/virtio/virtio.rst    | 274 ++++++++++++++++++
>  .../virtio/writing_virtio_drivers.rst         | 190 ++++++++++++
>  MAINTAINERS                                   |   1 +
>  include/linux/virtio.h                        |   6 +-
>  include/linux/virtio_config.h                 |   6 +-
>  include/uapi/linux/virtio_ring.h              |  16 +-
>  8 files changed, 494 insertions(+), 11 deletions(-)
>  create mode 100644 Documentation/driver-api/virtio/index.rst
>  create mode 100644 Documentation/driver-api/virtio/virtio.rst
>  create mode 100644 Documentation/driver-api/virtio/writing_virtio_drivers.rst
> 
> diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
> index d3a58f77328e..30a3de452b1d 100644
> --- a/Documentation/driver-api/index.rst
> +++ b/Documentation/driver-api/index.rst
> @@ -106,6 +106,7 @@ available subsections can be seen below.
>     vfio-mediated-device
>     vfio
>     vfio-pci-device-specific-driver-acceptance
> +   virtio/index
>     xilinx/index
>     xillybus
>     zorro
> diff --git a/Documentation/driver-api/virtio/index.rst b/Documentation/driver-api/virtio/index.rst
> new file mode 100644
> index 000000000000..528b14b291e3
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/index.rst
> @@ -0,0 +1,11 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +======
> +Virtio
> +======
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   virtio
> +   writing_virtio_drivers
> diff --git a/Documentation/driver-api/virtio/virtio.rst b/Documentation/driver-api/virtio/virtio.rst
> new file mode 100644
> index 000000000000..049a8aefad92
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/virtio.rst
> @@ -0,0 +1,274 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _virtio:
> +
> +===============
> +Virtio on Linux
> +===============
> +
> +Introduction
> +============
> +
> +Virtio is an open standard interface for virtual machines to access
> +paravirtualized devices, ie. devices that aren't emulated by a
> +hypervisor, but rather real host devices that are exposed by the
> +hypervisor to the guest to achieve native performance. In other words,
> +it provides a communication mechanism for a guest OS to use devices on
> +the host machine.
> +
> +The concrete hardware details of the real host devices are abstracted in the
> +hypervisor, which provides a set of simplified virtual devices that
> +implement the Virtio protocol. These devices are defined in the Virtio
> +spec [1] and they're the devices that the guest OS will ultimately
> +handle. So, in that regard, the guest OS knows it's running in a virtual
> +environment and that it needs to use the appropriate Virtio drivers to
> +handle the devices instead of the regular device drivers it'd use in a
> +native or purely virtual environment (with emulated devices).
> +
> +
> +Device - Driver communication: virtqueues
> +=========================================
> +
> +Although the Virtio devices are really an abstraction layer in the
> +hypervisor, they're exposed to the guest as if they are physical devices
> +either PCI or MMIO-based. We refer to that as the transport method and
> +it's orthogonal to the device itself. The Virtio spec defines these two
> +and other transport methods in detail, including device discovery,
> +capabilities and interrupt handling.
> +
> +The communication between the driver in the guest OS and the device in
> +the hypervisor is done through shared memory (that's what makes virtio
> +devices so efficient) using specialized data structures called
> +virtqueues. The use of these virtqueues for data transfers is referred
> +to as the data plane, while the process of setting them up and
> +coordinating the driver and the device during the setup stage is called
> +the control plane.
> +
> +Virtqueues, which are used to communicate the driver and the device
> +regardless of the transport method used, are actually ring buffers
> +[#f1]_ of buffer descriptors similar to the ones used in a network
> +device:
> +
> +.. kernel-doc:: include/uapi/linux/virtio_ring.h
> +    :identifiers: struct vring_desc
> +

So I'm okay with a high level overview up to this point.
Below starts getting into details and they are not up to date
and I don't think we want to maintain a copy here.


> +All the buffers the descriptors point to are allocated by the guest and
> +used by the host either for reading or for writing but not for both.
> +
> +Refer to [2] for an illustrated overview of virtqueues and how the host
> +device and the guest driver communicate.


> +
> +Each virtqueue defines three areas: the descriptor area (`desc`), which
> +is an array of descriptors as described above, and the avail and used
> +rings.
> +
> +The `avail` ring is where the driver puts the indexes of the descriptors
> +that it has set up for the device to consume. The `used` ring is used by
> +the device to return the consumed buffers (read or written) to the
> +driver. For each used descriptor, the ring also contains the used buffer
> +length in case it was written.

the part above is duplicating too much from the spec I think.


> +
> +These rings are laid out in the :c:type:`vring_virtqueue` struct
> +together with other necessary management data, including a pointer to a
> +transport-specific ``notify`` function that is used to let the host side
> +know when new buffers have been put in the virtqueue for
> +processing. Embedded in this struct is the :c:type:`virtqueue` struct,
> +which is the data structure that's ultimately used by virtio drivers:
> +
> +.. kernel-doc:: include/linux/virtio.h
> +    :identifiers: struct virtqueue
> +
> +The callback saved in this struct serves the same purpose as the
> +``notify`` function in :c:type:`vring_virtqueue` but in the
> +opposite direction. That is, the callback is triggered when the host has
> +used the provided buffers. More specifically, the trigger will be an
> +interrupt issued by the hypervisor (QEMU, for example). Interrupt
> +request handlers are registered for a virtqueue during the virtqueue
> +setup process::
> +
> +	static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs,
> +			struct virtqueue *vqs[], vq_callback_t *callbacks[],
> +			const char * const names[], const bool *ctx)
> +	{
> +		struct virtio_pci_device *vp_dev = to_vp_device(vdev);
> +		int i, err, queue_idx = 0;
> +
> +		vp_dev->vqs = kcalloc(nvqs, sizeof(*vp_dev->vqs), GFP_KERNEL);
> +		if (!vp_dev->vqs)
> +			return -ENOMEM;
> +
> +		err = request_irq(vp_dev->pci_dev->irq, vp_interrupt, IRQF_SHARED,
> +				dev_name(&vdev->dev), vp_dev);
> +		...
> +
> +In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
> +called and it will ultimately lead to a call to
> +:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
> +function::
> +
> +	irqreturn_t vring_interrupt(int irq, void *_vq)
> +	{
> +		struct vring_virtqueue *vq = to_vvq(_vq);
> +
> +		...
> +
> +		pr_debug("virtqueue callback for %p (%p)\n", vq, vq->vq.callback);
> +		if (vq->vq.callback)
> +			vq->vq.callback(&vq->vq);
> +
> +		return IRQ_HANDLED;
> +	}
> +
> +Virtqueues are allocated by the guest kernel and their memory addresses
> +are then communicated to the hypervisor so it can access them. In the
> +host side there could be some differences in this process depending on
> +who is acting as the hypervisor, but from the guest point of view this
> +communication depends on the transport method used. So for a PCI device
> +in QEMU, for example, the QEMU setups the PCI BARs (which are memory
> +regions defined in the virtual PCI device) and the guest kernel maps
> +them to virtual memory, so when it writes to them it's actually writing
> +to the host userspace memory, acting like some kind of guest-host IPC
> +mechanism on top of PCI::

The reference to BARs seems bogus.

> +
> +	/*
> +	 * vp_modern_queue_address - set the virtqueue address
> +	 * @mdev: the modern virtio-pci device
> +	 * @index: the queue index
> +	 * @desc_addr: address of the descriptor area
> +	 * @driver_addr: address of the driver area
> +	 * @device_addr: address of the device area
> +	 */
> +	void vp_modern_queue_address(struct virtio_pci_modern_device *mdev,
> +				     u16 index, u64 desc_addr, u64 driver_addr,
> +				     u64 device_addr)
> +	{
> +		struct virtio_pci_common_cfg __iomem *cfg = mdev->common;
> +
> +		vp_iowrite16(index, &cfg->queue_select);
> +
> +		vp_iowrite64_twopart(desc_addr, &cfg->queue_desc_lo,
> +				     &cfg->queue_desc_hi);
> +		vp_iowrite64_twopart(driver_addr, &cfg->queue_avail_lo,
> +				     &cfg->queue_avail_hi);
> +		vp_iowrite64_twopart(device_addr, &cfg->queue_used_lo,
> +				     &cfg->queue_used_hi);
> +	}
> +
> +
> +Device discovery and probing
> +============================
> +
> +How a virtio device is found and configured by the kernel depends on how
> +the hypervisor defines it. Taking the `QEMU virtio-console
> +<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/char/virtio-console.c>`__

why gitlab? why not https://git.qemu.org/ then?

> +device as an example, which uses PCI as a transport method, the device
> +will present itself in the PCI bus with vendor 0x1af4 (RedHat, Inc.) and
> +device id 0x1003 (Virtio console), as defined in the spec, so the kernel
> +will detect it as it would do with any other PCI device.
> +
> +In more detail:
> +
> +The virtio core subsystem is composed of multiple modules/drivers, among
> +them:
> +
> +- virtio.c: implements the virtio bus driver.
> +- virtio_mmio.c: implements the MMIO transport, this is a platform
> +  driver (virtio-mmio).
> +- virtio_pci_common.c and virtio_pci_modern.c (and, optionally,
> +  virtio_pci_legacy.c): implement the virtio-pci PCI driver
> +
> +Then there are individual virtio drivers for specific device types, such
> +as the virtio-console driver. These are registered to the virtio bus
> +driver.
> +
> +During the PCI enumeration process, if a device is found to match the
> +virtio-pci driver (according to the virtio-pci device table, any PCI
> +device with vendor id = 0x1af4)::
> +
> +	/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
> +	static const struct pci_device_id virtio_pci_id_table[] = {
> +		{ PCI_DEVICE(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_ANY_ID) },
> +		{ 0 }
> +	};
> +
> +then the virtio-pci driver is probed. This includes setting up the
> +device ``virtio_config_ops`` functions and its virtqueue handling
> +methods, such as :c:func:`setup_vq` to allocate and configure them, (but
> +this method is not called yet)::
> +
> +	int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev)
> +	{
> +		struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
> +		struct pci_dev *pci_dev = vp_dev->pci_dev;
> +		int err;
> +
> +		mdev->pci_dev = pci_dev;
> +
> +		err = vp_modern_probe(mdev);
> +		if (err)
> +			return err;
> +
> +		if (mdev->device)
> +			vp_dev->vdev.config = &virtio_pci_config_ops;
> +		else
> +			vp_dev->vdev.config = &virtio_pci_config_nodev_ops;
> +
> +		vp_dev->config_vector = vp_config_vector;
> +		vp_dev->setup_vq = setup_vq;
> +		vp_dev->del_vq = del_vq;
> +		vp_dev->isr = mdev->isr;
> +		vp_dev->vdev.id = mdev->id;
> +
> +		return 0;
> +	}
> +
> +If the probing goes well, the device is registered to the virtio bus::
> +
> +	static int virtio_pci_probe(struct pci_dev *pci_dev,
> +				    const struct pci_device_id *id)
> +	{
> +		...
> +
> +		if (force_legacy) {
> +			rc = virtio_pci_legacy_probe(vp_dev);
> +			/* Also try modern mode if we can't map BAR0 (no IO space). */
> +			if (rc == -ENODEV || rc == -ENOMEM)
> +				rc = virtio_pci_modern_probe(vp_dev);
> +			if (rc)
> +				goto err_probe;
> +		} else {
> +			rc = virtio_pci_modern_probe(vp_dev);
> +			if (rc == -ENODEV)
> +				rc = virtio_pci_legacy_probe(vp_dev);
> +			if (rc)
> +				goto err_probe;
> +		}
> +
> +		...
> +
> +		rc = register_virtio_device(&vp_dev->vdev);
> +
> +When the device is registered to the virtio bus the kernel will look
> +for a driver in the bus that can handle the device and call that
> +driver's ``probe`` method.
> +
> +It's at this stage that the virtqueues will be setup by calling the
> +appropriate `virtio_find` helper function, such as
> +:c:func:`virtio_find_single_vq` or :c:func:`virtio_find_vqs`, which will
> +end up calling the device ``find_vqs`` config op (transport-specific),
> +which will allocate the virtqueues and configure them. In the case of
> +virtio PCI devices, that's done by the ``setup_vq`` method seen above.
> +
> +
> +References
> +==========
> +
> +[1] Virtio Spec v1.1:
> +https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html

1.2 is out.

> +
> +[2] Virtqueues and virtio ring: How the data travels
> +https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
> +
> +.. rubric:: Footnotes
> +
> +.. [#f1] that's why they may be also referred as virtrings.
> diff --git a/Documentation/driver-api/virtio/writing_virtio_drivers.rst b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> new file mode 100644
> index 000000000000..5cb088b817ae
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> @@ -0,0 +1,190 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _writing_virtio_drivers:
> +
> +======================
> +Writing Virtio Drivers
> +======================
> +
> +Introduction
> +============
> +
> +The `Virtio spec
> +<https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1930005>`__
> +defines all the supported Virtio device types. Since these devices are,
> +by definition, meant as abstractions for a wide variety of real
> +hardware, the addition of new virtio drivers is not expected to be very
> +frequent. Still, this document serves as a basic guideline for driver
> +programmers that need to hack a new Virtio driver or understand the
> +essentials of the existing ones. See :ref:`Virtio on Linux <virtio>` for
> +a general overview of Virtio.
> +
> +
> +Driver boilerplate
> +==================
> +
> +As a bare minimum, a virtio driver should register in the virtio bus and
> +configure the virtqueues for the device according to its spec, the
> +configuration of the virtqueues in the driver side must match the
> +virtqueue definitions in the device. A basic driver skeleton could look
> +like this::
> +
> +	#include <linux/virtio.h>
> +	#include <linux/virtio_ids.h>
> +	#include <linux/virtio_config.h>
> +	#include <linux/module.h>
> +
> +	/* device private data (one per device) */
> +	struct virtio_dummy_dev {
> +		struct virtqueue *vq;
> +	};
> +
> +	static void virtio_dummy_recv_cb(struct virtqueue *vq)
> +	{
> +		struct virtio_dummy_dev *dev = vq->vdev->priv;
> +		char *buf;
> +		unsigned int len;
> +
> +		buf = virtqueue_get_buf(dev->vq, &len);
> +		/* spurious callback? */
> +		if (!buf)
> +			return;
> +
> +		/* Process the received data */
> +	}
> +
> +	static int virtio_dummy_probe(struct virtio_device *vdev)
> +	{
> +		struct virtio_dummy_dev *dev = NULL;
> +
> +		/* initialize device data */
> +		dev = kzalloc(sizeof(struct virtio_dummy_dev), GFP_KERNEL);
> +		if (!dev)
> +			return -ENOMEM;
> +
> +		/* the device has a single virtqueue */
> +		dev->vq = virtio_find_single_vq(vdev, virtio_dummy_recv_cb, "input");
> +		if (IS_ERR(dev->vq)) {
> +			kfree(dev);
> +			return PTR_ERR(dev->vq);
> +
> +		}
> +		vdev->priv = dev;
> +
> +		return 0;
> +	}
> +
> +	static void virtio_dummy_remove(struct virtio_device *vdev)
> +	{
> +		struct virtio_dummy_dev *dev = vdev->priv;
> +
> +		/*
> +		 * Disable vq interrupts: equivalent to
> +		 * vdev->config->reset(vdev)
> +		 */
> +		virtio_reset_device(vdev);
> +
> +		/* remove virtqueues */
> +		vdev->config->del_vqs(vdev);
> +
> +		kfree(dev);
> +	}
> +
> +	static const struct virtio_device_id id_table[] = {
> +		{ VIRTIO_ID_DUMMY, VIRTIO_DEV_ANY_ID },
> +		{ 0 },
> +	};
> +
> +	static struct virtio_driver virtio_dummy_driver = {
> +		.driver.name =	KBUILD_MODNAME,
> +		.driver.owner =	THIS_MODULE,
> +		.id_table =	id_table,
> +		.probe =	virtio_dummy_probe,
> +		.remove =	virtio_dummy_remove,
> +	};
> +
> +	module_virtio_driver(virtio_dummy_driver);
> +	MODULE_DEVICE_TABLE(virtio, id_table);
> +	MODULE_DESCRIPTION("Dummy virtio driver");
> +	MODULE_LICENSE("GPL");
> +
> +This assumes the device is of a new virtio device type not defined
> +before: ``VIRTIO_DEVICE_DUMMY``, which we should define in
> +include/uapi/linux/virtio_ids.h. The device has only one virtqueue which
> +is meant to be used to send data from the host to the guest.
> +
> +If your driver doesn't have to do anything special in its ``init`` and
> +``exit`` methods, you can use the :c:func:`module_virtio_driver` helper to
> +reduce the amount of boilerplate code.
> +
> +The ``probe`` method does the minimum driver setup in this case
> +(memory allocation for the device data) and initializes the
> +virtqueue. The virtqueues are automatically enabled after ``probe``
> +returns, sending the appropriate "DRIVER_OK" status signal to the
> +device. If the virtqueues need to be enabled before ``probe`` ends, they
> +can be manually enabled by calling :c:func:`virtio_device_ready`:
> +
> +.. kernel-doc:: include/linux/virtio_config.h
> +    :identifiers: virtio_device_ready
> +
> +
> +Sending and receiving data
> +==========================
> +
> +The :c:func:`virtio_dummy_recv_cb` callback in the code above will be
> +triggered when the device notifies the driver after it finishes
> +processing a descriptor or descriptor chain, either for reading or
> +writing. However, that's only the second half of the Virtio
> +device-driver communication process, as the communication is always
> +started by the driver regardless of the direction of the data transfer.
> +
> +To configure a buffer transfer from the driver to the device, first you
> +have to add the buffers -- packed as `scatterlists` -- to the
> +appropriate virtqueue using any of the :c:func:`virtqueue_add_inbuf`,
> +:c:func:`virtqueue_add_outbuf` or :c:func:`virtqueue_add_sgs`, depending
> +on whether you need to add one input `scatterlist` (for the device to
> +fill in), one output `scatterlist` (for the device to consume) or
> +multiple `scatterlists`, respectively. Then, once the virtqueue is set
> +up, a call to :c:func:`virtqueue_kick` sends a notification that will be
> +serviced by the hypervisor that implements the device::
> +
> +	struct scatterlist sg[1];
> +	sg_init_one(sg, buffer, BUFLEN);
> +	virtqueue_add_inbuf(dev->vq, sg, 1, buffer, GFP_ATOMIC);
> +	virtqueue_kick(dev->vq);
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_add_inbuf
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_add_outbuf
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_add_sgs
> +
> +Then, after the device has read or written the buffers prepared by the
> +driver and notifies it back, the driver can call
> +:c:func:`virtqueue_get_buf` to read the data produced by the device (if
> +the virtqueue was set up with input buffers) or simply to reclaim the
> +buffers if they were already consumed by the device:
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_get_buf_ctx
> +
> +The virtqueue callbacks can be disabled and re-enabled using the
> +:c:func:`virtqueue_disable_cb` and the family of
> +:c:func:`virtqueue_enable_cb` functions respectively. See
> +drivers/virtio/virtio_ring.c for more details:
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_disable_cb
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_enable_cb
> +
> +
> +References
> +==========
> +
> +[1] Virtio Spec v1.1:
> +https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
> diff --git a/MAINTAINERS b/MAINTAINERS
> index c0f958dfd289..1ea754e45e07 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -21465,6 +21465,7 @@ S:	Maintained
>  F:	Documentation/ABI/testing/sysfs-bus-vdpa
>  F:	Documentation/ABI/testing/sysfs-class-vduse
>  F:	Documentation/devicetree/bindings/virtio/
> +F:	Documentation/driver-api/virtio/
>  F:	drivers/block/virtio_blk.c
>  F:	drivers/crypto/virtio/
>  F:	drivers/net/virtio_net.c
> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
> index d8fdf170637c..fd8440e85933 100644
> --- a/include/linux/virtio.h
> +++ b/include/linux/virtio.h
> @@ -11,7 +11,7 @@
>  #include <linux/gfp.h>
>  
>  /**
> - * virtqueue - a queue to register buffers for sending or receiving.
> + * struct virtqueue - a queue to register buffers for sending or receiving.
>   * @list: the chain of virtqueues for this device
>   * @callback: the function to call when buffers are consumed (can be NULL).
>   * @name: the name of this virtqueue (mainly for debugging)
> @@ -90,7 +90,7 @@ dma_addr_t virtqueue_get_avail_addr(struct virtqueue *vq);
>  dma_addr_t virtqueue_get_used_addr(struct virtqueue *vq);
>  
>  /**
> - * virtio_device - representation of a device using virtio
> + * struct virtio_device - representation of a device using virtio
>   * @index: unique position on the virtio bus
>   * @failed: saved value for VIRTIO_CONFIG_S_FAILED bit (for restore)
>   * @config_enabled: configuration change reporting enabled
> @@ -146,7 +146,7 @@ size_t virtio_max_dma_size(struct virtio_device *vdev);
>  	list_for_each_entry(vq, &vdev->vqs, list)
>  
>  /**
> - * virtio_driver - operations for a virtio I/O driver
> + * struct virtio_driver - operations for a virtio I/O driver
>   * @driver: underlying device driver (populate name and owner).
>   * @id_table: the ids serviced by this driver.
>   * @feature_table: an array of feature numbers supported by this driver.
> diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> index b47c2e7ed0ee..997801018ae4 100644
> --- a/include/linux/virtio_config.h
> +++ b/include/linux/virtio_config.h
> @@ -225,7 +225,7 @@ int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs,
>  
>  /**
>   * virtio_synchronize_cbs - synchronize with virtqueue callbacks
> - * @vdev: the device
> + * @dev: the device
>   */
>  static inline
>  void virtio_synchronize_cbs(struct virtio_device *dev)
> @@ -244,7 +244,7 @@ void virtio_synchronize_cbs(struct virtio_device *dev)
>  
>  /**
>   * virtio_device_ready - enable vq use in probe function
> - * @vdev: the device
> + * @dev: the device
>   *
>   * Driver must call this to use vqs in the probe function.
>   *
> @@ -292,7 +292,7 @@ const char *virtio_bus_name(struct virtio_device *vdev)
>  /**
>   * virtqueue_set_affinity - setting affinity for a virtqueue
>   * @vq: the virtqueue
> - * @cpu: the cpu no.
> + * @cpu_mask: the cpu mask
>   *
>   * Pay attention the function are best-effort: the affinity hint may not be set
>   * due to config support, irq type and sharing.
> diff --git a/include/uapi/linux/virtio_ring.h b/include/uapi/linux/virtio_ring.h
> index 476d3e5c0fe7..f8c20d3de8da 100644
> --- a/include/uapi/linux/virtio_ring.h
> +++ b/include/uapi/linux/virtio_ring.h
> @@ -93,15 +93,21 @@
>  #define VRING_USED_ALIGN_SIZE 4
>  #define VRING_DESC_ALIGN_SIZE 16
>  
> -/* Virtio ring descriptors: 16 bytes.  These can chain together via "next". */
> +/**
> + * struct vring_desc - Virtio ring descriptors,
> + * 16 bytes long. These can chain together via @next.
> + *
> + * @addr: buffer address (guest-physical)
> + * @len: buffer length
> + * @flags: descriptor flags
> + * @next: index of the next descriptor in the chain,
> + *        if the VRING_DESC_F_NEXT flag is set. We chain unused
> + *        descriptors via this, too.
> + */
>  struct vring_desc {
> -	/* Address (guest-physical). */
>  	__virtio64 addr;
> -	/* Length. */
>  	__virtio32 len;
> -	/* The flags as indicated above. */
>  	__virtio16 flags;
> -	/* We chain unused descriptors via this, too */
>  	__virtio16 next;
>  };
>  
> -- 
> 2.25.1

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Michael S. Tsirkin]

On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
> Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
> Minor fixes to existing virtio kerneldocs.
> 
> Signed-off-by: Ricardo Cañuelo <ricardo.canuelo@collabora.com>
> ---
>  Documentation/driver-api/index.rst            |   1 +
>  Documentation/driver-api/virtio/index.rst     |  11 +
>  Documentation/driver-api/virtio/virtio.rst    | 274 ++++++++++++++++++
>  .../virtio/writing_virtio_drivers.rst         | 190 ++++++++++++
>  MAINTAINERS                                   |   1 +
>  include/linux/virtio.h                        |   6 +-
>  include/linux/virtio_config.h                 |   6 +-
>  include/uapi/linux/virtio_ring.h              |  16 +-
>  8 files changed, 494 insertions(+), 11 deletions(-)
>  create mode 100644 Documentation/driver-api/virtio/index.rst
>  create mode 100644 Documentation/driver-api/virtio/virtio.rst
>  create mode 100644 Documentation/driver-api/virtio/writing_virtio_drivers.rst
> 
> diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
> index d3a58f77328e..30a3de452b1d 100644
> --- a/Documentation/driver-api/index.rst
> +++ b/Documentation/driver-api/index.rst
> @@ -106,6 +106,7 @@ available subsections can be seen below.
>     vfio-mediated-device
>     vfio
>     vfio-pci-device-specific-driver-acceptance
> +   virtio/index
>     xilinx/index
>     xillybus
>     zorro
> diff --git a/Documentation/driver-api/virtio/index.rst b/Documentation/driver-api/virtio/index.rst
> new file mode 100644
> index 000000000000..528b14b291e3
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/index.rst
> @@ -0,0 +1,11 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +======
> +Virtio
> +======
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   virtio
> +   writing_virtio_drivers
> diff --git a/Documentation/driver-api/virtio/virtio.rst b/Documentation/driver-api/virtio/virtio.rst
> new file mode 100644
> index 000000000000..049a8aefad92
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/virtio.rst
> @@ -0,0 +1,274 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _virtio:
> +
> +===============
> +Virtio on Linux
> +===============
> +
> +Introduction
> +============
> +
> +Virtio is an open standard interface for virtual machines to access
> +paravirtualized devices, ie. devices that aren't emulated by a
> +hypervisor, but rather real host devices that are exposed by the
> +hypervisor to the guest to achieve native performance. In other words,
> +it provides a communication mechanism for a guest OS to use devices on
> +the host machine.
> +
> +The concrete hardware details of the real host devices are abstracted in the
> +hypervisor, which provides a set of simplified virtual devices that
> +implement the Virtio protocol. These devices are defined in the Virtio
> +spec [1] and they're the devices that the guest OS will ultimately
> +handle. So, in that regard, the guest OS knows it's running in a virtual
> +environment and that it needs to use the appropriate Virtio drivers to
> +handle the devices instead of the regular device drivers it'd use in a
> +native or purely virtual environment (with emulated devices).
> +
> +
> +Device - Driver communication: virtqueues
> +=========================================
> +
> +Although the Virtio devices are really an abstraction layer in the
> +hypervisor, they're exposed to the guest as if they are physical devices
> +either PCI or MMIO-based. We refer to that as the transport method and
> +it's orthogonal to the device itself. The Virtio spec defines these two
> +and other transport methods in detail, including device discovery,
> +capabilities and interrupt handling.
> +
> +The communication between the driver in the guest OS and the device in
> +the hypervisor is done through shared memory (that's what makes virtio
> +devices so efficient) using specialized data structures called
> +virtqueues. The use of these virtqueues for data transfers is referred
> +to as the data plane, while the process of setting them up and
> +coordinating the driver and the device during the setup stage is called
> +the control plane.
> +
> +Virtqueues, which are used to communicate the driver and the device
> +regardless of the transport method used, are actually ring buffers
> +[#f1]_ of buffer descriptors similar to the ones used in a network
> +device:
> +
> +.. kernel-doc:: include/uapi/linux/virtio_ring.h
> +    :identifiers: struct vring_desc
> +

So I'm okay with a high level overview up to this point.
Below starts getting into details and they are not up to date
and I don't think we want to maintain a copy here.


> +All the buffers the descriptors point to are allocated by the guest and
> +used by the host either for reading or for writing but not for both.
> +
> +Refer to [2] for an illustrated overview of virtqueues and how the host
> +device and the guest driver communicate.


> +
> +Each virtqueue defines three areas: the descriptor area (`desc`), which
> +is an array of descriptors as described above, and the avail and used
> +rings.
> +
> +The `avail` ring is where the driver puts the indexes of the descriptors
> +that it has set up for the device to consume. The `used` ring is used by
> +the device to return the consumed buffers (read or written) to the
> +driver. For each used descriptor, the ring also contains the used buffer
> +length in case it was written.

the part above is duplicating too much from the spec I think.


> +
> +These rings are laid out in the :c:type:`vring_virtqueue` struct
> +together with other necessary management data, including a pointer to a
> +transport-specific ``notify`` function that is used to let the host side
> +know when new buffers have been put in the virtqueue for
> +processing. Embedded in this struct is the :c:type:`virtqueue` struct,
> +which is the data structure that's ultimately used by virtio drivers:
> +
> +.. kernel-doc:: include/linux/virtio.h
> +    :identifiers: struct virtqueue
> +
> +The callback saved in this struct serves the same purpose as the
> +``notify`` function in :c:type:`vring_virtqueue` but in the
> +opposite direction. That is, the callback is triggered when the host has
> +used the provided buffers. More specifically, the trigger will be an
> +interrupt issued by the hypervisor (QEMU, for example). Interrupt
> +request handlers are registered for a virtqueue during the virtqueue
> +setup process::
> +
> +	static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs,
> +			struct virtqueue *vqs[], vq_callback_t *callbacks[],
> +			const char * const names[], const bool *ctx)
> +	{
> +		struct virtio_pci_device *vp_dev = to_vp_device(vdev);
> +		int i, err, queue_idx = 0;
> +
> +		vp_dev->vqs = kcalloc(nvqs, sizeof(*vp_dev->vqs), GFP_KERNEL);
> +		if (!vp_dev->vqs)
> +			return -ENOMEM;
> +
> +		err = request_irq(vp_dev->pci_dev->irq, vp_interrupt, IRQF_SHARED,
> +				dev_name(&vdev->dev), vp_dev);
> +		...
> +
> +In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
> +called and it will ultimately lead to a call to
> +:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
> +function::
> +
> +	irqreturn_t vring_interrupt(int irq, void *_vq)
> +	{
> +		struct vring_virtqueue *vq = to_vvq(_vq);
> +
> +		...
> +
> +		pr_debug("virtqueue callback for %p (%p)\n", vq, vq->vq.callback);
> +		if (vq->vq.callback)
> +			vq->vq.callback(&vq->vq);
> +
> +		return IRQ_HANDLED;
> +	}
> +
> +Virtqueues are allocated by the guest kernel and their memory addresses
> +are then communicated to the hypervisor so it can access them. In the
> +host side there could be some differences in this process depending on
> +who is acting as the hypervisor, but from the guest point of view this
> +communication depends on the transport method used. So for a PCI device
> +in QEMU, for example, the QEMU setups the PCI BARs (which are memory
> +regions defined in the virtual PCI device) and the guest kernel maps
> +them to virtual memory, so when it writes to them it's actually writing
> +to the host userspace memory, acting like some kind of guest-host IPC
> +mechanism on top of PCI::

The reference to BARs seems bogus.

> +
> +	/*
> +	 * vp_modern_queue_address - set the virtqueue address
> +	 * @mdev: the modern virtio-pci device
> +	 * @index: the queue index
> +	 * @desc_addr: address of the descriptor area
> +	 * @driver_addr: address of the driver area
> +	 * @device_addr: address of the device area
> +	 */
> +	void vp_modern_queue_address(struct virtio_pci_modern_device *mdev,
> +				     u16 index, u64 desc_addr, u64 driver_addr,
> +				     u64 device_addr)
> +	{
> +		struct virtio_pci_common_cfg __iomem *cfg = mdev->common;
> +
> +		vp_iowrite16(index, &cfg->queue_select);
> +
> +		vp_iowrite64_twopart(desc_addr, &cfg->queue_desc_lo,
> +				     &cfg->queue_desc_hi);
> +		vp_iowrite64_twopart(driver_addr, &cfg->queue_avail_lo,
> +				     &cfg->queue_avail_hi);
> +		vp_iowrite64_twopart(device_addr, &cfg->queue_used_lo,
> +				     &cfg->queue_used_hi);
> +	}
> +
> +
> +Device discovery and probing
> +============================
> +
> +How a virtio device is found and configured by the kernel depends on how
> +the hypervisor defines it. Taking the `QEMU virtio-console
> +<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/char/virtio-console.c>`__

why gitlab? why not https://git.qemu.org/ then?

> +device as an example, which uses PCI as a transport method, the device
> +will present itself in the PCI bus with vendor 0x1af4 (RedHat, Inc.) and
> +device id 0x1003 (Virtio console), as defined in the spec, so the kernel
> +will detect it as it would do with any other PCI device.
> +
> +In more detail:
> +
> +The virtio core subsystem is composed of multiple modules/drivers, among
> +them:
> +
> +- virtio.c: implements the virtio bus driver.
> +- virtio_mmio.c: implements the MMIO transport, this is a platform
> +  driver (virtio-mmio).
> +- virtio_pci_common.c and virtio_pci_modern.c (and, optionally,
> +  virtio_pci_legacy.c): implement the virtio-pci PCI driver
> +
> +Then there are individual virtio drivers for specific device types, such
> +as the virtio-console driver. These are registered to the virtio bus
> +driver.
> +
> +During the PCI enumeration process, if a device is found to match the
> +virtio-pci driver (according to the virtio-pci device table, any PCI
> +device with vendor id = 0x1af4)::
> +
> +	/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
> +	static const struct pci_device_id virtio_pci_id_table[] = {
> +		{ PCI_DEVICE(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_ANY_ID) },
> +		{ 0 }
> +	};
> +
> +then the virtio-pci driver is probed. This includes setting up the
> +device ``virtio_config_ops`` functions and its virtqueue handling
> +methods, such as :c:func:`setup_vq` to allocate and configure them, (but
> +this method is not called yet)::
> +
> +	int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev)
> +	{
> +		struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
> +		struct pci_dev *pci_dev = vp_dev->pci_dev;
> +		int err;
> +
> +		mdev->pci_dev = pci_dev;
> +
> +		err = vp_modern_probe(mdev);
> +		if (err)
> +			return err;
> +
> +		if (mdev->device)
> +			vp_dev->vdev.config = &virtio_pci_config_ops;
> +		else
> +			vp_dev->vdev.config = &virtio_pci_config_nodev_ops;
> +
> +		vp_dev->config_vector = vp_config_vector;
> +		vp_dev->setup_vq = setup_vq;
> +		vp_dev->del_vq = del_vq;
> +		vp_dev->isr = mdev->isr;
> +		vp_dev->vdev.id = mdev->id;
> +
> +		return 0;
> +	}
> +
> +If the probing goes well, the device is registered to the virtio bus::
> +
> +	static int virtio_pci_probe(struct pci_dev *pci_dev,
> +				    const struct pci_device_id *id)
> +	{
> +		...
> +
> +		if (force_legacy) {
> +			rc = virtio_pci_legacy_probe(vp_dev);
> +			/* Also try modern mode if we can't map BAR0 (no IO space). */
> +			if (rc == -ENODEV || rc == -ENOMEM)
> +				rc = virtio_pci_modern_probe(vp_dev);
> +			if (rc)
> +				goto err_probe;
> +		} else {
> +			rc = virtio_pci_modern_probe(vp_dev);
> +			if (rc == -ENODEV)
> +				rc = virtio_pci_legacy_probe(vp_dev);
> +			if (rc)
> +				goto err_probe;
> +		}
> +
> +		...
> +
> +		rc = register_virtio_device(&vp_dev->vdev);
> +
> +When the device is registered to the virtio bus the kernel will look
> +for a driver in the bus that can handle the device and call that
> +driver's ``probe`` method.
> +
> +It's at this stage that the virtqueues will be setup by calling the
> +appropriate `virtio_find` helper function, such as
> +:c:func:`virtio_find_single_vq` or :c:func:`virtio_find_vqs`, which will
> +end up calling the device ``find_vqs`` config op (transport-specific),
> +which will allocate the virtqueues and configure them. In the case of
> +virtio PCI devices, that's done by the ``setup_vq`` method seen above.
> +
> +
> +References
> +==========
> +
> +[1] Virtio Spec v1.1:
> +https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html

1.2 is out.

> +
> +[2] Virtqueues and virtio ring: How the data travels
> +https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
> +
> +.. rubric:: Footnotes
> +
> +.. [#f1] that's why they may be also referred as virtrings.
> diff --git a/Documentation/driver-api/virtio/writing_virtio_drivers.rst b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> new file mode 100644
> index 000000000000..5cb088b817ae
> --- /dev/null
> +++ b/Documentation/driver-api/virtio/writing_virtio_drivers.rst
> @@ -0,0 +1,190 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _writing_virtio_drivers:
> +
> +======================
> +Writing Virtio Drivers
> +======================
> +
> +Introduction
> +============
> +
> +The `Virtio spec
> +<https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1930005>`__
> +defines all the supported Virtio device types. Since these devices are,
> +by definition, meant as abstractions for a wide variety of real
> +hardware, the addition of new virtio drivers is not expected to be very
> +frequent. Still, this document serves as a basic guideline for driver
> +programmers that need to hack a new Virtio driver or understand the
> +essentials of the existing ones. See :ref:`Virtio on Linux <virtio>` for
> +a general overview of Virtio.
> +
> +
> +Driver boilerplate
> +==================
> +
> +As a bare minimum, a virtio driver should register in the virtio bus and
> +configure the virtqueues for the device according to its spec, the
> +configuration of the virtqueues in the driver side must match the
> +virtqueue definitions in the device. A basic driver skeleton could look
> +like this::
> +
> +	#include <linux/virtio.h>
> +	#include <linux/virtio_ids.h>
> +	#include <linux/virtio_config.h>
> +	#include <linux/module.h>
> +
> +	/* device private data (one per device) */
> +	struct virtio_dummy_dev {
> +		struct virtqueue *vq;
> +	};
> +
> +	static void virtio_dummy_recv_cb(struct virtqueue *vq)
> +	{
> +		struct virtio_dummy_dev *dev = vq->vdev->priv;
> +		char *buf;
> +		unsigned int len;
> +
> +		buf = virtqueue_get_buf(dev->vq, &len);
> +		/* spurious callback? */
> +		if (!buf)
> +			return;
> +
> +		/* Process the received data */
> +	}
> +
> +	static int virtio_dummy_probe(struct virtio_device *vdev)
> +	{
> +		struct virtio_dummy_dev *dev = NULL;
> +
> +		/* initialize device data */
> +		dev = kzalloc(sizeof(struct virtio_dummy_dev), GFP_KERNEL);
> +		if (!dev)
> +			return -ENOMEM;
> +
> +		/* the device has a single virtqueue */
> +		dev->vq = virtio_find_single_vq(vdev, virtio_dummy_recv_cb, "input");
> +		if (IS_ERR(dev->vq)) {
> +			kfree(dev);
> +			return PTR_ERR(dev->vq);
> +
> +		}
> +		vdev->priv = dev;
> +
> +		return 0;
> +	}
> +
> +	static void virtio_dummy_remove(struct virtio_device *vdev)
> +	{
> +		struct virtio_dummy_dev *dev = vdev->priv;
> +
> +		/*
> +		 * Disable vq interrupts: equivalent to
> +		 * vdev->config->reset(vdev)
> +		 */
> +		virtio_reset_device(vdev);
> +
> +		/* remove virtqueues */
> +		vdev->config->del_vqs(vdev);
> +
> +		kfree(dev);
> +	}
> +
> +	static const struct virtio_device_id id_table[] = {
> +		{ VIRTIO_ID_DUMMY, VIRTIO_DEV_ANY_ID },
> +		{ 0 },
> +	};
> +
> +	static struct virtio_driver virtio_dummy_driver = {
> +		.driver.name =	KBUILD_MODNAME,
> +		.driver.owner =	THIS_MODULE,
> +		.id_table =	id_table,
> +		.probe =	virtio_dummy_probe,
> +		.remove =	virtio_dummy_remove,
> +	};
> +
> +	module_virtio_driver(virtio_dummy_driver);
> +	MODULE_DEVICE_TABLE(virtio, id_table);
> +	MODULE_DESCRIPTION("Dummy virtio driver");
> +	MODULE_LICENSE("GPL");
> +
> +This assumes the device is of a new virtio device type not defined
> +before: ``VIRTIO_DEVICE_DUMMY``, which we should define in
> +include/uapi/linux/virtio_ids.h. The device has only one virtqueue which
> +is meant to be used to send data from the host to the guest.
> +
> +If your driver doesn't have to do anything special in its ``init`` and
> +``exit`` methods, you can use the :c:func:`module_virtio_driver` helper to
> +reduce the amount of boilerplate code.
> +
> +The ``probe`` method does the minimum driver setup in this case
> +(memory allocation for the device data) and initializes the
> +virtqueue. The virtqueues are automatically enabled after ``probe``
> +returns, sending the appropriate "DRIVER_OK" status signal to the
> +device. If the virtqueues need to be enabled before ``probe`` ends, they
> +can be manually enabled by calling :c:func:`virtio_device_ready`:
> +
> +.. kernel-doc:: include/linux/virtio_config.h
> +    :identifiers: virtio_device_ready
> +
> +
> +Sending and receiving data
> +==========================
> +
> +The :c:func:`virtio_dummy_recv_cb` callback in the code above will be
> +triggered when the device notifies the driver after it finishes
> +processing a descriptor or descriptor chain, either for reading or
> +writing. However, that's only the second half of the Virtio
> +device-driver communication process, as the communication is always
> +started by the driver regardless of the direction of the data transfer.
> +
> +To configure a buffer transfer from the driver to the device, first you
> +have to add the buffers -- packed as `scatterlists` -- to the
> +appropriate virtqueue using any of the :c:func:`virtqueue_add_inbuf`,
> +:c:func:`virtqueue_add_outbuf` or :c:func:`virtqueue_add_sgs`, depending
> +on whether you need to add one input `scatterlist` (for the device to
> +fill in), one output `scatterlist` (for the device to consume) or
> +multiple `scatterlists`, respectively. Then, once the virtqueue is set
> +up, a call to :c:func:`virtqueue_kick` sends a notification that will be
> +serviced by the hypervisor that implements the device::
> +
> +	struct scatterlist sg[1];
> +	sg_init_one(sg, buffer, BUFLEN);
> +	virtqueue_add_inbuf(dev->vq, sg, 1, buffer, GFP_ATOMIC);
> +	virtqueue_kick(dev->vq);
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_add_inbuf
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_add_outbuf
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_add_sgs
> +
> +Then, after the device has read or written the buffers prepared by the
> +driver and notifies it back, the driver can call
> +:c:func:`virtqueue_get_buf` to read the data produced by the device (if
> +the virtqueue was set up with input buffers) or simply to reclaim the
> +buffers if they were already consumed by the device:
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_get_buf_ctx
> +
> +The virtqueue callbacks can be disabled and re-enabled using the
> +:c:func:`virtqueue_disable_cb` and the family of
> +:c:func:`virtqueue_enable_cb` functions respectively. See
> +drivers/virtio/virtio_ring.c for more details:
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_disable_cb
> +
> +.. kernel-doc:: drivers/virtio/virtio_ring.c
> +    :identifiers: virtqueue_enable_cb
> +
> +
> +References
> +==========
> +
> +[1] Virtio Spec v1.1:
> +https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
> diff --git a/MAINTAINERS b/MAINTAINERS
> index c0f958dfd289..1ea754e45e07 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -21465,6 +21465,7 @@ S:	Maintained
>  F:	Documentation/ABI/testing/sysfs-bus-vdpa
>  F:	Documentation/ABI/testing/sysfs-class-vduse
>  F:	Documentation/devicetree/bindings/virtio/
> +F:	Documentation/driver-api/virtio/
>  F:	drivers/block/virtio_blk.c
>  F:	drivers/crypto/virtio/
>  F:	drivers/net/virtio_net.c
> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
> index d8fdf170637c..fd8440e85933 100644
> --- a/include/linux/virtio.h
> +++ b/include/linux/virtio.h
> @@ -11,7 +11,7 @@
>  #include <linux/gfp.h>
>  
>  /**
> - * virtqueue - a queue to register buffers for sending or receiving.
> + * struct virtqueue - a queue to register buffers for sending or receiving.
>   * @list: the chain of virtqueues for this device
>   * @callback: the function to call when buffers are consumed (can be NULL).
>   * @name: the name of this virtqueue (mainly for debugging)
> @@ -90,7 +90,7 @@ dma_addr_t virtqueue_get_avail_addr(struct virtqueue *vq);
>  dma_addr_t virtqueue_get_used_addr(struct virtqueue *vq);
>  
>  /**
> - * virtio_device - representation of a device using virtio
> + * struct virtio_device - representation of a device using virtio
>   * @index: unique position on the virtio bus
>   * @failed: saved value for VIRTIO_CONFIG_S_FAILED bit (for restore)
>   * @config_enabled: configuration change reporting enabled
> @@ -146,7 +146,7 @@ size_t virtio_max_dma_size(struct virtio_device *vdev);
>  	list_for_each_entry(vq, &vdev->vqs, list)
>  
>  /**
> - * virtio_driver - operations for a virtio I/O driver
> + * struct virtio_driver - operations for a virtio I/O driver
>   * @driver: underlying device driver (populate name and owner).
>   * @id_table: the ids serviced by this driver.
>   * @feature_table: an array of feature numbers supported by this driver.
> diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> index b47c2e7ed0ee..997801018ae4 100644
> --- a/include/linux/virtio_config.h
> +++ b/include/linux/virtio_config.h
> @@ -225,7 +225,7 @@ int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs,
>  
>  /**
>   * virtio_synchronize_cbs - synchronize with virtqueue callbacks
> - * @vdev: the device
> + * @dev: the device
>   */
>  static inline
>  void virtio_synchronize_cbs(struct virtio_device *dev)
> @@ -244,7 +244,7 @@ void virtio_synchronize_cbs(struct virtio_device *dev)
>  
>  /**
>   * virtio_device_ready - enable vq use in probe function
> - * @vdev: the device
> + * @dev: the device
>   *
>   * Driver must call this to use vqs in the probe function.
>   *
> @@ -292,7 +292,7 @@ const char *virtio_bus_name(struct virtio_device *vdev)
>  /**
>   * virtqueue_set_affinity - setting affinity for a virtqueue
>   * @vq: the virtqueue
> - * @cpu: the cpu no.
> + * @cpu_mask: the cpu mask
>   *
>   * Pay attention the function are best-effort: the affinity hint may not be set
>   * due to config support, irq type and sharing.
> diff --git a/include/uapi/linux/virtio_ring.h b/include/uapi/linux/virtio_ring.h
> index 476d3e5c0fe7..f8c20d3de8da 100644
> --- a/include/uapi/linux/virtio_ring.h
> +++ b/include/uapi/linux/virtio_ring.h
> @@ -93,15 +93,21 @@
>  #define VRING_USED_ALIGN_SIZE 4
>  #define VRING_DESC_ALIGN_SIZE 16
>  
> -/* Virtio ring descriptors: 16 bytes.  These can chain together via "next". */
> +/**
> + * struct vring_desc - Virtio ring descriptors,
> + * 16 bytes long. These can chain together via @next.
> + *
> + * @addr: buffer address (guest-physical)
> + * @len: buffer length
> + * @flags: descriptor flags
> + * @next: index of the next descriptor in the chain,
> + *        if the VRING_DESC_F_NEXT flag is set. We chain unused
> + *        descriptors via this, too.
> + */
>  struct vring_desc {
> -	/* Address (guest-physical). */
>  	__virtio64 addr;
> -	/* Length. */
>  	__virtio32 len;
> -	/* The flags as indicated above. */
>  	__virtio16 flags;
> -	/* We chain unused descriptors via this, too */
>  	__virtio16 next;
>  };
>  
> -- 
> 2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Matthew Wilcox <willy@infradead.org> writes:

> On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
>> +In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
>> +called and it will ultimately lead to a call to
>> +:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
>> +function::
>
> You don't need to use :c:func:`foo`.  You can just write foo() and the
> tooling will convert it into :c:func:`foo` for you.

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Matthew Wilcox <willy@infradead.org> writes:

> On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
>> +In this case, when the interrupt arrives :c:func:`vp_interrupt` will be
>> +called and it will ultimately lead to a call to
>> +:c:func:`vring_interrupt`, which ends up calling the virtqueue callback
>> +function::
>
> You don't need to use :c:func:`foo`.  You can just write foo() and the
> tooling will convert it into :c:func:`foo` for you.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Matthew,

On mar, ago 02 2022 at 16:56:48, Matthew Wilcox <willy@infradead.org> wrote:
> You don't need to use :c:func:`foo`.  You can just write foo() and the
> tooling will convert it into :c:func:`foo` for you.

Thanks for the tip. However, I did some tests and the results aren't
quite the same. For functions with kerneldocs that are referenced in the
same document (.. kernel-doc::) the tool does efectively link to the
generated documentation, but for all the other functions using
c:func:`foo` generates a different formatting than `foo`, which does no
formatting at all.

I don't know if the extra semantic info can be used by the tooling in
any other way, in case it doesn't the fancy formatting might not be
worth it in exchange for a noisier source text, but I've seen it been
used in other driver-api docs.

Cheers,
Ricardo

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Matthew,

On mar, ago 02 2022 at 16:56:48, Matthew Wilcox <willy@infradead.org> wrote:
> You don't need to use :c:func:`foo`.  You can just write foo() and the
> tooling will convert it into :c:func:`foo` for you.

Thanks for the tip. However, I did some tests and the results aren't
quite the same. For functions with kerneldocs that are referenced in the
same document (.. kernel-doc::) the tool does efectively link to the
generated documentation, but for all the other functions using
c:func:`foo` generates a different formatting than `foo`, which does no
formatting at all.

I don't know if the extra semantic info can be used by the tooling in
any other way, in case it doesn't the fancy formatting might not be
worth it in exchange for a noisier source text, but I've seen it been
used in other driver-api docs.

Cheers,
Ricardo
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Cornelia,

Thanks for the review. This is my first approach to virtio and I had to
make some assumptions myself from reading the code and the spec, so I
expected lots of corrections.

On mar, ago 02 2022 at 17:55:43, Cornelia Huck <cohuck@redhat.com> wrote:
> I think kerneldoc updates should be split out into a separate patch.

Ack, I'll split them in the next version.

> There are devices that implement virtio in hardware.
> ...
> Please also mention CCW (s390 channel devices), as Linux supports it as
> well.

I'll add that to v2.

> Should be the "virtio-console (...) device when using PCI...", I guess?

You're right, I was focusing on the concrete example I was testing
instead.

> v1.2 is out now :)
>
> (I think it would be better to refer to the base spec directory?)

Do you mean this? https://docs.oasis-open.org/virtio/virtio/
Or should I link the source repo instead?
https://github.com/oasis-tcs/virtio-spec

> Chapter 5 ("Device Types") in the virtio specification [1]
>
> (and add the same reference as above in this file)?

Got it.

> Very important point here: Don't add a device driver for a device that
> has no reserved device id in the spec. Preferably, don't add a device
> driver if no spec is available (it is useful to develop
> devices/drivers/spec at the same time to avoid incompatibilities/fixups
> later; and you can consider a device spec "final" if it has been
> accepted into the spec, even if it isn't in a released version yet.)
> ...
> Don't do that :) You need to make clear that VIRTIO_DEVICE_DUMMY is only
> a placeholder.

Thanks, I'll make sure to rework this and to mention that you shouldn't
add a new out-of-spec device id.

Cheers,
Ricardo

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Cornelia,

Thanks for the review. This is my first approach to virtio and I had to
make some assumptions myself from reading the code and the spec, so I
expected lots of corrections.

On mar, ago 02 2022 at 17:55:43, Cornelia Huck <cohuck@redhat.com> wrote:
> I think kerneldoc updates should be split out into a separate patch.

Ack, I'll split them in the next version.

> There are devices that implement virtio in hardware.
> ...
> Please also mention CCW (s390 channel devices), as Linux supports it as
> well.

I'll add that to v2.

> Should be the "virtio-console (...) device when using PCI...", I guess?

You're right, I was focusing on the concrete example I was testing
instead.

> v1.2 is out now :)
>
> (I think it would be better to refer to the base spec directory?)

Do you mean this? https://docs.oasis-open.org/virtio/virtio/
Or should I link the source repo instead?
https://github.com/oasis-tcs/virtio-spec

> Chapter 5 ("Device Types") in the virtio specification [1]
>
> (and add the same reference as above in this file)?

Got it.

> Very important point here: Don't add a device driver for a device that
> has no reserved device id in the spec. Preferably, don't add a device
> driver if no spec is available (it is useful to develop
> devices/drivers/spec at the same time to avoid incompatibilities/fixups
> later; and you can consider a device spec "final" if it has been
> accepted into the spec, even if it isn't in a released version yet.)
> ...
> Don't do that :) You need to make clear that VIRTIO_DEVICE_DUMMY is only
> a placeholder.

Thanks, I'll make sure to rework this and to mention that you shouldn't
add a new out-of-spec device id.

Cheers,
Ricardo
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Michael, thanks for reviewing the patch. Comments below:

On mié, ago 03 2022 at 02:54:28, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> So I'm okay with a high level overview up to this point.
> Below starts getting into details and they are not up to date
> and I don't think we want to maintain a copy here.

I agree I might have gone too much into code details, but I assumed
those were pretty much the elementary and "fixed" parts of the
subsystem, with a slow rate of changes in the fundamental stuff.

If the code is prone to future changes or is outdated it's true we don't
want it mentioned here but leaving this doc as a short intro to virtio
won't add much to what's already available in other resources.

The intention was mostly to illustrate the basic virtio concepts with
the concrete implementation in the kernel. I'll see if I can find a way
to do that without relying so much on specific code fragments.


> the part above is duplicating too much from the spec I think.

Ack.

> The reference to BARs seems bogus.

Got it.

> why gitlab? why not https://git.qemu.org/ then?

According to the QEMU website the official repo lives in gitlab.

> 1.2 is out.

Thanks, I didn't see it when searching for the spec.

Cheers,
Ricardo

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Ricardo Cañuelo]

Hi Michael, thanks for reviewing the patch. Comments below:

On mié, ago 03 2022 at 02:54:28, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> So I'm okay with a high level overview up to this point.
> Below starts getting into details and they are not up to date
> and I don't think we want to maintain a copy here.

I agree I might have gone too much into code details, but I assumed
those were pretty much the elementary and "fixed" parts of the
subsystem, with a slow rate of changes in the fundamental stuff.

If the code is prone to future changes or is outdated it's true we don't
want it mentioned here but leaving this doc as a short intro to virtio
won't add much to what's already available in other resources.

The intention was mostly to illustrate the basic virtio concepts with
the concrete implementation in the kernel. I'll see if I can find a way
to do that without relying so much on specific code fragments.


> the part above is duplicating too much from the spec I think.

Ack.

> The reference to BARs seems bogus.

Got it.

> why gitlab? why not https://git.qemu.org/ then?

According to the QEMU website the official repo lives in gitlab.

> 1.2 is out.

Thanks, I didn't see it when searching for the spec.

Cheers,
Ricardo
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Cornelia Huck]

On Wed, Aug 03 2022, Ricardo Cañuelo <ricardo.canuelo@collabora.com> wrote:

>> v1.2 is out now :)
>>
>> (I think it would be better to refer to the base spec directory?)
>
> Do you mean this? https://docs.oasis-open.org/virtio/virtio/
> Or should I link the source repo instead?
> https://github.com/oasis-tcs/virtio-spec

Hm... this should refer to the OASIS site, but the base directory
listing is a bit ugly. Maybe refer to v1.2, and add a note that there
might be later versions available?

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Cornelia Huck]

On Wed, Aug 03 2022, Ricardo Cañuelo <ricardo.canuelo@collabora.com> wrote:

>> v1.2 is out now :)
>>
>> (I think it would be better to refer to the base spec directory?)
>
> Do you mean this? https://docs.oasis-open.org/virtio/virtio/
> Or should I link the source repo instead?
> https://github.com/oasis-tcs/virtio-spec

Hm... this should refer to the OASIS site, but the base directory
listing is a bit ugly. Maybe refer to v1.2, and add a note that there
might be later versions available?

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Matthew Wilcox]

On Wed, Aug 03, 2022 at 09:24:49AM +0200, Ricardo Cañuelo wrote:
> Hi Matthew,
> 
> On mar, ago 02 2022 at 16:56:48, Matthew Wilcox <willy@infradead.org> wrote:
> > You don't need to use :c:func:`foo`.  You can just write foo() and the
> > tooling will convert it into :c:func:`foo` for you.
> 
> Thanks for the tip. However, I did some tests and the results aren't
> quite the same. For functions with kerneldocs that are referenced in the
> same document (.. kernel-doc::) the tool does efectively link to the
> generated documentation, but for all the other functions using
> c:func:`foo` generates a different formatting than `foo`, which does no
> formatting at all.

I didn't say `foo`, I said foo().  This is handled by
Documentation/sphinx/automarkup.py.  To quote the doc-guide:

Please note that there is no need to use ``c:func:`` to generate cross
references to function documentation.  Due to some Sphinx extension magic,
the documentation build system will automatically turn a reference to
``function()`` into a cross reference if an index entry for the given
function name exists.  If you see ``c:func:`` use in a kernel document,
please feel free to remove it.

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Matthew Wilcox]

On Wed, Aug 03, 2022 at 09:24:49AM +0200, Ricardo Cañuelo wrote:
> Hi Matthew,
> 
> On mar, ago 02 2022 at 16:56:48, Matthew Wilcox <willy@infradead.org> wrote:
> > You don't need to use :c:func:`foo`.  You can just write foo() and the
> > tooling will convert it into :c:func:`foo` for you.
> 
> Thanks for the tip. However, I did some tests and the results aren't
> quite the same. For functions with kerneldocs that are referenced in the
> same document (.. kernel-doc::) the tool does efectively link to the
> generated documentation, but for all the other functions using
> c:func:`foo` generates a different formatting than `foo`, which does no
> formatting at all.

I didn't say `foo`, I said foo().  This is handled by
Documentation/sphinx/automarkup.py.  To quote the doc-guide:

Please note that there is no need to use ``c:func:`` to generate cross
references to function documentation.  Due to some Sphinx extension magic,
the documentation build system will automatically turn a reference to
``function()`` into a cross reference if an index entry for the given
function name exists.  If you see ``c:func:`` use in a kernel document,
please feel free to remove it.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] docs: driver-api: virtio: virtio on Linux

[Bagas Sanjaya]

On Tue, Aug 02, 2022 at 02:42:22PM +0200, Ricardo Cañuelo wrote:
> Basic doc about Virtio on Linux and a short tutorial on Virtio drivers.
> Minor fixes to existing virtio kerneldocs.
> 
> Signed-off-by: Ricardo Cañuelo <ricardo.canuelo@collabora.com>

The documentation (htmldocs) with this patch builds successfully without
new warnings.

Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>

-- 
An old man doll... just what I always wanted! - Clara