[virtio] [PATCH v6 0/5] packed ring layout spec

[virtio] [PATCH v6 0/5] packed ring layout spec

[Michael S. Tsirkin]

OK, this is in a shape where we could include it in the spec.
Changes from v5:
- scope reductions (see below). We can add more
  features down the road, hopefully reduced scope will be enough
  to finalize spec soon.
- cleanup and integrate in the spec
- pseudo-code

Deferred features:
- dropped _F_DESC_LIST, 1.0 includes this unconditionally, we
  can do same
- dropped event structure change notifications - needed for
  efficient hardware implementations but let's add this on top

3 1st patches just move text around so all virtio 1.0
things are in the same place. 2 last ones add the new layout

Option to mark descriptors as not generating events isn't
yet implemented. Again, let's add this on top.

I also note that for hardware implementations, a different
set of memory barriers is needed. Again, let's add this on top

not linking into conformance sections, will add after spec
itself is approved.

Michael S. Tsirkin (5):
  content: move 1.0 queue format out to a separate section
  content: move ring text out to a separate file
  content: move virtqueue operation description
  packed virtqueues: more efficient virtqueue layout
  packed-ring: add in order request support

 conformance.tex |   4 +-
 content.tex     | 718 +++++---------------------------------------------------
 packed-ring.tex | 692 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 split-ring.tex  | 666 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 1423 insertions(+), 657 deletions(-)
 create mode 100644 packed-ring.tex
 create mode 100644 split-ring.tex

-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v6 1/5] content: move 1.0 queue format out to a separate section

[Michael S. Tsirkin]

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/content.tex b/content.tex
index c7ef7fd..4483a4b 100644
--- a/content.tex
+++ b/content.tex
@@ -230,7 +230,30 @@ result.
 The mechanism for bulk data transport on virtio devices is
 pretentiously called a virtqueue. Each device can have zero or more
 virtqueues\footnote{For example, the simplest network device has one virtqueue for
-transmit and one for receive.}.  Each queue has a 16-bit queue size
+transmit and one for receive.}.
+
+Driver makes requests available to device by adding
+an available buffer to the queue - i.e. adding a buffer
+describing the request to a virtqueue, and optionally triggering
+a driver event - i.e. sending a notification to the device.
+
+Device executes the requests and - when complete - adds
+a used buffer to the queue - i.e. lets the driver
+know by marking the buffer as used. Device can then trigger
+a device event - i.e. send an interrupt to the driver.
+
+For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
+
+\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
+The split virtqueue format is the original format used by legacy
+virtio devices.  The split virtqueue format separates the
+virtqueue into several parts, where each part is write-able by
+either the driver or the device, but not both. Multiple
+locations need to be updated when making a buffer available
+and when marking it as used.
+
+
+Each queue has a 16-bit queue size
 parameter, which sets the number of entries and implies the total size
 of the queue.
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 1/5] content: move 1.0 queue format out to a separate section

[Cornelia Huck]

On Wed, 10 Jan 2018 11:47:55 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 25 ++++++++++++++++++++++++-
>  1 file changed, 24 insertions(+), 1 deletion(-)
> 
> diff --git a/content.tex b/content.tex
> index c7ef7fd..4483a4b 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -230,7 +230,30 @@ result.
>  The mechanism for bulk data transport on virtio devices is
>  pretentiously called a virtqueue. Each device can have zero or more
>  virtqueues\footnote{For example, the simplest network device has one virtqueue for
> -transmit and one for receive.}.  Each queue has a 16-bit queue size
> +transmit and one for receive.}.
> +
> +Driver makes requests available to device by adding
> +an available buffer to the queue - i.e. adding a buffer
> +describing the request to a virtqueue, and optionally triggering
> +a driver event - i.e. sending a notification to the device.
> +
> +Device executes the requests and - when complete - adds
> +a used buffer to the queue - i.e. lets the driver
> +know by marking the buffer as used. Device can then trigger
> +a device event - i.e. send an interrupt to the driver.
> +
> +For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
> +
> +\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
> +The split virtqueue format is the original format used by legacy
> +virtio devices.  

I think the term 'legacy' is a bit confusing here. Elsewhere, 'legacy'
refers to pre-standard devices, and while they use a variant of this
layout, I don't think we should mix this up with 1.0 devices.

What about:

"The split virtqueue format is the original format used by devices
conforming to the 1.0 version of this standard (and a variant thereof
by legacy virtio devices)."

> The split virtqueue format separates the
> +virtqueue into several parts, where each part is write-able by
> +either the driver or the device, but not both. Multiple
> +locations need to be updated when making a buffer available
> +and when marking it as used.
> +
> +
> +Each queue has a 16-bit queue size
>  parameter, which sets the number of entries and implies the total size
>  of the queue.
>  


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v6 2/5] content: move ring text out to a separate file

[Michael S. Tsirkin]

Will be easier to manage this way.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex    | 499 +--------------------------------------------------------
 split-ring.tex | 498 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 499 insertions(+), 498 deletions(-)
 create mode 100644 split-ring.tex

diff --git a/content.tex b/content.tex
index 4483a4b..5b4c4e9 100644
--- a/content.tex
+++ b/content.tex
@@ -244,504 +244,7 @@ a device event - i.e. send an interrupt to the driver.
 
 For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
 
-\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
-The split virtqueue format is the original format used by legacy
-virtio devices.  The split virtqueue format separates the
-virtqueue into several parts, where each part is write-able by
-either the driver or the device, but not both. Multiple
-locations need to be updated when making a buffer available
-and when marking it as used.
-
-
-Each queue has a 16-bit queue size
-parameter, which sets the number of entries and implies the total size
-of the queue.
-
-Each virtqueue consists of three parts:
-
-\begin{itemize}
-\item Descriptor Table
-\item Available Ring
-\item Used Ring
-\end{itemize}
-
-where each part is physically-contiguous in guest memory,
-and has different alignment requirements.
-
-The memory aligment and size requirements, in bytes, of each part of the
-virtqueue are summarized in the following table:
-
-\begin{tabular}{|l|l|l|}
-\hline
-Virtqueue Part    & Alignment & Size \\
-\hline \hline
-Descriptor Table  & 16        & $16 * $(Queue Size) \\
-\hline
-Available Ring    & 2         & $6 + 2 * $(Queue Size) \\
- \hline
-Used Ring         & 4         & $6 + 8 * $(Queue Size) \\
- \hline
-\end{tabular}
-
-The Alignment column gives the minimum alignment for each part
-of the virtqueue.
-
-The Size column gives the total number of bytes for each
-part of the virtqueue.
-
-Queue Size corresponds to the maximum number of buffers in the
-virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
-can be queued at any given time.}.  Queue Size value is always a
-power of 2.  The maximum Queue Size value is 32768.  This value
-is specified in a bus-specific way.
-
-When the driver wants to send a buffer to the device, it fills in
-a slot in the descriptor table (or chains several together), and
-writes the descriptor index into the available ring.  It then
-notifies the device. When the device has finished a buffer, it
-writes the descriptor index into the used ring, and sends an interrupt.
-
-\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a Virtio Device / Virtqueues}
-The driver MUST ensure that the physical address of the first byte
-of each virtqueue part is a multiple of the specified alignment value
-in the above table.
-
-\subsection{Legacy Interfaces: A Note on Virtqueue Layout}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
-
-For Legacy Interfaces, several additional
-restrictions are placed on the virtqueue layout:
-
-Each virtqueue occupies two or more physically-contiguous pages
-(usually defined as 4096 bytes, but depending on the transport;
-henceforth referred to as Queue Align)
-and consists of three parts:
-
-\begin{tabular}{|l|l|l|}
-\hline
-Descriptor Table & Available Ring (\ldots padding\ldots) & Used Ring \\
-\hline
-\end{tabular}
-
-The bus-specific Queue Size field controls the total number of bytes
-for the virtqueue.
-When using the legacy interface, the transitional
-driver MUST retrieve the Queue Size field from the device
-and MUST allocate the total number of bytes for the virtqueue
-according to the following formula (Queue Align given in qalign and
-Queue Size given in qsz):
-
-\begin{lstlisting}
-#define ALIGN(x) (((x) + qalign) & ~qalign)
-static inline unsigned virtq_size(unsigned int qsz)
-{
-     return ALIGN(sizeof(struct virtq_desc)*qsz + sizeof(u16)*(3 + qsz))
-          + ALIGN(sizeof(u16)*3 + sizeof(struct virtq_used_elem)*qsz);
-}
-\end{lstlisting}
-
-This wastes some space with padding.
-When using the legacy interface, both transitional
-devices and drivers MUST use the following virtqueue layout
-structure to locate elements of the virtqueue:
-
-\begin{lstlisting}
-struct virtq {
-        // The actual descriptors (16 bytes each)
-        struct virtq_desc desc[ Queue Size ];
-
-        // A ring of available descriptor heads with free-running index.
-        struct virtq_avail avail;
-
-        // Padding to the next Queue Align boundary.
-        u8 pad[ Padding ];
-
-        // A ring of used descriptor heads with free-running index.
-        struct virtq_used used;
-};
-\end{lstlisting}
-
-\subsection{Legacy Interfaces: A Note on Virtqueue Endianness}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
-
-Note that when using the legacy interface, transitional
-devices and drivers MUST use the native
-endian of the guest as the endian of fields and in the virtqueue.
-This is opposed to little-endian for non-legacy interface as
-specified by this standard.
-It is assumed that the host is already aware of the guest endian.
-
-\subsection{Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing}
-The framing of messages with descriptors is
-independent of the contents of the buffers. For example, a network
-transmit buffer consists of a 12 byte header followed by the network
-packet. This could be most simply placed in the descriptor table as a
-12 byte output descriptor followed by a 1514 byte output descriptor,
-but it could also consist of a single 1526 byte output descriptor in
-the case where the header and packet are adjacent, or even three or
-more descriptors (possibly with loss of efficiency in that case).
-
-Note that, some device implementations have large-but-reasonable
-restrictions on total descriptor size (such as based on IOV_MAX in the
-host OS). This has not been a problem in practice: little sympathy
-will be given to drivers which create unreasonably-sized descriptors
-such as by dividing a network packet into 1500 single-byte
-descriptors!
-
-\devicenormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
-The device MUST NOT make assumptions about the particular arrangement
-of descriptors.  The device MAY have a reasonable limit of descriptors
-it will allow in a chain.
-
-\drivernormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
-The driver MUST place any device-writable descriptor elements after
-any device-readable descriptor elements.
-
-The driver SHOULD NOT use an excessive number of descriptors to
-describe a buffer.
-
-\subsubsection{Legacy Interface: Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
-
-Regrettably, initial driver implementations used simple layouts, and
-devices came to rely on it, despite this specification wording.  In
-addition, the specification for virtio_blk SCSI commands required
-intuiting field lengths from frame boundaries (see
- \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}~\nameref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation})
-
-Thus when using the legacy interface, the VIRTIO_F_ANY_LAYOUT
-feature indicates to both the device and the driver that no
-assumptions were made about framing.  Requirements for
-transitional drivers when this is not negotiated are included in
-each device section.
-
-\subsection{The Virtqueue Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-
-The descriptor table refers to the buffers the driver is using for
-the device. \field{addr} is a physical address, and the buffers
-can be chained via \field{next}. Each descriptor describes a
-buffer which is read-only for the device (``device-readable'') or write-only for the device (``device-writable''), but a chain of
-descriptors can contain both device-readable and device-writable buffers.
-
-The actual contents of the memory offered to the device depends on the
-device type.  Most common is to begin the data with a header
-(containing little-endian fields) for the device to read, and postfix
-it with a status tailer for the device to write.
-
-\begin{lstlisting}
-struct virtq_desc {
-        /* Address (guest-physical). */
-        le64 addr;
-        /* Length. */
-        le32 len;
-
-/* This marks a buffer as continuing via the next field. */
-#define VIRTQ_DESC_F_NEXT   1
-/* This marks a buffer as device write-only (otherwise device read-only). */
-#define VIRTQ_DESC_F_WRITE     2
-/* This means the buffer contains a list of buffer descriptors. */
-#define VIRTQ_DESC_F_INDIRECT   4
-        /* The flags as indicated above. */
-        le16 flags;
-        /* Next field if flags & NEXT */
-        le16 next;
-};
-\end{lstlisting}
-
-The number of descriptors in the table is defined by the queue size
-for this virtqueue: this is the maximum possible descriptor chain length.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to this structure as vring_desc, and the constants as
-VRING_DESC_F_NEXT, etc, but the layout and values were identical.
-\end{note}
-
-\devicenormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
-read a device-writable buffer (it MAY do so for debugging or diagnostic
-purposes).
-
-\drivernormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
-this implies that loops in the descriptor chain are forbidden!
-
-\subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-
-Some devices benefit by concurrently dispatching a large number
-of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this (see \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}). To increase
-ring capacity the driver can store a table of indirect
-descriptors anywhere in memory, and insert a descriptor in main
-virtqueue (with \field{flags}\&VIRTQ_DESC_F_INDIRECT on) that refers to memory buffer
-containing this indirect descriptor table; \field{addr} and \field{len}
-refer to the indirect table address and length in bytes,
-respectively.
-
-The indirect table layout structure looks like this
-(\field{len} is the length of the descriptor that refers to this table,
-which is a variable, so this code won't compile):
-
-\begin{lstlisting}
-struct indirect_descriptor_table {
-        /* The actual descriptors (16 bytes each) */
-        struct virtq_desc desc[len / 16];
-};
-\end{lstlisting}
-
-The first indirect descriptor is located at start of the indirect
-descriptor table (index 0), additional indirect descriptors are
-chained by \field{next}. An indirect descriptor without a valid \field{next}
-(with \field{flags}\&VIRTQ_DESC_F_NEXT off) signals the end of the descriptor.
-A single indirect descriptor
-table can include both device-readable and device-writable descriptors.
-
-\drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
-VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
-set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only
-one table per descriptor).
-
-A driver MUST NOT create a descriptor chain longer than the Queue Size of
-the device.
-
-A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
-in \field{flags}.
-
-\devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
-
-The device MUST handle the case of zero or more normal chained
-descriptors followed by a single descriptor with \field{flags}\&VIRTQ_DESC_F_INDIRECT.
-
-\begin{note}
-While unusual (most implementations either create a chain solely using
-non-indirect descriptors, or use a single indirect element), such a
-layout is valid.
-\end{note}
-
-\subsection{The Virtqueue Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
-
-\begin{lstlisting}
-struct virtq_avail {
-#define VIRTQ_AVAIL_F_NO_INTERRUPT      1
-        le16 flags;
-        le16 idx;
-        le16 ring[ /* Queue Size */ ];
-        le16 used_event; /* Only if VIRTIO_F_EVENT_IDX */
-};
-\end{lstlisting}
-
-The driver uses the available ring to offer buffers to the
-device: each ring entry refers to the head of a descriptor chain.  It is only
-written by the driver and read by the device.
-
-\field{idx} field indicates where the driver would put the next descriptor
-entry in the ring (modulo the queue size). This starts at 0, and increases.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to this structure as vring_avail, and the constant as
-VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
-\end{note}
-
-\subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
-the \field{flags} field in the available ring offers a crude mechanism for the driver to inform
-the device that it doesn't want interrupts when buffers are used.  Otherwise
-\field{used_event} is a more performant alternative where the driver
-specifies how far the device can progress before interrupting.
-
-Neither of these interrupt suppression methods are reliable, as they
-are not synchronized with the device, but they serve as
-useful optimizations.
-
-\drivernormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The driver MUST set \field{flags} to 0 or 1.
-\item The driver MAY set \field{flags} to 1 to advise
-the device that interrupts are not needed.
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The driver MUST set \field{flags} to 0.
-\item The driver MAY use \field{used_event} to advise the device that interrupts are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
-used ring will reach the value \field{used_event} + 1).
-\end{itemize}
-
-The driver MUST handle spurious interrupts from the device.
-
-\devicenormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The device MUST ignore the \field{used_event} value.
-\item After the device writes a descriptor index into the used ring:
-  \begin{itemize}
-  \item If \field{flags} is 1, the device SHOULD NOT send an interrupt.
-  \item If \field{flags} is 0, the device MUST send an interrupt.
-  \end{itemize}
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The device MUST ignore the lower bit of \field{flags}.
-\item After the device writes a descriptor index into the used ring:
-  \begin{itemize}
-  \item If the \field{idx} field in the used ring (which determined
-    where that descriptor index was placed) was equal to
-    \field{used_event}, the device MUST send an interrupt.
-  \item Otherwise the device SHOULD NOT send an interrupt.
-  \end{itemize}
-\end{itemize}
-
-\begin{note}
-For example, if \field{used_event} is 0, then a device using
-  VIRTIO_F_EVENT_IDX would interrupt after the first buffer is
-  used (and again after the 65536th buffer, etc).
-\end{note}
-
-\subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-\begin{lstlisting}
-struct virtq_used {
-#define VIRTQ_USED_F_NO_NOTIFY  1
-        le16 flags;
-        le16 idx;
-        struct virtq_used_elem ring[ /* Queue Size */];
-        le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
-};
-
-/* le32 is used here for ids for padding reasons. */
-struct virtq_used_elem {
-        /* Index of start of used descriptor chain. */
-        le32 id;
-        /* Total length of the descriptor chain which was used (written to) */
-        le32 len;
-};
-\end{lstlisting}
-
-The used ring is where the device returns buffers once it is done with
-them: it is only written to by the device, and read by the driver.
-
-Each entry in the ring is a pair: \field{id} indicates the head entry of the
-descriptor chain describing the buffer (this matches an entry
-placed in the available ring by the guest earlier), and \field{len} the total
-of bytes written into the buffer.
-
-\begin{note}
-\field{len} is particularly useful
-for drivers using untrusted buffers: if a driver does not know exactly
-how much has been written by the device, the driver would have to zero
-the buffer in advance to ensure no data leakage occurs.
-
-For example, a network driver may hand a received buffer directly to
-an unprivileged userspace application.  If the network device has not
-overwritten the bytes which were in that buffer, this could leak the
-contents of freed memory from other processes to the application.
-\end{note}
-
-\field{idx} field indicates where the driver would put the next descriptor
-entry in the ring (modulo the queue size). This starts at 0, and increases.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to these structures as vring_used and vring_used_elem, and
-the constant as VRING_USED_F_NO_NOTIFY, but the layout and value were
-identical.
-\end{note}
-
-\subsubsection{Legacy Interface: The Virtqueue Used
-Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues
-/ The Virtqueue Used Ring/ Legacy Interface: The Virtqueue Used
-Ring}
-
-Historically, many drivers ignored the \field{len} value, as a
-result, many devices set \field{len} incorrectly.  Thus, when
-using the legacy interface, it is generally a good idea to ignore
-the \field{len} value in used ring entries if possible.  Specific
-known issues are listed per device type.
-
-\devicenormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-The device MUST set \field{len} prior to updating the used \field{idx}.
-
-The device MUST write at least \field{len} bytes to descriptor,
-beginning at the first device-writable buffer,
-prior to updating the used \field{idx}.
-
-The device MAY write more than \field{len} bytes to descriptor.
-
-\begin{note}
-There are potential error cases where a device might not know what
-parts of the buffers have been written.  This is why \field{len} is
-permitted to be an underestimate: that's preferable to the driver believing
-that uninitialized memory has been overwritten when it has not.
-\end{note}
-
-\drivernormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-The driver MUST NOT make assumptions about data in device-writable buffers
-beyond the first \field{len} bytes, and SHOULD ignore this data.
-
-\subsection{Virtqueue Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-
-The device can suppress notifications in a manner analogous to the way
-drivers can suppress interrupts as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
-The device manipulates \field{flags} or \field{avail_event} in the used ring the
-same way the driver manipulates \field{flags} or \field{used_event} in the available ring.
-
-\drivernormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-
-The driver MUST initialize \field{flags} in the used ring to 0 when
-allocating the used ring.
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The driver MUST ignore the \field{avail_event} value.
-\item After the driver writes a descriptor index into the available ring:
-  \begin{itemize}
-        \item If \field{flags} is 1, the driver SHOULD NOT send a notification.
-        \item If \field{flags} is 0, the driver MUST send a notification.
-  \end{itemize}
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The driver MUST ignore the lower bit of \field{flags}.
-\item After the driver writes a descriptor index into the available ring:
-  \begin{itemize}
-        \item If the \field{idx} field in the available ring (which determined
-          where that descriptor index was placed) was equal to
-          \field{avail_event}, the driver MUST send a notification.
-        \item Otherwise the driver SHOULD NOT send a notification.
-  \end{itemize}
-\end{itemize}
-
-\devicenormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The device MUST set \field{flags} to 0 or 1.
-\item The device MAY set \field{flags} to 1 to advise
-the driver that notifications are not needed.
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The device MUST set \field{flags} to 0.
-\item The device MAY use \field{avail_event} to advise the driver that notifications are unnecessary until the driver writes entry with an index specified by \field{avail_event} into the available ring (equivalently, until \field{idx} in the
-available ring will reach the value \field{avail_event} + 1).
-\end{itemize}
-
-The device MUST handle spurious notifications from the driver.
-
-\subsection{Helpers for Operating Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Helpers for Operating Virtqueues}
-
-The Linux Kernel Source code contains the definitions above and
-helper routines in a more usable form, in
-include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
-and Red Hat under the (3-clause) BSD license so that it can be
-freely used by all other projects, and is reproduced (with slight
-variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
+\input{split-ring.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
diff --git a/split-ring.tex b/split-ring.tex
new file mode 100644
index 0000000..418f63d
--- /dev/null
+++ b/split-ring.tex
@@ -0,0 +1,498 @@
+\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
+The split virtqueue format is the original format used by legacy
+virtio devices.  The split virtqueue format separates the
+virtqueue into several parts, where each part is write-able by
+either the driver or the device, but not both. Multiple
+locations need to be updated when making a buffer available
+and when marking it as used.
+
+
+Each queue has a 16-bit queue size
+parameter, which sets the number of entries and implies the total size
+of the queue.
+
+Each virtqueue consists of three parts:
+
+\begin{itemize}
+\item Descriptor Table
+\item Available Ring
+\item Used Ring
+\end{itemize}
+
+where each part is physically-contiguous in guest memory,
+and has different alignment requirements.
+
+The memory aligment and size requirements, in bytes, of each part of the
+virtqueue are summarized in the following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Virtqueue Part    & Alignment & Size \\
+\hline \hline
+Descriptor Table  & 16        & $16 * $(Queue Size) \\
+\hline
+Available Ring    & 2         & $6 + 2 * $(Queue Size) \\
+ \hline
+Used Ring         & 4         & $6 + 8 * $(Queue Size) \\
+ \hline
+\end{tabular}
+
+The Alignment column gives the minimum alignment for each part
+of the virtqueue.
+
+The Size column gives the total number of bytes for each
+part of the virtqueue.
+
+Queue Size corresponds to the maximum number of buffers in the
+virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
+can be queued at any given time.}.  Queue Size value is always a
+power of 2.  The maximum Queue Size value is 32768.  This value
+is specified in a bus-specific way.
+
+When the driver wants to send a buffer to the device, it fills in
+a slot in the descriptor table (or chains several together), and
+writes the descriptor index into the available ring.  It then
+notifies the device. When the device has finished a buffer, it
+writes the descriptor index into the used ring, and sends an interrupt.
+
+\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a Virtio Device / Virtqueues}
+The driver MUST ensure that the physical address of the first byte
+of each virtqueue part is a multiple of the specified alignment value
+in the above table.
+
+\subsection{Legacy Interfaces: A Note on Virtqueue Layout}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
+
+For Legacy Interfaces, several additional
+restrictions are placed on the virtqueue layout:
+
+Each virtqueue occupies two or more physically-contiguous pages
+(usually defined as 4096 bytes, but depending on the transport;
+henceforth referred to as Queue Align)
+and consists of three parts:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Descriptor Table & Available Ring (\ldots padding\ldots) & Used Ring \\
+\hline
+\end{tabular}
+
+The bus-specific Queue Size field controls the total number of bytes
+for the virtqueue.
+When using the legacy interface, the transitional
+driver MUST retrieve the Queue Size field from the device
+and MUST allocate the total number of bytes for the virtqueue
+according to the following formula (Queue Align given in qalign and
+Queue Size given in qsz):
+
+\begin{lstlisting}
+#define ALIGN(x) (((x) + qalign) & ~qalign)
+static inline unsigned virtq_size(unsigned int qsz)
+{
+     return ALIGN(sizeof(struct virtq_desc)*qsz + sizeof(u16)*(3 + qsz))
+          + ALIGN(sizeof(u16)*3 + sizeof(struct virtq_used_elem)*qsz);
+}
+\end{lstlisting}
+
+This wastes some space with padding.
+When using the legacy interface, both transitional
+devices and drivers MUST use the following virtqueue layout
+structure to locate elements of the virtqueue:
+
+\begin{lstlisting}
+struct virtq {
+        // The actual descriptors (16 bytes each)
+        struct virtq_desc desc[ Queue Size ];
+
+        // A ring of available descriptor heads with free-running index.
+        struct virtq_avail avail;
+
+        // Padding to the next Queue Align boundary.
+        u8 pad[ Padding ];
+
+        // A ring of used descriptor heads with free-running index.
+        struct virtq_used used;
+};
+\end{lstlisting}
+
+\subsection{Legacy Interfaces: A Note on Virtqueue Endianness}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
+
+Note that when using the legacy interface, transitional
+devices and drivers MUST use the native
+endian of the guest as the endian of fields and in the virtqueue.
+This is opposed to little-endian for non-legacy interface as
+specified by this standard.
+It is assumed that the host is already aware of the guest endian.
+
+\subsection{Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing}
+The framing of messages with descriptors is
+independent of the contents of the buffers. For example, a network
+transmit buffer consists of a 12 byte header followed by the network
+packet. This could be most simply placed in the descriptor table as a
+12 byte output descriptor followed by a 1514 byte output descriptor,
+but it could also consist of a single 1526 byte output descriptor in
+the case where the header and packet are adjacent, or even three or
+more descriptors (possibly with loss of efficiency in that case).
+
+Note that, some device implementations have large-but-reasonable
+restrictions on total descriptor size (such as based on IOV_MAX in the
+host OS). This has not been a problem in practice: little sympathy
+will be given to drivers which create unreasonably-sized descriptors
+such as by dividing a network packet into 1500 single-byte
+descriptors!
+
+\devicenormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
+The device MUST NOT make assumptions about the particular arrangement
+of descriptors.  The device MAY have a reasonable limit of descriptors
+it will allow in a chain.
+
+\drivernormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
+The driver MUST place any device-writable descriptor elements after
+any device-readable descriptor elements.
+
+The driver SHOULD NOT use an excessive number of descriptors to
+describe a buffer.
+
+\subsubsection{Legacy Interface: Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
+
+Regrettably, initial driver implementations used simple layouts, and
+devices came to rely on it, despite this specification wording.  In
+addition, the specification for virtio_blk SCSI commands required
+intuiting field lengths from frame boundaries (see
+ \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}~\nameref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation})
+
+Thus when using the legacy interface, the VIRTIO_F_ANY_LAYOUT
+feature indicates to both the device and the driver that no
+assumptions were made about framing.  Requirements for
+transitional drivers when this is not negotiated are included in
+each device section.
+
+\subsection{The Virtqueue Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+
+The descriptor table refers to the buffers the driver is using for
+the device. \field{addr} is a physical address, and the buffers
+can be chained via \field{next}. Each descriptor describes a
+buffer which is read-only for the device (``device-readable'') or write-only for the device (``device-writable''), but a chain of
+descriptors can contain both device-readable and device-writable buffers.
+
+The actual contents of the memory offered to the device depends on the
+device type.  Most common is to begin the data with a header
+(containing little-endian fields) for the device to read, and postfix
+it with a status tailer for the device to write.
+
+\begin{lstlisting}
+struct virtq_desc {
+        /* Address (guest-physical). */
+        le64 addr;
+        /* Length. */
+        le32 len;
+
+/* This marks a buffer as continuing via the next field. */
+#define VIRTQ_DESC_F_NEXT   1
+/* This marks a buffer as device write-only (otherwise device read-only). */
+#define VIRTQ_DESC_F_WRITE     2
+/* This means the buffer contains a list of buffer descriptors. */
+#define VIRTQ_DESC_F_INDIRECT   4
+        /* The flags as indicated above. */
+        le16 flags;
+        /* Next field if flags & NEXT */
+        le16 next;
+};
+\end{lstlisting}
+
+The number of descriptors in the table is defined by the queue size
+for this virtqueue: this is the maximum possible descriptor chain length.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to this structure as vring_desc, and the constants as
+VRING_DESC_F_NEXT, etc, but the layout and values were identical.
+\end{note}
+
+\devicenormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
+read a device-writable buffer (it MAY do so for debugging or diagnostic
+purposes).
+
+\drivernormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
+this implies that loops in the descriptor chain are forbidden!
+
+\subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+
+Some devices benefit by concurrently dispatching a large number
+of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this (see \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}). To increase
+ring capacity the driver can store a table of indirect
+descriptors anywhere in memory, and insert a descriptor in main
+virtqueue (with \field{flags}\&VIRTQ_DESC_F_INDIRECT on) that refers to memory buffer
+containing this indirect descriptor table; \field{addr} and \field{len}
+refer to the indirect table address and length in bytes,
+respectively.
+
+The indirect table layout structure looks like this
+(\field{len} is the length of the descriptor that refers to this table,
+which is a variable, so this code won't compile):
+
+\begin{lstlisting}
+struct indirect_descriptor_table {
+        /* The actual descriptors (16 bytes each) */
+        struct virtq_desc desc[len / 16];
+};
+\end{lstlisting}
+
+The first indirect descriptor is located at start of the indirect
+descriptor table (index 0), additional indirect descriptors are
+chained by \field{next}. An indirect descriptor without a valid \field{next}
+(with \field{flags}\&VIRTQ_DESC_F_NEXT off) signals the end of the descriptor.
+A single indirect descriptor
+table can include both device-readable and device-writable descriptors.
+
+\drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
+VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
+set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only
+one table per descriptor).
+
+A driver MUST NOT create a descriptor chain longer than the Queue Size of
+the device.
+
+A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
+in \field{flags}.
+
+\devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
+
+The device MUST handle the case of zero or more normal chained
+descriptors followed by a single descriptor with \field{flags}\&VIRTQ_DESC_F_INDIRECT.
+
+\begin{note}
+While unusual (most implementations either create a chain solely using
+non-indirect descriptors, or use a single indirect element), such a
+layout is valid.
+\end{note}
+
+\subsection{The Virtqueue Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
+
+\begin{lstlisting}
+struct virtq_avail {
+#define VIRTQ_AVAIL_F_NO_INTERRUPT      1
+        le16 flags;
+        le16 idx;
+        le16 ring[ /* Queue Size */ ];
+        le16 used_event; /* Only if VIRTIO_F_EVENT_IDX */
+};
+\end{lstlisting}
+
+The driver uses the available ring to offer buffers to the
+device: each ring entry refers to the head of a descriptor chain.  It is only
+written by the driver and read by the device.
+
+\field{idx} field indicates where the driver would put the next descriptor
+entry in the ring (modulo the queue size). This starts at 0, and increases.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to this structure as vring_avail, and the constant as
+VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
+\end{note}
+
+\subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
+the \field{flags} field in the available ring offers a crude mechanism for the driver to inform
+the device that it doesn't want interrupts when buffers are used.  Otherwise
+\field{used_event} is a more performant alternative where the driver
+specifies how far the device can progress before interrupting.
+
+Neither of these interrupt suppression methods are reliable, as they
+are not synchronized with the device, but they serve as
+useful optimizations.
+
+\drivernormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The driver MUST set \field{flags} to 0 or 1.
+\item The driver MAY set \field{flags} to 1 to advise
+the device that interrupts are not needed.
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The driver MUST set \field{flags} to 0.
+\item The driver MAY use \field{used_event} to advise the device that interrupts are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
+used ring will reach the value \field{used_event} + 1).
+\end{itemize}
+
+The driver MUST handle spurious interrupts from the device.
+
+\devicenormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The device MUST ignore the \field{used_event} value.
+\item After the device writes a descriptor index into the used ring:
+  \begin{itemize}
+  \item If \field{flags} is 1, the device SHOULD NOT send an interrupt.
+  \item If \field{flags} is 0, the device MUST send an interrupt.
+  \end{itemize}
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The device MUST ignore the lower bit of \field{flags}.
+\item After the device writes a descriptor index into the used ring:
+  \begin{itemize}
+  \item If the \field{idx} field in the used ring (which determined
+    where that descriptor index was placed) was equal to
+    \field{used_event}, the device MUST send an interrupt.
+  \item Otherwise the device SHOULD NOT send an interrupt.
+  \end{itemize}
+\end{itemize}
+
+\begin{note}
+For example, if \field{used_event} is 0, then a device using
+  VIRTIO_F_EVENT_IDX would interrupt after the first buffer is
+  used (and again after the 65536th buffer, etc).
+\end{note}
+
+\subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+\begin{lstlisting}
+struct virtq_used {
+#define VIRTQ_USED_F_NO_NOTIFY  1
+        le16 flags;
+        le16 idx;
+        struct virtq_used_elem ring[ /* Queue Size */];
+        le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
+};
+
+/* le32 is used here for ids for padding reasons. */
+struct virtq_used_elem {
+        /* Index of start of used descriptor chain. */
+        le32 id;
+        /* Total length of the descriptor chain which was used (written to) */
+        le32 len;
+};
+\end{lstlisting}
+
+The used ring is where the device returns buffers once it is done with
+them: it is only written to by the device, and read by the driver.
+
+Each entry in the ring is a pair: \field{id} indicates the head entry of the
+descriptor chain describing the buffer (this matches an entry
+placed in the available ring by the guest earlier), and \field{len} the total
+of bytes written into the buffer.
+
+\begin{note}
+\field{len} is particularly useful
+for drivers using untrusted buffers: if a driver does not know exactly
+how much has been written by the device, the driver would have to zero
+the buffer in advance to ensure no data leakage occurs.
+
+For example, a network driver may hand a received buffer directly to
+an unprivileged userspace application.  If the network device has not
+overwritten the bytes which were in that buffer, this could leak the
+contents of freed memory from other processes to the application.
+\end{note}
+
+\field{idx} field indicates where the driver would put the next descriptor
+entry in the ring (modulo the queue size). This starts at 0, and increases.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to these structures as vring_used and vring_used_elem, and
+the constant as VRING_USED_F_NO_NOTIFY, but the layout and value were
+identical.
+\end{note}
+
+\subsubsection{Legacy Interface: The Virtqueue Used
+Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues
+/ The Virtqueue Used Ring/ Legacy Interface: The Virtqueue Used
+Ring}
+
+Historically, many drivers ignored the \field{len} value, as a
+result, many devices set \field{len} incorrectly.  Thus, when
+using the legacy interface, it is generally a good idea to ignore
+the \field{len} value in used ring entries if possible.  Specific
+known issues are listed per device type.
+
+\devicenormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+The device MUST set \field{len} prior to updating the used \field{idx}.
+
+The device MUST write at least \field{len} bytes to descriptor,
+beginning at the first device-writable buffer,
+prior to updating the used \field{idx}.
+
+The device MAY write more than \field{len} bytes to descriptor.
+
+\begin{note}
+There are potential error cases where a device might not know what
+parts of the buffers have been written.  This is why \field{len} is
+permitted to be an underestimate: that's preferable to the driver believing
+that uninitialized memory has been overwritten when it has not.
+\end{note}
+
+\drivernormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+The driver MUST NOT make assumptions about data in device-writable buffers
+beyond the first \field{len} bytes, and SHOULD ignore this data.
+
+\subsection{Virtqueue Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+
+The device can suppress notifications in a manner analogous to the way
+drivers can suppress interrupts as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+The device manipulates \field{flags} or \field{avail_event} in the used ring the
+same way the driver manipulates \field{flags} or \field{used_event} in the available ring.
+
+\drivernormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+
+The driver MUST initialize \field{flags} in the used ring to 0 when
+allocating the used ring.
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The driver MUST ignore the \field{avail_event} value.
+\item After the driver writes a descriptor index into the available ring:
+  \begin{itemize}
+        \item If \field{flags} is 1, the driver SHOULD NOT send a notification.
+        \item If \field{flags} is 0, the driver MUST send a notification.
+  \end{itemize}
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The driver MUST ignore the lower bit of \field{flags}.
+\item After the driver writes a descriptor index into the available ring:
+  \begin{itemize}
+        \item If the \field{idx} field in the available ring (which determined
+          where that descriptor index was placed) was equal to
+          \field{avail_event}, the driver MUST send a notification.
+        \item Otherwise the driver SHOULD NOT send a notification.
+  \end{itemize}
+\end{itemize}
+
+\devicenormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The device MUST set \field{flags} to 0 or 1.
+\item The device MAY set \field{flags} to 1 to advise
+the driver that notifications are not needed.
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The device MUST set \field{flags} to 0.
+\item The device MAY use \field{avail_event} to advise the driver that notifications are unnecessary until the driver writes entry with an index specified by \field{avail_event} into the available ring (equivalently, until \field{idx} in the
+available ring will reach the value \field{avail_event} + 1).
+\end{itemize}
+
+The device MUST handle spurious notifications from the driver.
+
+\subsection{Helpers for Operating Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Helpers for Operating Virtqueues}
+
+The Linux Kernel Source code contains the definitions above and
+helper routines in a more usable form, in
+include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
+and Red Hat under the (3-clause) BSD license so that it can be
+freely used by all other projects, and is reproduced (with slight
+variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
-- 
MST



---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 2/5] content: move ring text out to a separate file

[Cornelia Huck]

On Wed, 10 Jan 2018 11:47:56 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Will be easier to manage this way.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex    | 499 +--------------------------------------------------------
>  split-ring.tex | 498 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 499 insertions(+), 498 deletions(-)
>  create mode 100644 split-ring.tex

Makes sense.

Acked-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v6 3/5] content: move virtqueue operation description

[Michael S. Tsirkin]

virtqueue operation description is specific to the virtqueue
format. Move it out to split-ring.tex and update all
references.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 conformance.tex |   4 +-
 content.tex     | 171 +++-----------------------------------------------------
 split-ring.tex  | 168 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 178 insertions(+), 165 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index f59e360..55d17b4 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -40,9 +40,9 @@ A driver MUST conform to the following normative statements:
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization}
-\item \ref{drivernormative:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-\item \ref{drivernormative:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Cleanup}
 \item \ref{drivernormative:Reserved Feature Bits}
 \end{itemize}
diff --git a/content.tex b/content.tex
index 5b4c4e9..3b4579e 100644
--- a/content.tex
+++ b/content.tex
@@ -337,167 +337,14 @@ And Device Operation / Device Initialization / Set DRIVER-OK}.
 
 \section{Device Operation}\label{sec:General Initialization And Device Operation / Device Operation}
 
-There are two parts to device operation: supplying new buffers to
-the device, and processing used buffers from the device.
-
-\begin{note} As an
-example, the simplest virtio network device has two virtqueues: the
-transmit virtqueue and the receive virtqueue. The driver adds
-outgoing (device-readable) packets to the transmit virtqueue, and then
-frees them after they are used. Similarly, incoming (device-writable)
-buffers are added to the receive virtqueue, and processed after
-they are used.
-\end{note}
-
-\subsection{Supplying Buffers to The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device}
-
-The driver offers buffers to one of the device's virtqueues as follows:
-
-\begin{enumerate}
-\item\label{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Buffers} The driver places the buffer into free descriptor(s) in the
-   descriptor table, chaining as necessary (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}).
-
-\item\label{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Index} The driver places the index of the head of the descriptor chain
-   into the next ring entry of the available ring.
-
-\item Steps \ref{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Buffers} and \ref{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Index} MAY be performed repeatedly if batching
-  is possible.
-
-\item The driver performs suitable a memory barrier to ensure the device sees
-  the updated descriptor table and available ring before the next
-  step.
-
-\item The available \field{idx} is increased by the number of
-  descriptor chain heads added to the available ring.
-
-\item The driver performs a suitable memory barrier to ensure that it updates
-  the \field{idx} field before checking for notification suppression.
-
-\item If notifications are not suppressed, the driver notifies the device
-    of the new available buffers.
-\end{enumerate}
-
-Note that the above code does not take precautions against the
-available ring buffer wrapping around: this is not possible since
-the ring buffer is the same size as the descriptor table, so step
-(1) will prevent such a condition.
-
-In addition, the maximum queue size is 32768 (the highest power
-of 2 which fits in 16 bits), so the 16-bit \field{idx} value can always
-distinguish between a full and empty buffer.
+When operating the device, each field in the device configuration
+space can be changed by either the driver or the device.
 
-What follows is the requirements of each stage in more detail.
-
-\subsubsection{Placing Buffers Into The Descriptor Table}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Placing Buffers Into The Descriptor Table}
-
-A buffer consists of zero or more device-readable physically-contiguous
-elements followed by zero or more physically-contiguous
-device-writable elements (each has at least one element). This
-algorithm maps it into the descriptor table to form a descriptor
-chain:
-
-for each buffer element, b:
-
-\begin{enumerate}
-\item Get the next free descriptor table entry, d
-\item Set \field{d.addr} to the physical address of the start of b
-\item Set \field{d.len} to the length of b.
-\item If b is device-writable, set \field{d.flags} to VIRTQ_DESC_F_WRITE,
-    otherwise 0.
-\item If there is a buffer element after this:
-    \begin{enumerate}
-    \item Set \field{d.next} to the index of the next free descriptor
-      element.
-    \item Set the VIRTQ_DESC_F_NEXT bit in \field{d.flags}.
-    \end{enumerate}
-\end{enumerate}
-
-In practice, \field{d.next} is usually used to chain free
-descriptors, and a separate count kept to check there are enough
-free descriptors before beginning the mappings.
-
-\subsubsection{Updating The Available Ring}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating The Available Ring}
-
-The descriptor chain head is the first d in the algorithm
-above, ie. the index of the descriptor table entry referring to the first
-part of the buffer.  A naive driver implementation MAY do the following (with the
-appropriate conversion to-and-from little-endian assumed):
-
-\begin{lstlisting}
-avail->ring[avail->idx % qsz] = head;
-\end{lstlisting}
+Whenever such a configuration change is triggered by the device,
+driver is notified. This makes it possible for drivers to
+cache device configuration, avoiding expensive configuration
+reads unless notified.
 
-However, in general the driver MAY add many descriptor chains before it updates
-\field{idx} (at which point they become visible to the
-device), so it is common to keep a counter of how many the driver has added:
-
-\begin{lstlisting}
-avail->ring[(avail->idx + added++) % qsz] = head;
-\end{lstlisting}
-
-\subsubsection{Updating \field{idx}}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-
-\field{idx} always increments, and wraps naturally at
-65536:
-
-\begin{lstlisting}
-avail->idx += added;
-\end{lstlisting}
-
-Once available \field{idx} is updated by the driver, this exposes the
-descriptor and its contents.  The device MAY
-access the descriptor chains the driver created and the
-memory they refer to immediately.
-
-\drivernormative{\paragraph}{Updating idx}{General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-The driver MUST perform a suitable memory barrier before the \field{idx} update, to ensure the
-device sees the most up-to-date copy.
-
-\subsubsection{Notifying The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
-
-The actual method of device notification is bus-specific, but generally
-it can be expensive.  So the device MAY suppress such notifications if it
-doesn't need them, as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}.
-
-The driver has to be careful to expose the new \field{idx}
-value before checking if notifications are suppressed.
-
-\drivernormative{\paragraph}{Notifying The Device}{General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
-The driver MUST perform a suitable memory barrier before reading \field{flags} or
-\field{avail_event}, to avoid missing a notification.
-
-\subsection{Receiving Used Buffers From The Device}\label{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}
-
-Once the device has used buffers referred to by a descriptor (read from or written to them, or
-parts of both, depending on the nature of the virtqueue and the
-device), it interrupts the driver as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
-
-\begin{note}
-For optimal performance, a driver MAY disable interrupts while processing
-the used ring, but beware the problem of missing interrupts between
-emptying the ring and reenabling interrupts.  This is usually handled by
-re-checking for more used buffers after interrups are re-enabled:
-
-\begin{lstlisting}
-virtq_disable_interrupts(vq);
-
-for (;;) {
-        if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) {
-                virtq_enable_interrupts(vq);
-                mb();
-
-                if (vq->last_seen_used != le16_to_cpu(virtq->used.idx))
-                        break;
-
-                virtq_disable_interrupts(vq);
-        }
-
-        struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz];
-        process_buffer(e);
-        vq->last_seen_used++;
-}
-\end{lstlisting}
-\end{note}
 
 \subsection{Notification of Device Configuration Changes}\label{sec:General Initialization And Device Operation / Device Operation / Notification of Device Configuration Changes}
 
@@ -3017,9 +2864,7 @@ If VIRTIO_NET_HDR_F_NEEDS_CSUM is not set, the device MUST NOT
 rely on the packet checksum being correct.
 \paragraph{Packet Transmission Interrupt}\label{sec:Device Types / Network Device / Device Operation / Packet Transmission / Packet Transmission Interrupt}
 
-Often a driver will suppress transmission interrupts using the
-VIRTQ_AVAIL_F_NO_INTERRUPT flag
- (see \ref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}~\nameref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device})
+Often a driver will suppress transmission virtqueue interrupts
 and check for used packets in the transmit path of following
 packets.
 
@@ -3079,7 +2924,7 @@ if VIRTIO_NET_F_MRG_RXBUF is not negotiated.}
 
 When a packet is copied into a buffer in the receiveq, the
 optimal path is to disable further interrupts for the receiveq
-(see \ref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}~\nameref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}) and process
+and process
 packets until no more are found, then re-enable them.
 
 Processing incoming packets involves:
diff --git a/split-ring.tex b/split-ring.tex
index 418f63d..6da5cba 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -496,3 +496,171 @@ include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
 and Red Hat under the (3-clause) BSD license so that it can be
 freely used by all other projects, and is reproduced (with slight
 variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
+
+\subsection{Virtqueue Operation}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Operation}
+
+There are two parts to virtqueue operation: supplying new
+available buffers to the device, and processing used buffers from
+the device.
+
+\begin{note} As an
+example, the simplest virtio network device has two virtqueues: the
+transmit virtqueue and the receive virtqueue. The driver adds
+outgoing (device-readable) packets to the transmit virtqueue, and then
+frees them after they are used. Similarly, incoming (device-writable)
+buffers are added to the receive virtqueue, and processed after
+they are used.
+\end{note}
+
+What follows is the requirements of each of these two parts
+when using the split virtqueue format in more detail.
+
+\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device}
+
+The driver offers buffers to one of the device's virtqueues as follows:
+
+\begin{enumerate}
+\item\label{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Buffers} The driver places the buffer into free descriptor(s) in the
+   descriptor table, chaining as necessary (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}).
+
+\item\label{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Index} The driver places the index of the head of the descriptor chain
+   into the next ring entry of the available ring.
+
+\item Steps \ref{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Buffers} and \ref{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Index} MAY be performed repeatedly if batching
+  is possible.
+
+\item The driver performs suitable a memory barrier to ensure the device sees
+  the updated descriptor table and available ring before the next
+  step.
+
+\item The available \field{idx} is increased by the number of
+  descriptor chain heads added to the available ring.
+
+\item The driver performs a suitable memory barrier to ensure that it updates
+  the \field{idx} field before checking for notification suppression.
+
+\item If notifications are not suppressed, the driver notifies the device
+    of the new available buffers.
+\end{enumerate}
+
+Note that the above code does not take precautions against the
+available ring buffer wrapping around: this is not possible since
+the ring buffer is the same size as the descriptor table, so step
+(1) will prevent such a condition.
+
+In addition, the maximum queue size is 32768 (the highest power
+of 2 which fits in 16 bits), so the 16-bit \field{idx} value can always
+distinguish between a full and empty buffer.
+
+What follows is the requirements of each stage in more detail.
+
+\subsubsection{Placing Buffers Into The Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Buffers Into The Descriptor Table}
+
+A buffer consists of zero or more device-readable physically-contiguous
+elements followed by zero or more physically-contiguous
+device-writable elements (each has at least one element). This
+algorithm maps it into the descriptor table to form a descriptor
+chain:
+
+for each buffer element, b:
+
+\begin{enumerate}
+\item Get the next free descriptor table entry, d
+\item Set \field{d.addr} to the physical address of the start of b
+\item Set \field{d.len} to the length of b.
+\item If b is device-writable, set \field{d.flags} to VIRTQ_DESC_F_WRITE,
+    otherwise 0.
+\item If there is a buffer element after this:
+    \begin{enumerate}
+    \item Set \field{d.next} to the index of the next free descriptor
+      element.
+    \item Set the VIRTQ_DESC_F_NEXT bit in \field{d.flags}.
+    \end{enumerate}
+\end{enumerate}
+
+In practice, \field{d.next} is usually used to chain free
+descriptors, and a separate count kept to check there are enough
+free descriptors before beginning the mappings.
+
+\subsubsection{Updating The Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating The Available Ring}
+
+The descriptor chain head is the first d in the algorithm
+above, ie. the index of the descriptor table entry referring to the first
+part of the buffer.  A naive driver implementation MAY do the following (with the
+appropriate conversion to-and-from little-endian assumed):
+
+\begin{lstlisting}
+avail->ring[avail->idx % qsz] = head;
+\end{lstlisting}
+
+However, in general the driver MAY add many descriptor chains before it updates
+\field{idx} (at which point they become visible to the
+device), so it is common to keep a counter of how many the driver has added:
+
+\begin{lstlisting}
+avail->ring[(avail->idx + added++) % qsz] = head;
+\end{lstlisting}
+
+\subsubsection{Updating \field{idx}}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+
+\field{idx} always increments, and wraps naturally at
+65536:
+
+\begin{lstlisting}
+avail->idx += added;
+\end{lstlisting}
+
+Once available \field{idx} is updated by the driver, this exposes the
+descriptor and its contents.  The device MAY
+access the descriptor chains the driver created and the
+memory they refer to immediately.
+
+\drivernormative{\paragraph}{Updating idx}{Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+The driver MUST perform a suitable memory barrier before the \field{idx} update, to ensure the
+device sees the most up-to-date copy.
+
+\subsubsection{Notifying The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+
+The actual method of device notification is bus-specific, but generally
+it can be expensive.  So the device MAY suppress such notifications if it
+doesn't need them, as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}.
+
+The driver has to be careful to expose the new \field{idx}
+value before checking if notifications are suppressed.
+
+\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+The driver MUST perform a suitable memory barrier before reading \field{flags} or
+\field{avail_event}, to avoid missing a notification.
+
+\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Receiving Used Buffers From The Device}
+
+Once the device has used buffers referred to by a descriptor (read from or written to them, or
+parts of both, depending on the nature of the virtqueue and the
+device), it interrupts the driver as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+
+\begin{note}
+For optimal performance, a driver MAY disable interrupts while processing
+the used ring, but beware the problem of missing interrupts between
+emptying the ring and reenabling interrupts.  This is usually handled by
+re-checking for more used buffers after interrups are re-enabled:
+
+\begin{lstlisting}
+virtq_disable_interrupts(vq);
+
+for (;;) {
+        if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) {
+                virtq_enable_interrupts(vq);
+                mb();
+
+                if (vq->last_seen_used != le16_to_cpu(virtq->used.idx))
+                        break;
+
+                virtq_disable_interrupts(vq);
+        }
+
+        struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz];
+        process_buffer(e);
+        vq->last_seen_used++;
+}
+\end{lstlisting}
+\end{note}
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 3/5] content: move virtqueue operation description

[Cornelia Huck]

On Wed, 10 Jan 2018 11:47:57 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> virtqueue operation description is specific to the virtqueue
> format. Move it out to split-ring.tex and update all
> references.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  conformance.tex |   4 +-
>  content.tex     | 171 +++-----------------------------------------------------
>  split-ring.tex  | 168 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 178 insertions(+), 165 deletions(-)

Acked-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

Performance analysis of this is in my kvm forum 2016 presentation.  The
idea is to have a r/w descriptor in a ring structure, replacing the used
and available ring, index and descriptor buffer.

This is also easier for devices to implement than the 1.0 layout.
Several more enhancements will be necessary to actually make this
efficient for devices to use.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex     |  36 ++-
 packed-ring.tex | 668 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 701 insertions(+), 3 deletions(-)
 create mode 100644 packed-ring.tex

diff --git a/content.tex b/content.tex
index 3b4579e..3059bd3 100644
--- a/content.tex
+++ b/content.tex
@@ -242,10 +242,26 @@ a used buffer to the queue - i.e. lets the driver
 know by marking the buffer as used. Device can then trigger
 a device event - i.e. send an interrupt to the driver.
 
-For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
+Device is not generally required to use buffers in
+the same order in which they have been made available
+by the driver.
+
+Some devices always use descriptors in the same order in which
+they have been made available. These devices can offer the
+VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+might allow optimizations or simplify driver code.
+
+Two formats are supported: Split Virtqueues (see \ref{sec:Basic
+Facilities of a Virtio Device / Split
+Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
+Split Virtqueues}) and Packed Virtqueues (see \ref{sec:Basic
+Facilities of a Virtio Device / Packed
+Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
+Packed Virtqueues}).
 
 \input{split-ring.tex}
 
+\input{packed-ring.tex}
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -5199,10 +5215,19 @@ Currently these device-independent feature bits defined:
 \begin{description}
   \item[VIRTIO_F_RING_INDIRECT_DESC (28)] Negotiating this feature indicates
   that the driver can use descriptors with the VIRTQ_DESC_F_INDIRECT
-  flag set, as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}.
+  flag set, as described in \ref{sec:Basic Facilities of a Virtio
+Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
+Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device /
+Virtqueues / The Virtqueue Descriptor Table / Indirect
+Descriptors} and
+  \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
+\ref{sec:Packed Virtqueues / Driver and Device Event Suppression}.
 
   \item[VIRTIO_F_RING_EVENT_IDX(29)] This feature enables the \field{used_event}
-  and the \field{avail_event} fields as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression} and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}.
+  and the \field{avail_event} fields as described in
+\ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
+Virtqueue Interrupt Suppression}, \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+and .
 
   \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
     specification, giving a simple way to detect legacy devices or drivers.
@@ -5212,6 +5237,9 @@ Currently these device-independent feature bits defined:
   addresses in memory.  If this feature bit is set to 0, then the device emits
   physical addresses which are not translated further, even though an IOMMU
   may be present.
+  \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
+  support for the packed virtqueue layout as described in
+\ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5225,6 +5253,8 @@ passed to the device into physical addresses in memory.  If
 VIRTIO_F_IOMMU_PLATFORM is not offered, then a driver MUST pass only physical
 addresses to the device.
 
+A driver SHOULD accept VIRTIO_F_PACKED_RING if it is offered.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
diff --git a/packed-ring.tex b/packed-ring.tex
new file mode 100644
index 0000000..ffc699c
--- /dev/null
+++ b/packed-ring.tex
@@ -0,0 +1,668 @@
+\section{Packed Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}
+
+Packed virtqueues is an alternative compact virtqueue layout using
+read-write memory, that is memory that is both read and written
+by both host and guest.
+
+Use of packed virtqueues is enabled by the VIRTIO_F_PACKED_RING
+feature bit.
+
+Packed virtqueues support up to $2^{14}$ queues, with up to $2^{15}$
+entries each.
+
+With current transports, queues are located in guest memory
+allocated by driver.
+Each packed virtqueue consists of three parts:
+
+\begin{itemize}
+\item Descriptor Ring
+\item Device Event Suppression
+\item Driver Event Suppression
+\end{itemize}
+
+Where Descriptor Ring in turn consists of descriptors,
+and where each descriptor can contain the following parts:
+
+\begin{itemize}
+\item Buffer ID
+\item Buffer Address
+\item Buffer Length
+\item Flags
+\end{itemize}
+
+A buffer consists of zero or more device-readable physically-contiguous
+elements followed by zero or more physically-contiguous
+device-writable elements (each buffer has at least one element).
+
+When the driver wants to send such a buffer to the device, it
+writes at least one available descriptor describing elements of
+the buffer into the Descriptor Ring.  The descriptor(s) are
+associated with a buffer by means of a Buffer ID stored within
+the descriptor.
+
+Driver then notifies the device. When the device has finished
+processing the buffer, it writes a used device descriptor
+including the Buffer ID into the Descriptor Ring (overwriting a
+driver descriptor previously made available), and sends an
+interrupt.
+
+Descriptor Ring is used in a circular manner: driver writes
+descriptors into the ring in order. After reaching end of ring,
+the next descriptor is placed at head of the ring.  Once ring is
+full of driver descriptors, driver stops sending new requests and
+waits for device to start processing descriptors and to write out
+some used descriptors before making new driver descriptors
+available.
+
+Similarly, device reads descriptors from the ring in order and
+detects that a driver descriptor has been made available.  As
+processing of descriptors is completed used descriptors are
+written by the device back into the ring.
+
+Note: after reading driver descriptors and starting their
+processing in order, device might complete their processing out
+of order.  Used device descriptors are written in the order
+in which their processing is complete.
+
+Device Event Suppression data structure is read-only by the
+device. It includes information for reducing the number of
+device events - i.e. interrupts to driver.
+
+Driver Event Suppression data structure is write-only by the
+device. It includes information for reducing the number of
+driver events - i.e. notifications to device.
+
+\subsection{Available and Used Ring Wrap Counters}
+\label{sec:Packed Virtqueues / Available and Used Ring Wrap Counters}
+Each of the driver and the device are expected to maintain,
+internally, a single-bit ring wrap counter initialized to 1.
+
+The counter maintained by the driver is called the Available
+Ring Wrap Counter. Driver changes the value of this counter
+each time it makes available the
+last descriptor in the ring (after making the last descriptor
+available).
+
+The counter maintained by the device is called the Used Ring Wrap
+Counter.  Device changes the value of this counter
+each time it uses the last descriptor in
+the ring (after marking the last descriptor used).
+
+It is easy to see that the Available Ring Wrap Counter in the driver matches
+the Used Ring Wrap Counter in the device when both are processing the same
+descriptor, or when all available descriptors have been used.
+
+To mark a descriptor as available and used, both driver and
+device use the following two flags:
+\begin{lstlisting}
+#define VIRTQ_DESC_F_AVAIL     7
+#define VIRTQ_DESC_F_USED      15
+\end{lstlisting}
+
+To mark a descriptor as available, driver sets the
+VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Available
+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_USED bit to match the
+\emph{inverse} value.
+
+To mark a descriptor as used, device sets the
+VIRTQ_DESC_F_USED bit in Flags to match the internal Used
+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_AVAIL bit to match the
+\emph{same} value.
+
+Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different
+for an available descriptor and equal for a used descriptor.
+
+\subsection{Polling of available and used descriptors}
+\label{sec:Packed Virtqueues / Polling of available and used descriptors}
+
+Writes of device and driver descriptors can generally be
+reordered, but each side (driver and device) are only required to
+poll (or test) a single location in memory: next device descriptor after
+the one they processed previously, in circular order.
+
+Sometimes device needs to only write out a single used descriptor
+after processing a batch of multiple available descriptors.  As
+described in more detail below, this can happen when using
+descriptor chaining or with in-order
+use of descriptors.  In this case, device writes out a used
+descriptor with buffer id of the last descriptor in the group.
+After processing the used descriptor, both device and driver then
+skip forward in the ring the number of the remaining descriptors
+in the group until processing (reading for the driver and writing
+for the device) the next used descriptor.
+
+\subsection{Write Flag}
+\label{sec:Packed Virtqueues / Write Flag}
+
+In an available descriptor, VIRTQ_DESC_F_WRITE bit within Flags
+is used to mark a descriptor as corresponding to a write-only or
+read-only element of a buffer.
+
+\begin{lstlisting}
+/* This marks a buffer as device write-only (otherwise device read-only). */
+#define VIRTQ_DESC_F_WRITE     2
+\end{lstlisting}
+
+In a used descriptor, this bit it used to specify whether any
+data has been written by the device into any parts of the buffer.
+
+
+\subsection{Buffer Address and Length}
+\label{sec:Packed Virtqueues / Buffer Address and Length}
+
+In an available descriptor, Buffer Address corresponds to the
+physical address of the buffer. The length of the buffer assumed
+to be physically contigious is stored in Buffer Length.
+
+In a used descriptor, Buffer Address is unused. Buffer Length
+specifies the length of the buffer that has been initialized
+(written to) by the device.
+
+Buffer length is reserved for used descriptors without the
+VIRTQ_DESC_F_WRITE flag, and is ignored by drivers.
+
+\subsection{Scatter-Gather Support}
+\label{sec:Packed Virtqueues / Scatter-Gather Support}
+
+Some drivers need an ability to supply a list of multiple buffer
+elements (also known as a scatter/gather list) with a request.
+Two optional features support this: descriptor
+chaining and indirect descriptors.
+
+If neither feature has been negotiated, each buffer is
+physically-contigious, either read-only or write-only and is
+described completely by a single descriptor.
+
+While unusual (most implementations either create all lists
+solely using non-indirect descriptors, or always use a single
+indirect element), if both features have been negotiated, mixing
+direct and direct descriptors in a ring is valid, as long as each
+list only contains descriptors of a given type.
+
+Scatter/gather lists only apply to available descriptors. A
+single used descriptor corresponds to the whole list.
+
+The device limits the number of descriptors in a list through a
+transport-specific and/or device-specific value. If not limited,
+the maximum number of descriptors in a list is the virt queue
+size.
+
+\subsection{Next Flag: Descriptor Chaining}
+\label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
+
+The VIRTIO_F_LIST_DESC feature allows driver to supply
+a scatter/gather list to the device
+by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT in
+Flags for all but the last available descriptor.
+
+\begin{lstlisting}
+/* This marks a buffer as continuing. */
+#define VIRTQ_DESC_F_NEXT   1
+\end{lstlisting}
+
+Buffer ID is included in the last descriptor in the list.  
+
+The driver always makes the the first descriptor in the list
+available after the rest of the list has been written out into
+the ring. This guarantees that the device will never observe a
+partial scatter/gather list in the ring.
+
+Device only writes out a single used descriptor for the whole
+list. It then skips forward according to the number of
+descriptors in the list. Driver needs to keep track of the size
+of the list corresponding to each buffer ID, to be able to skip
+to where the next used descriptor is written by the device.
+
+For example, if descriptors are used in the same order in which
+they are made available, this will result in the used descriptor
+overwriting the first available descriptor in the list, the used
+descriptor for the next list overwriting the first available
+descriptor in the next list, etc.
+
+VIRTQ_DESC_F_NEXT is reserved in used descriptors, and
+should be ignored by drivers.
+
+\subsection{Indirect Flag: Scatter-Gather Support}
+\label{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
+
+Some devices benefit by concurrently dispatching a large number
+of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this. To increase
+ring capacity the driver can store a (read-only by the device) table of indirect
+descriptors anywhere in memory, and insert a descriptor in main
+virtqueue (with \field{Flags}\&VIRTQ_DESC_F_INDIRECT on) that refers to
+a memory buffer
+containing this indirect descriptor table; \field{addr} and \field{len}
+refer to the indirect table address and length in bytes,
+respectively.
+\begin{lstlisting}
+/* This means the buffer contains a table of buffer descriptors. */
+#define VIRTQ_DESC_F_INDIRECT   4
+\end{lstlisting}
+
+The indirect table layout structure looks like this
+(\field{len} is the Buffer Length of the descriptor that refers to this table,
+which is a variable, so this code won't compile):
+
+\begin{lstlisting}
+struct indirect_descriptor_table {
+        /* The actual descriptor structures (struct Desc each) */
+        struct Desc desc[len / sizeof(struct Desc)];
+};
+\end{lstlisting}
+
+The first descriptor is located at start of the indirect
+descriptor table, additional indirect descriptors come
+immediately afterwards. \field{Flags} \&VIRTQ_DESC_F_WRITE is the
+only valid flag for descriptors in the indirect table. Others
+are reserved and are ignored by the device. 
+Buffer ID is also reserved and is ignored by the device.
+
+In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
+is reserved and is ignored by the device.
+
+\subsection{Multi-buffer requests}
+\label{sec:Packed Virtqueues / Multi-descriptor batches}
+Some devices combine multiple buffers as part of processing of a
+single request.  These devices always make the first
+descriptor in the request available after the rest of the request
+has been written out request the ring. This guarantees that the
+driver will never observe a partial request in the ring.
+
+
+\subsection{Driver and Device Event Suppression}
+\label{sec:Packed Virtqueues / Driver and Device Event Suppression}
+In many systems driver and device notifications involve
+significant overhead. To mitigate this overhead,
+each virtqueue includes two identical structures used for
+controlling notifications between device and driver.
+
+Driver Event Suppression structure is read-only by the
+device and controls the events sent by the device
+(e.g. interrupts).
+
+Device Event Suppression structure is read-only by
+the driver and controls the events sent by the driver
+(e.g. IO).
+
+
+Each of these Event Suppression structures controls
+both Descriptor Ring events and structure events, and
+each includes the following fields:
+
+\begin{description}
+\item [Descriptor Ring Change Event Flags] Takes values:
+\begin{itemize}
+\item 00b reserved
+\item 01b enable events
+\item 11b disable events
+\item 10b enable events for a specific descriptor
+(as specified by Descriptor Ring Change Event Offset/Wrap Counter).
+Only valid if VIRTIO_F_RING_EVENT_IDX has been negotiated.
+\end{itemize}
+\item [Descriptor Ring Change Event Offset] If Event Flags set to descriptor
+specific event: offset within the ring (in units of descriptor
+size). Event will only trigger when this descriptor is
+made available/used respectively.
+\item [Descriptor Ring Change Event Wrap Counter] If Event Flags set to descriptor
+specific event: offset within the ring (in units of descriptor
+size). Event will only trigger when Ring Wrap Counter
+matches this value and a descriptor is
+made available/used respectively.
+\end{description}
+
+After writing out some descriptors, both device and driver
+are expected to consult the relevant structure to find out
+whether interrupt should be sent.
+
+\subsubsection{Driver notifications}
+\label{sec:Packed Virtqueues / Driver notifications}
+Whenever not suppressed by Device Event Suppression,
+driver is required to notify the device after
+making changes to the virtqueue.
+
+Some devices benefit from ability to find out the number of
+available descriptors in the ring, and whether to send
+interrupts to drivers without accessing virtqueue in memory:
+for efficiency or as a debugging aid.
+
+To help with these optimizations, driver notifications
+to the device include the following information:
+
+\begin{itemize}
+\item VQ number
+\item Flags - set to 00b
+\item Offset (in units of descriptor size) within the ring
+      where the next available descriptor will be written
+\item Available Ring Wrap Counter referring to the last available
+      descriptor
+\end{itemize}
+
+\subsubsection{Structure Size and Alignment}
+\label{sec:Packed Virtqueues / Structure Size and Alignment}
+
+Each part of the virtqueue is physically-contiguous in guest memory,
+and has different alignment requirements.
+
+The memory aligment and size requirements, in bytes, of each part of the
+virtqueue are summarized in the following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Virtqueue Part    & Alignment & Size \\
+\hline \hline
+Descriptor Ring  & 16        & $16 * $(Queue Size) \\
+\hline
+Device Event Suppression    & 4         & 4 \\
+ \hline
+Driver Event Suppression         & 4         & 4 \\
+ \hline
+\end{tabular}
+
+The Alignment column gives the minimum alignment for each part
+of the virtqueue.
+
+The Size column gives the total number of bytes for each
+part of the virtqueue.
+
+Queue Size corresponds to the maximum number of descriptors in the
+virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
+can be queued at any given time.}.  Queue Size value does not
+have to be a power of 2 unless enforced by the transport.
+
+\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a
+Virtio Device / Packed Virtqueues}
+The driver MUST ensure that the physical address of the first byte
+of each virtqueue part is a multiple of the specified alignment value
+in the above table.
+
+\devicenormative{\subsection}{Virtqueues}{Basic Facilities of a
+Virtio Device / Packed Virtqueues}
+The device MUST start processing driver descriptors in the order
+in which they appear in the ring.
+The device MUST start writing device descriptors into the ring in
+the order in which they complete.
+Device MAY reorder descriptor writes once they are started.
+
+\subsection{The Virtqueue Descriptor Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue
+Descriptor Format}
+
+The available descriptor refers to the buffers the driver is sending
+to the device. \field{addr} is a physical address, and the
+descriptor is identified with a buffer using the \field{id} field.
+
+\begin{lstlisting}
+struct virtq_desc {
+        /* Buffer Address. */
+        le64 addr;
+        /* Buffer Length. */
+        le32 len;
+        /* Buffer ID. */
+        le16 id;
+        /* The flags depending on descriptor type. */
+        le16 flags;
+};
+\end{lstlisting}
+
+The descriptor ring is zero-initialized.
+
+\subsection{Event Suppression Structure Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event Suppression Structure
+Format}
+
+The following structure is used to reduce the number of
+notifications sent between driver and device.
+
+\begin{lstlisting}
+__le16 desc_event_off : 15; /* Descriptor Event Offset */
+int    desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
+__le16 desc_event_flags : 2; /* Descriptor Event Flags */
+\end{lstlisting}
+
+\subsection{Driver Notification Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
+
+The following structure is used to notify device of available
+descriptors:
+
+\begin{lstlisting}
+__le16 vqn : 14;
+__le16 desc_event_flags : 2;
+__le16 desc_event_off : 15;
+int    desc_event_wrap : 1;
+\end{lstlisting}
+
+\devicenormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table}
+A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
+read a device-writable buffer.
+A device MUST NOT use a descriptor unless it observes
+VIRTQ_DESC_F_AVAIL bit in its \field{flags} being changed.
+A device MUST NOT change a descriptor after changing it's
+VIRTQ_DESC_F_USED bit in its \field{flags}.
+
+\drivernormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / PAcked Virtqueues / The Virtqueue Descriptor Table}
+A driver MUST NOT change a descriptor unless it observes
+VIRTQ_DESC_F_USED bit in its \field{flags} being changed.
+A driver MUST NOT change a descriptor after changing
+VIRTQ_DESC_F_USED bit in its \field{flags}.
+
+\drivernormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
+Virtio Device / Packed Virtqueues / Scatter-Gather Support}
+A driver MUST NOT create a descriptor list longer than allowed
+by the device.
+
+A driver MUST NOT create a descriptor list longer than the Queue
+Size.
+
+This implies that loops in the descriptor list are forbidden!
+
+The driver MUST place any device-writable descriptor elements after
+any device-readable descriptor elements.
+
+A driver MUST NOT depend on the device to use more descriptors
+to be able to write out all descriptors in a list. A driver
+MUST make sure there's enough space in the ring
+for the whole list before making the first descriptor in the list
+available to the device.
+
+A driver MUST NOT make the first descriptor in the list
+available before initializing the rest of the descriptors.
+
+\devicenormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
+Virtio Device / Packed Virtqueues / Scatter-Gather Support}
+The device MUST use descriptors in a list chained by the
+VIRTQ_DESC_F_NEXT flag in the same order that they
+were made available by the driver.
+
+The device MAY limit the number of buffers it will allow in a
+list.
+
+\drivernormative{\subsection}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The driver MUST NOT set the DESC_F_INDIRECT flag unless the
+VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
+set any flags except DESC_F_WRITE within an indirect descriptor.
+
+A driver MUST NOT create a descriptor chain longer than allowed
+by the device.
+
+A driver MUST NOT write direct descriptors with
+DESC_F_INDIRECT set in a scatter-gather list linked by
+VIRTQ_DESC_F_NEXT.
+\field{flags}.
+
+\subsection{Virtqueue Operation}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Virtqueue Operation}
+
+There are two parts to virtqueue operation: supplying new
+available buffers to the device, and processing used buffers from
+the device.
+
+What follows is the requirements of each of these two parts
+when using the packed virtqueue format in more detail.
+
+\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device}
+
+The driver offers buffers to one of the device's virtqueues as follows:
+
+\begin{enumerate}
+\item The driver places the buffer into free descriptor in the Descriptor Ring.
+
+\item The driver performs a suitable memory barrier to ensure that it updates
+  the descriptor(s) before checking for notification suppression.
+
+\item If notifications are not suppressed, the driver notifies the device
+    of the new available buffers.
+\end{enumerate}
+
+What follows is the requirements of each stage in more detail.
+
+\subsubsection{Placing Available Buffers Into The Descriptor Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Available Buffers Into The Descriptor Ring}
+
+For each buffer element, b:
+
+\begin{enumerate}
+\item Get the next descriptor table entry, d
+\item Get the next free buffer id value
+\item Set \field{d.addr} to the physical address of the start of b
+\item Set \field{d.len} to the length of b.
+\item Set \field{d.id} to the buffer id
+\item Calculate the flags as follows:
+\begin{enumerate}
+\item If b is device-writable, set the VIRTQ_DESC_F_WRITE bit to 1, otherwise 0
+\item Set VIRTQ_DESC_F_AVAIL bit to the current value of the Available Ring Wrap Counter
+\item Set VIRTQ_DESC_F_AVAIL bit to inverse value
+\end{enumerate}
+\item Perform a memory barrier to ensure that the descriptor has
+      been initialized
+\item Set \field{d.flags} to the calculated flags value
+\item If d is the last descriptor in the ring, toggle the
+      Available Ring Wrap Counter
+\item Otherwise, increment d to point at the next descriptor
+\end{enumerate}
+
+This makes a single descriptor buffer available. However, in
+general the driver MAY make use of a batch of descriptors as part
+of a single request. In that case, it defers updating
+the descriptor flags for the first descriptor
+(and the previous memory barrier) until after the rest of
+the descriptors have been initialized.
+
+Once the descriptor \field{flags} is updated by the driver, this exposes the
+descriptor and its contents.  The device MAY
+access the descriptor and any following descriptors the driver created and the
+memory they refer to immediately.
+
+\drivernormative{\paragraph}{Updating flags}{Basic Facilities of
+a Virtio Device / Packed Virtqueues / Supplying Buffers to The
+Device / Updating flags}
+The driver MUST perform a suitable memory barrier before the
+\field{flags} update, to ensure the
+device sees the most up-to-date copy.
+
+\subsubsection{Notifying The Device}\label{sec:Basic Facilities
+of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+
+The actual method of device notification is bus-specific, but generally
+it can be expensive.  So the device MAY suppress such notifications if it
+doesn't need them, using the Driver Event Suppression structure
+as detailed in section \ref{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event
+Suppression Structure Format}.
+
+The driver has to be careful to expose the new \field{flags}
+value before checking if notifications are suppressed.
+
+\subsubsection{Implementation Example}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Implementation Example}
+
+Below is an example driver code. It does not attempt to reduce
+the number of device interrupts, neither does it support
+the VIRTIO_F_RING_EVENT_IDX feature.
+
+\begin{lstlisting}
+
+first = vq->next_avail;
+id = alloc_id(vq);
+
+for (each buffer element b) {
+        vq->desc[vq->next_avail].address = get_addr(b);
+        vq->desc[vq->next_avail].len = get_len(b);
+        init_desc(vq->next_avail, b);
+        avail = vq->avail_wrap_count;
+        used = !vq->avail_wrap_count;
+        f = get_flags(b) | (avail << VIRTQ_DESC_F_AVAIL) | (used << VIRTQ_DESC_F_USED);
+        if (vq->next_avail == first) {
+                flags = f;
+        } else {
+                vq->desc[vq->next_avail].flags = f;
+        }
+
+}
+vq->desc[vq->next_avail].id = id;
+write_memory_barrier();
+vq->desc[first].flags = flags;
+
+memory_barrier();
+
+if (vq->driver_event.flags != 0x3) {
+        notify_device(vq, vq->next_avail, vq->avail_wrap_count);
+}
+
+vq->next_avail++;
+
+if (vq->next_avail > vq->size) {
+        vq->next_avail = 0;
+        vq->avail_wrap_count \^= 1;
+}
+
+\end{lstlisting}
+
+
+\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+The driver MUST perform a suitable memory barrier before reading
+the Driver Event Suppression structure, to avoid missing a notification.
+
+\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Receiving Used Buffers From The Device}
+
+Once the device has used buffers referred to by a descriptor (read from or written to them, or
+parts of both, depending on the nature of the virtqueue and the
+device), it interrupts the driver
+as detailed in section \ref{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event
+Suppression Structure Format}.
+
+\begin{note}
+For optimal performance, a driver MAY disable interrupts while processing
+the used buffers, but beware the problem of missing interrupts between
+emptying the ring and reenabling interrupts.  This is usually handled by
+re-checking for more used buffers after interrups are re-enabled:
+\end{note}
+
+\begin{lstlisting}
+vq->device_event.flags = 0x3;
+
+for (;;) {
+        flags = vq->desc[vq->next_used].flags;
+        bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
+        bool used = flags & (1 << VIRTQ_DESC_F_USED);
+
+        if (avail != used) {
+                vq->device_event.flags = 0x1;
+                mb();
+
+                flags = vq->desc[vq->next_used].flags;
+                bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
+                bool used = flags & (1 << VIRTQ_DESC_F_USED);
+                if (avail != used) {
+                        break;
+                }
+
+                vq->device_event.flags = 0x3;
+        }
+
+        struct virtq_desc *d = vq->desc[vq->next_used];
+        process_buffer(d);
+        vq->next_used++;
+        if (vq->next_used > vq->size) {
+                vq->next_used = 0;
+        }
+}
+\end{lstlisting}
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Cornelia Huck]

On Wed, 10 Jan 2018 11:47:58 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Performance analysis of this is in my kvm forum 2016 presentation.  The
> idea is to have a r/w descriptor in a ring structure, replacing the used
> and available ring, index and descriptor buffer.
> 
> This is also easier for devices to implement than the 1.0 layout.
> Several more enhancements will be necessary to actually make this
> efficient for devices to use.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex     |  36 ++-
>  packed-ring.tex | 668 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 701 insertions(+), 3 deletions(-)
>  create mode 100644 packed-ring.tex

Some minor gripes by git am:

.git/rebase-apply/patch:288: trailing whitespace.
Buffer ID is included in the last descriptor in the list.  
.git/rebase-apply/patch:342: trailing whitespace.
are reserved and are ignored by the device. 

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Jens Freimann]

On Wed, Jan 10, 2018 at 11:47:58AM +0200, Michael S. Tsirkin wrote:
[...]
>+\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device}
>+
>+The driver offers buffers to one of the device's virtqueues as follows:
>+
>+\begin{enumerate}
>+\item The driver places the buffer into free descriptor in the Descriptor Ring.
>+
>+\item The driver performs a suitable memory barrier to ensure that it updates
>+  the descriptor(s) before checking for notification suppression.
>+
>+\item If notifications are not suppressed, the driver notifies the device
>+    of the new available buffers.
>+\end{enumerate}
>+
>+What follows is the requirements of each stage in more detail.
>+
>+\subsubsection{Placing Available Buffers Into The Descriptor Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Available Buffers Into The Descriptor Ring}
>+
>+For each buffer element, b:
>+
>+\begin{enumerate}
>+\item Get the next descriptor table entry, d
>+\item Get the next free buffer id value
>+\item Set \field{d.addr} to the physical address of the start of b
>+\item Set \field{d.len} to the length of b.
>+\item Set \field{d.id} to the buffer id
>+\item Calculate the flags as follows:
>+\begin{enumerate}
>+\item If b is device-writable, set the VIRTQ_DESC_F_WRITE bit to 1, otherwise 0
>+\item Set VIRTQ_DESC_F_AVAIL bit to the current value of the Available Ring Wrap Counter
>+\item Set VIRTQ_DESC_F_AVAIL bit to inverse value

This should be "Set VIRTQ_DESC_F_USED bit to inverse value, no?

regards,
Jens 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [virtio-dev] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Wed, Jan 10, 2018 at 02:49:42PM +0100, Jens Freimann wrote:
> On Wed, Jan 10, 2018 at 11:47:58AM +0200, Michael S. Tsirkin wrote:
> [...]
> > +\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device}
> > +
> > +The driver offers buffers to one of the device's virtqueues as follows:
> > +
> > +\begin{enumerate}
> > +\item The driver places the buffer into free descriptor in the Descriptor Ring.
> > +
> > +\item The driver performs a suitable memory barrier to ensure that it updates
> > +  the descriptor(s) before checking for notification suppression.
> > +
> > +\item If notifications are not suppressed, the driver notifies the device
> > +    of the new available buffers.
> > +\end{enumerate}
> > +
> > +What follows is the requirements of each stage in more detail.
> > +
> > +\subsubsection{Placing Available Buffers Into The Descriptor Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Available Buffers Into The Descriptor Ring}
> > +
> > +For each buffer element, b:
> > +
> > +\begin{enumerate}
> > +\item Get the next descriptor table entry, d
> > +\item Get the next free buffer id value
> > +\item Set \field{d.addr} to the physical address of the start of b
> > +\item Set \field{d.len} to the length of b.
> > +\item Set \field{d.id} to the buffer id
> > +\item Calculate the flags as follows:
> > +\begin{enumerate}
> > +\item If b is device-writable, set the VIRTQ_DESC_F_WRITE bit to 1, otherwise 0
> > +\item Set VIRTQ_DESC_F_AVAIL bit to the current value of the Available Ring Wrap Counter
> > +\item Set VIRTQ_DESC_F_AVAIL bit to inverse value
> 
> This should be "Set VIRTQ_DESC_F_USED bit to inverse value, no?
> 
> regards,
> Jens

Right.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Tiwei Bie]

On Wed, Jan 10, 2018 at 11:47:58AM +0200, Michael S. Tsirkin wrote:
> Performance analysis of this is in my kvm forum 2016 presentation.  The
> idea is to have a r/w descriptor in a ring structure, replacing the used
> and available ring, index and descriptor buffer.
> 
> This is also easier for devices to implement than the 1.0 layout.
> Several more enhancements will be necessary to actually make this
> efficient for devices to use.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex     |  36 ++-
>  packed-ring.tex | 668 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 701 insertions(+), 3 deletions(-)
>  create mode 100644 packed-ring.tex
> 
> diff --git a/content.tex b/content.tex
> index 3b4579e..3059bd3 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -242,10 +242,26 @@ a used buffer to the queue - i.e. lets the driver
>  know by marking the buffer as used. Device can then trigger
>  a device event - i.e. send an interrupt to the driver.
>  
> -For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
> +Device is not generally required to use buffers in
> +the same order in which they have been made available
> +by the driver.
> +
> +Some devices always use descriptors in the same order in which
> +they have been made available. These devices can offer the
> +VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
> +might allow optimizations or simplify driver code.

Does this mean that for "Split Virtqueues" if VIRTIO_F_IN_ORDER
feature is negotiated, drivers won't be required to access the
id field of used_elem to figure out the desc idx when processing
the used ring?

> +
> +Two formats are supported: Split Virtqueues (see \ref{sec:Basic
> +Facilities of a Virtio Device / Split
> +Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
> +Split Virtqueues}) and Packed Virtqueues (see \ref{sec:Basic
> +Facilities of a Virtio Device / Packed
> +Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
> +Packed Virtqueues}).
>  
>  \input{split-ring.tex}
>  
> +\input{packed-ring.tex}
[...]
> +Below is an example driver code. It does not attempt to reduce
> +the number of device interrupts, neither does it support
> +the VIRTIO_F_RING_EVENT_IDX feature.
> +
> +\begin{lstlisting}
> +
> +first = vq->next_avail;
> +id = alloc_id(vq);
> +
> +for (each buffer element b) {
> +        vq->desc[vq->next_avail].address = get_addr(b);
> +        vq->desc[vq->next_avail].len = get_len(b);
> +        init_desc(vq->next_avail, b);
> +        avail = vq->avail_wrap_count;
> +        used = !vq->avail_wrap_count;
> +        f = get_flags(b) | (avail << VIRTQ_DESC_F_AVAIL) | (used << VIRTQ_DESC_F_USED);
> +        if (vq->next_avail == first) {
> +                flags = f;

This is to implement the batching? I.e. don't make the first
desc available to the device before other descs are ready?

> +        } else {
> +                vq->desc[vq->next_avail].flags = f;
> +        }

The vq->next_avail updating is missing in the loop?

> +
> +}
> +vq->desc[vq->next_avail].id = id;
> +write_memory_barrier();
> +vq->desc[first].flags = flags;
> +
> +memory_barrier();
> +
> +if (vq->driver_event.flags != 0x3) {
> +        notify_device(vq, vq->next_avail, vq->avail_wrap_count);
> +}
> +
> +vq->next_avail++;
> +
> +if (vq->next_avail > vq->size) {

Should be (vq->next_avail >= vq->size)?

> +        vq->next_avail = 0;
> +        vq->avail_wrap_count \^= 1;
> +}
> +
> +\end{lstlisting}
> +
[...]
> +\begin{lstlisting}
> +vq->device_event.flags = 0x3;
> +
> +for (;;) {
> +        flags = vq->desc[vq->next_used].flags;
> +        bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> +        bool used = flags & (1 << VIRTQ_DESC_F_USED);
> +
> +        if (avail != used) {
> +                vq->device_event.flags = 0x1;
> +                mb();
> +
> +                flags = vq->desc[vq->next_used].flags;
> +                bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> +                bool used = flags & (1 << VIRTQ_DESC_F_USED);
> +                if (avail != used) {
> +                        break;
> +                }
> +
> +                vq->device_event.flags = 0x3;
> +        }
> +
> +        struct virtq_desc *d = vq->desc[vq->next_used];
> +        process_buffer(d);
> +        vq->next_used++;
> +        if (vq->next_used > vq->size) {

Should be (vq->next_used >= vq->size)?

> +                vq->next_used = 0;
> +        }
> +}
> +\end{lstlisting}
> -- 
> MST

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [virtio-dev] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Wed, Jan 10, 2018 at 10:08:01PM +0800, Tiwei Bie wrote:
> On Wed, Jan 10, 2018 at 11:47:58AM +0200, Michael S. Tsirkin wrote:
> > Performance analysis of this is in my kvm forum 2016 presentation.  The
> > idea is to have a r/w descriptor in a ring structure, replacing the used
> > and available ring, index and descriptor buffer.
> > 
> > This is also easier for devices to implement than the 1.0 layout.
> > Several more enhancements will be necessary to actually make this
> > efficient for devices to use.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex     |  36 ++-
> >  packed-ring.tex | 668 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 701 insertions(+), 3 deletions(-)
> >  create mode 100644 packed-ring.tex
> > 
> > diff --git a/content.tex b/content.tex
> > index 3b4579e..3059bd3 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -242,10 +242,26 @@ a used buffer to the queue - i.e. lets the driver
> >  know by marking the buffer as used. Device can then trigger
> >  a device event - i.e. send an interrupt to the driver.
> >  
> > -For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
> > +Device is not generally required to use buffers in
> > +the same order in which they have been made available
> > +by the driver.
> > +
> > +Some devices always use descriptors in the same order in which
> > +they have been made available. These devices can offer the
> > +VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
> > +might allow optimizations or simplify driver code.
> 
> Does this mean that for "Split Virtqueues" if VIRTIO_F_IN_ORDER
> feature is negotiated, drivers won't be required to access the
> id field of used_elem to figure out the desc idx when processing
> the used ring?

The scope of this work is very big as is. For now the proposal limits
VIRTIO_F_IN_ORDER to packed ring format.  A patch on top can relax this
restriction if people have the time to work on it.

> > +
> > +Two formats are supported: Split Virtqueues (see \ref{sec:Basic
> > +Facilities of a Virtio Device / Split
> > +Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
> > +Split Virtqueues}) and Packed Virtqueues (see \ref{sec:Basic
> > +Facilities of a Virtio Device / Packed
> > +Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
> > +Packed Virtqueues}).
> >  
> >  \input{split-ring.tex}
> >  
> > +\input{packed-ring.tex}
> [...]
> > +Below is an example driver code. It does not attempt to reduce
> > +the number of device interrupts, neither does it support
> > +the VIRTIO_F_RING_EVENT_IDX feature.
> > +
> > +\begin{lstlisting}
> > +
> > +first = vq->next_avail;
> > +id = alloc_id(vq);
> > +
> > +for (each buffer element b) {
> > +        vq->desc[vq->next_avail].address = get_addr(b);
> > +        vq->desc[vq->next_avail].len = get_len(b);
> > +        init_desc(vq->next_avail, b);
> > +        avail = vq->avail_wrap_count;
> > +        used = !vq->avail_wrap_count;
> > +        f = get_flags(b) | (avail << VIRTQ_DESC_F_AVAIL) | (used << VIRTQ_DESC_F_USED);
> > +        if (vq->next_avail == first) {
> > +                flags = f;
> 
> This is to implement the batching? I.e. don't make the first
> desc available to the device before other descs are ready?


Exactly. Will add a comment.

> > +        } else {
> > +                vq->desc[vq->next_avail].flags = f;
> > +        }
> 
> The vq->next_avail updating is missing in the loop?

Right.

> > +
> > +}
> > +vq->desc[vq->next_avail].id = id;
> > +write_memory_barrier();
> > +vq->desc[first].flags = flags;
> > +
> > +memory_barrier();
> > +
> > +if (vq->driver_event.flags != 0x3) {
> > +        notify_device(vq, vq->next_avail, vq->avail_wrap_count);
> > +}
> > +
> > +vq->next_avail++;
> > +
> > +if (vq->next_avail > vq->size) {
> 
> Should be (vq->next_avail >= vq->size)?


Right.

> > +        vq->next_avail = 0;
> > +        vq->avail_wrap_count \^= 1;
> > +}
> > +
> > +\end{lstlisting}
> > +
> [...]
> > +\begin{lstlisting}
> > +vq->device_event.flags = 0x3;
> > +
> > +for (;;) {
> > +        flags = vq->desc[vq->next_used].flags;
> > +        bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> > +        bool used = flags & (1 << VIRTQ_DESC_F_USED);
> > +
> > +        if (avail != used) {
> > +                vq->device_event.flags = 0x1;
> > +                mb();
> > +
> > +                flags = vq->desc[vq->next_used].flags;
> > +                bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> > +                bool used = flags & (1 << VIRTQ_DESC_F_USED);
> > +                if (avail != used) {
> > +                        break;
> > +                }
> > +
> > +                vq->device_event.flags = 0x3;
> > +        }
> > +
> > +        struct virtq_desc *d = vq->desc[vq->next_used];
> > +        process_buffer(d);
> > +        vq->next_used++;
> > +        if (vq->next_used > vq->size) {
> 
> Should be (vq->next_used >= vq->size)?

Right.

> > +                vq->next_used = 0;
> > +        }
> > +}
> > +\end{lstlisting}
> > -- 
> > MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Cornelia Huck]

On Wed, 10 Jan 2018 11:47:58 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Performance analysis of this is in my kvm forum 2016 presentation.  The
> idea is to have a r/w descriptor in a ring structure, replacing the used
> and available ring, index and descriptor buffer.

Do we have a place to store pointers to such information?

> 
> This is also easier for devices to implement than the 1.0 layout.
> Several more enhancements will be necessary to actually make this
> efficient for devices to use.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex     |  36 ++-
>  packed-ring.tex | 668 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 701 insertions(+), 3 deletions(-)
>  create mode 100644 packed-ring.tex
> 
> diff --git a/content.tex b/content.tex
> index 3b4579e..3059bd3 100644
> --- a/content.tex
> +++ b/content.tex

(...)

> @@ -5199,10 +5215,19 @@ Currently these device-independent feature bits defined:
>  \begin{description}
>    \item[VIRTIO_F_RING_INDIRECT_DESC (28)] Negotiating this feature indicates
>    that the driver can use descriptors with the VIRTQ_DESC_F_INDIRECT
> -  flag set, as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}.
> +  flag set, as described in \ref{sec:Basic Facilities of a Virtio
> +Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
> +Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device /
> +Virtqueues / The Virtqueue Descriptor Table / Indirect
> +Descriptors} and
> +  \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
> +\ref{sec:Packed Virtqueues / Driver and Device Event Suppression}.

That one needs to go below (copy/paste error?).

>  
>    \item[VIRTIO_F_RING_EVENT_IDX(29)] This feature enables the \field{used_event}
> -  and the \field{avail_event} fields as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression} and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}.
> +  and the \field{avail_event} fields as described in
> +\ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> +Virtqueue Interrupt Suppression}, \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
> +and .

Here's where the ref above needs to go.

>  
>    \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
>      specification, giving a simple way to detect legacy devices or drivers.
> @@ -5212,6 +5237,9 @@ Currently these device-independent feature bits defined:
>    addresses in memory.  If this feature bit is set to 0, then the device emits
>    physical addresses which are not translated further, even though an IOMMU
>    may be present.
> +  \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
> +  support for the packed virtqueue layout as described in
> +\ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
>  \end{description}

I think the references should include a title as well (that makes the
resulting document easier to read).

>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 4/5] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Wed, Jan 10, 2018 at 03:15:38PM +0100, Cornelia Huck wrote:
> On Wed, 10 Jan 2018 11:47:58 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > Performance analysis of this is in my kvm forum 2016 presentation.  The
> > idea is to have a r/w descriptor in a ring structure, replacing the used
> > and available ring, index and descriptor buffer.
> 
> Do we have a place to store pointers to such information?
> 
> > 
> > This is also easier for devices to implement than the 1.0 layout.
> > Several more enhancements will be necessary to actually make this
> > efficient for devices to use.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex     |  36 ++-
> >  packed-ring.tex | 668 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 701 insertions(+), 3 deletions(-)
> >  create mode 100644 packed-ring.tex
> > 
> > diff --git a/content.tex b/content.tex
> > index 3b4579e..3059bd3 100644
> > --- a/content.tex
> > +++ b/content.tex
> 
> (...)
> 
> > @@ -5199,10 +5215,19 @@ Currently these device-independent feature bits defined:
> >  \begin{description}
> >    \item[VIRTIO_F_RING_INDIRECT_DESC (28)] Negotiating this feature indicates
> >    that the driver can use descriptors with the VIRTQ_DESC_F_INDIRECT
> > -  flag set, as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}.
> > +  flag set, as described in \ref{sec:Basic Facilities of a Virtio
> > +Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
> > +Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device /
> > +Virtqueues / The Virtqueue Descriptor Table / Indirect
> > +Descriptors} and
> > +  \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
> > +\ref{sec:Packed Virtqueues / Driver and Device Event Suppression}.
> 
> That one needs to go below (copy/paste error?).
> 
> >  
> >    \item[VIRTIO_F_RING_EVENT_IDX(29)] This feature enables the \field{used_event}
> > -  and the \field{avail_event} fields as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression} and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}.
> > +  and the \field{avail_event} fields as described in
> > +\ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> > +Virtqueue Interrupt Suppression}, \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
> > +and .
> 
> Here's where the ref above needs to go.
> 
> >  
> >    \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
> >      specification, giving a simple way to detect legacy devices or drivers.
> > @@ -5212,6 +5237,9 @@ Currently these device-independent feature bits defined:
> >    addresses in memory.  If this feature bit is set to 0, then the device emits
> >    physical addresses which are not translated further, even though an IOMMU
> >    may be present.
> > +  \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
> > +  support for the packed virtqueue layout as described in
> > +\ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
> >  \end{description}
> 
> I think the references should include a title as well (that makes the
> resulting document easier to read).

Want to add a macro including ref+nameref?
We'd use it for all sec: references then.

> >  
> >  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v6 5/5] packed-ring: add in order request support

[Michael S. Tsirkin]

support in-order requests for packed rings.  more work is needed to use
them efficiently for split rings (e.g. it makes sense to bypass
avail/used rings in this case), forbid that combination for now.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex     |  7 +++++++
 packed-ring.tex | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+)

diff --git a/content.tex b/content.tex
index 3059bd3..f478ae2 100644
--- a/content.tex
+++ b/content.tex
@@ -5240,6 +5240,10 @@ and .
   \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
   support for the packed virtqueue layout as described in
 \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
+  \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
+  that all buffers are used by the device in the same
+  order in which they have been made available.
+  This feature should only be enabled with VIRTIO_F_RING_PACKED.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5255,6 +5259,9 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_PACKED_RING if it is offered.
 
+A driver MUST only accept VIRTIO_F_IN_ORDER if it also accepts
+VIRTIO_F_RING_PACKED.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
diff --git a/packed-ring.tex b/packed-ring.tex
index ffc699c..88d121c 100644
--- a/packed-ring.tex
+++ b/packed-ring.tex
@@ -260,6 +260,30 @@ Buffer ID is also reserved and is ignored by the device.
 In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
 is reserved and is ignored by the device.
 
+\subsection{In-order use of descriptors}
+\label{sec:Packed Virtqueues / In-order use of descriptors}
+
+Some devices always use descriptors in the same order in which
+they have been made available. These devices can offer the
+VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch.
+
+Device then skips forward in the ring according to the size of
+the batch. Driver needs to look up the used Buffer ID and
+calculate the batch size to be able to advance to where the next
+used descriptor will be written by the device.
+
+This will result in the used descriptor overwriting the first
+available descriptor in the batch, the used descriptor for the
+next batch overwriting the first available descriptor in the next
+batch, etc.
+
+The skipped buffers (for which no used descriptor was written)
+are assumed to have been used (read or written) by the
+device completely.
+
 \subsection{Multi-buffer requests}
 \label{sec:Packed Virtqueues / Multi-descriptor batches}
 Some devices combine multiple buffers as part of processing of a
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 0/5] packed ring layout spec

[Cornelia Huck]

On Wed, 10 Jan 2018 11:47:53 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> OK, this is in a shape where we could include it in the spec.
> Changes from v5:
> - scope reductions (see below). We can add more
>   features down the road, hopefully reduced scope will be enough
>   to finalize spec soon.
> - cleanup and integrate in the spec
> - pseudo-code
> 
> Deferred features:
> - dropped _F_DESC_LIST, 1.0 includes this unconditionally, we
>   can do same
> - dropped event structure change notifications - needed for
>   efficient hardware implementations but let's add this on top
> 
> 3 1st patches just move text around so all virtio 1.0
> things are in the same place. 2 last ones add the new layout
> 
> Option to mark descriptors as not generating events isn't
> yet implemented. Again, let's add this on top.
> 
> I also note that for hardware implementations, a different
> set of memory barriers is needed. Again, let's add this on top
> 
> not linking into conformance sections, will add after spec
> itself is approved.
> 
> Michael S. Tsirkin (5):
>   content: move 1.0 queue format out to a separate section
>   content: move ring text out to a separate file
>   content: move virtqueue operation description
>   packed virtqueues: more efficient virtqueue layout
>   packed-ring: add in order request support
> 
>  conformance.tex |   4 +-
>  content.tex     | 718 +++++---------------------------------------------------
>  packed-ring.tex | 692 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  split-ring.tex  | 666 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 1423 insertions(+), 657 deletions(-)
>  create mode 100644 packed-ring.tex
>  create mode 100644 split-ring.tex
> 

One thing I just noticed: This does not seem to contain any changes for
transports. For ccw, we use different payloads in the SET_VQ command
for legacy (pre-virtio-1) and virtio-1 split layout. I think we would
need a new SET_VQ payload for the packed ring guarded by a new
revision, but I have not investigated it yet. I have not yet thought
through how this interacts with feature negotiation, either.

Otherwise, I don't see any reason why the packed ring would not work
fine for ccw as well.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 0/5] packed ring layout spec

[Michael S. Tsirkin]

On Wed, Jan 10, 2018 at 11:33:15AM +0100, Cornelia Huck wrote:
> On Wed, 10 Jan 2018 11:47:53 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > OK, this is in a shape where we could include it in the spec.
> > Changes from v5:
> > - scope reductions (see below). We can add more
> >   features down the road, hopefully reduced scope will be enough
> >   to finalize spec soon.
> > - cleanup and integrate in the spec
> > - pseudo-code
> > 
> > Deferred features:
> > - dropped _F_DESC_LIST, 1.0 includes this unconditionally, we
> >   can do same
> > - dropped event structure change notifications - needed for
> >   efficient hardware implementations but let's add this on top
> > 
> > 3 1st patches just move text around so all virtio 1.0
> > things are in the same place. 2 last ones add the new layout
> > 
> > Option to mark descriptors as not generating events isn't
> > yet implemented. Again, let's add this on top.
> > 
> > I also note that for hardware implementations, a different
> > set of memory barriers is needed. Again, let's add this on top
> > 
> > not linking into conformance sections, will add after spec
> > itself is approved.
> > 
> > Michael S. Tsirkin (5):
> >   content: move 1.0 queue format out to a separate section
> >   content: move ring text out to a separate file
> >   content: move virtqueue operation description
> >   packed virtqueues: more efficient virtqueue layout
> >   packed-ring: add in order request support
> > 
> >  conformance.tex |   4 +-
> >  content.tex     | 718 +++++---------------------------------------------------
> >  packed-ring.tex | 692 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  split-ring.tex  | 666 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  4 files changed, 1423 insertions(+), 657 deletions(-)
> >  create mode 100644 packed-ring.tex
> >  create mode 100644 split-ring.tex
> > 
> 
> One thing I just noticed: This does not seem to contain any changes for
> transports. For ccw, we use different payloads in the SET_VQ command
> for legacy (pre-virtio-1) and virtio-1 split layout. I think we would
> need a new SET_VQ payload for the packed ring guarded by a new
> revision, but I have not investigated it yet. I have not yet thought
> through how this interacts with feature negotiation, either.

Good point.
There's exactly same number of chunks (descriptor, device and driver
event suppression).

So the idea is to substitute descriptor for descriptor, device event
suppression for available ring and driver event suppression for
used ring.

That's why we don't need to change transports.

I will add text to clarify that.

> Otherwise, I don't see any reason why the packed ring would not work
> fine for ccw as well.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 0/5] packed ring layout spec

[Cornelia Huck]

On Wed, 10 Jan 2018 13:10:49 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Jan 10, 2018 at 11:33:15AM +0100, Cornelia Huck wrote:
> > On Wed, 10 Jan 2018 11:47:53 +0200
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > OK, this is in a shape where we could include it in the spec.
> > > Changes from v5:
> > > - scope reductions (see below). We can add more
> > >   features down the road, hopefully reduced scope will be enough
> > >   to finalize spec soon.
> > > - cleanup and integrate in the spec
> > > - pseudo-code
> > > 
> > > Deferred features:
> > > - dropped _F_DESC_LIST, 1.0 includes this unconditionally, we
> > >   can do same
> > > - dropped event structure change notifications - needed for
> > >   efficient hardware implementations but let's add this on top
> > > 
> > > 3 1st patches just move text around so all virtio 1.0
> > > things are in the same place. 2 last ones add the new layout
> > > 
> > > Option to mark descriptors as not generating events isn't
> > > yet implemented. Again, let's add this on top.
> > > 
> > > I also note that for hardware implementations, a different
> > > set of memory barriers is needed. Again, let's add this on top
> > > 
> > > not linking into conformance sections, will add after spec
> > > itself is approved.
> > > 
> > > Michael S. Tsirkin (5):
> > >   content: move 1.0 queue format out to a separate section
> > >   content: move ring text out to a separate file
> > >   content: move virtqueue operation description
> > >   packed virtqueues: more efficient virtqueue layout
> > >   packed-ring: add in order request support
> > > 
> > >  conformance.tex |   4 +-
> > >  content.tex     | 718 +++++---------------------------------------------------
> > >  packed-ring.tex | 692 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >  split-ring.tex  | 666 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >  4 files changed, 1423 insertions(+), 657 deletions(-)
> > >  create mode 100644 packed-ring.tex
> > >  create mode 100644 split-ring.tex
> > >   
> > 
> > One thing I just noticed: This does not seem to contain any changes for
> > transports. For ccw, we use different payloads in the SET_VQ command
> > for legacy (pre-virtio-1) and virtio-1 split layout. I think we would
> > need a new SET_VQ payload for the packed ring guarded by a new
> > revision, but I have not investigated it yet. I have not yet thought
> > through how this interacts with feature negotiation, either.  
> 
> Good point.
> There's exactly same number of chunks (descriptor, device and driver
> event suppression).
> 
> So the idea is to substitute descriptor for descriptor, device event
> suppression for available ring and driver event suppression for
> used ring.
> 
> That's why we don't need to change transports.
> 
> I will add text to clarify that.

I think we also need to update the text for the transports, as they
explicitly refer to avail/used.

> 
> > Otherwise, I don't see any reason why the packed ring would not work
> > fine for ccw as well.  


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v6 0/5] packed ring layout spec

[Michael S. Tsirkin]

On Wed, Jan 10, 2018 at 12:14:03PM +0100, Cornelia Huck wrote:
> On Wed, 10 Jan 2018 13:10:49 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Wed, Jan 10, 2018 at 11:33:15AM +0100, Cornelia Huck wrote:
> > > On Wed, 10 Jan 2018 11:47:53 +0200
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >   
> > > > OK, this is in a shape where we could include it in the spec.
> > > > Changes from v5:
> > > > - scope reductions (see below). We can add more
> > > >   features down the road, hopefully reduced scope will be enough
> > > >   to finalize spec soon.
> > > > - cleanup and integrate in the spec
> > > > - pseudo-code
> > > > 
> > > > Deferred features:
> > > > - dropped _F_DESC_LIST, 1.0 includes this unconditionally, we
> > > >   can do same
> > > > - dropped event structure change notifications - needed for
> > > >   efficient hardware implementations but let's add this on top
> > > > 
> > > > 3 1st patches just move text around so all virtio 1.0
> > > > things are in the same place. 2 last ones add the new layout
> > > > 
> > > > Option to mark descriptors as not generating events isn't
> > > > yet implemented. Again, let's add this on top.
> > > > 
> > > > I also note that for hardware implementations, a different
> > > > set of memory barriers is needed. Again, let's add this on top
> > > > 
> > > > not linking into conformance sections, will add after spec
> > > > itself is approved.
> > > > 
> > > > Michael S. Tsirkin (5):
> > > >   content: move 1.0 queue format out to a separate section
> > > >   content: move ring text out to a separate file
> > > >   content: move virtqueue operation description
> > > >   packed virtqueues: more efficient virtqueue layout
> > > >   packed-ring: add in order request support
> > > > 
> > > >  conformance.tex |   4 +-
> > > >  content.tex     | 718 +++++---------------------------------------------------
> > > >  packed-ring.tex | 692 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >  split-ring.tex  | 666 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >  4 files changed, 1423 insertions(+), 657 deletions(-)
> > > >  create mode 100644 packed-ring.tex
> > > >  create mode 100644 split-ring.tex
> > > >   
> > > 
> > > One thing I just noticed: This does not seem to contain any changes for
> > > transports. For ccw, we use different payloads in the SET_VQ command
> > > for legacy (pre-virtio-1) and virtio-1 split layout. I think we would
> > > need a new SET_VQ payload for the packed ring guarded by a new
> > > revision, but I have not investigated it yet. I have not yet thought
> > > through how this interacts with feature negotiation, either.  
> > 
> > Good point.
> > There's exactly same number of chunks (descriptor, device and driver
> > event suppression).
> > 
> > So the idea is to substitute descriptor for descriptor, device event
> > suppression for available ring and driver event suppression for
> > used ring.
> > 
> > That's why we don't need to change transports.
> > 
> > I will add text to clarify that.
> 
> I think we also need to update the text for the transports, as they
> explicitly refer to avail/used.

Exactly.

> > 
> > > Otherwise, I don't see any reason why the packed ring would not work
> > > fine for ccw as well.  

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 02/11] content: move ring text out to a separate file

[Michael S. Tsirkin]

Will be easier to manage this way.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex    | 499 +--------------------------------------------------------
 split-ring.tex | 498 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 499 insertions(+), 498 deletions(-)
 create mode 100644 split-ring.tex

diff --git a/content.tex b/content.tex
index 4483a4b..5b4c4e9 100644
--- a/content.tex
+++ b/content.tex
@@ -244,504 +244,7 @@ a device event - i.e. send an interrupt to the driver.
 
 For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
 
-\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
-The split virtqueue format is the original format used by legacy
-virtio devices.  The split virtqueue format separates the
-virtqueue into several parts, where each part is write-able by
-either the driver or the device, but not both. Multiple
-locations need to be updated when making a buffer available
-and when marking it as used.
-
-
-Each queue has a 16-bit queue size
-parameter, which sets the number of entries and implies the total size
-of the queue.
-
-Each virtqueue consists of three parts:
-
-\begin{itemize}
-\item Descriptor Table
-\item Available Ring
-\item Used Ring
-\end{itemize}
-
-where each part is physically-contiguous in guest memory,
-and has different alignment requirements.
-
-The memory aligment and size requirements, in bytes, of each part of the
-virtqueue are summarized in the following table:
-
-\begin{tabular}{|l|l|l|}
-\hline
-Virtqueue Part    & Alignment & Size \\
-\hline \hline
-Descriptor Table  & 16        & $16 * $(Queue Size) \\
-\hline
-Available Ring    & 2         & $6 + 2 * $(Queue Size) \\
- \hline
-Used Ring         & 4         & $6 + 8 * $(Queue Size) \\
- \hline
-\end{tabular}
-
-The Alignment column gives the minimum alignment for each part
-of the virtqueue.
-
-The Size column gives the total number of bytes for each
-part of the virtqueue.
-
-Queue Size corresponds to the maximum number of buffers in the
-virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
-can be queued at any given time.}.  Queue Size value is always a
-power of 2.  The maximum Queue Size value is 32768.  This value
-is specified in a bus-specific way.
-
-When the driver wants to send a buffer to the device, it fills in
-a slot in the descriptor table (or chains several together), and
-writes the descriptor index into the available ring.  It then
-notifies the device. When the device has finished a buffer, it
-writes the descriptor index into the used ring, and sends an interrupt.
-
-\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a Virtio Device / Virtqueues}
-The driver MUST ensure that the physical address of the first byte
-of each virtqueue part is a multiple of the specified alignment value
-in the above table.
-
-\subsection{Legacy Interfaces: A Note on Virtqueue Layout}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
-
-For Legacy Interfaces, several additional
-restrictions are placed on the virtqueue layout:
-
-Each virtqueue occupies two or more physically-contiguous pages
-(usually defined as 4096 bytes, but depending on the transport;
-henceforth referred to as Queue Align)
-and consists of three parts:
-
-\begin{tabular}{|l|l|l|}
-\hline
-Descriptor Table & Available Ring (\ldots padding\ldots) & Used Ring \\
-\hline
-\end{tabular}
-
-The bus-specific Queue Size field controls the total number of bytes
-for the virtqueue.
-When using the legacy interface, the transitional
-driver MUST retrieve the Queue Size field from the device
-and MUST allocate the total number of bytes for the virtqueue
-according to the following formula (Queue Align given in qalign and
-Queue Size given in qsz):
-
-\begin{lstlisting}
-#define ALIGN(x) (((x) + qalign) & ~qalign)
-static inline unsigned virtq_size(unsigned int qsz)
-{
-     return ALIGN(sizeof(struct virtq_desc)*qsz + sizeof(u16)*(3 + qsz))
-          + ALIGN(sizeof(u16)*3 + sizeof(struct virtq_used_elem)*qsz);
-}
-\end{lstlisting}
-
-This wastes some space with padding.
-When using the legacy interface, both transitional
-devices and drivers MUST use the following virtqueue layout
-structure to locate elements of the virtqueue:
-
-\begin{lstlisting}
-struct virtq {
-        // The actual descriptors (16 bytes each)
-        struct virtq_desc desc[ Queue Size ];
-
-        // A ring of available descriptor heads with free-running index.
-        struct virtq_avail avail;
-
-        // Padding to the next Queue Align boundary.
-        u8 pad[ Padding ];
-
-        // A ring of used descriptor heads with free-running index.
-        struct virtq_used used;
-};
-\end{lstlisting}
-
-\subsection{Legacy Interfaces: A Note on Virtqueue Endianness}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
-
-Note that when using the legacy interface, transitional
-devices and drivers MUST use the native
-endian of the guest as the endian of fields and in the virtqueue.
-This is opposed to little-endian for non-legacy interface as
-specified by this standard.
-It is assumed that the host is already aware of the guest endian.
-
-\subsection{Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing}
-The framing of messages with descriptors is
-independent of the contents of the buffers. For example, a network
-transmit buffer consists of a 12 byte header followed by the network
-packet. This could be most simply placed in the descriptor table as a
-12 byte output descriptor followed by a 1514 byte output descriptor,
-but it could also consist of a single 1526 byte output descriptor in
-the case where the header and packet are adjacent, or even three or
-more descriptors (possibly with loss of efficiency in that case).
-
-Note that, some device implementations have large-but-reasonable
-restrictions on total descriptor size (such as based on IOV_MAX in the
-host OS). This has not been a problem in practice: little sympathy
-will be given to drivers which create unreasonably-sized descriptors
-such as by dividing a network packet into 1500 single-byte
-descriptors!
-
-\devicenormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
-The device MUST NOT make assumptions about the particular arrangement
-of descriptors.  The device MAY have a reasonable limit of descriptors
-it will allow in a chain.
-
-\drivernormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
-The driver MUST place any device-writable descriptor elements after
-any device-readable descriptor elements.
-
-The driver SHOULD NOT use an excessive number of descriptors to
-describe a buffer.
-
-\subsubsection{Legacy Interface: Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
-
-Regrettably, initial driver implementations used simple layouts, and
-devices came to rely on it, despite this specification wording.  In
-addition, the specification for virtio_blk SCSI commands required
-intuiting field lengths from frame boundaries (see
- \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}~\nameref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation})
-
-Thus when using the legacy interface, the VIRTIO_F_ANY_LAYOUT
-feature indicates to both the device and the driver that no
-assumptions were made about framing.  Requirements for
-transitional drivers when this is not negotiated are included in
-each device section.
-
-\subsection{The Virtqueue Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-
-The descriptor table refers to the buffers the driver is using for
-the device. \field{addr} is a physical address, and the buffers
-can be chained via \field{next}. Each descriptor describes a
-buffer which is read-only for the device (``device-readable'') or write-only for the device (``device-writable''), but a chain of
-descriptors can contain both device-readable and device-writable buffers.
-
-The actual contents of the memory offered to the device depends on the
-device type.  Most common is to begin the data with a header
-(containing little-endian fields) for the device to read, and postfix
-it with a status tailer for the device to write.
-
-\begin{lstlisting}
-struct virtq_desc {
-        /* Address (guest-physical). */
-        le64 addr;
-        /* Length. */
-        le32 len;
-
-/* This marks a buffer as continuing via the next field. */
-#define VIRTQ_DESC_F_NEXT   1
-/* This marks a buffer as device write-only (otherwise device read-only). */
-#define VIRTQ_DESC_F_WRITE     2
-/* This means the buffer contains a list of buffer descriptors. */
-#define VIRTQ_DESC_F_INDIRECT   4
-        /* The flags as indicated above. */
-        le16 flags;
-        /* Next field if flags & NEXT */
-        le16 next;
-};
-\end{lstlisting}
-
-The number of descriptors in the table is defined by the queue size
-for this virtqueue: this is the maximum possible descriptor chain length.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to this structure as vring_desc, and the constants as
-VRING_DESC_F_NEXT, etc, but the layout and values were identical.
-\end{note}
-
-\devicenormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
-read a device-writable buffer (it MAY do so for debugging or diagnostic
-purposes).
-
-\drivernormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
-this implies that loops in the descriptor chain are forbidden!
-
-\subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-
-Some devices benefit by concurrently dispatching a large number
-of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this (see \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}). To increase
-ring capacity the driver can store a table of indirect
-descriptors anywhere in memory, and insert a descriptor in main
-virtqueue (with \field{flags}\&VIRTQ_DESC_F_INDIRECT on) that refers to memory buffer
-containing this indirect descriptor table; \field{addr} and \field{len}
-refer to the indirect table address and length in bytes,
-respectively.
-
-The indirect table layout structure looks like this
-(\field{len} is the length of the descriptor that refers to this table,
-which is a variable, so this code won't compile):
-
-\begin{lstlisting}
-struct indirect_descriptor_table {
-        /* The actual descriptors (16 bytes each) */
-        struct virtq_desc desc[len / 16];
-};
-\end{lstlisting}
-
-The first indirect descriptor is located at start of the indirect
-descriptor table (index 0), additional indirect descriptors are
-chained by \field{next}. An indirect descriptor without a valid \field{next}
-(with \field{flags}\&VIRTQ_DESC_F_NEXT off) signals the end of the descriptor.
-A single indirect descriptor
-table can include both device-readable and device-writable descriptors.
-
-\drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
-VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
-set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only
-one table per descriptor).
-
-A driver MUST NOT create a descriptor chain longer than the Queue Size of
-the device.
-
-A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
-in \field{flags}.
-
-\devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
-
-The device MUST handle the case of zero or more normal chained
-descriptors followed by a single descriptor with \field{flags}\&VIRTQ_DESC_F_INDIRECT.
-
-\begin{note}
-While unusual (most implementations either create a chain solely using
-non-indirect descriptors, or use a single indirect element), such a
-layout is valid.
-\end{note}
-
-\subsection{The Virtqueue Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
-
-\begin{lstlisting}
-struct virtq_avail {
-#define VIRTQ_AVAIL_F_NO_INTERRUPT      1
-        le16 flags;
-        le16 idx;
-        le16 ring[ /* Queue Size */ ];
-        le16 used_event; /* Only if VIRTIO_F_EVENT_IDX */
-};
-\end{lstlisting}
-
-The driver uses the available ring to offer buffers to the
-device: each ring entry refers to the head of a descriptor chain.  It is only
-written by the driver and read by the device.
-
-\field{idx} field indicates where the driver would put the next descriptor
-entry in the ring (modulo the queue size). This starts at 0, and increases.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to this structure as vring_avail, and the constant as
-VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
-\end{note}
-
-\subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
-the \field{flags} field in the available ring offers a crude mechanism for the driver to inform
-the device that it doesn't want interrupts when buffers are used.  Otherwise
-\field{used_event} is a more performant alternative where the driver
-specifies how far the device can progress before interrupting.
-
-Neither of these interrupt suppression methods are reliable, as they
-are not synchronized with the device, but they serve as
-useful optimizations.
-
-\drivernormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The driver MUST set \field{flags} to 0 or 1.
-\item The driver MAY set \field{flags} to 1 to advise
-the device that interrupts are not needed.
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The driver MUST set \field{flags} to 0.
-\item The driver MAY use \field{used_event} to advise the device that interrupts are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
-used ring will reach the value \field{used_event} + 1).
-\end{itemize}
-
-The driver MUST handle spurious interrupts from the device.
-
-\devicenormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The device MUST ignore the \field{used_event} value.
-\item After the device writes a descriptor index into the used ring:
-  \begin{itemize}
-  \item If \field{flags} is 1, the device SHOULD NOT send an interrupt.
-  \item If \field{flags} is 0, the device MUST send an interrupt.
-  \end{itemize}
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The device MUST ignore the lower bit of \field{flags}.
-\item After the device writes a descriptor index into the used ring:
-  \begin{itemize}
-  \item If the \field{idx} field in the used ring (which determined
-    where that descriptor index was placed) was equal to
-    \field{used_event}, the device MUST send an interrupt.
-  \item Otherwise the device SHOULD NOT send an interrupt.
-  \end{itemize}
-\end{itemize}
-
-\begin{note}
-For example, if \field{used_event} is 0, then a device using
-  VIRTIO_F_EVENT_IDX would interrupt after the first buffer is
-  used (and again after the 65536th buffer, etc).
-\end{note}
-
-\subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-\begin{lstlisting}
-struct virtq_used {
-#define VIRTQ_USED_F_NO_NOTIFY  1
-        le16 flags;
-        le16 idx;
-        struct virtq_used_elem ring[ /* Queue Size */];
-        le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
-};
-
-/* le32 is used here for ids for padding reasons. */
-struct virtq_used_elem {
-        /* Index of start of used descriptor chain. */
-        le32 id;
-        /* Total length of the descriptor chain which was used (written to) */
-        le32 len;
-};
-\end{lstlisting}
-
-The used ring is where the device returns buffers once it is done with
-them: it is only written to by the device, and read by the driver.
-
-Each entry in the ring is a pair: \field{id} indicates the head entry of the
-descriptor chain describing the buffer (this matches an entry
-placed in the available ring by the guest earlier), and \field{len} the total
-of bytes written into the buffer.
-
-\begin{note}
-\field{len} is particularly useful
-for drivers using untrusted buffers: if a driver does not know exactly
-how much has been written by the device, the driver would have to zero
-the buffer in advance to ensure no data leakage occurs.
-
-For example, a network driver may hand a received buffer directly to
-an unprivileged userspace application.  If the network device has not
-overwritten the bytes which were in that buffer, this could leak the
-contents of freed memory from other processes to the application.
-\end{note}
-
-\field{idx} field indicates where the driver would put the next descriptor
-entry in the ring (modulo the queue size). This starts at 0, and increases.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to these structures as vring_used and vring_used_elem, and
-the constant as VRING_USED_F_NO_NOTIFY, but the layout and value were
-identical.
-\end{note}
-
-\subsubsection{Legacy Interface: The Virtqueue Used
-Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues
-/ The Virtqueue Used Ring/ Legacy Interface: The Virtqueue Used
-Ring}
-
-Historically, many drivers ignored the \field{len} value, as a
-result, many devices set \field{len} incorrectly.  Thus, when
-using the legacy interface, it is generally a good idea to ignore
-the \field{len} value in used ring entries if possible.  Specific
-known issues are listed per device type.
-
-\devicenormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-The device MUST set \field{len} prior to updating the used \field{idx}.
-
-The device MUST write at least \field{len} bytes to descriptor,
-beginning at the first device-writable buffer,
-prior to updating the used \field{idx}.
-
-The device MAY write more than \field{len} bytes to descriptor.
-
-\begin{note}
-There are potential error cases where a device might not know what
-parts of the buffers have been written.  This is why \field{len} is
-permitted to be an underestimate: that's preferable to the driver believing
-that uninitialized memory has been overwritten when it has not.
-\end{note}
-
-\drivernormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-The driver MUST NOT make assumptions about data in device-writable buffers
-beyond the first \field{len} bytes, and SHOULD ignore this data.
-
-\subsection{Virtqueue Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-
-The device can suppress notifications in a manner analogous to the way
-drivers can suppress interrupts as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
-The device manipulates \field{flags} or \field{avail_event} in the used ring the
-same way the driver manipulates \field{flags} or \field{used_event} in the available ring.
-
-\drivernormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-
-The driver MUST initialize \field{flags} in the used ring to 0 when
-allocating the used ring.
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The driver MUST ignore the \field{avail_event} value.
-\item After the driver writes a descriptor index into the available ring:
-  \begin{itemize}
-        \item If \field{flags} is 1, the driver SHOULD NOT send a notification.
-        \item If \field{flags} is 0, the driver MUST send a notification.
-  \end{itemize}
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The driver MUST ignore the lower bit of \field{flags}.
-\item After the driver writes a descriptor index into the available ring:
-  \begin{itemize}
-        \item If the \field{idx} field in the available ring (which determined
-          where that descriptor index was placed) was equal to
-          \field{avail_event}, the driver MUST send a notification.
-        \item Otherwise the driver SHOULD NOT send a notification.
-  \end{itemize}
-\end{itemize}
-
-\devicenormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The device MUST set \field{flags} to 0 or 1.
-\item The device MAY set \field{flags} to 1 to advise
-the driver that notifications are not needed.
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The device MUST set \field{flags} to 0.
-\item The device MAY use \field{avail_event} to advise the driver that notifications are unnecessary until the driver writes entry with an index specified by \field{avail_event} into the available ring (equivalently, until \field{idx} in the
-available ring will reach the value \field{avail_event} + 1).
-\end{itemize}
-
-The device MUST handle spurious notifications from the driver.
-
-\subsection{Helpers for Operating Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Helpers for Operating Virtqueues}
-
-The Linux Kernel Source code contains the definitions above and
-helper routines in a more usable form, in
-include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
-and Red Hat under the (3-clause) BSD license so that it can be
-freely used by all other projects, and is reproduced (with slight
-variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
+\input{split-ring.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
diff --git a/split-ring.tex b/split-ring.tex
new file mode 100644
index 0000000..418f63d
--- /dev/null
+++ b/split-ring.tex
@@ -0,0 +1,498 @@
+\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
+The split virtqueue format is the original format used by legacy
+virtio devices.  The split virtqueue format separates the
+virtqueue into several parts, where each part is write-able by
+either the driver or the device, but not both. Multiple
+locations need to be updated when making a buffer available
+and when marking it as used.
+
+
+Each queue has a 16-bit queue size
+parameter, which sets the number of entries and implies the total size
+of the queue.
+
+Each virtqueue consists of three parts:
+
+\begin{itemize}
+\item Descriptor Table
+\item Available Ring
+\item Used Ring
+\end{itemize}
+
+where each part is physically-contiguous in guest memory,
+and has different alignment requirements.
+
+The memory aligment and size requirements, in bytes, of each part of the
+virtqueue are summarized in the following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Virtqueue Part    & Alignment & Size \\
+\hline \hline
+Descriptor Table  & 16        & $16 * $(Queue Size) \\
+\hline
+Available Ring    & 2         & $6 + 2 * $(Queue Size) \\
+ \hline
+Used Ring         & 4         & $6 + 8 * $(Queue Size) \\
+ \hline
+\end{tabular}
+
+The Alignment column gives the minimum alignment for each part
+of the virtqueue.
+
+The Size column gives the total number of bytes for each
+part of the virtqueue.
+
+Queue Size corresponds to the maximum number of buffers in the
+virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
+can be queued at any given time.}.  Queue Size value is always a
+power of 2.  The maximum Queue Size value is 32768.  This value
+is specified in a bus-specific way.
+
+When the driver wants to send a buffer to the device, it fills in
+a slot in the descriptor table (or chains several together), and
+writes the descriptor index into the available ring.  It then
+notifies the device. When the device has finished a buffer, it
+writes the descriptor index into the used ring, and sends an interrupt.
+
+\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a Virtio Device / Virtqueues}
+The driver MUST ensure that the physical address of the first byte
+of each virtqueue part is a multiple of the specified alignment value
+in the above table.
+
+\subsection{Legacy Interfaces: A Note on Virtqueue Layout}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
+
+For Legacy Interfaces, several additional
+restrictions are placed on the virtqueue layout:
+
+Each virtqueue occupies two or more physically-contiguous pages
+(usually defined as 4096 bytes, but depending on the transport;
+henceforth referred to as Queue Align)
+and consists of three parts:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Descriptor Table & Available Ring (\ldots padding\ldots) & Used Ring \\
+\hline
+\end{tabular}
+
+The bus-specific Queue Size field controls the total number of bytes
+for the virtqueue.
+When using the legacy interface, the transitional
+driver MUST retrieve the Queue Size field from the device
+and MUST allocate the total number of bytes for the virtqueue
+according to the following formula (Queue Align given in qalign and
+Queue Size given in qsz):
+
+\begin{lstlisting}
+#define ALIGN(x) (((x) + qalign) & ~qalign)
+static inline unsigned virtq_size(unsigned int qsz)
+{
+     return ALIGN(sizeof(struct virtq_desc)*qsz + sizeof(u16)*(3 + qsz))
+          + ALIGN(sizeof(u16)*3 + sizeof(struct virtq_used_elem)*qsz);
+}
+\end{lstlisting}
+
+This wastes some space with padding.
+When using the legacy interface, both transitional
+devices and drivers MUST use the following virtqueue layout
+structure to locate elements of the virtqueue:
+
+\begin{lstlisting}
+struct virtq {
+        // The actual descriptors (16 bytes each)
+        struct virtq_desc desc[ Queue Size ];
+
+        // A ring of available descriptor heads with free-running index.
+        struct virtq_avail avail;
+
+        // Padding to the next Queue Align boundary.
+        u8 pad[ Padding ];
+
+        // A ring of used descriptor heads with free-running index.
+        struct virtq_used used;
+};
+\end{lstlisting}
+
+\subsection{Legacy Interfaces: A Note on Virtqueue Endianness}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
+
+Note that when using the legacy interface, transitional
+devices and drivers MUST use the native
+endian of the guest as the endian of fields and in the virtqueue.
+This is opposed to little-endian for non-legacy interface as
+specified by this standard.
+It is assumed that the host is already aware of the guest endian.
+
+\subsection{Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing}
+The framing of messages with descriptors is
+independent of the contents of the buffers. For example, a network
+transmit buffer consists of a 12 byte header followed by the network
+packet. This could be most simply placed in the descriptor table as a
+12 byte output descriptor followed by a 1514 byte output descriptor,
+but it could also consist of a single 1526 byte output descriptor in
+the case where the header and packet are adjacent, or even three or
+more descriptors (possibly with loss of efficiency in that case).
+
+Note that, some device implementations have large-but-reasonable
+restrictions on total descriptor size (such as based on IOV_MAX in the
+host OS). This has not been a problem in practice: little sympathy
+will be given to drivers which create unreasonably-sized descriptors
+such as by dividing a network packet into 1500 single-byte
+descriptors!
+
+\devicenormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
+The device MUST NOT make assumptions about the particular arrangement
+of descriptors.  The device MAY have a reasonable limit of descriptors
+it will allow in a chain.
+
+\drivernormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
+The driver MUST place any device-writable descriptor elements after
+any device-readable descriptor elements.
+
+The driver SHOULD NOT use an excessive number of descriptors to
+describe a buffer.
+
+\subsubsection{Legacy Interface: Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
+
+Regrettably, initial driver implementations used simple layouts, and
+devices came to rely on it, despite this specification wording.  In
+addition, the specification for virtio_blk SCSI commands required
+intuiting field lengths from frame boundaries (see
+ \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}~\nameref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation})
+
+Thus when using the legacy interface, the VIRTIO_F_ANY_LAYOUT
+feature indicates to both the device and the driver that no
+assumptions were made about framing.  Requirements for
+transitional drivers when this is not negotiated are included in
+each device section.
+
+\subsection{The Virtqueue Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+
+The descriptor table refers to the buffers the driver is using for
+the device. \field{addr} is a physical address, and the buffers
+can be chained via \field{next}. Each descriptor describes a
+buffer which is read-only for the device (``device-readable'') or write-only for the device (``device-writable''), but a chain of
+descriptors can contain both device-readable and device-writable buffers.
+
+The actual contents of the memory offered to the device depends on the
+device type.  Most common is to begin the data with a header
+(containing little-endian fields) for the device to read, and postfix
+it with a status tailer for the device to write.
+
+\begin{lstlisting}
+struct virtq_desc {
+        /* Address (guest-physical). */
+        le64 addr;
+        /* Length. */
+        le32 len;
+
+/* This marks a buffer as continuing via the next field. */
+#define VIRTQ_DESC_F_NEXT   1
+/* This marks a buffer as device write-only (otherwise device read-only). */
+#define VIRTQ_DESC_F_WRITE     2
+/* This means the buffer contains a list of buffer descriptors. */
+#define VIRTQ_DESC_F_INDIRECT   4
+        /* The flags as indicated above. */
+        le16 flags;
+        /* Next field if flags & NEXT */
+        le16 next;
+};
+\end{lstlisting}
+
+The number of descriptors in the table is defined by the queue size
+for this virtqueue: this is the maximum possible descriptor chain length.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to this structure as vring_desc, and the constants as
+VRING_DESC_F_NEXT, etc, but the layout and values were identical.
+\end{note}
+
+\devicenormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
+read a device-writable buffer (it MAY do so for debugging or diagnostic
+purposes).
+
+\drivernormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
+this implies that loops in the descriptor chain are forbidden!
+
+\subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+
+Some devices benefit by concurrently dispatching a large number
+of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this (see \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}). To increase
+ring capacity the driver can store a table of indirect
+descriptors anywhere in memory, and insert a descriptor in main
+virtqueue (with \field{flags}\&VIRTQ_DESC_F_INDIRECT on) that refers to memory buffer
+containing this indirect descriptor table; \field{addr} and \field{len}
+refer to the indirect table address and length in bytes,
+respectively.
+
+The indirect table layout structure looks like this
+(\field{len} is the length of the descriptor that refers to this table,
+which is a variable, so this code won't compile):
+
+\begin{lstlisting}
+struct indirect_descriptor_table {
+        /* The actual descriptors (16 bytes each) */
+        struct virtq_desc desc[len / 16];
+};
+\end{lstlisting}
+
+The first indirect descriptor is located at start of the indirect
+descriptor table (index 0), additional indirect descriptors are
+chained by \field{next}. An indirect descriptor without a valid \field{next}
+(with \field{flags}\&VIRTQ_DESC_F_NEXT off) signals the end of the descriptor.
+A single indirect descriptor
+table can include both device-readable and device-writable descriptors.
+
+\drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
+VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
+set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only
+one table per descriptor).
+
+A driver MUST NOT create a descriptor chain longer than the Queue Size of
+the device.
+
+A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
+in \field{flags}.
+
+\devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
+
+The device MUST handle the case of zero or more normal chained
+descriptors followed by a single descriptor with \field{flags}\&VIRTQ_DESC_F_INDIRECT.
+
+\begin{note}
+While unusual (most implementations either create a chain solely using
+non-indirect descriptors, or use a single indirect element), such a
+layout is valid.
+\end{note}
+
+\subsection{The Virtqueue Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
+
+\begin{lstlisting}
+struct virtq_avail {
+#define VIRTQ_AVAIL_F_NO_INTERRUPT      1
+        le16 flags;
+        le16 idx;
+        le16 ring[ /* Queue Size */ ];
+        le16 used_event; /* Only if VIRTIO_F_EVENT_IDX */
+};
+\end{lstlisting}
+
+The driver uses the available ring to offer buffers to the
+device: each ring entry refers to the head of a descriptor chain.  It is only
+written by the driver and read by the device.
+
+\field{idx} field indicates where the driver would put the next descriptor
+entry in the ring (modulo the queue size). This starts at 0, and increases.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to this structure as vring_avail, and the constant as
+VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
+\end{note}
+
+\subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
+the \field{flags} field in the available ring offers a crude mechanism for the driver to inform
+the device that it doesn't want interrupts when buffers are used.  Otherwise
+\field{used_event} is a more performant alternative where the driver
+specifies how far the device can progress before interrupting.
+
+Neither of these interrupt suppression methods are reliable, as they
+are not synchronized with the device, but they serve as
+useful optimizations.
+
+\drivernormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The driver MUST set \field{flags} to 0 or 1.
+\item The driver MAY set \field{flags} to 1 to advise
+the device that interrupts are not needed.
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The driver MUST set \field{flags} to 0.
+\item The driver MAY use \field{used_event} to advise the device that interrupts are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
+used ring will reach the value \field{used_event} + 1).
+\end{itemize}
+
+The driver MUST handle spurious interrupts from the device.
+
+\devicenormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The device MUST ignore the \field{used_event} value.
+\item After the device writes a descriptor index into the used ring:
+  \begin{itemize}
+  \item If \field{flags} is 1, the device SHOULD NOT send an interrupt.
+  \item If \field{flags} is 0, the device MUST send an interrupt.
+  \end{itemize}
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The device MUST ignore the lower bit of \field{flags}.
+\item After the device writes a descriptor index into the used ring:
+  \begin{itemize}
+  \item If the \field{idx} field in the used ring (which determined
+    where that descriptor index was placed) was equal to
+    \field{used_event}, the device MUST send an interrupt.
+  \item Otherwise the device SHOULD NOT send an interrupt.
+  \end{itemize}
+\end{itemize}
+
+\begin{note}
+For example, if \field{used_event} is 0, then a device using
+  VIRTIO_F_EVENT_IDX would interrupt after the first buffer is
+  used (and again after the 65536th buffer, etc).
+\end{note}
+
+\subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+\begin{lstlisting}
+struct virtq_used {
+#define VIRTQ_USED_F_NO_NOTIFY  1
+        le16 flags;
+        le16 idx;
+        struct virtq_used_elem ring[ /* Queue Size */];
+        le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
+};
+
+/* le32 is used here for ids for padding reasons. */
+struct virtq_used_elem {
+        /* Index of start of used descriptor chain. */
+        le32 id;
+        /* Total length of the descriptor chain which was used (written to) */
+        le32 len;
+};
+\end{lstlisting}
+
+The used ring is where the device returns buffers once it is done with
+them: it is only written to by the device, and read by the driver.
+
+Each entry in the ring is a pair: \field{id} indicates the head entry of the
+descriptor chain describing the buffer (this matches an entry
+placed in the available ring by the guest earlier), and \field{len} the total
+of bytes written into the buffer.
+
+\begin{note}
+\field{len} is particularly useful
+for drivers using untrusted buffers: if a driver does not know exactly
+how much has been written by the device, the driver would have to zero
+the buffer in advance to ensure no data leakage occurs.
+
+For example, a network driver may hand a received buffer directly to
+an unprivileged userspace application.  If the network device has not
+overwritten the bytes which were in that buffer, this could leak the
+contents of freed memory from other processes to the application.
+\end{note}
+
+\field{idx} field indicates where the driver would put the next descriptor
+entry in the ring (modulo the queue size). This starts at 0, and increases.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to these structures as vring_used and vring_used_elem, and
+the constant as VRING_USED_F_NO_NOTIFY, but the layout and value were
+identical.
+\end{note}
+
+\subsubsection{Legacy Interface: The Virtqueue Used
+Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues
+/ The Virtqueue Used Ring/ Legacy Interface: The Virtqueue Used
+Ring}
+
+Historically, many drivers ignored the \field{len} value, as a
+result, many devices set \field{len} incorrectly.  Thus, when
+using the legacy interface, it is generally a good idea to ignore
+the \field{len} value in used ring entries if possible.  Specific
+known issues are listed per device type.
+
+\devicenormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+The device MUST set \field{len} prior to updating the used \field{idx}.
+
+The device MUST write at least \field{len} bytes to descriptor,
+beginning at the first device-writable buffer,
+prior to updating the used \field{idx}.
+
+The device MAY write more than \field{len} bytes to descriptor.
+
+\begin{note}
+There are potential error cases where a device might not know what
+parts of the buffers have been written.  This is why \field{len} is
+permitted to be an underestimate: that's preferable to the driver believing
+that uninitialized memory has been overwritten when it has not.
+\end{note}
+
+\drivernormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+The driver MUST NOT make assumptions about data in device-writable buffers
+beyond the first \field{len} bytes, and SHOULD ignore this data.
+
+\subsection{Virtqueue Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+
+The device can suppress notifications in a manner analogous to the way
+drivers can suppress interrupts as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+The device manipulates \field{flags} or \field{avail_event} in the used ring the
+same way the driver manipulates \field{flags} or \field{used_event} in the available ring.
+
+\drivernormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+
+The driver MUST initialize \field{flags} in the used ring to 0 when
+allocating the used ring.
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The driver MUST ignore the \field{avail_event} value.
+\item After the driver writes a descriptor index into the available ring:
+  \begin{itemize}
+        \item If \field{flags} is 1, the driver SHOULD NOT send a notification.
+        \item If \field{flags} is 0, the driver MUST send a notification.
+  \end{itemize}
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The driver MUST ignore the lower bit of \field{flags}.
+\item After the driver writes a descriptor index into the available ring:
+  \begin{itemize}
+        \item If the \field{idx} field in the available ring (which determined
+          where that descriptor index was placed) was equal to
+          \field{avail_event}, the driver MUST send a notification.
+        \item Otherwise the driver SHOULD NOT send a notification.
+  \end{itemize}
+\end{itemize}
+
+\devicenormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The device MUST set \field{flags} to 0 or 1.
+\item The device MAY set \field{flags} to 1 to advise
+the driver that notifications are not needed.
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The device MUST set \field{flags} to 0.
+\item The device MAY use \field{avail_event} to advise the driver that notifications are unnecessary until the driver writes entry with an index specified by \field{avail_event} into the available ring (equivalently, until \field{idx} in the
+available ring will reach the value \field{avail_event} + 1).
+\end{itemize}
+
+The device MUST handle spurious notifications from the driver.
+
+\subsection{Helpers for Operating Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Helpers for Operating Virtqueues}
+
+The Linux Kernel Source code contains the definitions above and
+helper routines in a more usable form, in
+include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
+and Red Hat under the (3-clause) BSD license so that it can be
+freely used by all other projects, and is reproduced (with slight
+variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
-- 
MST



---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 02/11] content: move ring text out to a separate file

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:01 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Will be easier to manage this way.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex    | 499 +--------------------------------------------------------
>  split-ring.tex | 498 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 499 insertions(+), 498 deletions(-)
>  create mode 100644 split-ring.tex

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Michael S. Tsirkin]

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/content.tex b/content.tex
index c7ef7fd..4483a4b 100644
--- a/content.tex
+++ b/content.tex
@@ -230,7 +230,30 @@ result.
 The mechanism for bulk data transport on virtio devices is
 pretentiously called a virtqueue. Each device can have zero or more
 virtqueues\footnote{For example, the simplest network device has one virtqueue for
-transmit and one for receive.}.  Each queue has a 16-bit queue size
+transmit and one for receive.}.
+
+Driver makes requests available to device by adding
+an available buffer to the queue - i.e. adding a buffer
+describing the request to a virtqueue, and optionally triggering
+a driver event - i.e. sending a notification to the device.
+
+Device executes the requests and - when complete - adds
+a used buffer to the queue - i.e. lets the driver
+know by marking the buffer as used. Device can then trigger
+a device event - i.e. send an interrupt to the driver.
+
+For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
+
+\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
+The split virtqueue format is the original format used by legacy
+virtio devices.  The split virtqueue format separates the
+virtqueue into several parts, where each part is write-able by
+either the driver or the device, but not both. Multiple
+locations need to be updated when making a buffer available
+and when marking it as used.
+
+
+Each queue has a 16-bit queue size
 parameter, which sets the number of entries and implies the total size
 of the queue.
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:01 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 25 ++++++++++++++++++++++++-
>  1 file changed, 24 insertions(+), 1 deletion(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Halil Pasic]

On 01/23/2018 01:01 AM, Michael S. Tsirkin wrote:
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 25 ++++++++++++++++++++++++-
>  1 file changed, 24 insertions(+), 1 deletion(-)
> 
> diff --git a/content.tex b/content.tex
> index c7ef7fd..4483a4b 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -230,7 +230,30 @@ result.
>  The mechanism for bulk data transport on virtio devices is
>  pretentiously called a virtqueue. Each device can have zero or more
>  virtqueues\footnote{For example, the simplest network device has one virtqueue for
> -transmit and one for receive.}.  Each queue has a 16-bit queue size
> +transmit and one for receive.}.
> +
> +Driver makes requests available to device by adding
> +an available buffer to the queue - i.e. adding a buffer
> +describing the request to a virtqueue, and optionally triggering
> +a driver event - i.e. sending a notification to the device.
> +
> +Device executes the requests and - when complete - adds
> +a used buffer to the queue - i.e. lets the driver
> +know by marking the buffer as used. Device can then trigger
> +a device event - i.e. send an interrupt to the driver.
> +

Here I seem to recognize my suggestion about describing the
relationship between virtqueue buffers and requests. But none
of the terms your are using are defined yet, so assuming linear
reading, this may not be the best place to establish that relationship.

Furthermore I think the usage of the term 'buffer' got even messier
that in v1.0. I will elaborate on that later.


> +For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
> +
> +\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
> +The split virtqueue format is the original format used by legacy
> +virtio devices.

All v1.0 devices and drivers are using split too not only legacy (aka pre v1.0). 

>  The split virtqueue format separates the
> +virtqueue into several parts, where each part is write-able by
> +either the driver or the device, but not both. Multiple
> +locations need to be updated when making a buffer available
> +and when marking it as used.
> +

If we assume 3 parts (available ring, used ring and descriptor table),
then the two last sentences are contradictory: as one of three would have
to be updated by both the device and the driver. Or did I misunderstand
something?

I think, the purpose of this paragraph is to distinguish the split
form the packed. We probably don't need these additions to understand
'split'. I would rather see a discussion on the two formats in the
common (2.4) virtqueue section.

[..]

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Michael S. Tsirkin]

On Mon, Feb 05, 2018 at 11:54:52PM +0100, Halil Pasic wrote:
> 
> 
> On 01/23/2018 01:01 AM, Michael S. Tsirkin wrote:
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex | 25 ++++++++++++++++++++++++-
> >  1 file changed, 24 insertions(+), 1 deletion(-)
> > 
> > diff --git a/content.tex b/content.tex
> > index c7ef7fd..4483a4b 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -230,7 +230,30 @@ result.
> >  The mechanism for bulk data transport on virtio devices is
> >  pretentiously called a virtqueue. Each device can have zero or more
> >  virtqueues\footnote{For example, the simplest network device has one virtqueue for
> > -transmit and one for receive.}.  Each queue has a 16-bit queue size
> > +transmit and one for receive.}.
> > +
> > +Driver makes requests available to device by adding
> > +an available buffer to the queue - i.e. adding a buffer
> > +describing the request to a virtqueue, and optionally triggering
> > +a driver event - i.e. sending a notification to the device.
> > +
> > +Device executes the requests and - when complete - adds
> > +a used buffer to the queue - i.e. lets the driver
> > +know by marking the buffer as used. Device can then trigger
> > +a device event - i.e. send an interrupt to the driver.
> > +
> 
> Here I seem to recognize my suggestion about describing the
> relationship between virtqueue buffers and requests. But none
> of the terms your are using are defined yet, so assuming linear
> reading, this may not be the best place to establish that relationship.
> 
> Furthermore I think the usage of the term 'buffer' got even messier
> that in v1.0. I will elaborate on that later.
> 

Sounds like a subject for a rework unrelated to this specific
project?

> > +For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
> > +
> > +\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
> > +The split virtqueue format is the original format used by legacy
> > +virtio devices.
> 
> All v1.0 devices and drivers are using split too not only legacy (aka pre v1.0). 

Yes but there's no versioning in virtio. IOW there is not need to
introduce a concept of "1.0 device". Devices just either do or
do not support the packed format.

> >  The split virtqueue format separates the
> > +virtqueue into several parts, where each part is write-able by
> > +either the driver or the device, but not both. Multiple
> > +locations need to be updated when making a buffer available
> > +and when marking it as used.
> > +
> 
> If we assume 3 parts (available ring, used ring and descriptor table),
> then the two last sentences are contradictory: as one of three would have
> to be updated by both the device and the driver. Or did I misunderstand
> something?

I don't see a contradiction.
Split rings only have RO and WO parts. There are 

> I think, the purpose of this paragraph is to distinguish the split
> form the packed. We probably don't need these additions to understand
> 'split'. I would rather see a discussion on the two formats in the
> common (2.4) virtqueue section.
> 
> [..]
> 
> Regards,
> Halil

This doesn't really scale - if we have a 3rd format we do not
want to mix them all in a common section.
So description of split format goes into split section, and so on.
I'd be fine to add a format comparison section if that will
make things easier.



---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Cornelia Huck]

On Tue, 6 Feb 2018 02:05:11 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Feb 05, 2018 at 11:54:52PM +0100, Halil Pasic wrote:
> > 
> > 
> > On 01/23/2018 01:01 AM, Michael S. Tsirkin wrote:  

> > > +\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
> > > +The split virtqueue format is the original format used by legacy
> > > +virtio devices.  
> > 
> > All v1.0 devices and drivers are using split too not only legacy (aka pre v1.0).   
> 
> Yes but there's no versioning in virtio. IOW there is not need to
> introduce a concept of "1.0 device". Devices just either do or
> do not support the packed format.

Maybe just drop "used by legacy virtio devices"?

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Halil Pasic]

On 02/06/2018 01:05 AM, Michael S. Tsirkin wrote:
> On Mon, Feb 05, 2018 at 11:54:52PM +0100, Halil Pasic wrote:
>>
>>
>> On 01/23/2018 01:01 AM, Michael S. Tsirkin wrote:
>>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>> ---
>>>  content.tex | 25 ++++++++++++++++++++++++-
>>>  1 file changed, 24 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/content.tex b/content.tex
>>> index c7ef7fd..4483a4b 100644
>>> --- a/content.tex
>>> +++ b/content.tex
>>> @@ -230,7 +230,30 @@ result.
>>>  The mechanism for bulk data transport on virtio devices is
>>>  pretentiously called a virtqueue. Each device can have zero or more
>>>  virtqueues\footnote{For example, the simplest network device has one virtqueue for
>>> -transmit and one for receive.}.  Each queue has a 16-bit queue size
>>> +transmit and one for receive.}.
>>> +
>>> +Driver makes requests available to device by adding
>>> +an available buffer to the queue - i.e. adding a buffer
>>> +describing the request to a virtqueue, and optionally triggering
>>> +a driver event - i.e. sending a notification to the device.
>>> +
>>> +Device executes the requests and - when complete - adds
>>> +a used buffer to the queue - i.e. lets the driver
>>> +know by marking the buffer as used. Device can then trigger
>>> +a device event - i.e. send an interrupt to the driver.
>>> +
>>
>> Here I seem to recognize my suggestion about describing the
>> relationship between virtqueue buffers and requests. But none
>> of the terms your are using are defined yet, so assuming linear
>> reading, this may not be the best place to establish that relationship.
>>
>> Furthermore I think the usage of the term 'buffer' got even messier
>> that in v1.0. I will elaborate on that later.
>>
> 
> Sounds like a subject for a rework unrelated to this specific
> project?
> 

I tend to agree, but it does have some impact on the clarity and
thus on the quality of what is the focus of this project IMHO. Please
see my comments on #8. 

>>> +For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
>>> +
>>> +\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
>>> +The split virtqueue format is the original format used by legacy
>>> +virtio devices.
>>
>> All v1.0 devices and drivers are using split too not only legacy (aka pre v1.0). 
> 
> Yes but there's no versioning in virtio. IOW there is not need to
> introduce a concept of "1.0 device". Devices just either do or
> do not support the packed format.
>


I agree with what Connie proposed (drop 'used by legacy virtio devices').
My point is that this legacy can lead to confusion.

Regarding no versioning in virtio: I agree only partially. We have
the VIRTIO_F_VERSION_1 feature bit and we have a version number in
the title. But I think, I understand what you mean. This non-egsistence
of versioning in virtio is probably trivial for anybody working on
virtio for years. But is it for a new hire who just got trough the spec?

In the CIO transport we have an explicit mention of virtio 1.0 (explains
revision 1). I wonder if that is still appropriate. Shouldn't that just
be virtio 1?
 
>>>  The split virtqueue format separates the
>>> +virtqueue into several parts, where each part is write-able by
>>> +either the driver or the device, but not both. Multiple
>>> +locations need to be updated when making a buffer available
>>> +and when marking it as used.
>>> +
>>
>> If we assume 3 parts (available ring, used ring and descriptor table),
>> then the two last sentences are contradictory: as one of three would have
>> to be updated by both the device and the driver. Or did I misunderstand
>> something?
> 
> I don't see a contradiction.
> Split rings only have RO and WO parts. There are 
> 

This sentence seems unfinished. I was probably wrong. I assumed 'location'
means 'area' in this context. What does location mean in this context
(e.g. same location is equivalent to same byte)?

>> I think, the purpose of this paragraph is to distinguish the split
>> form the packed. We probably don't need these additions to understand
>> 'split'. I would rather see a discussion on the two formats in the
>> common (2.4) virtqueue section.
>>
>> [..]
>>
>> Regards,
>> Halil
> 
> This doesn't really scale - if we have a 3rd format we do not
> want to mix them all in a common section.
> So description of split format goes into split section, and so on.
> I'd be fine to add a format comparison section if that will
> make things easier.

OK. I need to think about the structure a bit more myself. Let's
just go with what we have.

Thanks for your answers!

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Cornelia Huck]

On Tue, 6 Feb 2018 12:10:20 +0100
Halil Pasic <pasic@linux.vnet.ibm.com> wrote:

> On 02/06/2018 01:05 AM, Michael S. Tsirkin wrote:
> > On Mon, Feb 05, 2018 at 11:54:52PM +0100, Halil Pasic wrote:  
> >>
> >>
> >> On 01/23/2018 01:01 AM, Michael S. Tsirkin wrote:  
> >>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

> >>> +\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
> >>> +The split virtqueue format is the original format used by legacy
> >>> +virtio devices.  
> >>
> >> All v1.0 devices and drivers are using split too not only legacy (aka pre v1.0).   
> > 
> > Yes but there's no versioning in virtio. IOW there is not need to
> > introduce a concept of "1.0 device". Devices just either do or
> > do not support the packed format.
> >  
> 
> 
> I agree with what Connie proposed (drop 'used by legacy virtio devices').
> My point is that this legacy can lead to confusion.
> 
> Regarding no versioning in virtio: I agree only partially. We have
> the VIRTIO_F_VERSION_1 feature bit and we have a version number in
> the title. But I think, I understand what you mean. This non-egsistence
> of versioning in virtio is probably trivial for anybody working on
> virtio for years. But is it for a new hire who just got trough the spec?
> 
> In the CIO transport we have an explicit mention of virtio 1.0 (explains
> revision 1). I wonder if that is still appropriate. Shouldn't that just
> be virtio 1?

We can certainly do s/1.0/1/ in the ccw transport (mind doing a patch?)

For the greater picture: As I see it, any implementation conforming to
1.1 is also conforming to 1.0. To conform to 1.1, it only needs the
split layout. I think we can continue with that for any further
iterations of 1.n (and keep VERSION_1 as it is now.)

What we want to do with version 2.n (should that ever come up) is up
for discussion (can an implementation conform to version 2 but not to
version 1?). But I prefer to cross that bridge when we come to it.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Halil Pasic]

On 02/06/2018 12:20 PM, Cornelia Huck wrote:
> On Tue, 6 Feb 2018 12:10:20 +0100
> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
> 
>> On 02/06/2018 01:05 AM, Michael S. Tsirkin wrote:
>>> On Mon, Feb 05, 2018 at 11:54:52PM +0100, Halil Pasic wrote:  
[..]
>> In the CIO transport we have an explicit mention of virtio 1.0 (explains
>> revision 1). I wonder if that is still appropriate. Shouldn't that just
>> be virtio 1?
> 
> We can certainly do s/1.0/1/ in the ccw transport (mind doing a patch?)
> 

Will do.

> For the greater picture: As I see it, any implementation conforming to
> 1.1 is also conforming to 1.0. To conform to 1.1, it only needs the
> split layout. I think we can continue with that for any further
> iterations of 1.n (and keep VERSION_1 as it is now.)
>

Agreed. I will think a bit if there is an elegant way to codify this in
the spec.
 
> What we want to do with version 2.n (should that ever come up) is up
> for discussion (can an implementation conform to version 2 but not to
> version 1?). But I prefer to cross that bridge when we come to it.
> 

I agree. There is no reason to ponder about this right now.

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 01/11] content: move 1.0 queue format out to a separate section

[Michael S. Tsirkin]

On Tue, Feb 06, 2018 at 12:10:20PM +0100, Halil Pasic wrote:
> I agree with what Connie proposed (drop 'used by legacy virtio devices').
> My point is that this legacy can lead to confusion.

It's a good point.
I guess it's better to just go the regular route and add a
separate section
explaining that legacy devices always use a split ring.

> Regarding no versioning in virtio: I agree only partially. We have
> the VIRTIO_F_VERSION_1 feature bit and we have a version number in
> the title. But I think, I understand what you mean. This non-egsistence
> of versioning in virtio is probably trivial for anybody working on
> virtio for years. But is it for a new hire who just got trough the spec?

It's not easy to write a spec that is both formally correct and
easy to read. It might be a good idea to extend the introduction
some more and explain how does virtio handle compatibility
without versioning.

> In the CIO transport we have an explicit mention of virtio 1.0 (explains
> revision 1). I wonder if that is still appropriate. Shouldn't that just
> be virtio 1?

Some transports chose to have their own transport versioning.

> >>>  The split virtqueue format separates the
> >>> +virtqueue into several parts, where each part is write-able by
> >>> +either the driver or the device, but not both. Multiple
> >>> +locations need to be updated when making a buffer available
> >>> +and when marking it as used.
> >>> +
> >>
> >> If we assume 3 parts (available ring, used ring and descriptor table),
> >> then the two last sentences are contradictory: as one of three would have
> >> to be updated by both the device and the driver. Or did I misunderstand
> >> something?
> > 
> > I don't see a contradiction.
> > Split rings only have RO and WO parts. There are 
> > 
> 
> This sentence seems unfinished. I was probably wrong. I assumed 'location'
> means 'area' in this context. What does location mean in this context
> (e.g. same location is equivalent to same byte)?

Yes, that's what I meant. Maybe "Multiple parts and/or locations within
a part"?


> >> I think, the purpose of this paragraph is to distinguish the split
> >> form the packed. We probably don't need these additions to understand
> >> 'split'. I would rather see a discussion on the two formats in the
> >> common (2.4) virtqueue section.
> >>
> >> [..]
> >>
> >> Regards,
> >> Halil
> > 
> > This doesn't really scale - if we have a 3rd format we do not
> > want to mix them all in a common section.
> > So description of split format goes into split section, and so on.
> > I'd be fine to add a format comparison section if that will
> > make things easier.
> 
> OK. I need to think about the structure a bit more myself. Let's
> just go with what we have.
> 
> Thanks for your answers!
> 
> Regards,
> Halil

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 03/11] content: move virtqueue operation description

[Michael S. Tsirkin]

virtqueue operation description is specific to the virtqueue
format. Move it out to split-ring.tex and update all
references.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 conformance.tex |   4 +-
 content.tex     | 171 +++---------------------------------------------------
 split-ring.tex  | 175 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 168 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index f59e360..55d17b4 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -40,9 +40,9 @@ A driver MUST conform to the following normative statements:
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization}
-\item \ref{drivernormative:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-\item \ref{drivernormative:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Cleanup}
 \item \ref{drivernormative:Reserved Feature Bits}
 \end{itemize}
diff --git a/content.tex b/content.tex
index 5b4c4e9..3b4579e 100644
--- a/content.tex
+++ b/content.tex
@@ -337,167 +337,14 @@ And Device Operation / Device Initialization / Set DRIVER-OK}.
 
 \section{Device Operation}\label{sec:General Initialization And Device Operation / Device Operation}
 
-There are two parts to device operation: supplying new buffers to
-the device, and processing used buffers from the device.
-
-\begin{note} As an
-example, the simplest virtio network device has two virtqueues: the
-transmit virtqueue and the receive virtqueue. The driver adds
-outgoing (device-readable) packets to the transmit virtqueue, and then
-frees them after they are used. Similarly, incoming (device-writable)
-buffers are added to the receive virtqueue, and processed after
-they are used.
-\end{note}
-
-\subsection{Supplying Buffers to The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device}
-
-The driver offers buffers to one of the device's virtqueues as follows:
-
-\begin{enumerate}
-\item\label{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Buffers} The driver places the buffer into free descriptor(s) in the
-   descriptor table, chaining as necessary (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}).
-
-\item\label{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Index} The driver places the index of the head of the descriptor chain
-   into the next ring entry of the available ring.
-
-\item Steps \ref{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Buffers} and \ref{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Index} MAY be performed repeatedly if batching
-  is possible.
-
-\item The driver performs suitable a memory barrier to ensure the device sees
-  the updated descriptor table and available ring before the next
-  step.
-
-\item The available \field{idx} is increased by the number of
-  descriptor chain heads added to the available ring.
-
-\item The driver performs a suitable memory barrier to ensure that it updates
-  the \field{idx} field before checking for notification suppression.
-
-\item If notifications are not suppressed, the driver notifies the device
-    of the new available buffers.
-\end{enumerate}
-
-Note that the above code does not take precautions against the
-available ring buffer wrapping around: this is not possible since
-the ring buffer is the same size as the descriptor table, so step
-(1) will prevent such a condition.
-
-In addition, the maximum queue size is 32768 (the highest power
-of 2 which fits in 16 bits), so the 16-bit \field{idx} value can always
-distinguish between a full and empty buffer.
+When operating the device, each field in the device configuration
+space can be changed by either the driver or the device.
 
-What follows is the requirements of each stage in more detail.
-
-\subsubsection{Placing Buffers Into The Descriptor Table}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Placing Buffers Into The Descriptor Table}
-
-A buffer consists of zero or more device-readable physically-contiguous
-elements followed by zero or more physically-contiguous
-device-writable elements (each has at least one element). This
-algorithm maps it into the descriptor table to form a descriptor
-chain:
-
-for each buffer element, b:
-
-\begin{enumerate}
-\item Get the next free descriptor table entry, d
-\item Set \field{d.addr} to the physical address of the start of b
-\item Set \field{d.len} to the length of b.
-\item If b is device-writable, set \field{d.flags} to VIRTQ_DESC_F_WRITE,
-    otherwise 0.
-\item If there is a buffer element after this:
-    \begin{enumerate}
-    \item Set \field{d.next} to the index of the next free descriptor
-      element.
-    \item Set the VIRTQ_DESC_F_NEXT bit in \field{d.flags}.
-    \end{enumerate}
-\end{enumerate}
-
-In practice, \field{d.next} is usually used to chain free
-descriptors, and a separate count kept to check there are enough
-free descriptors before beginning the mappings.
-
-\subsubsection{Updating The Available Ring}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating The Available Ring}
-
-The descriptor chain head is the first d in the algorithm
-above, ie. the index of the descriptor table entry referring to the first
-part of the buffer.  A naive driver implementation MAY do the following (with the
-appropriate conversion to-and-from little-endian assumed):
-
-\begin{lstlisting}
-avail->ring[avail->idx % qsz] = head;
-\end{lstlisting}
+Whenever such a configuration change is triggered by the device,
+driver is notified. This makes it possible for drivers to
+cache device configuration, avoiding expensive configuration
+reads unless notified.
 
-However, in general the driver MAY add many descriptor chains before it updates
-\field{idx} (at which point they become visible to the
-device), so it is common to keep a counter of how many the driver has added:
-
-\begin{lstlisting}
-avail->ring[(avail->idx + added++) % qsz] = head;
-\end{lstlisting}
-
-\subsubsection{Updating \field{idx}}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-
-\field{idx} always increments, and wraps naturally at
-65536:
-
-\begin{lstlisting}
-avail->idx += added;
-\end{lstlisting}
-
-Once available \field{idx} is updated by the driver, this exposes the
-descriptor and its contents.  The device MAY
-access the descriptor chains the driver created and the
-memory they refer to immediately.
-
-\drivernormative{\paragraph}{Updating idx}{General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-The driver MUST perform a suitable memory barrier before the \field{idx} update, to ensure the
-device sees the most up-to-date copy.
-
-\subsubsection{Notifying The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
-
-The actual method of device notification is bus-specific, but generally
-it can be expensive.  So the device MAY suppress such notifications if it
-doesn't need them, as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}.
-
-The driver has to be careful to expose the new \field{idx}
-value before checking if notifications are suppressed.
-
-\drivernormative{\paragraph}{Notifying The Device}{General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
-The driver MUST perform a suitable memory barrier before reading \field{flags} or
-\field{avail_event}, to avoid missing a notification.
-
-\subsection{Receiving Used Buffers From The Device}\label{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}
-
-Once the device has used buffers referred to by a descriptor (read from or written to them, or
-parts of both, depending on the nature of the virtqueue and the
-device), it interrupts the driver as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
-
-\begin{note}
-For optimal performance, a driver MAY disable interrupts while processing
-the used ring, but beware the problem of missing interrupts between
-emptying the ring and reenabling interrupts.  This is usually handled by
-re-checking for more used buffers after interrups are re-enabled:
-
-\begin{lstlisting}
-virtq_disable_interrupts(vq);
-
-for (;;) {
-        if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) {
-                virtq_enable_interrupts(vq);
-                mb();
-
-                if (vq->last_seen_used != le16_to_cpu(virtq->used.idx))
-                        break;
-
-                virtq_disable_interrupts(vq);
-        }
-
-        struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz];
-        process_buffer(e);
-        vq->last_seen_used++;
-}
-\end{lstlisting}
-\end{note}
 
 \subsection{Notification of Device Configuration Changes}\label{sec:General Initialization And Device Operation / Device Operation / Notification of Device Configuration Changes}
 
@@ -3017,9 +2864,7 @@ If VIRTIO_NET_HDR_F_NEEDS_CSUM is not set, the device MUST NOT
 rely on the packet checksum being correct.
 \paragraph{Packet Transmission Interrupt}\label{sec:Device Types / Network Device / Device Operation / Packet Transmission / Packet Transmission Interrupt}
 
-Often a driver will suppress transmission interrupts using the
-VIRTQ_AVAIL_F_NO_INTERRUPT flag
- (see \ref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}~\nameref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device})
+Often a driver will suppress transmission virtqueue interrupts
 and check for used packets in the transmit path of following
 packets.
 
@@ -3079,7 +2924,7 @@ if VIRTIO_NET_F_MRG_RXBUF is not negotiated.}
 
 When a packet is copied into a buffer in the receiveq, the
 optimal path is to disable further interrupts for the receiveq
-(see \ref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}~\nameref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}) and process
+and process
 packets until no more are found, then re-enable them.
 
 Processing incoming packets involves:
diff --git a/split-ring.tex b/split-ring.tex
index 418f63d..724bb82 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -1,12 +1,13 @@
 \section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
-The split virtqueue format is the original format used by legacy
-virtio devices.  The split virtqueue format separates the
+The split virtqueue format is the original format used by devices
+conforming to the 1.0 version of this standard (and a variant thereof
+by legacy virtio devices).
+The split virtqueue format separates the
 virtqueue into several parts, where each part is write-able by
 either the driver or the device, but not both. Multiple
 locations need to be updated when making a buffer available
 and when marking it as used.
 
-
 Each queue has a 16-bit queue size
 parameter, which sets the number of entries and implies the total size
 of the queue.
@@ -496,3 +497,171 @@ include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
 and Red Hat under the (3-clause) BSD license so that it can be
 freely used by all other projects, and is reproduced (with slight
 variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
+
+\subsection{Virtqueue Operation}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Operation}
+
+There are two parts to virtqueue operation: supplying new
+available buffers to the device, and processing used buffers from
+the device.
+
+\begin{note} As an
+example, the simplest virtio network device has two virtqueues: the
+transmit virtqueue and the receive virtqueue. The driver adds
+outgoing (device-readable) packets to the transmit virtqueue, and then
+frees them after they are used. Similarly, incoming (device-writable)
+buffers are added to the receive virtqueue, and processed after
+they are used.
+\end{note}
+
+What follows is the requirements of each of these two parts
+when using the split virtqueue format in more detail.
+
+\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device}
+
+The driver offers buffers to one of the device's virtqueues as follows:
+
+\begin{enumerate}
+\item\label{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Buffers} The driver places the buffer into free descriptor(s) in the
+   descriptor table, chaining as necessary (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}).
+
+\item\label{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Index} The driver places the index of the head of the descriptor chain
+   into the next ring entry of the available ring.
+
+\item Steps \ref{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Buffers} and \ref{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Index} MAY be performed repeatedly if batching
+  is possible.
+
+\item The driver performs suitable a memory barrier to ensure the device sees
+  the updated descriptor table and available ring before the next
+  step.
+
+\item The available \field{idx} is increased by the number of
+  descriptor chain heads added to the available ring.
+
+\item The driver performs a suitable memory barrier to ensure that it updates
+  the \field{idx} field before checking for notification suppression.
+
+\item If notifications are not suppressed, the driver notifies the device
+    of the new available buffers.
+\end{enumerate}
+
+Note that the above code does not take precautions against the
+available ring buffer wrapping around: this is not possible since
+the ring buffer is the same size as the descriptor table, so step
+(1) will prevent such a condition.
+
+In addition, the maximum queue size is 32768 (the highest power
+of 2 which fits in 16 bits), so the 16-bit \field{idx} value can always
+distinguish between a full and empty buffer.
+
+What follows is the requirements of each stage in more detail.
+
+\subsubsection{Placing Buffers Into The Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Buffers Into The Descriptor Table}
+
+A buffer consists of zero or more device-readable physically-contiguous
+elements followed by zero or more physically-contiguous
+device-writable elements (each has at least one element). This
+algorithm maps it into the descriptor table to form a descriptor
+chain:
+
+for each buffer element, b:
+
+\begin{enumerate}
+\item Get the next free descriptor table entry, d
+\item Set \field{d.addr} to the physical address of the start of b
+\item Set \field{d.len} to the length of b.
+\item If b is device-writable, set \field{d.flags} to VIRTQ_DESC_F_WRITE,
+    otherwise 0.
+\item If there is a buffer element after this:
+    \begin{enumerate}
+    \item Set \field{d.next} to the index of the next free descriptor
+      element.
+    \item Set the VIRTQ_DESC_F_NEXT bit in \field{d.flags}.
+    \end{enumerate}
+\end{enumerate}
+
+In practice, \field{d.next} is usually used to chain free
+descriptors, and a separate count kept to check there are enough
+free descriptors before beginning the mappings.
+
+\subsubsection{Updating The Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating The Available Ring}
+
+The descriptor chain head is the first d in the algorithm
+above, ie. the index of the descriptor table entry referring to the first
+part of the buffer.  A naive driver implementation MAY do the following (with the
+appropriate conversion to-and-from little-endian assumed):
+
+\begin{lstlisting}
+avail->ring[avail->idx % qsz] = head;
+\end{lstlisting}
+
+However, in general the driver MAY add many descriptor chains before it updates
+\field{idx} (at which point they become visible to the
+device), so it is common to keep a counter of how many the driver has added:
+
+\begin{lstlisting}
+avail->ring[(avail->idx + added++) % qsz] = head;
+\end{lstlisting}
+
+\subsubsection{Updating \field{idx}}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+
+\field{idx} always increments, and wraps naturally at
+65536:
+
+\begin{lstlisting}
+avail->idx += added;
+\end{lstlisting}
+
+Once available \field{idx} is updated by the driver, this exposes the
+descriptor and its contents.  The device MAY
+access the descriptor chains the driver created and the
+memory they refer to immediately.
+
+\drivernormative{\paragraph}{Updating idx}{Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+The driver MUST perform a suitable memory barrier before the \field{idx} update, to ensure the
+device sees the most up-to-date copy.
+
+\subsubsection{Notifying The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+
+The actual method of device notification is bus-specific, but generally
+it can be expensive.  So the device MAY suppress such notifications if it
+doesn't need them, as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}.
+
+The driver has to be careful to expose the new \field{idx}
+value before checking if notifications are suppressed.
+
+\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+The driver MUST perform a suitable memory barrier before reading \field{flags} or
+\field{avail_event}, to avoid missing a notification.
+
+\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Receiving Used Buffers From The Device}
+
+Once the device has used buffers referred to by a descriptor (read from or written to them, or
+parts of both, depending on the nature of the virtqueue and the
+device), it interrupts the driver as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+
+\begin{note}
+For optimal performance, a driver MAY disable interrupts while processing
+the used ring, but beware the problem of missing interrupts between
+emptying the ring and reenabling interrupts.  This is usually handled by
+re-checking for more used buffers after interrups are re-enabled:
+
+\begin{lstlisting}
+virtq_disable_interrupts(vq);
+
+for (;;) {
+        if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) {
+                virtq_enable_interrupts(vq);
+                mb();
+
+                if (vq->last_seen_used != le16_to_cpu(virtq->used.idx))
+                        break;
+
+                virtq_disable_interrupts(vq);
+        }
+
+        struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz];
+        process_buffer(e);
+        vq->last_seen_used++;
+}
+\end{lstlisting}
+\end{note}
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 03/11] content: move virtqueue operation description

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:02 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> virtqueue operation description is specific to the virtqueue
> format. Move it out to split-ring.tex and update all
> references.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  conformance.tex |   4 +-
>  content.tex     | 171 +++---------------------------------------------------
>  split-ring.tex  | 175 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 182 insertions(+), 168 deletions(-)

> diff --git a/split-ring.tex b/split-ring.tex
> index 418f63d..724bb82 100644
> --- a/split-ring.tex
> +++ b/split-ring.tex
> @@ -1,12 +1,13 @@
>  \section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
> -The split virtqueue format is the original format used by legacy
> -virtio devices.  The split virtqueue format separates the
> +The split virtqueue format is the original format used by devices
> +conforming to the 1.0 version of this standard (and a variant thereof
> +by legacy virtio devices).

Ah, just noticed this hunk. Might make sense to move this to a prior
patch, although it does not matter in the end.

> +The split virtqueue format separates the
>  virtqueue into several parts, where each part is write-able by
>  either the driver or the device, but not both. Multiple
>  locations need to be updated when making a buffer available
>  and when marking it as used.
>  
> -

Dito here (should probably go to a prior patch).

>  Each queue has a 16-bit queue size
>  parameter, which sets the number of entries and implies the total size
>  of the queue.

Else:

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 04/11] content: replace mentions of len with used length

[Michael S. Tsirkin]

Document buffer used len and use that terminology everywhere in the
generic section.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 39 ++++++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 19 deletions(-)

diff --git a/content.tex b/content.tex
index 3b4579e..4350ecf 100644
--- a/content.tex
+++ b/content.tex
@@ -242,7 +242,8 @@ a used buffer to the queue - i.e. lets the driver
 know by marking the buffer as used. Device can then trigger
 a device event - i.e. send an interrupt to the driver.
 
-For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
+Device reports the number of bytes it has written to memory for
+each buffer it uses. This is referred to as ``used length''.
 
 \input{split-ring.tex}
 
@@ -1304,7 +1305,7 @@ The driver interrupt handler would typically:
     \begin{itemize}
       \item Read the ISR Status field, which will reset it to zero.
       \item If the lower bit is set:
-        look through the used rings of all virtqueues for the
+        look through all virtqueues for the
         device, to see if any progress has been made by the device
         which requires servicing.
       \item If the second lower bit is set:
@@ -1313,8 +1314,7 @@ The driver interrupt handler would typically:
   \item If MSI-X capability is enabled:
     \begin{itemize}
       \item
-        Look through the used rings of
-        all virtqueues mapped to that MSI-X vector for the
+        Look through all virtqueues mapped to that MSI-X vector for the
         device, to see if any progress has been made by the device
         which requires servicing.
       \item
@@ -1728,8 +1728,7 @@ nor behaviour:
   }
   \hline
   \mmioreg{QueueNum}{Virtual queue size}{0x038}{W}{%
-    Queue size is the number of elements in the queue, therefore size
-    of the descriptor table and both available and used rings.
+    Queue size is the number of elements in the queue.
     Writing to this register notifies the device what size of the
     queue the driver will use. This applies to the queue selected by
     writing to \field{QueueSel}.
@@ -2709,7 +2708,7 @@ when VIRTIO_NET_F_MRG_RXBUF was negotiated; without that feature the
 structure was 2 bytes shorter.
 
 When using the legacy interface, the driver SHOULD ignore the
-\field{len} value in used ring entries for the transmit queues
+used length for the transmit queues
 and the controlq queue.
 \begin{note}
 Historically, some devices put
@@ -2868,8 +2867,8 @@ Often a driver will suppress transmission virtqueue interrupts
 and check for used packets in the transmit path of following
 packets.
 
-The normal behavior in this interrupt handler is to retrieve and
-new descriptors from the used ring and free the corresponding
+The normal behavior in this interrupt handler is to retrieve
+used buffers from the virtqueue and free the corresponding
 headers and packets.
 
 \subsubsection{Setting Up Receive Buffers}\label{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}
@@ -2936,7 +2935,7 @@ Processing incoming packets involves:
   This allows receipt of large packets without having to allocate large
   buffers: a packet that does not fit in a single buffer can flow
   over to the next buffer, and so on. In this case, there will be
-  at least \field{num_buffers} in the used ring, and the device
+  at least \field{num_buffers} used buffers in the virtqueue, and the device
   chains them together to form a single packet in a way similar to
   how it would store it in a single buffer spread over multiple
   descriptors.
@@ -2990,8 +2989,8 @@ MUST use all buffers but the last (i.e. the first $num_buffers -
 supplied by the driver.
 
 The device MUST use all buffers used by a single receive
-packet together, by atomically incrementing \field{idx} in the
-used ring by the \field{num_buffers} value.
+packet together, such that at least \field{num_buffers} are
+observed by driver as used.
 
 If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device MUST set
 \field{flags} to zero and SHOULD supply a fully checksummed
@@ -3378,7 +3377,8 @@ The device MUST queue packets only on any receiveq1 before the
 VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command.
 
 The device MUST NOT queue packets on receive queues greater than
-\field{virtqueue_pairs} once it has placed the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in the used ring.
+\field{virtqueue_pairs} once it has placed the
+VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in a used buffer.
 
 \subparagraph{Legacy Interface: Automatic receive steering in multiqueue mode}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Legacy Interface: Automatic receive steering in multiqueue mode}
 When using the legacy interface, transitional devices and drivers
@@ -3744,7 +3744,7 @@ according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
 When using the legacy interface, transitional drivers
-SHOULD ignore the \field{len} value in used ring entries.
+SHOULD ignore the used length values.
 \begin{note}
 Historically, some devices put the total descriptor length,
 or the total length of device-writable buffers there,
@@ -4081,7 +4081,7 @@ according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
 When using the legacy interface, the driver SHOULD ignore the
-\field{len} value in used ring entries for the transmit queues
+used length values for the transmit queues
 and the control transmitq.
 \begin{note}
 Historically, some devices put the total descriptor length there,
@@ -4337,7 +4337,8 @@ and acknowledging the deflate request.
 \paragraph{Legacy Interface: Device Operation}\label{sec:Device
 Types / Memory Balloon Device / Device Operation / Legacy
 Interface: Device Operation}
-When using the legacy interface, the driver SHOULD ignore the \field{len} value in used ring entries.
+When using the legacy interface, the driver SHOULD ignore the
+used length values.
 \begin{note}
 Historically, some devices put the total descriptor length there,
 even though no data was actually written.
@@ -4370,8 +4371,7 @@ and notifies the device. A request for memory statistics proceeds
 as follows:
 
 \begin{enumerate}
-\item The device pushes the buffer onto the used ring and sends an
-  interrupt.
+\item The device uses the buffer and sends an interrupt.
 
 \item The driver pops the used buffer and discards it.
 
@@ -4632,7 +4632,8 @@ queue and the event queue.
 \paragraph{Legacy Interface: Device Operation}\label{sec:Device
 Types / SCSI Host Device / Device Operation / Legacy
 Interface: Device Operation}
-When using the legacy interface, the driver SHOULD ignore the \field{len} value in used ring entries.
+When using the legacy interface, the driver SHOULD ignore the
+used length values.
 \begin{note}
 Historically, devices put the total descriptor length,
 or the total length of device-writable buffers there,
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 04/11] content: replace mentions of len with used length

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:03 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Document buffer used len and use that terminology everywhere in the
> generic section.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 39 ++++++++++++++++++++-------------------
>  1 file changed, 20 insertions(+), 19 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index 3b4579e..4350ecf 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -242,7 +242,8 @@ a used buffer to the queue - i.e. lets the driver
>  know by marking the buffer as used. Device can then trigger
>  a device event - i.e. send an interrupt to the driver.
>  
> -For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.

Why are you dropping this?

> +Device reports the number of bytes it has written to memory for
> +each buffer it uses. This is referred to as ``used length''.
>  
>  \input{split-ring.tex}
>  

Otherwise, using the term "used length" looks good.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 04/11] content: replace mentions of len with used length

[Michael S. Tsirkin]

On Tue, Jan 30, 2018 at 11:16:59AM +0100, Cornelia Huck wrote:
> On Tue, 23 Jan 2018 02:01:03 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > Document buffer used len and use that terminology everywhere in the
> > generic section.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex | 39 ++++++++++++++++++++-------------------
> >  1 file changed, 20 insertions(+), 19 deletions(-)
> > 
> > diff --git a/content.tex b/content.tex
> > index 3b4579e..4350ecf 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -242,7 +242,8 @@ a used buffer to the queue - i.e. lets the driver
> >  know by marking the buffer as used. Device can then trigger
> >  a device event - i.e. send an interrupt to the driver.
> >  
> > -For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
> 
> Why are you dropping this?

That chapter follows immediately so the link is not helpful and it just looks weird:
see split virtqueues - split virtqueues. I'll add text to description
to mention this.

> > +Device reports the number of bytes it has written to memory for
> > +each buffer it uses. This is referred to as ``used length''.
> >  
> >  \input{split-ring.tex}
> >  
> 
> Otherwise, using the term "used length" looks good.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 05/11] content: generalize transport ring part naming

[Michael S. Tsirkin]

Replace descriptor table/available ring/used ring
with descriptor area/driver area/device area
in all transports.

Document what's in which area.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex    | 61 ++++++++++++++++++++++++++++++++++++++--------------------
 split-ring.tex |  6 +++---
 2 files changed, 43 insertions(+), 24 deletions(-)

diff --git a/content.tex b/content.tex
index 4350ecf..9fc9673 100644
--- a/content.tex
+++ b/content.tex
@@ -245,6 +245,24 @@ a device event - i.e. send an interrupt to the driver.
 Device reports the number of bytes it has written to memory for
 each buffer it uses. This is referred to as ``used length''.
 
+Each virtqueue can consist of up to 3 parts:
+\begin{itemize}
+\item Descriptor Area - used for describing buffers
+\item Driver Area - extra data supplied by driver to the device
+\item Device Area - extra data supplied by device to driver
+\end{itemize}
+
+\begin{note}
+Note that previous versions of this spec used different names for
+these parts (following \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}):
+\begin{itemize}
+\item Descriptor Table - for the Descriptor Area
+\item Available Ring - for the Driver Area
+\item Used Ring - for the Device Area
+\end{itemize}
+
+\end{note}
+
 \input{split-ring.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
@@ -667,8 +685,8 @@ struct virtio_pci_common_cfg {
         le16 queue_enable;              /* read-write */
         le16 queue_notify_off;          /* read-only for driver */
         le64 queue_desc;                /* read-write */
-        le64 queue_avail;               /* read-write */
-        le64 queue_used;                /* read-write */
+        le64 queue_driver;              /* read-write */
+        le64 queue_device;              /* read-write */
 };
 \end{lstlisting}
 
@@ -728,13 +746,13 @@ struct virtio_pci_common_cfg {
         \end{note}
 
 \item[\field{queue_desc}]
-        The driver writes the physical address of Descriptor Table here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
+        The driver writes the physical address of Descriptor Area here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
 
-\item[\field{queue_avail}]
-        The driver writes the physical address of Available Ring here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
+\item[\field{queue_driver}]
+        The driver writes the physical address of Driver Area here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
 
-\item[\field{queue_used}]
-        The driver writes the physical address of Used Ring here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
+\item[\field{queue_device}]
+        The driver writes the physical address of Device Area here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1496,24 +1514,24 @@ All register values are organized as Little Endian.
     See also p. \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}~\nameref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}.
   }
   \hline 
-  \mmiodreg{QueueDescLow}{QueueDescHigh}{Virtual queue's Descriptor Table 64 bit long physical address}{0x080}{0x084}{W}{%
+  \mmiodreg{QueueDescLow}{QueueDescHigh}{Virtual queue's Descriptor Area 64 bit long physical address}{0x080}{0x084}{W}{%
     Writing to these two registers (lower 32 bits of the address
     to \field{QueueDescLow}, higher 32 bits to \field{QueueDescHigh}) notifies
-    the device about location of the Descriptor Table of the queue
+    the device about location of the Descriptor Area of the queue
     selected by writing to \field{QueueSel} register.
   }
   \hline 
-  \mmiodreg{QueueAvailLow}{QueueAvailHigh}{Virtual queue's Available Ring 64 bit long physical address}{0x090}{0x094}{W}{%
+  \mmiodreg{QueueDriverLow}{QueueDriverHigh}{Virtual queue's Driver Area 64 bit long physical address}{0x090}{0x094}{W}{%
     Writing to these two registers (lower 32 bits of the address
     to \field{QueueAvailLow}, higher 32 bits to \field{QueueAvailHigh}) notifies
-    the device about location of the Available Ring of the queue
+    the device about location of the Driver Area of the queue
     selected by writing to \field{QueueSel}.
   }
   \hline 
-  \mmiodreg{QueueUsedLow}{QueueUsedHigh}{Virtual queue's Used Ring 64 bit long physical address}{0x0a0}{0x0a4}{W}{%
+  \mmiodreg{QueueDeviceLow}{QueueDeviceHigh}{Virtual queue's Device Area 64 bit long physical address}{0x0a0}{0x0a4}{W}{%
     Writing to these two registers (lower 32 bits of the address
     to \field{QueueUsedLow}, higher 32 bits to \field{QueueUsedHigh}) notifies
-    the device about location of the Used Ring of the queue
+    the device about location of the Device Area of the queue
     selected by writing to \field{QueueSel}.
   }
   \hline 
@@ -1631,11 +1649,11 @@ The driver will typically initialize the virtual queue in the following way:
 \item Notify the device about the queue size by writing the size to
    \field{QueueNum}.
 
-\item Write physical addresses of the queue's Descriptor Table,
-   Available Ring and Used Ring to (respectively) the
+\item Write physical addresses of the queue's Descriptor Area,
+   Driver Area and Device Area to (respectively) the
    \field{QueueDescLow}/\field{QueueDescHigh},
-   \field{QueueAvailLow}/\field{QueueAvailHigh} and
-   \field{QueueUsedLow}/\field{QueueUsedHigh} register pairs.
+   \field{QueueDriverLow}/\field{QueueDriverHigh} and
+   \field{QueueDeviceLow}/\field{QueueDeviceHigh} register pairs.
 
 \item Write 0x1 to \field{QueueReady}.
 \end{enumerate}
@@ -2025,13 +2043,14 @@ struct vq_info_block {
         be32 res0;
         be16 index;
         be16 num;
-        be64 avail;
-        be64 used;
+        be64 driver;
+        be64 device;
 };
 \end{lstlisting}
 
-\field{desc}, \field{avail} and \field{used} contain the guest addresses for the descriptor table,
-available ring and used ring for queue \field{index}, respectively. The actual
+\field{desc}, \field{driver} and \field{device} contain the guest
+addresses for the descriptor area,
+available area and used area for queue \field{index}, respectively. The actual
 virtqueue size (number of allocated buffers) is transmitted in \field{num}.
 
 \devicenormative{\paragraph}{Configuring a Virtqueue}{Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue}
diff --git a/split-ring.tex b/split-ring.tex
index 724bb82..acdee7d 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -15,9 +15,9 @@ of the queue.
 Each virtqueue consists of three parts:
 
 \begin{itemize}
-\item Descriptor Table
-\item Available Ring
-\item Used Ring
+\item Descriptor Table - occupies the Descriptor Area
+\item Available Ring - occupies the Driver Area
+\item Used Ring - occupies the Device Area
 \end{itemize}
 
 where each part is physically-contiguous in guest memory,
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 05/11] content: generalize transport ring part naming

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:04 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Replace descriptor table/available ring/used ring
> with descriptor area/driver area/device area
> in all transports.
> 
> Document what's in which area.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex    | 61 ++++++++++++++++++++++++++++++++++++++--------------------
>  split-ring.tex |  6 +++---
>  2 files changed, 43 insertions(+), 24 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 06/11] content: generalize rest of text

[Michael S. Tsirkin]

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/content.tex b/content.tex
index 9fc9673..5634c7d 100644
--- a/content.tex
+++ b/content.tex
@@ -1467,8 +1467,7 @@ All register values are organized as Little Endian.
   }
   \hline 
   \mmioreg{QueueNum}{Virtual queue size}{0x038}{W}{%
-    Queue size is the number of elements in the queue, therefore in each
-    of the Descriptor Table, the Available Ring and the Used Ring.
+    Queue size is the number of elements in the queue.
     Writing to this register notifies the device what size of the
     queue the driver will use. This applies to the queue selected by
     writing to \field{QueueSel}.
@@ -1491,9 +1490,9 @@ All register values are organized as Little Endian.
     caused the device interrupt to be asserted.
     The following events are possible:
     \begin{description}
-      \item[Used Ring Update] - bit 0 - the interrupt was asserted
-        because the device has updated the Used
-        Ring in at least one of the active virtual queues.
+      \item[Used Buffer Update] - bit 0 - the interrupt was asserted
+        because the device has used a buffer
+        in at least one of the active virtual queues.
       \item [Configuration Change] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
     \end{description}
@@ -1642,9 +1641,8 @@ The driver will typically initialize the virtual queue in the following way:
    \field{QueueNumMax}. If the returned value is zero (0x0) the
    queue is not available.
 
-\item Allocate and zero the queue pages, making sure the memory
-   is physically contiguous. It is recommended to align the
-   Used Ring to an optimal boundary (usually the page size).
+\item Allocate and zero the queue memory, making sure the memory
+   is physically contiguous.
 
 \item Notify the device about the queue size by writing the size to
    \field{QueueNum}.
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 06/11] content: generalize rest of text

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:05 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 14 ++++++--------
>  1 file changed, 6 insertions(+), 8 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index 9fc9673..5634c7d 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -1467,8 +1467,7 @@ All register values are organized as Little Endian.
>    }
>    \hline 
>    \mmioreg{QueueNum}{Virtual queue size}{0x038}{W}{%
> -    Queue size is the number of elements in the queue, therefore in each
> -    of the Descriptor Table, the Available Ring and the Used Ring.
> +    Queue size is the number of elements in the queue.
>      Writing to this register notifies the device what size of the
>      queue the driver will use. This applies to the queue selected by
>      writing to \field{QueueSel}.
> @@ -1491,9 +1490,9 @@ All register values are organized as Little Endian.
>      caused the device interrupt to be asserted.
>      The following events are possible:
>      \begin{description}
> -      \item[Used Ring Update] - bit 0 - the interrupt was asserted
> -        because the device has updated the Used
> -        Ring in at least one of the active virtual queues.
> +      \item[Used Buffer Update] - bit 0 - the interrupt was asserted
> +        because the device has used a buffer
> +        in at least one of the active virtual queues.
>        \item [Configuration Change] - bit 1 - the interrupt was
>          asserted because the configuration of the device has changed.
>      \end{description}
> @@ -1642,9 +1641,8 @@ The driver will typically initialize the virtual queue in the following way:
>     \field{QueueNumMax}. If the returned value is zero (0x0) the
>     queue is not available.
>  
> -\item Allocate and zero the queue pages, making sure the memory
> -   is physically contiguous. It is recommended to align the
> -   Used Ring to an optimal boundary (usually the page size).
> +\item Allocate and zero the queue memory, making sure the memory
> +   is physically contiguous.

What about recommendations of that type? Will we want some kind of
"best practice" suggestions (layout and/or transport specific?)

>  
>  \item Notify the device about the queue size by writing the size to
>     \field{QueueNum}.

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 06/11] content: generalize rest of text

[Michael S. Tsirkin]

On Tue, Jan 30, 2018 at 11:31:18AM +0100, Cornelia Huck wrote:
> On Tue, 23 Jan 2018 02:01:05 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex | 14 ++++++--------
> >  1 file changed, 6 insertions(+), 8 deletions(-)
> > 
> > diff --git a/content.tex b/content.tex
> > index 9fc9673..5634c7d 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -1467,8 +1467,7 @@ All register values are organized as Little Endian.
> >    }
> >    \hline 
> >    \mmioreg{QueueNum}{Virtual queue size}{0x038}{W}{%
> > -    Queue size is the number of elements in the queue, therefore in each
> > -    of the Descriptor Table, the Available Ring and the Used Ring.
> > +    Queue size is the number of elements in the queue.
> >      Writing to this register notifies the device what size of the
> >      queue the driver will use. This applies to the queue selected by
> >      writing to \field{QueueSel}.
> > @@ -1491,9 +1490,9 @@ All register values are organized as Little Endian.
> >      caused the device interrupt to be asserted.
> >      The following events are possible:
> >      \begin{description}
> > -      \item[Used Ring Update] - bit 0 - the interrupt was asserted
> > -        because the device has updated the Used
> > -        Ring in at least one of the active virtual queues.
> > +      \item[Used Buffer Update] - bit 0 - the interrupt was asserted
> > +        because the device has used a buffer
> > +        in at least one of the active virtual queues.
> >        \item [Configuration Change] - bit 1 - the interrupt was
> >          asserted because the configuration of the device has changed.
> >      \end{description}
> > @@ -1642,9 +1641,8 @@ The driver will typically initialize the virtual queue in the following way:
> >     \field{QueueNumMax}. If the returned value is zero (0x0) the
> >     queue is not available.
> >  
> > -\item Allocate and zero the queue pages, making sure the memory
> > -   is physically contiguous. It is recommended to align the
> > -   Used Ring to an optimal boundary (usually the page size).
> > +\item Allocate and zero the queue memory, making sure the memory
> > +   is physically contiguous.
> 
> What about recommendations of that type? Will we want some kind of
> "best practice" suggestions (layout and/or transport specific?)

I don't really know what was this in aid of. When is page size an
optimal boundary? I suspect it never is.  I suspect cache line alignment
is a good idea generally but I'm not sure.  Want to write something up?
It's also unlikely to be transport specific.

> >  
> >  \item Notify the device about the queue size by writing the size to
> >     \field{QueueNum}.
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 07/11] split-ring: generalize text

[Michael S. Tsirkin]

Update generic text to talk about available/used buffers, not rings.
Move some split-ring specific text to the correct section.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex    | 12 +++++-------
 split-ring.tex |  4 ++++
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/content.tex b/content.tex
index 5634c7d..0f7c2b9 100644
--- a/content.tex
+++ b/content.tex
@@ -381,12 +381,10 @@ of a device are live once the device has been reset.
 
 \drivernormative{\subsection}{Device Cleanup}{General Initialization And Device Operation / Device Cleanup}
 
-A driver MUST NOT alter descriptor table entries which have been
-exposed in the available ring (and not marked consumed by the device
-in the used ring) of a live virtqueue.
-
-A driver MUST NOT decrement the available \field{idx} on a live virtqueue (ie.
-there is no way to ``unexpose'' buffers).
+A driver MUST NOT alter virtqueue entries for exposed buffers -
+i.e. buffers which have been
+made available to the device (and not been used by the device)
+of a live virtqueue.
 
 Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
 
@@ -4652,7 +4650,7 @@ Interface: Device Operation}
 When using the legacy interface, the driver SHOULD ignore the
 used length values.
 \begin{note}
-Historically, devices put the total descriptor length,
+Historically, devices put the total length,
 or the total length of device-writable buffers there,
 even when only part of the buffers were actually written.
 \end{note}
diff --git a/split-ring.tex b/split-ring.tex
index acdee7d..f976e45 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -296,6 +296,10 @@ referred to this structure as vring_avail, and the constant as
 VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
 \end{note}
 
+\drivernormative{\subsubsection}{The Virtqueue Available Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
+A driver MUST NOT decrement the available \field{idx} on a virtqueue (ie.
+there is no way to ``unexpose'' buffers).
+
 \subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
 
 If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 07/11] split-ring: generalize text

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:06 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Update generic text to talk about available/used buffers, not rings.
> Move some split-ring specific text to the correct section.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex    | 12 +++++-------
>  split-ring.tex |  4 ++++
>  2 files changed, 9 insertions(+), 7 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index 5634c7d..0f7c2b9 100644
> --- a/content.tex
> +++ b/content.tex

> @@ -4652,7 +4650,7 @@ Interface: Device Operation}
>  When using the legacy interface, the driver SHOULD ignore the
>  used length values.
>  \begin{note}
> -Historically, devices put the total descriptor length,
> +Historically, devices put the total length,
>  or the total length of device-writable buffers there,
>  even when only part of the buffers were actually written.

This sentence reads a bit weirdly now. Total length of what?

>  \end{note}
> diff --git a/split-ring.tex b/split-ring.tex
> index acdee7d..f976e45 100644
> --- a/split-ring.tex
> +++ b/split-ring.tex
> @@ -296,6 +296,10 @@ referred to this structure as vring_avail, and the constant as
>  VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
>  \end{note}
>  
> +\drivernormative{\subsubsection}{The Virtqueue Available Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
> +A driver MUST NOT decrement the available \field{idx} on a virtqueue (ie.
> +there is no way to ``unexpose'' buffers).

Don't you need to reference that drivernormative section?

(It is a bit confusing with all those interim changes; maybe I should
only review the end result...)

> +
>  \subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
>  
>  If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 07/11] split-ring: generalize text

[Michael S. Tsirkin]

On Tue, Jan 30, 2018 at 11:45:35AM +0100, Cornelia Huck wrote:
> On Tue, 23 Jan 2018 02:01:06 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > Update generic text to talk about available/used buffers, not rings.
> > Move some split-ring specific text to the correct section.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex    | 12 +++++-------
> >  split-ring.tex |  4 ++++
> >  2 files changed, 9 insertions(+), 7 deletions(-)
> > 
> > diff --git a/content.tex b/content.tex
> > index 5634c7d..0f7c2b9 100644
> > --- a/content.tex
> > +++ b/content.tex
> 
> > @@ -4652,7 +4650,7 @@ Interface: Device Operation}
> >  When using the legacy interface, the driver SHOULD ignore the
> >  used length values.
> >  \begin{note}
> > -Historically, devices put the total descriptor length,
> > +Historically, devices put the total length,
> >  or the total length of device-writable buffers there,
> >  even when only part of the buffers were actually written.
> 
> This sentence reads a bit weirdly now. Total length of what?

Of readable and writeable buffers.

> >  \end{note}
> > diff --git a/split-ring.tex b/split-ring.tex
> > index acdee7d..f976e45 100644
> > --- a/split-ring.tex
> > +++ b/split-ring.tex
> > @@ -296,6 +296,10 @@ referred to this structure as vring_avail, and the constant as
> >  VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
> >  \end{note}
> >  
> > +\drivernormative{\subsubsection}{The Virtqueue Available Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
> > +A driver MUST NOT decrement the available \field{idx} on a virtqueue (ie.
> > +there is no way to ``unexpose'' buffers).
> 
> Don't you need to reference that drivernormative section?

Right, I forgot that part. Will do.

> 
> (It is a bit confusing with all those interim changes; maybe I should
> only review the end result...)


> > +
> >  \subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
> >  
> >  If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

Performance analysis of this is in my kvm forum 2016 presentation.  The
idea is to have a r/w descriptor in a ring structure, replacing the used
and available ring, index and descriptor buffer.

This is also easier for devices to implement than the 1.0 layout.
Several more enhancements will be necessary to actually make this
efficient for devices to use.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex     |  25 ++-
 packed-ring.tex | 678 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 700 insertions(+), 3 deletions(-)
 create mode 100644 packed-ring.tex

diff --git a/content.tex b/content.tex
index 0f7c2b9..4d522cc 100644
--- a/content.tex
+++ b/content.tex
@@ -263,8 +263,17 @@ these parts (following \ref{sec:Basic Facilities of a Virtio Device / Split Virt
 
 \end{note}
 
+Two formats are supported: Split Virtqueues (see \ref{sec:Basic
+Facilities of a Virtio Device / Split
+Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
+Split Virtqueues}) and Packed Virtqueues (see \ref{sec:Basic
+Facilities of a Virtio Device / Packed
+Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
+Packed Virtqueues}).
+
 \input{split-ring.tex}
 
+\input{packed-ring.tex}
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -5215,10 +5224,15 @@ Currently these device-independent feature bits defined:
 \begin{description}
   \item[VIRTIO_F_RING_INDIRECT_DESC (28)] Negotiating this feature indicates
   that the driver can use descriptors with the VIRTQ_DESC_F_INDIRECT
-  flag set, as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}.
-
+  flag set, as described in \ref{sec:Basic Facilities of a Virtio
+Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
+Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device /
+Virtqueues / The Virtqueue Descriptor Table / Indirect
+Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}~\nameref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}.
   \item[VIRTIO_F_RING_EVENT_IDX(29)] This feature enables the \field{used_event}
-  and the \field{avail_event} fields as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression} and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}.
+  and the \field{avail_event} fields as described in
+\ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}, \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring} and \ref{sec:Packed Virtqueues / Driver and Device Event Suppression}.
+
 
   \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
     specification, giving a simple way to detect legacy devices or drivers.
@@ -5228,6 +5242,9 @@ Currently these device-independent feature bits defined:
   addresses in memory.  If this feature bit is set to 0, then the device emits
   physical addresses which are not translated further, even though an IOMMU
   may be present.
+  \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
+  support for the packed virtqueue layout as described in
+  \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5241,6 +5258,8 @@ passed to the device into physical addresses in memory.  If
 VIRTIO_F_IOMMU_PLATFORM is not offered, then a driver MUST pass only physical
 addresses to the device.
 
+A driver SHOULD accept VIRTIO_F_PACKED_RING if it is offered.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
diff --git a/packed-ring.tex b/packed-ring.tex
new file mode 100644
index 0000000..b6cb979
--- /dev/null
+++ b/packed-ring.tex
@@ -0,0 +1,678 @@
+\section{Packed Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}
+
+Packed virtqueues is an alternative compact virtqueue layout using
+read-write memory, that is memory that is both read and written
+by both host and guest.
+
+Use of packed virtqueues is enabled by the VIRTIO_F_PACKED_RING
+feature bit.
+
+Packed virtqueues support up to $2^{15}$ entries each.
+
+With current transports, virtqueues are located in guest memory
+allocated by driver.
+Each packed virtqueue consists of three parts:
+
+\begin{itemize}
+\item Descriptor Ring - occupies the Descriptor Area
+\item Driver Event Suppression - occupies the Driver Area
+\item Device Event Suppression - occupies the Device Area
+\end{itemize}
+
+Where Descriptor Ring in turn consists of descriptors,
+and where each descriptor can contain the following parts:
+
+\begin{itemize}
+\item Buffer ID
+\item Buffer Address
+\item Buffer Length
+\item Flags
+\end{itemize}
+
+A buffer consists of zero or more device-readable physically-contiguous
+elements followed by zero or more physically-contiguous
+device-writable elements (each buffer has at least one element).
+
+When the driver wants to send such a buffer to the device, it
+writes at least one available descriptor describing elements of
+the buffer into the Descriptor Ring.  The descriptor(s) are
+associated with a buffer by means of a Buffer ID stored within
+the descriptor.
+
+Driver then notifies the device. When the device has finished
+processing the buffer, it writes a used device descriptor
+including the Buffer ID into the Descriptor Ring (overwriting a
+driver descriptor previously made available), and sends an
+interrupt.
+
+Descriptor Ring is used in a circular manner: driver writes
+descriptors into the ring in order. After reaching end of ring,
+the next descriptor is placed at head of the ring.  Once ring is
+full of driver descriptors, driver stops sending new requests and
+waits for device to start processing descriptors and to write out
+some used descriptors before making new driver descriptors
+available.
+
+Similarly, device reads descriptors from the ring in order and
+detects that a driver descriptor has been made available.  As
+processing of descriptors is completed used descriptors are
+written by the device back into the ring.
+
+Note: after reading driver descriptors and starting their
+processing in order, device might complete their processing out
+of order.  Used device descriptors are written in the order
+in which their processing is complete.
+
+Device Event Suppression data structure is write-only by the
+device. It includes information for reducing the number of
+device events - i.e. driver notifications to device.
+
+Driver Event Suppression data structure is read-only by the
+device. It includes information for reducing the number of
+driver events - i.e. device interrupts to driver.
+
+\subsection{Available and Used Ring Wrap Counters}
+\label{sec:Packed Virtqueues / Available and Used Ring Wrap Counters}
+Each of the driver and the device are expected to maintain,
+internally, a single-bit ring wrap counter initialized to 1.
+
+The counter maintained by the driver is called the Available
+Ring Wrap Counter. Driver changes the value of this counter
+each time it makes available the
+last descriptor in the ring (after making the last descriptor
+available).
+
+The counter maintained by the device is called the Used Ring Wrap
+Counter.  Device changes the value of this counter
+each time it uses the last descriptor in
+the ring (after marking the last descriptor used).
+
+It is easy to see that the Available Ring Wrap Counter in the driver matches
+the Used Ring Wrap Counter in the device when both are processing the same
+descriptor, or when all available descriptors have been used.
+
+To mark a descriptor as available and used, both driver and
+device use the following two flags:
+\begin{lstlisting}
+#define VIRTQ_DESC_F_AVAIL     7
+#define VIRTQ_DESC_F_USED      15
+\end{lstlisting}
+
+To mark a descriptor as available, driver sets the
+VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Available
+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_USED bit to match the
+\emph{inverse} value.
+
+To mark a descriptor as used, device sets the
+VIRTQ_DESC_F_USED bit in Flags to match the internal Used
+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_AVAIL bit to match the
+\emph{same} value.
+
+Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different
+for an available descriptor and equal for a used descriptor.
+
+\subsection{Polling of available and used descriptors}
+\label{sec:Packed Virtqueues / Polling of available and used descriptors}
+
+Writes of device and driver descriptors can generally be
+reordered, but each side (driver and device) are only required to
+poll (or test) a single location in memory: next device descriptor after
+the one they processed previously, in circular order.
+
+Sometimes device needs to only write out a single used descriptor
+after processing a batch of multiple available descriptors.  As
+described in more detail below, this can happen when using
+descriptor chaining or with in-order
+use of descriptors.  In this case, device writes out a used
+descriptor with buffer id of the last descriptor in the group.
+After processing the used descriptor, both device and driver then
+skip forward in the ring the number of the remaining descriptors
+in the group until processing (reading for the driver and writing
+for the device) the next used descriptor.
+
+\subsection{Write Flag}
+\label{sec:Packed Virtqueues / Write Flag}
+
+In an available descriptor, VIRTQ_DESC_F_WRITE bit within Flags
+is used to mark a descriptor as corresponding to a write-only or
+read-only element of a buffer.
+
+\begin{lstlisting}
+/* This marks a buffer as device write-only (otherwise device read-only). */
+#define VIRTQ_DESC_F_WRITE     2
+\end{lstlisting}
+
+In a used descriptor, this bit it used to specify whether any
+data has been written by the device into any parts of the buffer.
+
+
+\subsection{Buffer Address and Length}
+\label{sec:Packed Virtqueues / Buffer Address and Length}
+
+In an available descriptor, Buffer Address corresponds to the
+physical address of the buffer. The length of the buffer assumed
+to be physically contigious is stored in Buffer Length.
+
+In a used descriptor, Buffer Address is unused. Buffer Length
+specifies the length of the buffer that has been initialized
+(written to) by the device.
+
+Buffer length is reserved for used descriptors without the
+VIRTQ_DESC_F_WRITE flag, and is ignored by drivers.
+
+\subsection{Scatter-Gather Support}
+\label{sec:Packed Virtqueues / Scatter-Gather Support}
+
+Some drivers need an ability to supply a list of multiple buffer
+elements (also known as a scatter/gather list) with a request.
+Two optional features support this: descriptor
+chaining and indirect descriptors.
+
+If neither feature has been negotiated, each buffer is
+physically-contigious, either read-only or write-only and is
+described completely by a single descriptor.
+
+While unusual (most implementations either create all lists
+solely using non-indirect descriptors, or always use a single
+indirect element), if both features have been negotiated, mixing
+direct and direct descriptors in a ring is valid, as long as each
+list only contains descriptors of a given type.
+
+Scatter/gather lists only apply to available descriptors. A
+single used descriptor corresponds to the whole list.
+
+The device limits the number of descriptors in a list through a
+transport-specific and/or device-specific value. If not limited,
+the maximum number of descriptors in a list is the virt queue
+size.
+
+\subsection{Next Flag: Descriptor Chaining}
+\label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
+
+The VIRTIO_F_LIST_DESC feature allows driver to supply
+a scatter/gather list to the device
+by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT in
+Flags for all but the last available descriptor.
+
+\begin{lstlisting}
+/* This marks a buffer as continuing. */
+#define VIRTQ_DESC_F_NEXT   1
+\end{lstlisting}
+
+Buffer ID is included in the last descriptor in the list.
+
+The driver always makes the the first descriptor in the list
+available after the rest of the list has been written out into
+the ring. This guarantees that the device will never observe a
+partial scatter/gather list in the ring.
+
+Device only writes out a single used descriptor for the whole
+list. It then skips forward according to the number of
+descriptors in the list. Driver needs to keep track of the size
+of the list corresponding to each buffer ID, to be able to skip
+to where the next used descriptor is written by the device.
+
+For example, if descriptors are used in the same order in which
+they are made available, this will result in the used descriptor
+overwriting the first available descriptor in the list, the used
+descriptor for the next list overwriting the first available
+descriptor in the next list, etc.
+
+VIRTQ_DESC_F_NEXT is reserved in used descriptors, and
+should be ignored by drivers.
+
+\subsection{Indirect Flag: Scatter-Gather Support}
+\label{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
+
+Some devices benefit by concurrently dispatching a large number
+of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this. To increase
+ring capacity the driver can store a (read-only by the device) table of indirect
+descriptors anywhere in memory, and insert a descriptor in main
+virtqueue (with \field{Flags} bit VIRTQ_DESC_F_INDIRECT on) that refers to
+a memory buffer
+containing this indirect descriptor table; \field{addr} and \field{len}
+refer to the indirect table address and length in bytes,
+respectively.
+\begin{lstlisting}
+/* This means the buffer contains a table of buffer descriptors. */
+#define VIRTQ_DESC_F_INDIRECT   4
+\end{lstlisting}
+
+The indirect table layout structure looks like this
+(\field{len} is the Buffer Length of the descriptor that refers to this table,
+which is a variable, so this code won't compile):
+
+\begin{lstlisting}
+struct indirect_descriptor_table {
+        /* The actual descriptor structures (struct Desc each) */
+        struct Desc desc[len / sizeof(struct Desc)];
+};
+\end{lstlisting}
+
+The first descriptor is located at start of the indirect
+descriptor table, additional indirect descriptors come
+immediately afterwards. \field{Flags} bit VIRTQ_DESC_F_WRITE is the
+only valid flag for descriptors in the indirect table. Others
+are reserved and are ignored by the device.
+Buffer ID is also reserved and is ignored by the device.
+
+In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
+is reserved and is ignored by the device.
+
+\subsection{Multi-buffer requests}
+\label{sec:Packed Virtqueues / Multi-descriptor batches}
+Some devices combine multiple buffers as part of processing of a
+single request.  These devices always make the first
+descriptor in the request available after the rest of the request
+has been written out request the ring. This guarantees that the
+driver will never observe a partial request in the ring.
+
+
+\subsection{Driver and Device Event Suppression}
+\label{sec:Packed Virtqueues / Driver and Device Event Suppression}
+In many systems driver and device notifications involve
+significant overhead. To mitigate this overhead,
+each virtqueue includes two identical structures used for
+controlling notifications between device and driver.
+
+Driver Event Suppression structure is read-only by the
+device and controls the events sent by the device
+to the driver (e.g. interrupts).
+
+Device Event Suppression structure is read-only by
+the driver and controls the events sent by the driver
+to the device (e.g. IO).
+
+Each of these Event Suppression structures controls
+both Descriptor Ring events and structure events, and
+each includes the following fields:
+
+\begin{description}
+\item [Descriptor Ring Change Event Flags] Takes values:
+\begin{itemize}
+\item 00b enable events
+\item 01b disable events
+\item 10b enable events for a specific descriptor
+(as specified by Descriptor Ring Change Event Offset/Wrap Counter).
+Only valid if VIRTIO_F_RING_EVENT_IDX has been negotiated.
+\item 11b reserved
+\end{itemize}
+\item [Descriptor Ring Change Event Offset] If Event Flags set to descriptor
+specific event: offset within the ring (in units of descriptor
+size). Event will only trigger when this descriptor is
+made available/used respectively.
+\item [Descriptor Ring Change Event Wrap Counter] If Event Flags set to descriptor
+specific event: offset within the ring (in units of descriptor
+size). Event will only trigger when Ring Wrap Counter
+matches this value and a descriptor is
+made available/used respectively.
+\end{description}
+
+After writing out some descriptors, both device and driver
+are expected to consult the relevant structure to find out
+whether interrupt/notification should be sent.
+
+\subsubsection{Driver notifications}
+\label{sec:Packed Virtqueues / Driver notifications}
+Whenever not suppressed by Device Event Suppression,
+driver is required to notify the device after
+making changes to the virtqueue.
+
+Some devices benefit from ability to find out the number of
+available descriptors in the ring, and whether to send
+interrupts to drivers without accessing virtqueue in memory:
+for efficiency or as a debugging aid.
+
+To help with these optimizations, driver notifications
+to the device include the following information:
+
+\begin{itemize}
+\item VQ number
+\item Offset (in units of descriptor size) within the ring
+      where the next available descriptor will be written
+\item Wrap Counter referring to the next available
+      descriptor
+\end{itemize}
+
+Note that driver can trigger multiple notifications even without
+making any more changes to the ring. These would then have
+identical \field{Offset} and \field{Wrap Counter} values.
+
+\subsubsection{Structure Size and Alignment}
+\label{sec:Packed Virtqueues / Structure Size and Alignment}
+
+Each part of the virtqueue is physically-contiguous in guest memory,
+and has different alignment requirements.
+
+The memory aligment and size requirements, in bytes, of each part of the
+virtqueue are summarized in the following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Virtqueue Part    & Alignment & Size \\
+\hline \hline
+Descriptor Ring  & 16        & $16 * $(Queue Size) \\
+\hline
+Device Event Suppression    & 4         & 4 \\
+ \hline
+Driver Event Suppression         & 4         & 4 \\
+ \hline
+\end{tabular}
+
+The Alignment column gives the minimum alignment for each part
+of the virtqueue.
+
+The Size column gives the total number of bytes for each
+part of the virtqueue.
+
+Queue Size corresponds to the maximum number of descriptors in the
+virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
+can be queued at any given time.}.  Queue Size value does not
+have to be a power of 2 unless enforced by the transport.
+
+\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a
+Virtio Device / Packed Virtqueues}
+The driver MUST ensure that the physical address of the first byte
+of each virtqueue part is a multiple of the specified alignment value
+in the above table.
+
+\devicenormative{\subsection}{Virtqueues}{Basic Facilities of a
+Virtio Device / Packed Virtqueues}
+The device MUST start processing driver descriptors in the order
+in which they appear in the ring.
+The device MUST start writing device descriptors into the ring in
+the order in which they complete.
+Device MAY reorder descriptor writes once they are started.
+
+\subsection{The Virtqueue Descriptor Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue
+Descriptor Format}
+
+The available descriptor refers to the buffers the driver is sending
+to the device. \field{addr} is a physical address, and the
+descriptor is identified with a buffer using the \field{id} field.
+
+\begin{lstlisting}
+struct virtq_desc {
+        /* Buffer Address. */
+        le64 addr;
+        /* Buffer Length. */
+        le32 len;
+        /* Buffer ID. */
+        le16 id;
+        /* The flags depending on descriptor type. */
+        le16 flags;
+};
+\end{lstlisting}
+
+The descriptor ring is zero-initialized.
+
+\subsection{Event Suppression Structure Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event Suppression Structure
+Format}
+
+The following structure is used to reduce the number of
+notifications sent between driver and device.
+
+\begin{lstlisting}
+__le16 desc_event_off : 15; /* Descriptor Event Offset */
+int    desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
+__le16 desc_event_flags : 2; /* Descriptor Event Flags */
+\end{lstlisting}
+
+\subsection{Driver Notification Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
+
+The following structure is used to notify device of
+device events - i.e. available descriptors:
+
+\begin{lstlisting}
+__le16 vqn;
+__le16 next_off : 15;
+int    next_wrap : 1;
+\end{lstlisting}
+
+\devicenormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table}
+A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
+read a device-writable buffer.
+A device MUST NOT use a descriptor unless it observes
+VIRTQ_DESC_F_AVAIL bit in its \field{flags} being changed.
+A device MUST NOT change a descriptor after changing it's
+VIRTQ_DESC_F_USED bit in its \field{flags}.
+
+\drivernormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / PAcked Virtqueues / The Virtqueue Descriptor Table}
+A driver MUST NOT change a descriptor unless it observes
+VIRTQ_DESC_F_USED bit in its \field{flags} being changed.
+A driver MUST NOT change a descriptor after changing
+VIRTQ_DESC_F_USED bit in its \field{flags}.
+When notifying the device, driver MUST set
+\field{next_off} and
+\field{next_wrap} to match the next descriptor
+not yet made available to the device.
+A driver MAY send multiple notifications without making
+any new descriptors available to the device.
+
+\drivernormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
+Virtio Device / Packed Virtqueues / Scatter-Gather Support}
+A driver MUST NOT create a descriptor list longer than allowed
+by the device.
+
+A driver MUST NOT create a descriptor list longer than the Queue
+Size.
+
+This implies that loops in the descriptor list are forbidden!
+
+The driver MUST place any device-writable descriptor elements after
+any device-readable descriptor elements.
+
+A driver MUST NOT depend on the device to use more descriptors
+to be able to write out all descriptors in a list. A driver
+MUST make sure there's enough space in the ring
+for the whole list before making the first descriptor in the list
+available to the device.
+
+A driver MUST NOT make the first descriptor in the list
+available before initializing the rest of the descriptors.
+
+\devicenormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
+Virtio Device / Packed Virtqueues / Scatter-Gather Support}
+The device MUST use descriptors in a list chained by the
+VIRTQ_DESC_F_NEXT flag in the same order that they
+were made available by the driver.
+
+The device MAY limit the number of buffers it will allow in a
+list.
+
+\drivernormative{\subsection}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The driver MUST NOT set the DESC_F_INDIRECT flag unless the
+VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
+set any flags except DESC_F_WRITE within an indirect descriptor.
+
+A driver MUST NOT create a descriptor chain longer than allowed
+by the device.
+
+A driver MUST NOT write direct descriptors with
+DESC_F_INDIRECT set in a scatter-gather list linked by
+VIRTQ_DESC_F_NEXT.
+\field{flags}.
+
+\subsection{Virtqueue Operation}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Virtqueue Operation}
+
+There are two parts to virtqueue operation: supplying new
+available buffers to the device, and processing used buffers from
+the device.
+
+What follows is the requirements of each of these two parts
+when using the packed virtqueue format in more detail.
+
+\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device}
+
+The driver offers buffers to one of the device's virtqueues as follows:
+
+\begin{enumerate}
+\item The driver places the buffer into free descriptor in the Descriptor Ring.
+
+\item The driver performs a suitable memory barrier to ensure that it updates
+  the descriptor(s) before checking for notification suppression.
+
+\item If notifications are not suppressed, the driver notifies the device
+    of the new available buffers.
+\end{enumerate}
+
+What follows is the requirements of each stage in more detail.
+
+\subsubsection{Placing Available Buffers Into The Descriptor Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Available Buffers Into The Descriptor Ring}
+
+For each buffer element, b:
+
+\begin{enumerate}
+\item Get the next descriptor table entry, d
+\item Get the next free buffer id value
+\item Set \field{d.addr} to the physical address of the start of b
+\item Set \field{d.len} to the length of b.
+\item Set \field{d.id} to the buffer id
+\item Calculate the flags as follows:
+\begin{enumerate}
+\item If b is device-writable, set the VIRTQ_DESC_F_WRITE bit to 1, otherwise 0
+\item Set VIRTQ_DESC_F_AVAIL bit to the current value of the Available Ring Wrap Counter
+\item Set VIRTQ_DESC_F_USED bit to inverse value
+\end{enumerate}
+\item Perform a memory barrier to ensure that the descriptor has
+      been initialized
+\item Set \field{d.flags} to the calculated flags value
+\item If d is the last descriptor in the ring, toggle the
+      Available Ring Wrap Counter
+\item Otherwise, increment d to point at the next descriptor
+\end{enumerate}
+
+This makes a single descriptor buffer available. However, in
+general the driver MAY make use of a batch of descriptors as part
+of a single request. In that case, it defers updating
+the descriptor flags for the first descriptor
+(and the previous memory barrier) until after the rest of
+the descriptors have been initialized.
+
+Once the descriptor \field{flags} is updated by the driver, this exposes the
+descriptor and its contents.  The device MAY
+access the descriptor and any following descriptors the driver created and the
+memory they refer to immediately.
+
+\drivernormative{\paragraph}{Updating flags}{Basic Facilities of
+a Virtio Device / Packed Virtqueues / Supplying Buffers to The
+Device / Updating flags}
+The driver MUST perform a suitable memory barrier before the
+\field{flags} update, to ensure the
+device sees the most up-to-date copy.
+
+\subsubsection{Notifying The Device}\label{sec:Basic Facilities
+of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+
+The actual method of device notification is bus-specific, but generally
+it can be expensive.  So the device MAY suppress such notifications if it
+doesn't need them, using the Driver Event Suppression structure
+as detailed in section \ref{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event
+Suppression Structure Format}.
+
+The driver has to be careful to expose the new \field{flags}
+value before checking if notifications are suppressed.
+
+\subsubsection{Implementation Example}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Implementation Example}
+
+Below is an example driver code. It does not attempt to reduce
+the number of device interrupts, neither does it support
+the VIRTIO_F_RING_EVENT_IDX feature.
+
+\begin{lstlisting}
+
+first = vq->next_avail;
+id = alloc_id(vq);
+
+for (each buffer element b) {
+        vq->desc[vq->next_avail].address = get_addr(b);
+        vq->desc[vq->next_avail].len = get_len(b);
+        init_desc(vq->next_avail, b);
+        avail = vq->avail_wrap_count;
+        used = !vq->avail_wrap_count;
+        f = get_flags(b) | (avail << VIRTQ_DESC_F_AVAIL) | (used << VIRTQ_DESC_F_USED);
+	/* Don't mark the 1st descriptor available until all of them are ready. */
+        if (vq->next_avail == first) {
+                flags = f;
+        } else {
+                vq->desc[vq->next_avail].flags = f;
+        }
+
+	vq->next_avail++;
+
+	if (vq->next_avail > vq->size) {
+		vq->next_avail = 0;
+		vq->avail_wrap_count \^= 1;
+	}
+
+
+}
+vq->desc[vq->next_avail].id = id;
+write_memory_barrier();
+vq->desc[first].flags = flags;
+
+memory_barrier();
+
+if (vq->device_event.flags != 0x2) {
+        notify_device(vq, vq->next_avail, vq->avail_wrap_count);
+}
+
+\end{lstlisting}
+
+
+\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+The driver MUST perform a suitable memory barrier before reading
+the Driver Event Suppression structure, to avoid missing a notification.
+
+\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Receiving Used Buffers From The Device}
+
+Once the device has used buffers referred to by a descriptor (read from or written to them, or
+parts of both, depending on the nature of the virtqueue and the
+device), it interrupts the driver
+as detailed in section \ref{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event
+Suppression Structure Format}.
+
+\begin{note}
+For optimal performance, a driver MAY disable interrupts while processing
+the used buffers, but beware the problem of missing interrupts between
+emptying the ring and reenabling interrupts.  This is usually handled by
+re-checking for more used buffers after interrups are re-enabled:
+\end{note}
+
+\begin{lstlisting}
+vq->driver_event.flags = 0x2;
+
+for (;;) {
+        struct virtq_desc *d = vq->desc[vq->next_used];
+
+        flags = d->flags;
+        bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
+        bool used = flags & (1 << VIRTQ_DESC_F_USED);
+
+        if (avail != used) {
+                vq->driver_event.flags = 0x1;
+                memory_barrier();
+
+                flags = d->flags;
+                bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
+                bool used = flags & (1 << VIRTQ_DESC_F_USED);
+                if (avail != used) {
+                        break;
+                }
+
+                vq->driver_event.flags = 0x2;
+        }
+
+	read_memory_barrier();
+        process_buffer(d);
+        vq->next_used++;
+        if (vq->next_used > vq->size) {
+                vq->next_used = 0;
+        }
+}
+\end{lstlisting}
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Tiwei Bie]

On Tue, Jan 23, 2018 at 02:01:07AM +0200, Michael S. Tsirkin wrote:
[...]
> +\subsection{Available and Used Ring Wrap Counters}
> +\label{sec:Packed Virtqueues / Available and Used Ring Wrap Counters}
> +Each of the driver and the device are expected to maintain,
> +internally, a single-bit ring wrap counter initialized to 1.
> +
> +The counter maintained by the driver is called the Available
> +Ring Wrap Counter. Driver changes the value of this counter
> +each time it makes available the
> +last descriptor in the ring (after making the last descriptor
> +available).
> +
> +The counter maintained by the device is called the Used Ring Wrap
> +Counter.  Device changes the value of this counter
> +each time it uses the last descriptor in
> +the ring (after marking the last descriptor used).
> +
> +It is easy to see that the Available Ring Wrap Counter in the driver matches
> +the Used Ring Wrap Counter in the device when both are processing the same
> +descriptor, or when all available descriptors have been used.
> +
> +To mark a descriptor as available and used, both driver and
> +device use the following two flags:
> +\begin{lstlisting}
> +#define VIRTQ_DESC_F_AVAIL     7
> +#define VIRTQ_DESC_F_USED      15

I think above two definitions are not consistent with
below definitions:

#define VIRTQ_DESC_F_NEXT       1
#define VIRTQ_DESC_F_WRITE      2
#define VIRTQ_DESC_F_INDIRECT   4

Maybe it'd be better to define them as:

#define VIRTQ_DESC_F_AVAIL(b)     (b << 7)
#define VIRTQ_DESC_F_USED(b)      (b << 15)

> +\end{lstlisting}
> +
> +To mark a descriptor as available, driver sets the
> +VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Available
> +Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_USED bit to match the
> +\emph{inverse} value.
> +
> +To mark a descriptor as used, device sets the
> +VIRTQ_DESC_F_USED bit in Flags to match the internal Used
> +Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_AVAIL bit to match the
> +\emph{same} value.
> +
> +Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different
> +for an available descriptor and equal for a used descriptor.
> +
> +\subsection{Polling of available and used descriptors}
> +\label{sec:Packed Virtqueues / Polling of available and used descriptors}
> +
> +Writes of device and driver descriptors can generally be
> +reordered, but each side (driver and device) are only required to
> +poll (or test) a single location in memory: next device descriptor after
> +the one they processed previously, in circular order.
> +
> +Sometimes device needs to only write out a single used descriptor
> +after processing a batch of multiple available descriptors.  As
> +described in more detail below, this can happen when using
> +descriptor chaining or with in-order
> +use of descriptors.  In this case, device writes out a used
> +descriptor with buffer id of the last descriptor in the group.
> +After processing the used descriptor, both device and driver then
> +skip forward in the ring the number of the remaining descriptors
> +in the group until processing (reading for the driver and writing
> +for the device) the next used descriptor.
> +
> +\subsection{Write Flag}
> +\label{sec:Packed Virtqueues / Write Flag}
> +
> +In an available descriptor, VIRTQ_DESC_F_WRITE bit within Flags
> +is used to mark a descriptor as corresponding to a write-only or
> +read-only element of a buffer.
> +
> +\begin{lstlisting}
> +/* This marks a buffer as device write-only (otherwise device read-only). */
> +#define VIRTQ_DESC_F_WRITE     2
> +\end{lstlisting}
> +
> +In a used descriptor, this bit it used to specify whether any

Typo: s/it used/is used/

> +data has been written by the device into any parts of the buffer.
[...]
> +\subsection{Next Flag: Descriptor Chaining}
> +\label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
> +
> +The VIRTIO_F_LIST_DESC feature allows driver to supply
> +a scatter/gather list to the device
> +by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT in
> +Flags for all but the last available descriptor.
> +
> +\begin{lstlisting}
> +/* This marks a buffer as continuing. */
> +#define VIRTQ_DESC_F_NEXT   1
> +\end{lstlisting}
> +
> +Buffer ID is included in the last descriptor in the list.
> +
> +The driver always makes the the first descriptor in the list
> +available

Typo: s/the the/the/

The driver always and only makes the first descriptor in the list
available?

If my above understanding is correct (i.e. only the
first desc in a desc list will be made available by
driver -- update the VIRTQ_DESC_F_AVAIL/USED flags),
I think it would be better to make the description
more explicit.

Besides, does driver just need to set or clear the
VIRTQ_DESC_F_WRITE bit for the first desc in a list?

> after the rest of the list has been written out into
> +the ring. This guarantees that the device will never observe a
> +partial scatter/gather list in the ring.
> +
> +Device only writes out a single used descriptor for the whole
> +list. It then skips forward according to the number of
> +descriptors in the list. Driver needs to keep track of the size
> +of the list corresponding to each buffer ID, to be able to skip
> +to where the next used descriptor is written by the device.
> +
> +For example, if descriptors are used in the same order in which
> +they are made available, this will result in the used descriptor
> +overwriting the first available descriptor in the list, the used
> +descriptor for the next list overwriting the first available
> +descriptor in the next list, etc.
> +
> +VIRTQ_DESC_F_NEXT is reserved in used descriptors, and
> +should be ignored by drivers.
> +
[...]
> +\subsubsection{Implementation Example}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Implementation Example}
> +
> +Below is an example driver code. It does not attempt to reduce
> +the number of device interrupts, neither does it support
> +the VIRTIO_F_RING_EVENT_IDX feature.
> +
> +\begin{lstlisting}
> +
> +first = vq->next_avail;
> +id = alloc_id(vq);
> +
> +for (each buffer element b) {
> +        vq->desc[vq->next_avail].address = get_addr(b);
> +        vq->desc[vq->next_avail].len = get_len(b);
> +        init_desc(vq->next_avail, b);
> +        avail = vq->avail_wrap_count;
> +        used = !vq->avail_wrap_count;
> +        f = get_flags(b) | (avail << VIRTQ_DESC_F_AVAIL) | (used << VIRTQ_DESC_F_USED);
> +	/* Don't mark the 1st descriptor available until all of them are ready. */
> +        if (vq->next_avail == first) {
> +                flags = f;
> +        } else {
> +                vq->desc[vq->next_avail].flags = f;
> +        }
> +
> +	vq->next_avail++;
> +
> +	if (vq->next_avail > vq->size) {

Typo: vq->next_avail >= vq->size

> +		vq->next_avail = 0;
> +		vq->avail_wrap_count \^= 1;
> +	}
> +
> +
> +}
> +vq->desc[vq->next_avail].id = id;

I think if we need to update desc.id then the desc.id
needs to be updated for each desc in the loop.

> +write_memory_barrier();
> +vq->desc[first].flags = flags;
> +
> +memory_barrier();
> +
> +if (vq->device_event.flags != 0x2) {
> +        notify_device(vq, vq->next_avail, vq->avail_wrap_count);
> +}
> +
> +\end{lstlisting}
> +
> +
> +\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> +The driver MUST perform a suitable memory barrier before reading
> +the Driver Event Suppression structure, to avoid missing a notification.
> +
> +\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Receiving Used Buffers From The Device}
> +
> +Once the device has used buffers referred to by a descriptor (read from or written to them, or
> +parts of both, depending on the nature of the virtqueue and the
> +device), it interrupts the driver
> +as detailed in section \ref{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Event
> +Suppression Structure Format}.
> +
> +\begin{note}
> +For optimal performance, a driver MAY disable interrupts while processing
> +the used buffers, but beware the problem of missing interrupts between
> +emptying the ring and reenabling interrupts.  This is usually handled by
> +re-checking for more used buffers after interrups are re-enabled:
> +\end{note}
> +
> +\begin{lstlisting}
> +vq->driver_event.flags = 0x2;
> +
> +for (;;) {
> +        struct virtq_desc *d = vq->desc[vq->next_used];
> +
> +        flags = d->flags;
> +        bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> +        bool used = flags & (1 << VIRTQ_DESC_F_USED);
> +
> +        if (avail != used) {
> +                vq->driver_event.flags = 0x1;
> +                memory_barrier();
> +
> +                flags = d->flags;
> +                bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> +                bool used = flags & (1 << VIRTQ_DESC_F_USED);
> +                if (avail != used) {
> +                        break;
> +                }
> +
> +                vq->driver_event.flags = 0x2;
> +        }
> +
> +	read_memory_barrier();
> +        process_buffer(d);
> +        vq->next_used++;
> +        if (vq->next_used > vq->size) {

Typo: vq->next_used >= vq->size

> +                vq->next_used = 0;
> +        }
> +}
> +\end{lstlisting}
> -- 
> MST
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [virtio-dev] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Tue, Jan 30, 2018 at 03:16:41PM +0800, Tiwei Bie wrote:
> On Tue, Jan 23, 2018 at 02:01:07AM +0200, Michael S. Tsirkin wrote:
> [...]
> > +\subsection{Available and Used Ring Wrap Counters}
> > +\label{sec:Packed Virtqueues / Available and Used Ring Wrap Counters}
> > +Each of the driver and the device are expected to maintain,
> > +internally, a single-bit ring wrap counter initialized to 1.
> > +
> > +The counter maintained by the driver is called the Available
> > +Ring Wrap Counter. Driver changes the value of this counter
> > +each time it makes available the
> > +last descriptor in the ring (after making the last descriptor
> > +available).
> > +
> > +The counter maintained by the device is called the Used Ring Wrap
> > +Counter.  Device changes the value of this counter
> > +each time it uses the last descriptor in
> > +the ring (after marking the last descriptor used).
> > +
> > +It is easy to see that the Available Ring Wrap Counter in the driver matches
> > +the Used Ring Wrap Counter in the device when both are processing the same
> > +descriptor, or when all available descriptors have been used.
> > +
> > +To mark a descriptor as available and used, both driver and
> > +device use the following two flags:
> > +\begin{lstlisting}
> > +#define VIRTQ_DESC_F_AVAIL     7
> > +#define VIRTQ_DESC_F_USED      15
> 
> I think above two definitions are not consistent with
> below definitions:
> 
> #define VIRTQ_DESC_F_NEXT       1
> #define VIRTQ_DESC_F_WRITE      2
> #define VIRTQ_DESC_F_INDIRECT   4
> 
> Maybe it'd be better to define them as:
> 
> #define VIRTQ_DESC_F_AVAIL(b)     (b << 7)
> #define VIRTQ_DESC_F_USED(b)      (b << 15)

Good catch, thanks a lot!

> > +\end{lstlisting}
> > +
> > +To mark a descriptor as available, driver sets the
> > +VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Available
> > +Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_USED bit to match the
> > +\emph{inverse} value.
> > +
> > +To mark a descriptor as used, device sets the
> > +VIRTQ_DESC_F_USED bit in Flags to match the internal Used
> > +Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_AVAIL bit to match the
> > +\emph{same} value.
> > +
> > +Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different
> > +for an available descriptor and equal for a used descriptor.
> > +
> > +\subsection{Polling of available and used descriptors}
> > +\label{sec:Packed Virtqueues / Polling of available and used descriptors}
> > +
> > +Writes of device and driver descriptors can generally be
> > +reordered, but each side (driver and device) are only required to
> > +poll (or test) a single location in memory: next device descriptor after
> > +the one they processed previously, in circular order.
> > +
> > +Sometimes device needs to only write out a single used descriptor
> > +after processing a batch of multiple available descriptors.  As
> > +described in more detail below, this can happen when using
> > +descriptor chaining or with in-order
> > +use of descriptors.  In this case, device writes out a used
> > +descriptor with buffer id of the last descriptor in the group.
> > +After processing the used descriptor, both device and driver then
> > +skip forward in the ring the number of the remaining descriptors
> > +in the group until processing (reading for the driver and writing
> > +for the device) the next used descriptor.
> > +
> > +\subsection{Write Flag}
> > +\label{sec:Packed Virtqueues / Write Flag}
> > +
> > +In an available descriptor, VIRTQ_DESC_F_WRITE bit within Flags
> > +is used to mark a descriptor as corresponding to a write-only or
> > +read-only element of a buffer.
> > +
> > +\begin{lstlisting}
> > +/* This marks a buffer as device write-only (otherwise device read-only). */
> > +#define VIRTQ_DESC_F_WRITE     2
> > +\end{lstlisting}
> > +
> > +In a used descriptor, this bit it used to specify whether any
> 
> Typo: s/it used/is used/
> 
> > +data has been written by the device into any parts of the buffer.
> [...]
> > +\subsection{Next Flag: Descriptor Chaining}
> > +\label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
> > +
> > +The VIRTIO_F_LIST_DESC feature allows driver to supply
> > +a scatter/gather list to the device
> > +by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT in
> > +Flags for all but the last available descriptor.
> > +
> > +\begin{lstlisting}
> > +/* This marks a buffer as continuing. */
> > +#define VIRTQ_DESC_F_NEXT   1
> > +\end{lstlisting}
> > +
> > +Buffer ID is included in the last descriptor in the list.
> > +
> > +The driver always makes the the first descriptor in the list
> > +available
> 
> Typo: s/the the/the/
> 
> The driver always and only makes the first descriptor in the list
> available?

No - it makes all of them available - the head
of the list is made available the last.

> If my above understanding is correct (i.e. only the
> first desc in a desc list will be made available by
> driver -- update the VIRTQ_DESC_F_AVAIL/USED flags),
> I think it would be better to make the description
> more explicit.

I'll try to be more explicit.

> Besides, does driver just need to set or clear the
> VIRTQ_DESC_F_WRITE bit for the first desc in a list?

No because a list includes a mix of write and read descriptors.

> > after the rest of the list has been written out into
> > +the ring. This guarantees that the device will never observe a
> > +partial scatter/gather list in the ring.
> > +
> > +Device only writes out a single used descriptor for the whole
> > +list. It then skips forward according to the number of
> > +descriptors in the list. Driver needs to keep track of the size
> > +of the list corresponding to each buffer ID, to be able to skip
> > +to where the next used descriptor is written by the device.
> > +
> > +For example, if descriptors are used in the same order in which
> > +they are made available, this will result in the used descriptor
> > +overwriting the first available descriptor in the list, the used
> > +descriptor for the next list overwriting the first available
> > +descriptor in the next list, etc.
> > +
> > +VIRTQ_DESC_F_NEXT is reserved in used descriptors, and
> > +should be ignored by drivers.
> > +
> [...]
> > +\subsubsection{Implementation Example}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Implementation Example}
> > +
> > +Below is an example driver code. It does not attempt to reduce
> > +the number of device interrupts, neither does it support
> > +the VIRTIO_F_RING_EVENT_IDX feature.
> > +
> > +\begin{lstlisting}
> > +
> > +first = vq->next_avail;
> > +id = alloc_id(vq);
> > +
> > +for (each buffer element b) {
> > +        vq->desc[vq->next_avail].address = get_addr(b);
> > +        vq->desc[vq->next_avail].len = get_len(b);
> > +        init_desc(vq->next_avail, b);
> > +        avail = vq->avail_wrap_count;
> > +        used = !vq->avail_wrap_count;
> > +        f = get_flags(b) | (avail << VIRTQ_DESC_F_AVAIL) | (used << VIRTQ_DESC_F_USED);
> > +	/* Don't mark the 1st descriptor available until all of them are ready. */
> > +        if (vq->next_avail == first) {
> > +                flags = f;
> > +        } else {
> > +                vq->desc[vq->next_avail].flags = f;
> > +        }
> > +
> > +	vq->next_avail++;
> > +
> > +	if (vq->next_avail > vq->size) {
> 
> Typo: vq->next_avail >= vq->size

True.

> > +		vq->next_avail = 0;
> > +		vq->avail_wrap_count \^= 1;
> > +	}
> > +
> > +
> > +}
> > +vq->desc[vq->next_avail].id = id;
> 
> I think if we need to update desc.id then the desc.id
> needs to be updated for each desc in the loop.

You are right. Will fix.

> > +write_memory_barrier();
> > +vq->desc[first].flags = flags;
> > +
> > +memory_barrier();
> > +
> > +if (vq->device_event.flags != 0x2) {
> > +        notify_device(vq, vq->next_avail, vq->avail_wrap_count);
> > +}
> > +
> > +\end{lstlisting}
> > +
> > +
> > +\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> > +The driver MUST perform a suitable memory barrier before reading
> > +the Driver Event Suppression structure, to avoid missing a notification.
> > +
> > +\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Receiving Used Buffers From The Device}
> > +
> > +Once the device has used buffers referred to by a descriptor (read from or written to them, or
> > +parts of both, depending on the nature of the virtqueue and the
> > +device), it interrupts the driver
> > +as detailed in section \ref{sec:Basic
> > +Facilities of a Virtio Device / Packed Virtqueues / Event
> > +Suppression Structure Format}.
> > +
> > +\begin{note}
> > +For optimal performance, a driver MAY disable interrupts while processing
> > +the used buffers, but beware the problem of missing interrupts between
> > +emptying the ring and reenabling interrupts.  This is usually handled by
> > +re-checking for more used buffers after interrups are re-enabled:
> > +\end{note}
> > +
> > +\begin{lstlisting}
> > +vq->driver_event.flags = 0x2;
> > +
> > +for (;;) {
> > +        struct virtq_desc *d = vq->desc[vq->next_used];
> > +
> > +        flags = d->flags;
> > +        bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> > +        bool used = flags & (1 << VIRTQ_DESC_F_USED);
> > +
> > +        if (avail != used) {
> > +                vq->driver_event.flags = 0x1;
> > +                memory_barrier();
> > +
> > +                flags = d->flags;
> > +                bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> > +                bool used = flags & (1 << VIRTQ_DESC_F_USED);
> > +                if (avail != used) {
> > +                        break;
> > +                }
> > +
> > +                vq->driver_event.flags = 0x2;
> > +        }
> > +
> > +	read_memory_barrier();
> > +        process_buffer(d);
> > +        vq->next_used++;
> > +        if (vq->next_used > vq->size) {
> 
> Typo: vq->next_used >= vq->size

True.

> > +                vq->next_used = 0;
> > +        }
> > +}
> > +\end{lstlisting}
> > -- 
> > MST
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > 

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Jens Freimann]

On Tue, Jan 23, 2018 at 02:01:07AM +0200, Michael S. Tsirkin wrote:
[...]
>+
>+With current transports, virtqueues are located in guest memory
>+allocated by driver.
>+Each packed virtqueue consists of three parts:
>+
>+\begin{itemize}
>+\item Descriptor Ring - occupies the Descriptor Area
>+\item Driver Event Suppression - occupies the Driver Area
>+\item Device Event Suppression - occupies the Device Area
>+\end{itemize}
>+
>+Where Descriptor Ring in turn consists of descriptors,
>+and where each descriptor can contain the following parts:
>+
>+\begin{itemize}
>+\item Buffer ID
>+\item Buffer Address
>+\item Buffer Length
>+\item Flags
>+\end{itemize}
>+
>+A buffer consists of zero or more device-readable physically-contiguous
>+elements followed by zero or more physically-contiguous
>+device-writable elements (each buffer has at least one element).
>+
>+When the driver wants to send such a buffer to the device, it
>+writes at least one available descriptor describing elements of
>+the buffer into the Descriptor Ring.  The descriptor(s) are
>+associated with a buffer by means of a Buffer ID stored within
>+the descriptor.
>+
>+Driver then notifies the device. When the device has finished
>+processing the buffer, it writes a used device descriptor
>+including the Buffer ID into the Descriptor Ring (overwriting a
>+driver descriptor previously made available), and sends an
>+interrupt.
>+
>+Descriptor Ring is used in a circular manner: driver writes
>+descriptors into the ring in order. After reaching end of ring,
>+the next descriptor is placed at head of the ring.  Once ring is
>+full of driver descriptors, driver stops sending new requests and
>+waits for device to start processing descriptors and to write out
>+some used descriptors before making new driver descriptors
>+available.
>+
>+Similarly, device reads descriptors from the ring in order and
>+detects that a driver descriptor has been made available.  As
>+processing of descriptors is completed used descriptors are
>+written by the device back into the ring.
>+
>+Note: after reading driver descriptors and starting their
>+processing in order, device might complete their processing out
>+of order.  Used device descriptors are written in the order
>+in which their processing is complete.
>+
>+Device Event Suppression data structure is write-only by the
>+device. It includes information for reducing the number of
>+device events - i.e. driver notifications to device.
>+
>+Driver Event Suppression data structure is read-only by the
>+device. It includes information for reducing the number of
>+driver events - i.e. device interrupts to driver.
>+
>+\subsection{Available and Used Ring Wrap Counters}
>+\label{sec:Packed Virtqueues / Available and Used Ring Wrap Counters}
>+Each of the driver and the device are expected to maintain,
>+internally, a single-bit ring wrap counter initialized to 1.
>+
>+The counter maintained by the driver is called the Available
>+Ring Wrap Counter. Driver changes the value of this counter
>+each time it makes available the
>+last descriptor in the ring (after making the last descriptor
>+available).
>+
>+The counter maintained by the device is called the Used Ring Wrap
>+Counter.  Device changes the value of this counter
>+each time it uses the last descriptor in
>+the ring (after marking the last descriptor used).
>+
>+It is easy to see that the Available Ring Wrap Counter in the driver matches
>+the Used Ring Wrap Counter in the device when both are processing the same
>+descriptor, or when all available descriptors have been used.
>+
>+To mark a descriptor as available and used, both driver and
>+device use the following two flags:
>+\begin{lstlisting}
>+#define VIRTQ_DESC_F_AVAIL     7
>+#define VIRTQ_DESC_F_USED      15
>+\end{lstlisting}
>+
>+To mark a descriptor as available, driver sets the
>+VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Available
>+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_USED bit to match the
>+\emph{inverse} value.
>+
>+To mark a descriptor as used, device sets the
>+VIRTQ_DESC_F_USED bit in Flags to match the internal Used
>+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_AVAIL bit to match the
>+\emph{same} value.
>+
>+Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different
>+for an available descriptor and equal for a used descriptor.
>+
>+\subsection{Polling of available and used descriptors}
>+\label{sec:Packed Virtqueues / Polling of available and used descriptors}
>+
>+Writes of device and driver descriptors can generally be
>+reordered, but each side (driver and device) are only required to
>+poll (or test) a single location in memory: next device descriptor after
>+the one they processed previously, in circular order.
>+
>+Sometimes device needs to only write out a single used descriptor
>+after processing a batch of multiple available descriptors.  As
>+described in more detail below, this can happen when using
>+descriptor chaining or with in-order
>+use of descriptors.  In this case, device writes out a used
>+descriptor with buffer id of the last descriptor in the group.
>+After processing the used descriptor, both device and driver then
>+skip forward in the ring the number of the remaining descriptors
>+in the group until processing (reading for the driver and writing
>+for the device) the next used descriptor.
>+
>+\subsection{Write Flag}
>+\label{sec:Packed Virtqueues / Write Flag}
>+
>+In an available descriptor, VIRTQ_DESC_F_WRITE bit within Flags
>+is used to mark a descriptor as corresponding to a write-only or
>+read-only element of a buffer.
>+
>+\begin{lstlisting}
>+/* This marks a buffer as device write-only (otherwise device read-only). */
>+#define VIRTQ_DESC_F_WRITE     2
>+\end{lstlisting}
>+
>+In a used descriptor, this bit it used to specify whether any
>+data has been written by the device into any parts of the buffer.
>+
>+
>+\subsection{Buffer Address and Length}
>+\label{sec:Packed Virtqueues / Buffer Address and Length}
>+
>+In an available descriptor, Buffer Address corresponds to the
>+physical address of the buffer. The length of the buffer assumed
>+to be physically contigious is stored in Buffer Length.
>+
>+In a used descriptor, Buffer Address is unused. Buffer Length
>+specifies the length of the buffer that has been initialized
>+(written to) by the device.
>+
>+Buffer length is reserved for used descriptors without the
>+VIRTQ_DESC_F_WRITE flag, and is ignored by drivers.
>+
>+\subsection{Scatter-Gather Support}
>+\label{sec:Packed Virtqueues / Scatter-Gather Support}
>+
>+Some drivers need an ability to supply a list of multiple buffer
>+elements (also known as a scatter/gather list) with a request.
>+Two optional features support this: descriptor
>+chaining and indirect descriptors.
>+
>+If neither feature has been negotiated, each buffer is
>+physically-contigious, either read-only or write-only and is
>+described completely by a single descriptor.
>+
>+While unusual (most implementations either create all lists
>+solely using non-indirect descriptors, or always use a single
>+indirect element), if both features have been negotiated, mixing
>+direct and direct descriptors in a ring is valid, as long as each
>+list only contains descriptors of a given type.
>+
>+Scatter/gather lists only apply to available descriptors. A
>+single used descriptor corresponds to the whole list.
>+
>+The device limits the number of descriptors in a list through a
>+transport-specific and/or device-specific value. If not limited,
>+the maximum number of descriptors in a list is the virt queue
>+size.
>+
>+\subsection{Next Flag: Descriptor Chaining}
>+\label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
>+
>+The VIRTIO_F_LIST_DESC feature allows driver to supply
>+a scatter/gather list to the device
>+by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT in
>+Flags for all but the last available descriptor.
>+
>+\begin{lstlisting}
>+/* This marks a buffer as continuing. */
>+#define VIRTQ_DESC_F_NEXT   1
>+\end{lstlisting}
>+
>+Buffer ID is included in the last descriptor in the list.
>+
>+The driver always makes the the first descriptor in the list
>+available after the rest of the list has been written out into
>+the ring. This guarantees that the device will never observe a
>+partial scatter/gather list in the ring.
>+
>+Device only writes out a single used descriptor for the whole
>+list. It then skips forward according to the number of
>+descriptors in the list. Driver needs to keep track of the size
>+of the list corresponding to each buffer ID, to be able to skip
>+to where the next used descriptor is written by the device.
>+
>+For example, if descriptors are used in the same order in which
>+they are made available, this will result in the used descriptor
>+overwriting the first available descriptor in the list, the used
>+descriptor for the next list overwriting the first available
>+descriptor in the next list, etc.
>+
>+VIRTQ_DESC_F_NEXT is reserved in used descriptors, and
>+should be ignored by drivers.
>+
>+\subsection{Indirect Flag: Scatter-Gather Support}
>+\label{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
>+
>+Some devices benefit by concurrently dispatching a large number
>+of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this. To increase
>+ring capacity the driver can store a (read-only by the device) table of indirect
>+descriptors anywhere in memory, and insert a descriptor in main
>+virtqueue (with \field{Flags} bit VIRTQ_DESC_F_INDIRECT on) that refers to
>+a memory buffer
>+containing this indirect descriptor table; \field{addr} and \field{len}
>+refer to the indirect table address and length in bytes,
>+respectively.
>+\begin{lstlisting}
>+/* This means the buffer contains a table of buffer descriptors. */
>+#define VIRTQ_DESC_F_INDIRECT   4
>+\end{lstlisting}
>+
>+The indirect table layout structure looks like this
>+(\field{len} is the Buffer Length of the descriptor that refers to this table,
>+which is a variable, so this code won't compile):

It is pseudo-code, so I'm not sure if this remark is necessary. 

>+
>+\begin{lstlisting}
>+struct indirect_descriptor_table {
>+        /* The actual descriptor structures (struct Desc each) */
>+        struct Desc desc[len / sizeof(struct Desc)];
>+};
>+\end{lstlisting}
>+
>+The first descriptor is located at start of the indirect
>+descriptor table, additional indirect descriptors come
>+immediately afterwards. \field{Flags} bit VIRTQ_DESC_F_WRITE is the
>+only valid flag for descriptors in the indirect table. Others
>+are reserved and are ignored by the device.
>+Buffer ID is also reserved and is ignored by the device.
>+
>+In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
>+is reserved and is ignored by the device.
>+
>+\subsection{Multi-buffer requests}
>+\label{sec:Packed Virtqueues / Multi-descriptor batches}
>+Some devices combine multiple buffers as part of processing of a
>+single request.  These devices always make the first
>+descriptor in the request available after the rest of the request

maybe I don't understand it correctly, but how about "mark the first
descriptor as used after the rest.."

>+has been written out request the ring. 

I can parse this sentence. Should probably be "written out to the
ring"?


regards,
Jens 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:07 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Performance analysis of this is in my kvm forum 2016 presentation.  The
> idea is to have a r/w descriptor in a ring structure, replacing the used
> and available ring, index and descriptor buffer.
> 
> This is also easier for devices to implement than the 1.0 layout.
> Several more enhancements will be necessary to actually make this
> efficient for devices to use.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex     |  25 ++-
>  packed-ring.tex | 678 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 700 insertions(+), 3 deletions(-)
>  create mode 100644 packed-ring.tex

(...)

> +\subsubsection{Driver notifications}
> +\label{sec:Packed Virtqueues / Driver notifications}
> +Whenever not suppressed by Device Event Suppression,
> +driver is required to notify the device after
> +making changes to the virtqueue.
> +
> +Some devices benefit from ability to find out the number of
> +available descriptors in the ring, and whether to send
> +interrupts to drivers without accessing virtqueue in memory:
> +for efficiency or as a debugging aid.
> +
> +To help with these optimizations, driver notifications
> +to the device include the following information:
> +
> +\begin{itemize}
> +\item VQ number
> +\item Offset (in units of descriptor size) within the ring
> +      where the next available descriptor will be written
> +\item Wrap Counter referring to the next available
> +      descriptor
> +\end{itemize}
> +
> +Note that driver can trigger multiple notifications even without
> +making any more changes to the ring. These would then have
> +identical \field{Offset} and \field{Wrap Counter} values.

(...)

> +\subsection{Driver Notification Format}\label{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
> +
> +The following structure is used to notify device of
> +device events - i.e. available descriptors:
> +
> +\begin{lstlisting}
> +__le16 vqn;
> +__le16 next_off : 15;
> +int    next_wrap : 1;
> +\end{lstlisting}

(...)

> +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> +
> +The actual method of device notification is bus-specific, but generally
> +it can be expensive.  So the device MAY suppress such notifications if it
> +doesn't need them, using the Driver Event Suppression structure
> +as detailed in section \ref{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Event
> +Suppression Structure Format}.
> +
> +The driver has to be careful to expose the new \field{flags}
> +value before checking if notifications are suppressed.

This is all I could find regarding notifications, and it leaves me
puzzled how notifications are actually supposed to work; especially,
where that driver notification structure is supposed to be relayed.

I'm obviously coming from a ccw perspective, but I don't think that pci
is all that different (well, hopefully).

Up to now, we notified for a certain virtqueue -- i.e., the device
driver notified the device that there is something to process for a
certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
pci seems to use a write to the config space IIUC. With the packed
layout, we have more payload per notification. We should be able to put
it in the same gpr than the virtqueue for ccw (if needed, with some
compat magic, or with a new hypercall, which would be ugly but doable).
Not sure how this is supposed to work with pci.

Has there been any prototyping done to implement this in qemu + KVM?
I'm unsure how this will work with ioeventfds, which just trigger.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Tue, Jan 30, 2018 at 02:50:44PM +0100, Cornelia Huck wrote:
> On Tue, 23 Jan 2018 02:01:07 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > Performance analysis of this is in my kvm forum 2016 presentation.  The
> > idea is to have a r/w descriptor in a ring structure, replacing the used
> > and available ring, index and descriptor buffer.
> > 
> > This is also easier for devices to implement than the 1.0 layout.
> > Several more enhancements will be necessary to actually make this
> > efficient for devices to use.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex     |  25 ++-
> >  packed-ring.tex | 678 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 700 insertions(+), 3 deletions(-)
> >  create mode 100644 packed-ring.tex
> 
> (...)
> 
> > +\subsubsection{Driver notifications}
> > +\label{sec:Packed Virtqueues / Driver notifications}
> > +Whenever not suppressed by Device Event Suppression,
> > +driver is required to notify the device after
> > +making changes to the virtqueue.
> > +
> > +Some devices benefit from ability to find out the number of
> > +available descriptors in the ring, and whether to send
> > +interrupts to drivers without accessing virtqueue in memory:
> > +for efficiency or as a debugging aid.
> > +
> > +To help with these optimizations, driver notifications
> > +to the device include the following information:
> > +
> > +\begin{itemize}
> > +\item VQ number
> > +\item Offset (in units of descriptor size) within the ring
> > +      where the next available descriptor will be written
> > +\item Wrap Counter referring to the next available
> > +      descriptor
> > +\end{itemize}
> > +
> > +Note that driver can trigger multiple notifications even without
> > +making any more changes to the ring. These would then have
> > +identical \field{Offset} and \field{Wrap Counter} values.
> 
> (...)
> 
> > +\subsection{Driver Notification Format}\label{sec:Basic
> > +Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
> > +
> > +The following structure is used to notify device of
> > +device events - i.e. available descriptors:
> > +
> > +\begin{lstlisting}
> > +__le16 vqn;
> > +__le16 next_off : 15;
> > +int    next_wrap : 1;
> > +\end{lstlisting}
> 
> (...)
> 
> > +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> > +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> > +
> > +The actual method of device notification is bus-specific, but generally
> > +it can be expensive.  So the device MAY suppress such notifications if it
> > +doesn't need them, using the Driver Event Suppression structure
> > +as detailed in section \ref{sec:Basic
> > +Facilities of a Virtio Device / Packed Virtqueues / Event
> > +Suppression Structure Format}.
> > +
> > +The driver has to be careful to expose the new \field{flags}
> > +value before checking if notifications are suppressed.
> 
> This is all I could find regarding notifications, and it leaves me
> puzzled how notifications are actually supposed to work; especially,
> where that driver notification structure is supposed to be relayed.
> 
> I'm obviously coming from a ccw perspective, but I don't think that pci
> is all that different (well, hopefully).
> 
> Up to now, we notified for a certain virtqueue -- i.e., the device
> driver notified the device that there is something to process for a
> certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
> pci seems to use a write to the config space IIUC. With the packed
> layout, we have more payload per notification. We should be able to put
> it in the same gpr than the virtqueue for ccw (if needed, with some
> compat magic, or with a new hypercall, which would be ugly but doable).
> Not sure how this is supposed to work with pci.
> 
> Has there been any prototyping done to implement this in qemu + KVM?
> I'm unsure how this will work with ioeventfds, which just trigger.

The PCI MMIO version would just trigger on access to a specific
address, ignoring all data in there. PIO would need something
like a data mask so it can ignore everything except the vq #.

This is helpful for hardware offloads but I'm open to
making this PCI specific or deferring until we have
explicit support for hardware offloads.

What do you think?

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Tiwei Bie]

On Tue, Jan 30, 2018 at 09:40:35PM +0200, Michael S. Tsirkin wrote:
> On Tue, Jan 30, 2018 at 02:50:44PM +0100, Cornelia Huck wrote:
> > On Tue, 23 Jan 2018 02:01:07 +0200
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > Performance analysis of this is in my kvm forum 2016 presentation.  The
> > > idea is to have a r/w descriptor in a ring structure, replacing the used
> > > and available ring, index and descriptor buffer.
> > > 
> > > This is also easier for devices to implement than the 1.0 layout.
> > > Several more enhancements will be necessary to actually make this
> > > efficient for devices to use.
> > > 
> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > ---
> > >  content.tex     |  25 ++-
> > >  packed-ring.tex | 678 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 700 insertions(+), 3 deletions(-)
> > >  create mode 100644 packed-ring.tex
> > 
> > (...)
> > 
> > > +\subsubsection{Driver notifications}
> > > +\label{sec:Packed Virtqueues / Driver notifications}
> > > +Whenever not suppressed by Device Event Suppression,
> > > +driver is required to notify the device after
> > > +making changes to the virtqueue.
> > > +
> > > +Some devices benefit from ability to find out the number of
> > > +available descriptors in the ring, and whether to send
> > > +interrupts to drivers without accessing virtqueue in memory:
> > > +for efficiency or as a debugging aid.
> > > +
> > > +To help with these optimizations, driver notifications
> > > +to the device include the following information:
> > > +
> > > +\begin{itemize}
> > > +\item VQ number
> > > +\item Offset (in units of descriptor size) within the ring
> > > +      where the next available descriptor will be written
> > > +\item Wrap Counter referring to the next available
> > > +      descriptor
> > > +\end{itemize}
> > > +
> > > +Note that driver can trigger multiple notifications even without
> > > +making any more changes to the ring. These would then have
> > > +identical \field{Offset} and \field{Wrap Counter} values.
> > 
> > (...)
> > 
> > > +\subsection{Driver Notification Format}\label{sec:Basic
> > > +Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
> > > +
> > > +The following structure is used to notify device of
> > > +device events - i.e. available descriptors:
> > > +
> > > +\begin{lstlisting}
> > > +__le16 vqn;
> > > +__le16 next_off : 15;
> > > +int    next_wrap : 1;
> > > +\end{lstlisting}
> > 
> > (...)
> > 
> > > +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> > > +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> > > +
> > > +The actual method of device notification is bus-specific, but generally
> > > +it can be expensive.  So the device MAY suppress such notifications if it
> > > +doesn't need them, using the Driver Event Suppression structure
> > > +as detailed in section \ref{sec:Basic
> > > +Facilities of a Virtio Device / Packed Virtqueues / Event
> > > +Suppression Structure Format}.
> > > +
> > > +The driver has to be careful to expose the new \field{flags}
> > > +value before checking if notifications are suppressed.
> > 
> > This is all I could find regarding notifications, and it leaves me
> > puzzled how notifications are actually supposed to work; especially,
> > where that driver notification structure is supposed to be relayed.
> > 
> > I'm obviously coming from a ccw perspective, but I don't think that pci
> > is all that different (well, hopefully).
> > 
> > Up to now, we notified for a certain virtqueue -- i.e., the device
> > driver notified the device that there is something to process for a
> > certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
> > pci seems to use a write to the config space IIUC. With the packed
> > layout, we have more payload per notification. We should be able to put
> > it in the same gpr than the virtqueue for ccw (if needed, with some
> > compat magic, or with a new hypercall, which would be ugly but doable).
> > Not sure how this is supposed to work with pci.
> > 
> > Has there been any prototyping done to implement this in qemu + KVM?
> > I'm unsure how this will work with ioeventfds, which just trigger.
> 
> The PCI MMIO version would just trigger on access to a specific
> address, ignoring all data in there. PIO would need something
> like a data mask so it can ignore everything except the vq #.
> 
> This is helpful for hardware offloads but I'm open to
> making this PCI specific or deferring until we have
> explicit support for hardware offloads.
> 
> What do you think?
> 

Hi,

I prefer to keep it (at least for PCI) and refine it if
necessary.

Because one of the important goals of packed ring is to
be hardware friendly. Supporting tail pointer is one of
the important things to make it hardware friendly. More
details could be found in Kully's below mail (I've done
some slight reformatting):

----- START -----

why tail pointer is good for hardware implementation:

Assuming no tail pointer:

1. Hardware would have to speculatively read descriptors
   and check their validity by checking that DESC_HW=1.

1.1 Yes Hardware could request a large number of
    descriptors at a time, making the PCIe read
    response transfer (i.e. read descriptors) an
    efficient PCIe transfer.

The problems are as follows:

2. Issue 1: Wasting PCIe bandwidth

2.1 Although the PCIe read responses may be efficient
    transfers, if they contain invalid descriptors
    (DESC_HW=0), we have wasted PCIe bandwidth. This
    can be a problem when trying to maximize the
    performance possible from a design.

3. Issue 2: Wasting Hardware memory resources

3.1 When issuing PCIe read requests for descriptors,
    the hardware must reserve inadvance memory to
    store the descriptors.

3.2 Given PCIe read latencies can be in the order
    of 1us, this memory is reserved for that length
    of time.

3.3 For hardware, 1us is a very long time and for
    FPGAs, memory is not as plentiful/cheap as in
    a PC.

3.4 So reserving memory for descriptors that may
    end up being invalid is a waste. Ultimately,
    this could effect performance if a large number
    of invalid descriptors are being read.

So it is better for hardware to know which queues
(and hence guests) have descriptors available and
fetch those only.

The argument above is biased towards Tx (transfer
of packets from guest to device) but does also apply
for Rx.

Tail pointer resides in the hardware and so the
hardware always knows how many descriptors are
available for each queue (no need to waste PCIe
bandwidth to determine this) and so can fetch
only those valid descriptors.

----- END -----

Best regards,
Tiwei Bie

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Cornelia Huck]

On Thu, 1 Feb 2018 11:05:35 +0800
Tiwei Bie <tiwei.bie@intel.com> wrote:

> On Tue, Jan 30, 2018 at 09:40:35PM +0200, Michael S. Tsirkin wrote:
> > On Tue, Jan 30, 2018 at 02:50:44PM +0100, Cornelia Huck wrote:  
> > > On Tue, 23 Jan 2018 02:01:07 +0200
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:

> > > > +\subsubsection{Driver notifications}
> > > > +\label{sec:Packed Virtqueues / Driver notifications}
> > > > +Whenever not suppressed by Device Event Suppression,
> > > > +driver is required to notify the device after
> > > > +making changes to the virtqueue.
> > > > +
> > > > +Some devices benefit from ability to find out the number of
> > > > +available descriptors in the ring, and whether to send
> > > > +interrupts to drivers without accessing virtqueue in memory:
> > > > +for efficiency or as a debugging aid.
> > > > +
> > > > +To help with these optimizations, driver notifications
> > > > +to the device include the following information:
> > > > +
> > > > +\begin{itemize}
> > > > +\item VQ number
> > > > +\item Offset (in units of descriptor size) within the ring
> > > > +      where the next available descriptor will be written
> > > > +\item Wrap Counter referring to the next available
> > > > +      descriptor
> > > > +\end{itemize}
> > > > +
> > > > +Note that driver can trigger multiple notifications even without
> > > > +making any more changes to the ring. These would then have
> > > > +identical \field{Offset} and \field{Wrap Counter} values.  
> > > 
> > > (...)
> > >   
> > > > +\subsection{Driver Notification Format}\label{sec:Basic
> > > > +Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
> > > > +
> > > > +The following structure is used to notify device of
> > > > +device events - i.e. available descriptors:
> > > > +
> > > > +\begin{lstlisting}
> > > > +__le16 vqn;
> > > > +__le16 next_off : 15;
> > > > +int    next_wrap : 1;
> > > > +\end{lstlisting}  
> > > 
> > > (...)
> > >   
> > > > +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> > > > +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> > > > +
> > > > +The actual method of device notification is bus-specific, but generally
> > > > +it can be expensive.  So the device MAY suppress such notifications if it
> > > > +doesn't need them, using the Driver Event Suppression structure
> > > > +as detailed in section \ref{sec:Basic
> > > > +Facilities of a Virtio Device / Packed Virtqueues / Event
> > > > +Suppression Structure Format}.
> > > > +
> > > > +The driver has to be careful to expose the new \field{flags}
> > > > +value before checking if notifications are suppressed.  
> > > 
> > > This is all I could find regarding notifications, and it leaves me
> > > puzzled how notifications are actually supposed to work; especially,
> > > where that driver notification structure is supposed to be relayed.
> > > 
> > > I'm obviously coming from a ccw perspective, but I don't think that pci
> > > is all that different (well, hopefully).
> > > 
> > > Up to now, we notified for a certain virtqueue -- i.e., the device
> > > driver notified the device that there is something to process for a
> > > certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
> > > pci seems to use a write to the config space IIUC. With the packed
> > > layout, we have more payload per notification. We should be able to put
> > > it in the same gpr than the virtqueue for ccw (if needed, with some
> > > compat magic, or with a new hypercall, which would be ugly but doable).
> > > Not sure how this is supposed to work with pci.
> > > 
> > > Has there been any prototyping done to implement this in qemu + KVM?
> > > I'm unsure how this will work with ioeventfds, which just trigger.  
> > 
> > The PCI MMIO version would just trigger on access to a specific
> > address, ignoring all data in there. PIO would need something
> > like a data mask so it can ignore everything except the vq #.
> > 
> > This is helpful for hardware offloads but I'm open to
> > making this PCI specific or deferring until we have
> > explicit support for hardware offloads.
> > 
> > What do you think?
> >   
> 
> Hi,
> 
> I prefer to keep it (at least for PCI) and refine it if
> necessary.
> 
> Because one of the important goals of packed ring is to
> be hardware friendly. Supporting tail pointer is one of
> the important things to make it hardware friendly. More
> details could be found in Kully's below mail (I've done
> some slight reformatting):
> 
> ----- START -----

<thanks for the explanation>

> ----- END -----

So, my takeaway is here:

- Having this information (or a variant of it) available on
  notification is useful.
- The specifics on how to convey the info are still a bit unsettled.

I think this should be optionally available to any transport (i.e. not
pci-specific). What about the following wording:

"Driver notifications to the device include the virtqueue number. To
help with these optimizations, they also may include the following
information: ..."

(With some MUST/MAY wording in the normative sections, I guess.)

Also, I think the notification structure should not include any
endianness requirements. For ccw, we notify via a hypercall with the
payload in the GPRs, which are big endian. I would like to avoid
conversions in that case. Maybe make the details of how the information
is included entirely transport-specific?

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 11:11:28AM +0100, Cornelia Huck wrote:
> On Thu, 1 Feb 2018 11:05:35 +0800
> Tiwei Bie <tiwei.bie@intel.com> wrote:
> 
> > On Tue, Jan 30, 2018 at 09:40:35PM +0200, Michael S. Tsirkin wrote:
> > > On Tue, Jan 30, 2018 at 02:50:44PM +0100, Cornelia Huck wrote:  
> > > > On Tue, 23 Jan 2018 02:01:07 +0200
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > > > > +\subsubsection{Driver notifications}
> > > > > +\label{sec:Packed Virtqueues / Driver notifications}
> > > > > +Whenever not suppressed by Device Event Suppression,
> > > > > +driver is required to notify the device after
> > > > > +making changes to the virtqueue.
> > > > > +
> > > > > +Some devices benefit from ability to find out the number of
> > > > > +available descriptors in the ring, and whether to send
> > > > > +interrupts to drivers without accessing virtqueue in memory:
> > > > > +for efficiency or as a debugging aid.
> > > > > +
> > > > > +To help with these optimizations, driver notifications
> > > > > +to the device include the following information:
> > > > > +
> > > > > +\begin{itemize}
> > > > > +\item VQ number
> > > > > +\item Offset (in units of descriptor size) within the ring
> > > > > +      where the next available descriptor will be written
> > > > > +\item Wrap Counter referring to the next available
> > > > > +      descriptor
> > > > > +\end{itemize}
> > > > > +
> > > > > +Note that driver can trigger multiple notifications even without
> > > > > +making any more changes to the ring. These would then have
> > > > > +identical \field{Offset} and \field{Wrap Counter} values.  
> > > > 
> > > > (...)
> > > >   
> > > > > +\subsection{Driver Notification Format}\label{sec:Basic
> > > > > +Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
> > > > > +
> > > > > +The following structure is used to notify device of
> > > > > +device events - i.e. available descriptors:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +__le16 vqn;
> > > > > +__le16 next_off : 15;
> > > > > +int    next_wrap : 1;
> > > > > +\end{lstlisting}  
> > > > 
> > > > (...)
> > > >   
> > > > > +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> > > > > +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> > > > > +
> > > > > +The actual method of device notification is bus-specific, but generally
> > > > > +it can be expensive.  So the device MAY suppress such notifications if it
> > > > > +doesn't need them, using the Driver Event Suppression structure
> > > > > +as detailed in section \ref{sec:Basic
> > > > > +Facilities of a Virtio Device / Packed Virtqueues / Event
> > > > > +Suppression Structure Format}.
> > > > > +
> > > > > +The driver has to be careful to expose the new \field{flags}
> > > > > +value before checking if notifications are suppressed.  
> > > > 
> > > > This is all I could find regarding notifications, and it leaves me
> > > > puzzled how notifications are actually supposed to work; especially,
> > > > where that driver notification structure is supposed to be relayed.
> > > > 
> > > > I'm obviously coming from a ccw perspective, but I don't think that pci
> > > > is all that different (well, hopefully).
> > > > 
> > > > Up to now, we notified for a certain virtqueue -- i.e., the device
> > > > driver notified the device that there is something to process for a
> > > > certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
> > > > pci seems to use a write to the config space IIUC. With the packed
> > > > layout, we have more payload per notification. We should be able to put
> > > > it in the same gpr than the virtqueue for ccw (if needed, with some
> > > > compat magic, or with a new hypercall, which would be ugly but doable).
> > > > Not sure how this is supposed to work with pci.
> > > > 
> > > > Has there been any prototyping done to implement this in qemu + KVM?
> > > > I'm unsure how this will work with ioeventfds, which just trigger.  
> > > 
> > > The PCI MMIO version would just trigger on access to a specific
> > > address, ignoring all data in there. PIO would need something
> > > like a data mask so it can ignore everything except the vq #.
> > > 
> > > This is helpful for hardware offloads but I'm open to
> > > making this PCI specific or deferring until we have
> > > explicit support for hardware offloads.
> > > 
> > > What do you think?
> > >   
> > 
> > Hi,
> > 
> > I prefer to keep it (at least for PCI) and refine it if
> > necessary.
> > 
> > Because one of the important goals of packed ring is to
> > be hardware friendly. Supporting tail pointer is one of
> > the important things to make it hardware friendly. More
> > details could be found in Kully's below mail (I've done
> > some slight reformatting):
> > 
> > ----- START -----
> 
> <thanks for the explanation>
> 
> > ----- END -----
> 
> So, my takeaway is here:
> 
> - Having this information (or a variant of it) available on
>   notification is useful.
> - The specifics on how to convey the info are still a bit unsettled.
> 
> I think this should be optionally available to any transport (i.e. not
> pci-specific). What about the following wording:
> 
> "Driver notifications to the device include the virtqueue number. To
> help with these optimizations, they also may include the following
> information: ..."
> 
> (With some MUST/MAY wording in the normative sections, I guess.)
> 
> Also, I think the notification structure should not include any
> endianness requirements. For ccw, we notify via a hypercall with the
> payload in the GPRs, which are big endian. I would like to avoid
> conversions in that case. Maybe make the details of how the information
> is included entirely transport-specific?

Makes sense, thanks for the suggestions.

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Halil Pasic]

On 01/30/2018 02:50 PM, Cornelia Huck wrote:
> On Tue, 23 Jan 2018 02:01:07 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
>> Performance analysis of this is in my kvm forum 2016 presentation.  The
>> idea is to have a r/w descriptor in a ring structure, replacing the used
>> and available ring, index and descriptor buffer.
>>
>> This is also easier for devices to implement than the 1.0 layout.
>> Several more enhancements will be necessary to actually make this
>> efficient for devices to use.
>>
>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> ---
[..]
> 
>> +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
>> +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
>> +
>> +The actual method of device notification is bus-specific, but generally
>> +it can be expensive.  So the device MAY suppress such notifications if it
>> +doesn't need them, using the Driver Event Suppression structure
>> +as detailed in section \ref{sec:Basic
>> +Facilities of a Virtio Device / Packed Virtqueues / Event
>> +Suppression Structure Format}.
>> +
>> +The driver has to be careful to expose the new \field{flags}
>> +value before checking if notifications are suppressed.
> 
> This is all I could find regarding notifications, and it leaves me
> puzzled how notifications are actually supposed to work; especially,
> where that driver notification structure is supposed to be relayed.
> 
> I'm obviously coming from a ccw perspective, but I don't think that pci
> is all that different (well, hopefully).
> 
> Up to now, we notified for a certain virtqueue -- i.e., the device
> driver notified the device that there is something to process for a
> certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
> pci seems to use a write to the config space IIUC. With the packed
> layout, we have more payload per notification. We should be able to put
> it in the same gpr than the virtqueue for ccw (if needed, with some
> compat magic, or with a new hypercall, which would be ugly but doable).
> Not sure how this is supposed to work with pci.
> 
> Has there been any prototyping done to implement this in qemu + KVM?
> I'm unsure how this will work with ioeventfds, which just trigger.
> 


I'm also interested in an answer to Connie's question regarding a QEMU +
KVM prototype. IMHO we should definitively have at least a such an
prototype (preferably a reasonable implementation) before voting about
the changes envisioned by this series.

[META]

Unfortunately I have skipped v6 altogether (that is not even lurker
mode). I'm a bit overwhelmed. I'm also in doubt about how to articulate
my feelings and opinions. Maybe I will wait for v8 with my comments. You
seem to have received enough comments for v7 already.

Anyway, I'm happy to see virtio version 1.1 is slowly materializing.

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Mon, Feb 05, 2018 at 12:54:41PM +0100, Halil Pasic wrote:
> 
> 
> On 01/30/2018 02:50 PM, Cornelia Huck wrote:
> > On Tue, 23 Jan 2018 02:01:07 +0200
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> >> Performance analysis of this is in my kvm forum 2016 presentation.  The
> >> idea is to have a r/w descriptor in a ring structure, replacing the used
> >> and available ring, index and descriptor buffer.
> >>
> >> This is also easier for devices to implement than the 1.0 layout.
> >> Several more enhancements will be necessary to actually make this
> >> efficient for devices to use.
> >>
> >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >> ---
> [..]
> > 
> >> +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> >> +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> >> +
> >> +The actual method of device notification is bus-specific, but generally
> >> +it can be expensive.  So the device MAY suppress such notifications if it
> >> +doesn't need them, using the Driver Event Suppression structure
> >> +as detailed in section \ref{sec:Basic
> >> +Facilities of a Virtio Device / Packed Virtqueues / Event
> >> +Suppression Structure Format}.
> >> +
> >> +The driver has to be careful to expose the new \field{flags}
> >> +value before checking if notifications are suppressed.
> > 
> > This is all I could find regarding notifications, and it leaves me
> > puzzled how notifications are actually supposed to work; especially,
> > where that driver notification structure is supposed to be relayed.
> > 
> > I'm obviously coming from a ccw perspective, but I don't think that pci
> > is all that different (well, hopefully).
> > 
> > Up to now, we notified for a certain virtqueue -- i.e., the device
> > driver notified the device that there is something to process for a
> > certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
> > pci seems to use a write to the config space IIUC. With the packed
> > layout, we have more payload per notification. We should be able to put
> > it in the same gpr than the virtqueue for ccw (if needed, with some
> > compat magic, or with a new hypercall, which would be ugly but doable).
> > Not sure how this is supposed to work with pci.
> > 
> > Has there been any prototyping done to implement this in qemu + KVM?
> > I'm unsure how this will work with ioeventfds, which just trigger.
> > 
> 
> 
> I'm also interested in an answer to Connie's question regarding a QEMU +
> KVM prototype. IMHO we should definitively have at least a such an
> prototype (preferably a reasonable implementation) before voting about
> the changes envisioned by this series.

This is certainly not how we did it for v1.0, and not how
oasis process works generally. Implementations are required
to move to an oasis standard change. We are working on
a committee standard deliverables.

I don't yet plan to work on an implementation yet: it's a bit of a
chicked and egg problem. People are reluctant to work on what's not in
the spec. We can always make changes as long as there are no
implementations.

> [META]
> 
> Unfortunately I have skipped v6 altogether (that is not even lurker
> mode). I'm a bit overwhelmed. I'm also in doubt about how to articulate
> my feelings and opinions. Maybe I will wait for v8 with my comments. You
> seem to have received enough comments for v7 already.

It's been under review for a very long time and only s390 related
changes are planned so I hope to move to voting after v8.

> Anyway, I'm happy to see virtio version 1.1 is slowly materializing.
> 
> Regards,
> Halil

If people first wait for all *other* comments to be addressed,
then post their own, it will take months to merge anything.
so please do not do that.

If someone specific does not have time to review that's ok - one can
always abstain in a vote. Also, with main changes merged it will be
easier to tweak wording - people can just post small patches with
suggested wording.

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Halil Pasic]

On 02/05/2018 03:33 PM, Michael S. Tsirkin wrote:
> On Mon, Feb 05, 2018 at 12:54:41PM +0100, Halil Pasic wrote:
>>
>>
>> On 01/30/2018 02:50 PM, Cornelia Huck wrote:
>>> On Tue, 23 Jan 2018 02:01:07 +0200
>>> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>
>>>> Performance analysis of this is in my kvm forum 2016 presentation.  The
>>>> idea is to have a r/w descriptor in a ring structure, replacing the used
>>>> and available ring, index and descriptor buffer.
>>>>
>>>> This is also easier for devices to implement than the 1.0 layout.
>>>> Several more enhancements will be necessary to actually make this
>>>> efficient for devices to use.
>>>>
>>>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>> ---
>> [..]
>>>
>>>> +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
>>>> +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
>>>> +
>>>> +The actual method of device notification is bus-specific, but generally
>>>> +it can be expensive.  So the device MAY suppress such notifications if it
>>>> +doesn't need them, using the Driver Event Suppression structure
>>>> +as detailed in section \ref{sec:Basic
>>>> +Facilities of a Virtio Device / Packed Virtqueues / Event
>>>> +Suppression Structure Format}.
>>>> +
>>>> +The driver has to be careful to expose the new \field{flags}
>>>> +value before checking if notifications are suppressed.
>>>
>>> This is all I could find regarding notifications, and it leaves me
>>> puzzled how notifications are actually supposed to work; especially,
>>> where that driver notification structure is supposed to be relayed.
>>>
>>> I'm obviously coming from a ccw perspective, but I don't think that pci
>>> is all that different (well, hopefully).
>>>
>>> Up to now, we notified for a certain virtqueue -- i.e., the device
>>> driver notified the device that there is something to process for a
>>> certain queue. ccw uses the virtqueue number in a gpr for a hypercall,
>>> pci seems to use a write to the config space IIUC. With the packed
>>> layout, we have more payload per notification. We should be able to put
>>> it in the same gpr than the virtqueue for ccw (if needed, with some
>>> compat magic, or with a new hypercall, which would be ugly but doable).
>>> Not sure how this is supposed to work with pci.
>>>
>>> Has there been any prototyping done to implement this in qemu + KVM?
>>> I'm unsure how this will work with ioeventfds, which just trigger.
>>>
>>
>>
>> I'm also interested in an answer to Connie's question regarding a QEMU +
>> KVM prototype. IMHO we should definitively have at least a such an
>> prototype (preferably a reasonable implementation) before voting about
>> the changes envisioned by this series.
> 
> This is certainly not how we did it for v1.0, and not how
> oasis process works generally. Implementations are required
> to move to an oasis standard change. We are working on
> a committee standard deliverables.
> 
> I don't yet plan to work on an implementation yet: it's a bit of a
> chicked and egg problem. People are reluctant to work on what's not in
> the spec. We can always make changes as long as there are no
> implementations.
> 

'We can always make changes as long as there are no implementations'
comes very surprising to me. I believed, once a committee specification
is released the requirements for changes are given, and don't depend
on known implementations.

Does this imply that one should be reluctant about implementing
a virtio specification that still has no implementation (because it ain't
stable, and may change, because there is no implementation yet)?

I've re-read some of the OASIS documents (e.g. TC Process, Naming Directives),
and not surprisingly I did not find any requirements on compatibility
between e.g. two versions of the same OASIS Committee Specification. So
I guess, technically we can change anything.

With that said, in my reading your v1.1 proposal does not make the
relationship between v1.0 and v1.1 obvious (not even to the extent
to which the relationship between legacy and v1.0 is detailed).

I assume v1.1 is not a breaking change: every device/driver conforming
the v1.0 specification is per se conforming v1.1 too. Or am I wrong?


>> [META]
>>
>> Unfortunately I have skipped v6 altogether (that is not even lurker
>> mode). I'm a bit overwhelmed. I'm also in doubt about how to articulate
>> my feelings and opinions. Maybe I will wait for v8 with my comments. You
>> seem to have received enough comments for v7 already.
> 
> It's been under review for a very long time and only s390 related
> changes are planned so I hope to move to voting after v8.
> 

Thanks for the information. 

>> Anyway, I'm happy to see virtio version 1.1 is slowly materializing.
>>
>> Regards,
>> Halil
> 
> If people first wait for all *other* comments to be addressed,
> then post their own, it will take months to merge anything.
> so please do not do that.
> 

Point taken.

> If someone specific does not have time to review that's ok - one can
> always abstain in a vote.

OK. I may end up doing just that. Nevertheless, I'm going to try to give you
some of my perspective ASAP -- it's not that I don't have one.

> Also, with main changes merged it will be
> easier to tweak wording - people can just post small patches with
> suggested wording.
> 

I only agree with your point for changes that are really minor (typos,
grammar bugs, etc.). For example, consider your Used Ring -> Device
Area and Available Ring -> Driver Area change.

I think it's much simpler if we get as much as possible right from
the beginning (that is release). But that's only my very own opinion.

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Paolo Bonzini]

On 05/02/2018 17:57, Halil Pasic wrote:
>> This is certainly not how we did it for v1.0, and not how
>> oasis process works generally. Implementations are required
>> to move to an oasis standard change. We are working on
>> a committee standard deliverables.
>>
>> I don't yet plan to work on an implementation yet: it's a bit of a
>> chicked and egg problem. People are reluctant to work on what's not in
>> the spec. We can always make changes as long as there are no
>> implementations.
>>
> 'We can always make changes as long as there are no implementations'
> comes very surprising to me. I believed, once a committee specification
> is released the requirements for changes are given, and don't depend
> on known implementations.
> 
> Does this imply that one should be reluctant about implementing
> a virtio specification that still has no implementation (because it ain't
> stable, and may change, because there is no implementation yet)?

I agree that this doesn't seem optimal.  This is a much bigger change
than anything between virtio 0.9 and in virtio 1.0, because it affects
the data path directly.

Paolo

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Cornelia Huck]

On Mon, 5 Feb 2018 18:00:07 +0100
Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 05/02/2018 17:57, Halil Pasic wrote:
> >> This is certainly not how we did it for v1.0, and not how
> >> oasis process works generally. Implementations are required
> >> to move to an oasis standard change. We are working on
> >> a committee standard deliverables.
> >>
> >> I don't yet plan to work on an implementation yet: it's a bit of a
> >> chicked and egg problem. People are reluctant to work on what's not in
> >> the spec. We can always make changes as long as there are no
> >> implementations.
> >>  
> > 'We can always make changes as long as there are no implementations'
> > comes very surprising to me. I believed, once a committee specification
> > is released the requirements for changes are given, and don't depend
> > on known implementations.
> > 
> > Does this imply that one should be reluctant about implementing
> > a virtio specification that still has no implementation (because it ain't
> > stable, and may change, because there is no implementation yet)?  
> 
> I agree that this doesn't seem optimal.  This is a much bigger change
> than anything between virtio 0.9 and in virtio 1.0, because it affects
> the data path directly.

Nod. It looks in pretty good shape to me, once we fixed up the things I
noticed during this round, but I still feel a bit uncomfortable without
a prototype for non-pci.

One issue for me is that all work has been done with pci as the
transport, and with a view as to what could be helpful for implementers
of pci cards. There's nothing inherently bad in that, but it does
introduce a chance that other transports may run into problems when
they try to implement it.

Not an insurmountable problem, but something that should be kept in
mind.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Mon, Feb 05, 2018 at 07:16:54PM +0100, Cornelia Huck wrote:
> > I agree that this doesn't seem optimal.  This is a much bigger change
> > than anything between virtio 0.9 and in virtio 1.0, because it affects
> > the data path directly.
> 
> Nod. It looks in pretty good shape to me, once we fixed up the things I
> noticed during this round, but I still feel a bit uncomfortable without
> a prototype for non-pci.
> 
> One issue for me is that all work has been done with pci as the
> transport, and with a view as to what could be helpful for implementers
> of pci cards. There's nothing inherently bad in that, but it does
> introduce a chance that other transports may run into problems when
> they try to implement it.
> 
> Not an insurmountable problem, but something that should be kept in
> mind.

What do you suggest then?

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Cornelia Huck]

On Mon, 5 Feb 2018 20:21:12 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Feb 05, 2018 at 07:16:54PM +0100, Cornelia Huck wrote:
> > > I agree that this doesn't seem optimal.  This is a much bigger change
> > > than anything between virtio 0.9 and in virtio 1.0, because it affects
> > > the data path directly.  
> > 
> > Nod. It looks in pretty good shape to me, once we fixed up the things I
> > noticed during this round, but I still feel a bit uncomfortable without
> > a prototype for non-pci.
> > 
> > One issue for me is that all work has been done with pci as the
> > transport, and with a view as to what could be helpful for implementers
> > of pci cards. There's nothing inherently bad in that, but it does
> > introduce a chance that other transports may run into problems when
> > they try to implement it.
> > 
> > Not an insurmountable problem, but something that should be kept in
> > mind.  
> 
> What do you suggest then?
> 

Do at least a quick qemu prototype. That gives you some idea where
things won't work, as it's the only way to cover all transports without
having to be aware of architecture details.

[The real solution is to have people with enough time, but that's a
pipe dream.]

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Mon, Feb 05, 2018 at 05:57:26PM +0100, Halil Pasic wrote:
> I think it's much simpler if we get as much as possible right from
> the beginning (that is release). But that's only my very own opinion.

I agree here. If nothing else, if we can avoid making material changes
after the first public review period, we can put out a release faster,
and if delaying for another couple of days can help us avoid extra
public review periods, it's worth it.

Having said that, it doesn't make sense to wait for months just on an
off chance we'll get some comments - that's what the public review
period is for.

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] [PATCH v7 08/11] packed virtqueues: more efficient virtqueue layout

[Halil Pasic]

Hi! I've tried to not repeat the points raised by the other reviewers.
If I failed, please point me to the answer ;).


On 01/23/2018 01:01 AM, Michael S. Tsirkin wrote:
> Performance analysis of this is in my kvm forum 2016 presentation.  The
> idea is to have a r/w descriptor in a ring structure, replacing the used
> and available ring, index and descriptor buffer.
> 
> This is also easier for devices to implement than the 1.0 layout.
> Several more enhancements will be necessary to actually make this
> efficient for devices to use.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex     |  25 ++-
>  packed-ring.tex | 678 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 700 insertions(+), 3 deletions(-)
>  create mode 100644 packed-ring.tex
> 
> diff --git a/content.tex b/content.tex
> index 0f7c2b9..4d522cc 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -263,8 +263,17 @@ these parts (following \ref{sec:Basic Facilities of a Virtio Device / Split Virt
> 
>  \end{note}
> 
> +Two formats are supported: Split Virtqueues (see \ref{sec:Basic
> +Facilities of a Virtio Device / Split
> +Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
> +Split Virtqueues}) and Packed Virtqueues (see \ref{sec:Basic
> +Facilities of a Virtio Device / Packed
> +Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
> +Packed Virtqueues}).
> +

I guess, a driver which does not support packed remains a conforming
virtio (1.1) driver.

A complete device (that is a device and a driver pair) is using packed
layout for all the virtqueues iff VIRTIO_F_PACKED_RING was negotiated (that
is the device offered it and the driver accepted it. Otherwise split format
is used.

I could not find this specified explicitly. 
 
>  \input{split-ring.tex}
> 
> +\input{packed-ring.tex}
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}

[..]

> new file mode 100644
> index 0000000..b6cb979
> --- /dev/null
> +++ b/packed-ring.tex
> @@ -0,0 +1,678 @@
> +\section{Packed Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}
> +
> +Packed virtqueues is an alternative compact virtqueue layout using
> +read-write memory, that is memory that is both read and written
> +by both host and guest.
> +
> +Use of packed virtqueues is enabled by the VIRTIO_F_PACKED_RING
> +feature bit.

See above. Would prefer s/enabled by/negotiated via/

> +
> +Packed virtqueues support up to $2^{15}$ entries each.
> +
> +With current transports, virtqueues are located in guest memory
> +allocated by driver.
> +Each packed virtqueue consists of three parts:
> +
> +\begin{itemize}
> +\item Descriptor Ring - occupies the Descriptor Area
> +\item Driver Event Suppression - occupies the Driver Area
> +\item Device Event Suppression - occupies the Device Area
> +\end{itemize}
> +
> +Where Descriptor Ring in turn consists of descriptors,
> +and where each descriptor can contain the following parts:
> +
> +\begin{itemize}
> +\item Buffer ID

AFAIU this is on 'request' basis. That is, it corresponds to a
chain of descriptors (where chain length can be 1). Let's call
this one 'red buffer'. 

> +\item Buffer Address

This 'Buffer' as a different color. Here the 'buffer' stands
for 'buffer element'. That is corresponds to a single descriptor
and a single guest physically continuous chunk of memory.
Let's call this one 'blue buffer'.

> +\item Buffer Length

Same here.

> +\item Flags
> +\end{itemize}


> +
> +A buffer consists of zero or more device-readable physically-contiguous
     (that is 'red buffer')
> +elements followed by zero or more physically-contiguous
  (that is 'blue buffer')
> +device-writable elements (each buffer has at least one element).
                   (that is 'blue buffer')
> +
> +When the driver wants to send such a buffer to the device, it
> +writes at least one available descriptor describing elements of
> +the buffer into the Descriptor Ring.  The descriptor(s) are
> +associated with a buffer by means of a Buffer ID stored within
> +the descriptor.
> +
> +Driver then notifies the device. When the device has finished
> +processing the buffer, it writes a used device descriptor
> +including the Buffer ID into the Descriptor Ring (overwriting a
> +driver descriptor previously made available), and sends an
> +interrupt.
> +
> +Descriptor Ring is used in a circular manner: driver writes
> +descriptors into the ring in order. After reaching end of ring,
> +the next descriptor is placed at head of the ring.  Once ring is
> +full of driver descriptors, driver stops sending new requests and
> +waits for device to start processing descriptors and to write out
> +some used descriptors before making new driver descriptors
> +available.
> +
> +Similarly, device reads descriptors from the ring in order and
> +detects that a driver descriptor has been made available.  As
> +processing of descriptors is completed used descriptors are
> +written by the device back into the ring.
> +
> +Note: after reading driver descriptors and starting their
> +processing in order, device might complete their processing out
> +of order.  Used device descriptors are written in the order
> +in which their processing is complete.
> +
> +Device Event Suppression data structure is write-only by the
> +device. It includes information for reducing the number of
> +device events - i.e. driver notifications to device.
> +
> +Driver Event Suppression data structure is read-only by the
> +device. It includes information for reducing the number of
> +driver events - i.e. device interrupts to driver.
> +
> +\subsection{Available and Used Ring Wrap Counters}
> +\label{sec:Packed Virtqueues / Available and Used Ring Wrap Counters}

I find the names a bit unfortunate: it's clear that  it is a
available ring-wrap counter and not an available-ring wrap counter,
but still if I read available ring I kind of think of the available
ring at the moment (which does not exist for packed). 

Could we call these device's ring wrap counter and driver's
ring wrap counter?

> +Each of the driver and the device are expected to maintain,
> +internally, a single-bit ring wrap counter initialized to 1.
> +
> +The counter maintained by the driver is called the Available
> +Ring Wrap Counter. Driver changes the value of this counter
> +each time it makes available the
> +last descriptor in the ring (after making the last descriptor
> +available).
> +
> +The counter maintained by the device is called the Used Ring Wrap
> +Counter.  Device changes the value of this counter
> +each time it uses the last descriptor in
> +the ring (after marking the last descriptor used).
> +
> +It is easy to see that the Available Ring Wrap Counter in the driver matches
> +the Used Ring Wrap Counter in the device when both are processing the same
> +descriptor, or when all available descriptors have been used.
> +
> +To mark a descriptor as available and used, both driver and
> +device use the following two flags:
> +\begin{lstlisting}
> +#define VIRTQ_DESC_F_AVAIL     7
> +#define VIRTQ_DESC_F_USED      15
> +\end{lstlisting}
> +
> +To mark a descriptor as available, driver sets the
> +VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Available
> +Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_USED bit to match the
> +\emph{inverse} value.

I find inverse a bit problematic (as a half mathematician). Inverse is
defined in respect to an operation. If I think modulo arithmetic then
it does not add up. Maybe 'to not match'?

> +
> +To mark a descriptor as used, device sets the
> +VIRTQ_DESC_F_USED bit in Flags to match the internal Used
> +Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_AVAIL bit to match the
> +\emph{same} value.
> +
> +Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different
> +for an available descriptor and equal for a used descriptor.

We cant' turn it around: VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED
different is a necessary but not a sufficient pre-condition for
a descriptor being available; VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED
equal is a necessary but not a sufficient pre-condition for
a descriptor being used. Right?


> +
> +\subsection{Polling of available and used descriptors}
> +\label{sec:Packed Virtqueues / Polling of available and used descriptors}
> +
> +Writes of device and driver descriptors can generally be
> +reordered, but each side (driver and device) are only required to
> +poll (or test) a single location in memory: next device descriptor after
> +the one they processed previously, in circular order.
> +
> +Sometimes device needs to only write out a single used descriptor
> +after processing a batch of multiple available descriptors.  As
> +described in more detail below, this can happen when using
> +descriptor chaining or with in-order
> +use of descriptors.  In this case, device writes out a used
> +descriptor with buffer id of the last descriptor in the group.
> +After processing the used descriptor, both device and driver then
> +skip forward in the ring the number of the remaining descriptors
> +in the group until processing (reading for the driver and writing
> +for the device) the next used descriptor.
> +
> +\subsection{Write Flag}
> +\label{sec:Packed Virtqueues / Write Flag}
> +
> +In an available descriptor, VIRTQ_DESC_F_WRITE bit within Flags
> +is used to mark a descriptor as corresponding to a write-only or
> +read-only element of a buffer.
> +
> +\begin{lstlisting}
> +/* This marks a buffer as device write-only (otherwise device read-only). */

Above you use 'element of the buffer', here (in the C-comment) you
use just 'buffer'.

> +#define VIRTQ_DESC_F_WRITE     2
> +\end{lstlisting}
> +
> +In a used descriptor, this bit it used to specify whether any
> +data has been written by the device into any parts of the buffer.
> +
> +
> +\subsection{Buffer Address and Length}
> +\label{sec:Packed Virtqueues / Buffer Address and Length}
> +
> +In an available descriptor, Buffer Address corresponds to the
> +physical address of the buffer. The length of the buffer assumed
> +to be physically contigious is stored in Buffer Length.

These 'buffer's are again 'blue buffers', that is buffer elements.

> +
> +In a used descriptor, Buffer Address is unused. Buffer Length
> +specifies the length of the buffer that has been initialized
> +(written to) by the device.

I'm confused here. Which color buffer is it now?

> +
> +Buffer length is reserved for used descriptors without the
> +VIRTQ_DESC_F_WRITE flag, and is ignored by drivers.
> +
> +\subsection{Scatter-Gather Support}

[Consistent wording] Both types of virtqueues support scatter-gather
but the term is used only for packed. Maybe we could unify the wording.

> +\label{sec:Packed Virtqueues / Scatter-Gather Support}
> +
> +Some drivers need an ability to supply a list of multiple buffer
> +elements (also known as a scatter/gather list) with a request.
> +Two optional features support this: descriptor
> +chaining and indirect descriptors.
> +
> +If neither feature has been negotiated, each buffer is
> +physically-contigious, either read-only or write-only and is
> +described completely by a single descriptor.
> +

This seems different than split where chaining support is mandatory.
Is there a reason for making both optional?

> +While unusual (most implementations either create all lists
> +solely using non-indirect descriptors, or always use a single
> +indirect element), if both features have been negotiated, mixing
> +direct and direct descriptors in a ring is valid, as long as each
> +list only contains descriptors of a given type.
> +
> +Scatter/gather lists only apply to available descriptors. A
> +single used descriptor corresponds to the whole list.
> +
> +The device limits the number of descriptors in a list through a
> +transport-specific and/or device-specific value. If not limited,
> +the maximum number of descriptors in a list is the virt queue
> +size.
> +
> +\subsection{Next Flag: Descriptor Chaining}
> +\label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
> +
> +The VIRTIO_F_LIST_DESC feature allows driver to supply

This feature does not seem to appear anywhere else in the entire document.

> +a scatter/gather list to the device
> +by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT in
> +Flags for all but the last available descriptor.
> +
> +\begin{lstlisting}
> +/* This marks a buffer as continuing. */
> +#define VIRTQ_DESC_F_NEXT   1
> +\end{lstlisting}
> +
> +Buffer ID is included in the last descriptor in the list.
> +
> +The driver always makes the the first descriptor in the list
> +available after the rest of the list has been written out into
> +the ring. This guarantees that the device will never observe a
> +partial scatter/gather list in the ring.
> +
> +Device only writes out a single used descriptor for the whole
> +list. It then skips forward according to the number of
> +descriptors in the list. Driver needs to keep track of the size
> +of the list corresponding to each buffer ID, to be able to skip
> +to where the next used descriptor is written by the device.
> +
> +For example, if descriptors are used in the same order in which
> +they are made available, this will result in the used descriptor
> +overwriting the first available descriptor in the list, the used
> +descriptor for the next list overwriting the first available
> +descriptor in the next list, etc.
> +
> +VIRTQ_DESC_F_NEXT is reserved in used descriptors, and
> +should be ignored by drivers.
> +
> +\subsection{Indirect Flag: Scatter-Gather Support}
> +\label{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
> +
> +Some devices benefit by concurrently dispatching a large number
> +of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this. To increase
> +ring capacity the driver can store a (read-only by the device) table of indirect
> +descriptors anywhere in memory, and insert a descriptor in main
> +virtqueue (with \field{Flags} bit VIRTQ_DESC_F_INDIRECT on) that refers to
> +a memory buffer

This is again a blueish buffer.

> +containing this indirect descriptor table; \field{addr} and \field{len}
> +refer to the indirect table address and length in bytes,
> +respectively.
> +\begin{lstlisting}
> +/* This means the buffer contains a table of buffer descriptors. */

'a table of buffer descriptors' is a new term. 

> +#define VIRTQ_DESC_F_INDIRECT   4
> +\end{lstlisting}
> +
> +The indirect table layout structure looks like this
> +(\field{len} is the Buffer Length of the descriptor that refers to this table,
> +which is a variable, so this code won't compile):
> +
> +\begin{lstlisting}
> +struct indirect_descriptor_table {
> +        /* The actual descriptor structures (struct Desc each) */
> +        struct Desc desc[len / sizeof(struct Desc)];

Could not find struct Desc. Was it supposed to be  struct virtq_desc?


> +};
> +\end{lstlisting}
> +
> +The first descriptor is located at start of the indirect
> +descriptor table, additional indirect descriptors come
> +immediately afterwards. \field{Flags} bit VIRTQ_DESC_F_WRITE is the
> +only valid flag for descriptors in the indirect table. Others
> +are reserved and are ignored by the device.
> +Buffer ID is also reserved and is ignored by the device.
> +
> +In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
> +is reserved and is ignored by the device.
> +
> +\subsection{Multi-buffer requests}
> +\label{sec:Packed Virtqueues / Multi-descriptor batches}
> +Some devices combine multiple buffers as part of processing of a
> +single request.  These devices always make the first
> +descriptor in the request available after the rest of the request
> +has been written out request the ring. This guarantees that the
> +driver will never observe a partial request in the ring.
> +

Why does it have to be multiple buffers (I suppose red ones) then?

You are making a statement about devices (probably actually drivers
as we talk about 'making available') behavior AFAIU so I'm curious
how does this translate to split virtqueues?

> +
> +\subsection{Driver and Device Event Suppression}
> +\label{sec:Packed Virtqueues / Driver and Device Event Suppression}
> +In many systems driver and device notifications involve
> +significant overhead. To mitigate this overhead,
> +each virtqueue includes two identical structures used for
> +controlling notifications between device and driver.
> +
> +Driver Event Suppression structure is read-only by the
> +device and controls the events sent by the device
> +to the driver (e.g. interrupts).
> +
> +Device Event Suppression structure is read-only by
> +the driver and controls the events sent by the driver
> +to the device (e.g. IO).
> +
> +Each of these Event Suppression structures controls
> +both Descriptor Ring events and structure events, and
> +each includes the following fields:
> +
> +\begin{description}
> +\item [Descriptor Ring Change Event Flags] Takes values:
> +\begin{itemize}
> +\item 00b enable events
> +\item 01b disable events
> +\item 10b enable events for a specific descriptor
> +(as specified by Descriptor Ring Change Event Offset/Wrap Counter).
> +Only valid if VIRTIO_F_RING_EVENT_IDX has been negotiated.
> +\item 11b reserved
> +\end{itemize}
> +\item [Descriptor Ring Change Event Offset] If Event Flags set to descriptor
> +specific event: offset within the ring (in units of descriptor
> +size). Event will only trigger when this descriptor is
> +made available/used respectively.
> +\item [Descriptor Ring Change Event Wrap Counter] If Event Flags set to descriptor
> +specific event: offset within the ring (in units of descriptor
> +size). Event will only trigger when Ring Wrap Counter
> +matches this value and a descriptor is
> +made available/used respectively.
> +\end{description}
> +
> +After writing out some descriptors, both device and driver
> +are expected to consult the relevant structure to find out
> +whether interrupt/notification should be sent.
> +
> +\subsubsection{Driver notifications}
> +\label{sec:Packed Virtqueues / Driver notifications}
> +Whenever not suppressed by Device Event Suppression,
> +driver is required to notify the device after
> +making changes to the virtqueue.
> +
> +Some devices benefit from ability to find out the number of
> +available descriptors in the ring, and whether to send
> +interrupts to drivers without accessing virtqueue in memory:
> +for efficiency or as a debugging aid.
> +
> +To help with these optimizations, driver notifications
> +to the device include the following information:
> +
> +\begin{itemize}
> +\item VQ number
> +\item Offset (in units of descriptor size) within the ring
> +      where the next available descriptor will be written
> +\item Wrap Counter referring to the next available
> +      descriptor
> +\end{itemize}
> +
> +Note that driver can trigger multiple notifications even without
> +making any more changes to the ring. These would then have
> +identical \field{Offset} and \field{Wrap Counter} values.
> +
> +\subsubsection{Structure Size and Alignment}
> +\label{sec:Packed Virtqueues / Structure Size and Alignment}
> +
> +Each part of the virtqueue is physically-contiguous in guest memory,
> +and has different alignment requirements.
> +
> +The memory aligment and size requirements, in bytes, of each part of the
> +virtqueue are summarized in the following table:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Virtqueue Part    & Alignment & Size \\
> +\hline \hline
> +Descriptor Ring  & 16        & $16 * $(Queue Size) \\
> +\hline
> +Device Event Suppression    & 4         & 4 \\
> + \hline
> +Driver Event Suppression         & 4         & 4 \\
> + \hline
> +\end{tabular}
> +
> +The Alignment column gives the minimum alignment for each part
> +of the virtqueue.
> +
> +The Size column gives the total number of bytes for each
> +part of the virtqueue.
> +
> +Queue Size corresponds to the maximum number of descriptors in the
> +virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
> +can be queued at any given time.}.  Queue Size value does not
> +have to be a power of 2 unless enforced by the transport.
> +
> +\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a
> +Virtio Device / Packed Virtqueues}
> +The driver MUST ensure that the physical address of the first byte
> +of each virtqueue part is a multiple of the specified alignment value
> +in the above table.
> +
> +\devicenormative{\subsection}{Virtqueues}{Basic Facilities of a
> +Virtio Device / Packed Virtqueues}
> +The device MUST start processing driver descriptors in the order
> +in which they appear in the ring.
> +The device MUST start writing device descriptors into the ring in
> +the order in which they complete.
> +Device MAY reorder descriptor writes once they are started.
> +
> +\subsection{The Virtqueue Descriptor Format}\label{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue
> +Descriptor Format}
> +
> +The available descriptor refers to the buffers the driver is sending

Don't get the plural. This is a 'blue buffer' I guess.

> +to the device. \field{addr} is a physical address, and the
> +descriptor is identified with a buffer using the \field{id} field.

Reads strange. And this buffer is probably 'red buffer', but then it
does not make sense.

> +
> +\begin{lstlisting}
> +struct virtq_desc {
> +        /* Buffer Address. */
> +        le64 addr;
> +        /* Buffer Length. */
> +        le32 len;
> +        /* Buffer ID. */
> +        le16 id;
> +        /* The flags depending on descriptor type. */
> +        le16 flags;
> +};
> +\end{lstlisting}
> +
> +The descriptor ring is zero-initialized.
> +
> +\subsection{Event Suppression Structure Format}\label{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Event Suppression Structure
> +Format}
> +
> +The following structure is used to reduce the number of
> +notifications sent between driver and device.
> +
> +\begin{lstlisting}
> +__le16 desc_event_off : 15; /* Descriptor Event Offset */
> +int    desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
> +__le16 desc_event_flags : 2; /* Descriptor Event Flags */
> +\end{lstlisting}
> +
> +\subsection{Driver Notification Format}\label{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Driver Notification Format}
> +
> +The following structure is used to notify device of
> +device events - i.e. available descriptors:
> +
> +\begin{lstlisting}
> +__le16 vqn;
> +__le16 next_off : 15;
> +int    next_wrap : 1;
> +\end{lstlisting}
> +
> +\devicenormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table}

s/Descriptor Table/Descriptor Ring/ ?

> +A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
> +read a device-writable buffer.

These are again 'blue buffers' aka 'buffer elements'.

> +A device MUST NOT use a descriptor unless it observes
> +VIRTQ_DESC_F_AVAIL bit in its \field{flags} being changed.
> +A device MUST NOT change a descriptor after changing it's
> +VIRTQ_DESC_F_USED bit in its \field{flags}.
> +
> +\drivernormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / PAcked Virtqueues / The Virtqueue Descriptor Table}

s/Descriptor Table/Descriptor Ring/ ?

> +A driver MUST NOT change a descriptor unless it observes
> +VIRTQ_DESC_F_USED bit in its \field{flags} being changed.
> +A driver MUST NOT change a descriptor after changing
> +VIRTQ_DESC_F_USED bit in its \field{flags}.
> +When notifying the device, driver MUST set
> +\field{next_off} and
> +\field{next_wrap} to match the next descriptor
> +not yet made available to the device.
> +A driver MAY send multiple notifications without making
> +any new descriptors available to the device.
> +
> +\drivernormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
> +Virtio Device / Packed Virtqueues / Scatter-Gather Support}
> +A driver MUST NOT create a descriptor list longer than allowed
> +by the device.
> +
> +A driver MUST NOT create a descriptor list longer than the Queue
> +Size.
> +
> +This implies that loops in the descriptor list are forbidden!
> +
> +The driver MUST place any device-writable descriptor elements after
> +any device-readable descriptor elements.
> +
> +A driver MUST NOT depend on the device to use more descriptors
> +to be able to write out all descriptors in a list. A driver
> +MUST make sure there's enough space in the ring
> +for the whole list before making the first descriptor in the list
> +available to the device.
> +
> +A driver MUST NOT make the first descriptor in the list
> +available before initializing the rest of the descriptors.
> +
> +\devicenormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
> +Virtio Device / Packed Virtqueues / Scatter-Gather Support}
> +The device MUST use descriptors in a list chained by the
> +VIRTQ_DESC_F_NEXT flag in the same order that they
> +were made available by the driver.
> +
> +The device MAY limit the number of buffers it will allow in a
> +list.
> +
> +\drivernormative{\subsection}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}

s/Descriptor Table/Descriptor Ring/ ?

> +The driver MUST NOT set the DESC_F_INDIRECT flag unless the
> +VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
> +set any flags except DESC_F_WRITE within an indirect descriptor.
> +
> +A driver MUST NOT create a descriptor chain longer than allowed
> +by the device.
> +
> +A driver MUST NOT write direct descriptors with
> +DESC_F_INDIRECT set in a scatter-gather list linked by
> +VIRTQ_DESC_F_NEXT.
> +\field{flags}.
> +
> +\subsection{Virtqueue Operation}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Virtqueue Operation}
> +
> +There are two parts to virtqueue operation: supplying new
> +available buffers to the device, and processing used buffers from
> +the device.
> +
> +What follows is the requirements of each of these two parts
> +when using the packed virtqueue format in more detail.
> +
> +\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device}
> +
> +The driver offers buffers to one of the device's virtqueues as follows:

This is probably a 'red buffer'

> +
> +\begin{enumerate}
> +\item The driver places the buffer into free descriptor in the Descriptor Ring.

What is a free descriptor? s/free/next?

This is probably a 'blue buffer' as a 'red buffer' is not necessarily expressible
by a single 'blue buffer'.

> +
> +\item The driver performs a suitable memory barrier to ensure that it updates
> +  the descriptor(s) before checking for notification suppression.
> +
> +\item If notifications are not suppressed, the driver notifies the device
> +    of the new available buffers.
> +\end{enumerate}
> +
> +What follows is the requirements of each stage in more detail.
> +
> +\subsubsection{Placing Available Buffers Into The Descriptor Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Available Buffers Into The Descriptor Ring}
> +
> +For each buffer element, b:
> +
> +\begin{enumerate}
> +\item Get the next descriptor table entry, d

s/descriptor table/descriptor ring/

> +\item Get the next free buffer id value
> +\item Set \field{d.addr} to the physical address of the start of b
> +\item Set \field{d.len} to the length of b.
> +\item Set \field{d.id} to the buffer id
> +\item Calculate the flags as follows:
> +\begin{enumerate}
> +\item If b is device-writable, set the VIRTQ_DESC_F_WRITE bit to 1, otherwise 0
> +\item Set VIRTQ_DESC_F_AVAIL bit to the current value of the Available Ring Wrap Counter
> +\item Set VIRTQ_DESC_F_USED bit to inverse value
> +\end{enumerate}
> +\item Perform a memory barrier to ensure that the descriptor has
> +      been initialized
> +\item Set \field{d.flags} to the calculated flags value
> +\item If d is the last descriptor in the ring, toggle the
> +      Available Ring Wrap Counter
> +\item Otherwise, increment d to point at the next descriptor
> +\end{enumerate}
> +
> +This makes a single descriptor buffer available. However, in
> +general the driver MAY make use of a batch of descriptors as part
> +of a single request. In that case, it defers updating
> +the descriptor flags for the first descriptor
> +(and the previous memory barrier) until after the rest of
> +the descriptors have been initialized.
> +
> +Once the descriptor \field{flags} is updated by the driver, this exposes the
> +descriptor and its contents.  The device MAY
> +access the descriptor and any following descriptors the driver created and the
> +memory they refer to immediately.
> +
> +\drivernormative{\paragraph}{Updating flags}{Basic Facilities of
> +a Virtio Device / Packed Virtqueues / Supplying Buffers to The
> +Device / Updating flags}
> +The driver MUST perform a suitable memory barrier before the
> +\field{flags} update, to ensure the
> +device sees the most up-to-date copy.

Necessary only for the first 'blue buffer' whose flags are set last?

> +
> +\subsubsection{Notifying The Device}\label{sec:Basic Facilities
> +of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> +
> +The actual method of device notification is bus-specific, but generally
> +it can be expensive.  So the device MAY suppress such notifications if it
> +doesn't need them, using the Driver Event Suppression structure
> +as detailed in section \ref{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Event
> +Suppression Structure Format}.
> +
> +The driver has to be careful to expose the new \field{flags}
> +value before checking if notifications are suppressed.
> +
> +\subsubsection{Implementation Example}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Implementation Example}
> +
> +Below is an example driver code. It does not attempt to reduce
> +the number of device interrupts, neither does it support
> +the VIRTIO_F_RING_EVENT_IDX feature.
> +
> +\begin{lstlisting}
> +
> +first = vq->next_avail;
> +id = alloc_id(vq);
> +
> +for (each buffer element b) {
> +        vq->desc[vq->next_avail].address = get_addr(b);
> +        vq->desc[vq->next_avail].len = get_len(b);
> +        init_desc(vq->next_avail, b);

What is init_desc? Can't find it elsewhere.

> +        avail = vq->avail_wrap_count;
> +        used = !vq->avail_wrap_count;
> +        f = get_flags(b) | (avail << VIRTQ_DESC_F_AVAIL) | (used << VIRTQ_DESC_F_USED);
> +	/* Don't mark the 1st descriptor available until all of them are ready. */
> +        if (vq->next_avail == first) {
> +                flags = f;
> +        } else {
> +                vq->desc[vq->next_avail].flags = f;
> +        }
> +
> +	vq->next_avail++;
> +
> +	if (vq->next_avail > vq->size) {
> +		vq->next_avail = 0;
> +		vq->avail_wrap_count \^= 1;
> +	}
> +
> +
> +}
> +vq->desc[vq->next_avail].id = id;
> +write_memory_barrier();
> +vq->desc[first].flags = flags;
> +
> +memory_barrier();
> +
> +if (vq->device_event.flags != 0x2) {
> +        notify_device(vq, vq->next_avail, vq->avail_wrap_count);
> +}
> +
> +\end{lstlisting}
> +
> +
> +\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
> +The driver MUST perform a suitable memory barrier before reading
> +the Driver Event Suppression structure, to avoid missing a notification.
> +
> +\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Receiving Used Buffers From The Device}
> +
> +Once the device has used buffers referred to by a descriptor (read from or written to them, or
> +parts of both, depending on the nature of the virtqueue and the
> +device), it interrupts the driver
> +as detailed in section \ref{sec:Basic
> +Facilities of a Virtio Device / Packed Virtqueues / Event
> +Suppression Structure Format}.
> +
> +\begin{note}
> +For optimal performance, a driver MAY disable interrupts while processing
> +the used buffers, but beware the problem of missing interrupts between
> +emptying the ring and reenabling interrupts.  This is usually handled by
> +re-checking for more used buffers after interrups are re-enabled:
> +\end{note}
> +
> +\begin{lstlisting}
> +vq->driver_event.flags = 0x2;
> +
> +for (;;) {
> +        struct virtq_desc *d = vq->desc[vq->next_used];
> +
> +        flags = d->flags;
> +        bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> +        bool used = flags & (1 << VIRTQ_DESC_F_USED);
> +
> +        if (avail != used) {
> +                vq->driver_event.flags = 0x1;
> +                memory_barrier();
> +
> +                flags = d->flags;
> +                bool avail = flags & (1 << VIRTQ_DESC_F_AVAIL);
> +                bool used = flags & (1 << VIRTQ_DESC_F_USED);
> +                if (avail != used) {
> +                        break;
> +                }
> +
> +                vq->driver_event.flags = 0x2;
> +        }
> +
> +	read_memory_barrier();
> +        process_buffer(d);
> +        vq->next_used++;
> +        if (vq->next_used > vq->size) {
> +                vq->next_used = 0;
> +        }
> +}

I would have expected avail_wrap_count showing up here somewhere. Was I
wrong?

> +\end{lstlisting}
> 

Pff, it ended up being a mix of me being petty about wording and
hopefully more productive complaints. I hope it's still bearable.

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v7 09/11] content: in-order buffer use

[Michael S. Tsirkin]

Using descriptors in-order is sometimes benefitial.  Add an option for
that - per-format detail allowing more optimizations will be added by
follow-up patches.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/content.tex b/content.tex
index 4d522cc..f3e5f9b 100644
--- a/content.tex
+++ b/content.tex
@@ -245,6 +245,15 @@ a device event - i.e. send an interrupt to the driver.
 Device reports the number of bytes it has written to memory for
 each buffer it uses. This is referred to as ``used length''.
 
+Device is not generally required to use buffers in
+the same order in which they have been made available
+by the driver.
+
+Some devices always use descriptors in the same order in which
+they have been made available. These devices can offer the
+VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+might allow optimizations or simplify driver and/or device code.
+
 Each virtqueue can consist of up to 3 parts:
 \begin{itemize}
 \item Descriptor Area - used for describing buffers
@@ -5245,6 +5254,9 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
   support for the packed virtqueue layout as described in
   \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
+  \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
+  that all buffers are used by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5270,6 +5282,9 @@ translates bus addresses from the device into physical addresses in memory.
 A device MAY fail to operate further if VIRTIO_F_IOMMU_PLATFORM is not
 accepted.
 
+If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
+buffers in the same order in which they have been available.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 09/11] content: in-order buffer use

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:08 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Using descriptors in-order is sometimes benefitial.  Add an option for

s/benefitial/beneficial/

> that - per-format detail allowing more optimizations will be added by
> follow-up patches.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 09/11] content: in-order buffer use

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 431 bytes --]

On Tue, Jan 23, 2018 at 02:01:08AM +0200, Michael S. Tsirkin wrote:
> Using descriptors in-order is sometimes benefitial.  Add an option for
> that - per-format detail allowing more optimizations will be added by
> follow-up patches.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio] [PATCH v7 11/11] split-ring: in order feature

[Michael S. Tsirkin]

For a split ring, require that drivers use descriptors in order too.
This allows devices to skip reading the available ring.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 split-ring.tex | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/split-ring.tex b/split-ring.tex
index f976e45..43c496c 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -203,6 +203,10 @@ struct virtq_desc {
 The number of descriptors in the table is defined by the queue size
 for this virtqueue: this is the maximum possible descriptor chain length.
 
+If VIRTIO_F_IN_ORDER has been negotiated, driver uses
+descriptors in ring order: starting from offset 0 in the table,
+and wrapping around at the end of the table.
+
 \begin{note}
 The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
 referred to this structure as vring_desc, and the constants as
@@ -218,6 +222,12 @@ purposes).
 Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
 this implies that loops in the descriptor chain are forbidden!
 
+If VIRTIO_F_IN_ORDER has been negotiated, and when making a
+descriptor with VRING_DESC_F_NEXT set in \field{flags} at offset
+$x$ in the table available to the device, driver MUST set
+\field{next} to $0$ for the last descriptor in the table
+(where $x = queue_size - 1$) and to $x + 1$ for the rest of the descriptors.
+
 \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
 
 Some devices benefit by concurrently dispatching a large number
@@ -247,6 +257,10 @@ chained by \field{next}. An indirect descriptor without a valid \field{next}
 A single indirect descriptor
 table can include both device-readable and device-writable descriptors.
 
+If VIRTIO_F_IN_ORDER has been negotiated, indirect descriptors
+use sequential indices, in-order: index 0 followed by index 1
+followed by index 2, etc.
+
 \drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
 The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
 VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
@@ -259,6 +273,10 @@ the device.
 A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
 in \field{flags}.
 
+If VIRTIO_F_IN_ORDER has been negotiated, indirect descriptors
+MUST appear sequentially, with \field{next} taking the value
+of 1 for the 1st descriptor, 2 for the 2nd one, etc.
+
 \devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
 The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 11/11] split-ring: in order feature

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:09 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> For a split ring, require that drivers use descriptors in order too.
> This allows devices to skip reading the available ring.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  split-ring.tex | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/split-ring.tex b/split-ring.tex
> index f976e45..43c496c 100644
> --- a/split-ring.tex
> +++ b/split-ring.tex

> @@ -218,6 +222,12 @@ purposes).
>  Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
>  this implies that loops in the descriptor chain are forbidden!
>  
> +If VIRTIO_F_IN_ORDER has been negotiated, and when making a
> +descriptor with VRING_DESC_F_NEXT set in \field{flags} at offset
> +$x$ in the table available to the device, driver MUST set
> +\field{next} to $0$ for the last descriptor in the table
> +(where $x = queue_size - 1$) and to $x + 1$ for the rest of the descriptors.

The underscore in queue_size makes this a subscript s, which is
probably not what you want :)

> +
>  \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
>  
>  Some devices benefit by concurrently dispatching a large number

Otherwise,

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 11/11] split-ring: in order feature

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

On Tue, Jan 23, 2018 at 02:01:09AM +0200, Michael S. Tsirkin wrote:
> For a split ring, require that drivers use descriptors in order too.
> This allows devices to skip reading the available ring.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  split-ring.tex | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio] [PATCH v7 10/11] packed-ring: add in order support

[Michael S. Tsirkin]

Support in-order requests for packed rings.
This allows selective write-out of used descriptors.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 packed-ring.tex | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/packed-ring.tex b/packed-ring.tex
index b6cb979..3bbde5b 100644
--- a/packed-ring.tex
+++ b/packed-ring.tex
@@ -259,6 +259,30 @@ Buffer ID is also reserved and is ignored by the device.
 In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
 is reserved and is ignored by the device.
 
+\subsection{In-order use of descriptors}
+\label{sec:Packed Virtqueues / In-order use of descriptors}
+
+Some devices always use descriptors in the same order in which
+they have been made available. These devices can offer the
+VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch.
+
+Device then skips forward in the ring according to the size of
+the batch. Driver needs to look up the used Buffer ID and
+calculate the batch size to be able to advance to where the next
+used descriptor will be written by the device.
+
+This will result in the used descriptor overwriting the first
+available descriptor in the batch, the used descriptor for the
+next batch overwriting the first available descriptor in the next
+batch, etc.
+
+The skipped buffers (for which no used descriptor was written)
+are assumed to have been used (read or written) by the
+device completely.
+
 \subsection{Multi-buffer requests}
 \label{sec:Packed Virtqueues / Multi-descriptor batches}
 Some devices combine multiple buffers as part of processing of a
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 10/11] packed-ring: add in order support

[Cornelia Huck]

On Tue, 23 Jan 2018 02:01:09 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Support in-order requests for packed rings.
> This allows selective write-out of used descriptors.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  packed-ring.tex | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH v7 10/11] packed-ring: add in order support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 378 bytes --]

On Tue, Jan 23, 2018 at 02:01:09AM +0200, Michael S. Tsirkin wrote:
> Support in-order requests for packed rings.
> This allows selective write-out of used descriptors.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  packed-ring.tex | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]