[virtio] [PATCH v9 00/16] packed ring layout spec

[virtio] [PATCH v9 00/16] packed ring layout spec

[Michael S. Tsirkin]

This addresses comments on v8.

Thanks a lot to all reviewers of v8 - I hope we are
finally there or almost there.


A compiled version can
be found under https://github.com/oasis-tcs/virtio-docs.git

virtio-v1.1-packed-wd09-diff.pdf  virtio-v1.1-packed-wd09.pdf

for redline and clean versions, respectively.
If you are interested in changes from v8, that's in

virtio-v1.1-w08-to-wd09-diff.pdf

in this directory.

Note: please do not try to edit the pdf and post comments
in the edited file. Please post comments in a text
format, as pdfs are not archived with the list.

TODO: support for actual passthrough devices will likely
require more new features, such as requirement for
stronger memory barriers.

Note: should this proposal be accepted and approved, one or more
      claims disclosed to the TC admin and listed on the Virtio TC
      IPR page https://github.com/oasis-tcs/virtio-admin/blob/master/IPR.md
      might become Essential Claims.

Changes from v8:

 packed-ring: clarify wording on s/g conformance
 split-ring: typo: aligment
 packed-ring: typo: aligment
 packed-ring: clarify multi-buffer requests
 packed-ring: necessary, not sufficient condition
 packed-ring: support skipping used descritors
 packed-ring: fix pseudo code
 packed-ring: format pseudocode using spaces
 packed-ring: drop << in pseudocode
 packed-ring: clarify event suppression format
 packed-ring: chained descriptors are always on
 VIRTIO_F_NOTIFICATION_DATA: switch to bitfields
 introduction: document bitfield notation
 VIRTIO_F_NOTIFICATION_DATA: minor cleanups
 content: VIRTIO_F_NOTIFICATION_DATA is option
 VIRTIO_F_NOTIFICATION_DATA: improve formatting

Changes from v7:
- new notitfication_data feature, supported for all
  transports and formats
- addressed all outstanding comments

Changes from v6:
- isolate in-order feature to a separate set of patches
  (reduces scope in case there's more discussion around it)
- support in-order option for split rings
- update all references to available/used ring in spec
  to a format-agnostic terminology
- minor changes to event suppression format
- minor changes to notification format
- lots of new conformance clauses

Changes from v5:
- scope reductions (see below). We can add more
  features down the road, hopefully reduced scope will be enough
  to finalize spec soon.
- cleanup and integrate in the spec
- pseudo-code

Deferred features:
- dropped _F_DESC_LIST, 1.0 includes this unconditionally, we
  can do same
- dropped event structure change notifications - needed for
  efficient hardware implementations but let's add this on top

3 1st patches just move text around so all virtio 1.0
things are in the same place. 2 last ones add the new layout

Option to mark descriptors as not generating events isn't
yet implemented. Again, let's add this on top.

I also note that for hardware implementations, a different
set of memory barriers is needed. Again, let's add this on top

Michael S. Tsirkin (16):
  introduction: document bitfield notation
  content: move 1.0 queue format out to a separate section
  content: move ring text out to a separate file
  content: move virtqueue operation description
  content: len -> used length, used ring -> vq
  content: generalize transport ring part naming
  content: generalize rest of text
  split-ring: generalize text
  split-ring: typo: aligment
  packed virtqueues: more efficient virtqueue layout
  content: in-order buffer use
  packed-ring: add in order support
  split-ring: in order feature
  VIRTIO_F_NOTIFICATION_DATA: extra data to devices
  makediff: update to show diff from master
  REVISION: set to 1.1 packed wd09

 REVISION         |   2 +-
 conformance.tex  |   5 +-
 content.tex      | 919 ++++++++++++++-----------------------------------------
 introduction.tex |  18 ++
 makediff.sh      |   3 +-
 notifications.c  |   3 +
 packed-ring.tex  | 704 ++++++++++++++++++++++++++++++++++++++++++
 split-ring.tex   | 689 +++++++++++++++++++++++++++++++++++++++++
 8 files changed, 1655 insertions(+), 688 deletions(-)
 create mode 100644 notifications.c
 create mode 100644 packed-ring.tex
 create mode 100644 split-ring.tex

-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 01/16] introduction: document bitfield notation

[Michael S. Tsirkin]

Bitfields are a useful and familiar way to specify sub-byte structure
layout. The only issue is that bitfield order isn't portable across
architectures.  Document that we list bitfields from least to
most significant one, and warn about portability issues.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 introduction.tex | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/introduction.tex b/introduction.tex
index 979881e..3cb7a70 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -157,5 +157,23 @@ in little-endian byte order.
 in big-endian byte order.
 \end{description}
 
+When documenting sub-byte data fields, C-like bitfield notation
+is used. Fields within an integer are always listed in order,
+from the least significant to the most significant bit.
+
+For example:
+\begin{lstlisting}
+be16 A : 15;
+be16 B : 1;
+\end{lstlisting}
+documents the value A stored in the low 15 bit of a 16 bit
+integer and the value B stored in the high bit of the 16 bit
+integer, the integer in turn using the big-endian byte order.
+
+Note that this notation typically matches the way bitfields are
+packed by C compilers on little-endian architectures but not the
+way bitfields are packed by C compilers on big-endian
+architectures.
+
 \newpage
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 02/16] content: move 1.0 queue format out to a separate section

[Michael S. Tsirkin]

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
---
 content.tex | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/content.tex b/content.tex
index c7ef7fd..4483a4b 100644
--- a/content.tex
+++ b/content.tex
@@ -230,7 +230,30 @@ result.
 The mechanism for bulk data transport on virtio devices is
 pretentiously called a virtqueue. Each device can have zero or more
 virtqueues\footnote{For example, the simplest network device has one virtqueue for
-transmit and one for receive.}.  Each queue has a 16-bit queue size
+transmit and one for receive.}.
+
+Driver makes requests available to device by adding
+an available buffer to the queue - i.e. adding a buffer
+describing the request to a virtqueue, and optionally triggering
+a driver event - i.e. sending a notification to the device.
+
+Device executes the requests and - when complete - adds
+a used buffer to the queue - i.e. lets the driver
+know by marking the buffer as used. Device can then trigger
+a device event - i.e. send an interrupt to the driver.
+
+For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
+
+\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
+The split virtqueue format is the original format used by legacy
+virtio devices.  The split virtqueue format separates the
+virtqueue into several parts, where each part is write-able by
+either the driver or the device, but not both. Multiple
+locations need to be updated when making a buffer available
+and when marking it as used.
+
+
+Each queue has a 16-bit queue size
 parameter, which sets the number of entries and implies the total size
 of the queue.
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 03/16] content: move ring text out to a separate file

[Michael S. Tsirkin]

Will be easier to manage this way.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
---
 content.tex    | 499 +--------------------------------------------------------
 split-ring.tex | 498 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 499 insertions(+), 498 deletions(-)
 create mode 100644 split-ring.tex

diff --git a/content.tex b/content.tex
index 4483a4b..5b4c4e9 100644
--- a/content.tex
+++ b/content.tex
@@ -244,504 +244,7 @@ a device event - i.e. send an interrupt to the driver.
 
 For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
 
-\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
-The split virtqueue format is the original format used by legacy
-virtio devices.  The split virtqueue format separates the
-virtqueue into several parts, where each part is write-able by
-either the driver or the device, but not both. Multiple
-locations need to be updated when making a buffer available
-and when marking it as used.
-
-
-Each queue has a 16-bit queue size
-parameter, which sets the number of entries and implies the total size
-of the queue.
-
-Each virtqueue consists of three parts:
-
-\begin{itemize}
-\item Descriptor Table
-\item Available Ring
-\item Used Ring
-\end{itemize}
-
-where each part is physically-contiguous in guest memory,
-and has different alignment requirements.
-
-The memory aligment and size requirements, in bytes, of each part of the
-virtqueue are summarized in the following table:
-
-\begin{tabular}{|l|l|l|}
-\hline
-Virtqueue Part    & Alignment & Size \\
-\hline \hline
-Descriptor Table  & 16        & $16 * $(Queue Size) \\
-\hline
-Available Ring    & 2         & $6 + 2 * $(Queue Size) \\
- \hline
-Used Ring         & 4         & $6 + 8 * $(Queue Size) \\
- \hline
-\end{tabular}
-
-The Alignment column gives the minimum alignment for each part
-of the virtqueue.
-
-The Size column gives the total number of bytes for each
-part of the virtqueue.
-
-Queue Size corresponds to the maximum number of buffers in the
-virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
-can be queued at any given time.}.  Queue Size value is always a
-power of 2.  The maximum Queue Size value is 32768.  This value
-is specified in a bus-specific way.
-
-When the driver wants to send a buffer to the device, it fills in
-a slot in the descriptor table (or chains several together), and
-writes the descriptor index into the available ring.  It then
-notifies the device. When the device has finished a buffer, it
-writes the descriptor index into the used ring, and sends an interrupt.
-
-\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a Virtio Device / Virtqueues}
-The driver MUST ensure that the physical address of the first byte
-of each virtqueue part is a multiple of the specified alignment value
-in the above table.
-
-\subsection{Legacy Interfaces: A Note on Virtqueue Layout}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
-
-For Legacy Interfaces, several additional
-restrictions are placed on the virtqueue layout:
-
-Each virtqueue occupies two or more physically-contiguous pages
-(usually defined as 4096 bytes, but depending on the transport;
-henceforth referred to as Queue Align)
-and consists of three parts:
-
-\begin{tabular}{|l|l|l|}
-\hline
-Descriptor Table & Available Ring (\ldots padding\ldots) & Used Ring \\
-\hline
-\end{tabular}
-
-The bus-specific Queue Size field controls the total number of bytes
-for the virtqueue.
-When using the legacy interface, the transitional
-driver MUST retrieve the Queue Size field from the device
-and MUST allocate the total number of bytes for the virtqueue
-according to the following formula (Queue Align given in qalign and
-Queue Size given in qsz):
-
-\begin{lstlisting}
-#define ALIGN(x) (((x) + qalign) & ~qalign)
-static inline unsigned virtq_size(unsigned int qsz)
-{
-     return ALIGN(sizeof(struct virtq_desc)*qsz + sizeof(u16)*(3 + qsz))
-          + ALIGN(sizeof(u16)*3 + sizeof(struct virtq_used_elem)*qsz);
-}
-\end{lstlisting}
-
-This wastes some space with padding.
-When using the legacy interface, both transitional
-devices and drivers MUST use the following virtqueue layout
-structure to locate elements of the virtqueue:
-
-\begin{lstlisting}
-struct virtq {
-        // The actual descriptors (16 bytes each)
-        struct virtq_desc desc[ Queue Size ];
-
-        // A ring of available descriptor heads with free-running index.
-        struct virtq_avail avail;
-
-        // Padding to the next Queue Align boundary.
-        u8 pad[ Padding ];
-
-        // A ring of used descriptor heads with free-running index.
-        struct virtq_used used;
-};
-\end{lstlisting}
-
-\subsection{Legacy Interfaces: A Note on Virtqueue Endianness}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
-
-Note that when using the legacy interface, transitional
-devices and drivers MUST use the native
-endian of the guest as the endian of fields and in the virtqueue.
-This is opposed to little-endian for non-legacy interface as
-specified by this standard.
-It is assumed that the host is already aware of the guest endian.
-
-\subsection{Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing}
-The framing of messages with descriptors is
-independent of the contents of the buffers. For example, a network
-transmit buffer consists of a 12 byte header followed by the network
-packet. This could be most simply placed in the descriptor table as a
-12 byte output descriptor followed by a 1514 byte output descriptor,
-but it could also consist of a single 1526 byte output descriptor in
-the case where the header and packet are adjacent, or even three or
-more descriptors (possibly with loss of efficiency in that case).
-
-Note that, some device implementations have large-but-reasonable
-restrictions on total descriptor size (such as based on IOV_MAX in the
-host OS). This has not been a problem in practice: little sympathy
-will be given to drivers which create unreasonably-sized descriptors
-such as by dividing a network packet into 1500 single-byte
-descriptors!
-
-\devicenormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
-The device MUST NOT make assumptions about the particular arrangement
-of descriptors.  The device MAY have a reasonable limit of descriptors
-it will allow in a chain.
-
-\drivernormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
-The driver MUST place any device-writable descriptor elements after
-any device-readable descriptor elements.
-
-The driver SHOULD NOT use an excessive number of descriptors to
-describe a buffer.
-
-\subsubsection{Legacy Interface: Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
-
-Regrettably, initial driver implementations used simple layouts, and
-devices came to rely on it, despite this specification wording.  In
-addition, the specification for virtio_blk SCSI commands required
-intuiting field lengths from frame boundaries (see
- \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}~\nameref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation})
-
-Thus when using the legacy interface, the VIRTIO_F_ANY_LAYOUT
-feature indicates to both the device and the driver that no
-assumptions were made about framing.  Requirements for
-transitional drivers when this is not negotiated are included in
-each device section.
-
-\subsection{The Virtqueue Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-
-The descriptor table refers to the buffers the driver is using for
-the device. \field{addr} is a physical address, and the buffers
-can be chained via \field{next}. Each descriptor describes a
-buffer which is read-only for the device (``device-readable'') or write-only for the device (``device-writable''), but a chain of
-descriptors can contain both device-readable and device-writable buffers.
-
-The actual contents of the memory offered to the device depends on the
-device type.  Most common is to begin the data with a header
-(containing little-endian fields) for the device to read, and postfix
-it with a status tailer for the device to write.
-
-\begin{lstlisting}
-struct virtq_desc {
-        /* Address (guest-physical). */
-        le64 addr;
-        /* Length. */
-        le32 len;
-
-/* This marks a buffer as continuing via the next field. */
-#define VIRTQ_DESC_F_NEXT   1
-/* This marks a buffer as device write-only (otherwise device read-only). */
-#define VIRTQ_DESC_F_WRITE     2
-/* This means the buffer contains a list of buffer descriptors. */
-#define VIRTQ_DESC_F_INDIRECT   4
-        /* The flags as indicated above. */
-        le16 flags;
-        /* Next field if flags & NEXT */
-        le16 next;
-};
-\end{lstlisting}
-
-The number of descriptors in the table is defined by the queue size
-for this virtqueue: this is the maximum possible descriptor chain length.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to this structure as vring_desc, and the constants as
-VRING_DESC_F_NEXT, etc, but the layout and values were identical.
-\end{note}
-
-\devicenormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
-read a device-writable buffer (it MAY do so for debugging or diagnostic
-purposes).
-
-\drivernormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
-Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
-this implies that loops in the descriptor chain are forbidden!
-
-\subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-
-Some devices benefit by concurrently dispatching a large number
-of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this (see \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}). To increase
-ring capacity the driver can store a table of indirect
-descriptors anywhere in memory, and insert a descriptor in main
-virtqueue (with \field{flags}\&VIRTQ_DESC_F_INDIRECT on) that refers to memory buffer
-containing this indirect descriptor table; \field{addr} and \field{len}
-refer to the indirect table address and length in bytes,
-respectively.
-
-The indirect table layout structure looks like this
-(\field{len} is the length of the descriptor that refers to this table,
-which is a variable, so this code won't compile):
-
-\begin{lstlisting}
-struct indirect_descriptor_table {
-        /* The actual descriptors (16 bytes each) */
-        struct virtq_desc desc[len / 16];
-};
-\end{lstlisting}
-
-The first indirect descriptor is located at start of the indirect
-descriptor table (index 0), additional indirect descriptors are
-chained by \field{next}. An indirect descriptor without a valid \field{next}
-(with \field{flags}\&VIRTQ_DESC_F_NEXT off) signals the end of the descriptor.
-A single indirect descriptor
-table can include both device-readable and device-writable descriptors.
-
-\drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
-VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
-set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only
-one table per descriptor).
-
-A driver MUST NOT create a descriptor chain longer than the Queue Size of
-the device.
-
-A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
-in \field{flags}.
-
-\devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
-
-The device MUST handle the case of zero or more normal chained
-descriptors followed by a single descriptor with \field{flags}\&VIRTQ_DESC_F_INDIRECT.
-
-\begin{note}
-While unusual (most implementations either create a chain solely using
-non-indirect descriptors, or use a single indirect element), such a
-layout is valid.
-\end{note}
-
-\subsection{The Virtqueue Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
-
-\begin{lstlisting}
-struct virtq_avail {
-#define VIRTQ_AVAIL_F_NO_INTERRUPT      1
-        le16 flags;
-        le16 idx;
-        le16 ring[ /* Queue Size */ ];
-        le16 used_event; /* Only if VIRTIO_F_EVENT_IDX */
-};
-\end{lstlisting}
-
-The driver uses the available ring to offer buffers to the
-device: each ring entry refers to the head of a descriptor chain.  It is only
-written by the driver and read by the device.
-
-\field{idx} field indicates where the driver would put the next descriptor
-entry in the ring (modulo the queue size). This starts at 0, and increases.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to this structure as vring_avail, and the constant as
-VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
-\end{note}
-
-\subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
-the \field{flags} field in the available ring offers a crude mechanism for the driver to inform
-the device that it doesn't want interrupts when buffers are used.  Otherwise
-\field{used_event} is a more performant alternative where the driver
-specifies how far the device can progress before interrupting.
-
-Neither of these interrupt suppression methods are reliable, as they
-are not synchronized with the device, but they serve as
-useful optimizations.
-
-\drivernormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The driver MUST set \field{flags} to 0 or 1.
-\item The driver MAY set \field{flags} to 1 to advise
-the device that interrupts are not needed.
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The driver MUST set \field{flags} to 0.
-\item The driver MAY use \field{used_event} to advise the device that interrupts are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
-used ring will reach the value \field{used_event} + 1).
-\end{itemize}
-
-The driver MUST handle spurious interrupts from the device.
-
-\devicenormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The device MUST ignore the \field{used_event} value.
-\item After the device writes a descriptor index into the used ring:
-  \begin{itemize}
-  \item If \field{flags} is 1, the device SHOULD NOT send an interrupt.
-  \item If \field{flags} is 0, the device MUST send an interrupt.
-  \end{itemize}
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The device MUST ignore the lower bit of \field{flags}.
-\item After the device writes a descriptor index into the used ring:
-  \begin{itemize}
-  \item If the \field{idx} field in the used ring (which determined
-    where that descriptor index was placed) was equal to
-    \field{used_event}, the device MUST send an interrupt.
-  \item Otherwise the device SHOULD NOT send an interrupt.
-  \end{itemize}
-\end{itemize}
-
-\begin{note}
-For example, if \field{used_event} is 0, then a device using
-  VIRTIO_F_EVENT_IDX would interrupt after the first buffer is
-  used (and again after the 65536th buffer, etc).
-\end{note}
-
-\subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-\begin{lstlisting}
-struct virtq_used {
-#define VIRTQ_USED_F_NO_NOTIFY  1
-        le16 flags;
-        le16 idx;
-        struct virtq_used_elem ring[ /* Queue Size */];
-        le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
-};
-
-/* le32 is used here for ids for padding reasons. */
-struct virtq_used_elem {
-        /* Index of start of used descriptor chain. */
-        le32 id;
-        /* Total length of the descriptor chain which was used (written to) */
-        le32 len;
-};
-\end{lstlisting}
-
-The used ring is where the device returns buffers once it is done with
-them: it is only written to by the device, and read by the driver.
-
-Each entry in the ring is a pair: \field{id} indicates the head entry of the
-descriptor chain describing the buffer (this matches an entry
-placed in the available ring by the guest earlier), and \field{len} the total
-of bytes written into the buffer.
-
-\begin{note}
-\field{len} is particularly useful
-for drivers using untrusted buffers: if a driver does not know exactly
-how much has been written by the device, the driver would have to zero
-the buffer in advance to ensure no data leakage occurs.
-
-For example, a network driver may hand a received buffer directly to
-an unprivileged userspace application.  If the network device has not
-overwritten the bytes which were in that buffer, this could leak the
-contents of freed memory from other processes to the application.
-\end{note}
-
-\field{idx} field indicates where the driver would put the next descriptor
-entry in the ring (modulo the queue size). This starts at 0, and increases.
-
-\begin{note}
-The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
-referred to these structures as vring_used and vring_used_elem, and
-the constant as VRING_USED_F_NO_NOTIFY, but the layout and value were
-identical.
-\end{note}
-
-\subsubsection{Legacy Interface: The Virtqueue Used
-Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues
-/ The Virtqueue Used Ring/ Legacy Interface: The Virtqueue Used
-Ring}
-
-Historically, many drivers ignored the \field{len} value, as a
-result, many devices set \field{len} incorrectly.  Thus, when
-using the legacy interface, it is generally a good idea to ignore
-the \field{len} value in used ring entries if possible.  Specific
-known issues are listed per device type.
-
-\devicenormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-The device MUST set \field{len} prior to updating the used \field{idx}.
-
-The device MUST write at least \field{len} bytes to descriptor,
-beginning at the first device-writable buffer,
-prior to updating the used \field{idx}.
-
-The device MAY write more than \field{len} bytes to descriptor.
-
-\begin{note}
-There are potential error cases where a device might not know what
-parts of the buffers have been written.  This is why \field{len} is
-permitted to be an underestimate: that's preferable to the driver believing
-that uninitialized memory has been overwritten when it has not.
-\end{note}
-
-\drivernormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-
-The driver MUST NOT make assumptions about data in device-writable buffers
-beyond the first \field{len} bytes, and SHOULD ignore this data.
-
-\subsection{Virtqueue Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-
-The device can suppress notifications in a manner analogous to the way
-drivers can suppress interrupts as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
-The device manipulates \field{flags} or \field{avail_event} in the used ring the
-same way the driver manipulates \field{flags} or \field{used_event} in the available ring.
-
-\drivernormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-
-The driver MUST initialize \field{flags} in the used ring to 0 when
-allocating the used ring.
-
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The driver MUST ignore the \field{avail_event} value.
-\item After the driver writes a descriptor index into the available ring:
-  \begin{itemize}
-        \item If \field{flags} is 1, the driver SHOULD NOT send a notification.
-        \item If \field{flags} is 0, the driver MUST send a notification.
-  \end{itemize}
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The driver MUST ignore the lower bit of \field{flags}.
-\item After the driver writes a descriptor index into the available ring:
-  \begin{itemize}
-        \item If the \field{idx} field in the available ring (which determined
-          where that descriptor index was placed) was equal to
-          \field{avail_event}, the driver MUST send a notification.
-        \item Otherwise the driver SHOULD NOT send a notification.
-  \end{itemize}
-\end{itemize}
-
-\devicenormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
-If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
-\begin{itemize}
-\item The device MUST set \field{flags} to 0 or 1.
-\item The device MAY set \field{flags} to 1 to advise
-the driver that notifications are not needed.
-\end{itemize}
-
-Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
-\begin{itemize}
-\item The device MUST set \field{flags} to 0.
-\item The device MAY use \field{avail_event} to advise the driver that notifications are unnecessary until the driver writes entry with an index specified by \field{avail_event} into the available ring (equivalently, until \field{idx} in the
-available ring will reach the value \field{avail_event} + 1).
-\end{itemize}
-
-The device MUST handle spurious notifications from the driver.
-
-\subsection{Helpers for Operating Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Helpers for Operating Virtqueues}
-
-The Linux Kernel Source code contains the definitions above and
-helper routines in a more usable form, in
-include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
-and Red Hat under the (3-clause) BSD license so that it can be
-freely used by all other projects, and is reproduced (with slight
-variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
+\input{split-ring.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
diff --git a/split-ring.tex b/split-ring.tex
new file mode 100644
index 0000000..418f63d
--- /dev/null
+++ b/split-ring.tex
@@ -0,0 +1,498 @@
+\section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
+The split virtqueue format is the original format used by legacy
+virtio devices.  The split virtqueue format separates the
+virtqueue into several parts, where each part is write-able by
+either the driver or the device, but not both. Multiple
+locations need to be updated when making a buffer available
+and when marking it as used.
+
+
+Each queue has a 16-bit queue size
+parameter, which sets the number of entries and implies the total size
+of the queue.
+
+Each virtqueue consists of three parts:
+
+\begin{itemize}
+\item Descriptor Table
+\item Available Ring
+\item Used Ring
+\end{itemize}
+
+where each part is physically-contiguous in guest memory,
+and has different alignment requirements.
+
+The memory aligment and size requirements, in bytes, of each part of the
+virtqueue are summarized in the following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Virtqueue Part    & Alignment & Size \\
+\hline \hline
+Descriptor Table  & 16        & $16 * $(Queue Size) \\
+\hline
+Available Ring    & 2         & $6 + 2 * $(Queue Size) \\
+ \hline
+Used Ring         & 4         & $6 + 8 * $(Queue Size) \\
+ \hline
+\end{tabular}
+
+The Alignment column gives the minimum alignment for each part
+of the virtqueue.
+
+The Size column gives the total number of bytes for each
+part of the virtqueue.
+
+Queue Size corresponds to the maximum number of buffers in the
+virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
+can be queued at any given time.}.  Queue Size value is always a
+power of 2.  The maximum Queue Size value is 32768.  This value
+is specified in a bus-specific way.
+
+When the driver wants to send a buffer to the device, it fills in
+a slot in the descriptor table (or chains several together), and
+writes the descriptor index into the available ring.  It then
+notifies the device. When the device has finished a buffer, it
+writes the descriptor index into the used ring, and sends an interrupt.
+
+\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a Virtio Device / Virtqueues}
+The driver MUST ensure that the physical address of the first byte
+of each virtqueue part is a multiple of the specified alignment value
+in the above table.
+
+\subsection{Legacy Interfaces: A Note on Virtqueue Layout}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
+
+For Legacy Interfaces, several additional
+restrictions are placed on the virtqueue layout:
+
+Each virtqueue occupies two or more physically-contiguous pages
+(usually defined as 4096 bytes, but depending on the transport;
+henceforth referred to as Queue Align)
+and consists of three parts:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Descriptor Table & Available Ring (\ldots padding\ldots) & Used Ring \\
+\hline
+\end{tabular}
+
+The bus-specific Queue Size field controls the total number of bytes
+for the virtqueue.
+When using the legacy interface, the transitional
+driver MUST retrieve the Queue Size field from the device
+and MUST allocate the total number of bytes for the virtqueue
+according to the following formula (Queue Align given in qalign and
+Queue Size given in qsz):
+
+\begin{lstlisting}
+#define ALIGN(x) (((x) + qalign) & ~qalign)
+static inline unsigned virtq_size(unsigned int qsz)
+{
+     return ALIGN(sizeof(struct virtq_desc)*qsz + sizeof(u16)*(3 + qsz))
+          + ALIGN(sizeof(u16)*3 + sizeof(struct virtq_used_elem)*qsz);
+}
+\end{lstlisting}
+
+This wastes some space with padding.
+When using the legacy interface, both transitional
+devices and drivers MUST use the following virtqueue layout
+structure to locate elements of the virtqueue:
+
+\begin{lstlisting}
+struct virtq {
+        // The actual descriptors (16 bytes each)
+        struct virtq_desc desc[ Queue Size ];
+
+        // A ring of available descriptor heads with free-running index.
+        struct virtq_avail avail;
+
+        // Padding to the next Queue Align boundary.
+        u8 pad[ Padding ];
+
+        // A ring of used descriptor heads with free-running index.
+        struct virtq_used used;
+};
+\end{lstlisting}
+
+\subsection{Legacy Interfaces: A Note on Virtqueue Endianness}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
+
+Note that when using the legacy interface, transitional
+devices and drivers MUST use the native
+endian of the guest as the endian of fields and in the virtqueue.
+This is opposed to little-endian for non-legacy interface as
+specified by this standard.
+It is assumed that the host is already aware of the guest endian.
+
+\subsection{Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing}
+The framing of messages with descriptors is
+independent of the contents of the buffers. For example, a network
+transmit buffer consists of a 12 byte header followed by the network
+packet. This could be most simply placed in the descriptor table as a
+12 byte output descriptor followed by a 1514 byte output descriptor,
+but it could also consist of a single 1526 byte output descriptor in
+the case where the header and packet are adjacent, or even three or
+more descriptors (possibly with loss of efficiency in that case).
+
+Note that, some device implementations have large-but-reasonable
+restrictions on total descriptor size (such as based on IOV_MAX in the
+host OS). This has not been a problem in practice: little sympathy
+will be given to drivers which create unreasonably-sized descriptors
+such as by dividing a network packet into 1500 single-byte
+descriptors!
+
+\devicenormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
+The device MUST NOT make assumptions about the particular arrangement
+of descriptors.  The device MAY have a reasonable limit of descriptors
+it will allow in a chain.
+
+\drivernormative{\subsubsection}{Message Framing}{Basic Facilities of a Virtio Device / Message Framing}
+The driver MUST place any device-writable descriptor elements after
+any device-readable descriptor elements.
+
+The driver SHOULD NOT use an excessive number of descriptors to
+describe a buffer.
+
+\subsubsection{Legacy Interface: Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
+
+Regrettably, initial driver implementations used simple layouts, and
+devices came to rely on it, despite this specification wording.  In
+addition, the specification for virtio_blk SCSI commands required
+intuiting field lengths from frame boundaries (see
+ \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}~\nameref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation})
+
+Thus when using the legacy interface, the VIRTIO_F_ANY_LAYOUT
+feature indicates to both the device and the driver that no
+assumptions were made about framing.  Requirements for
+transitional drivers when this is not negotiated are included in
+each device section.
+
+\subsection{The Virtqueue Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+
+The descriptor table refers to the buffers the driver is using for
+the device. \field{addr} is a physical address, and the buffers
+can be chained via \field{next}. Each descriptor describes a
+buffer which is read-only for the device (``device-readable'') or write-only for the device (``device-writable''), but a chain of
+descriptors can contain both device-readable and device-writable buffers.
+
+The actual contents of the memory offered to the device depends on the
+device type.  Most common is to begin the data with a header
+(containing little-endian fields) for the device to read, and postfix
+it with a status tailer for the device to write.
+
+\begin{lstlisting}
+struct virtq_desc {
+        /* Address (guest-physical). */
+        le64 addr;
+        /* Length. */
+        le32 len;
+
+/* This marks a buffer as continuing via the next field. */
+#define VIRTQ_DESC_F_NEXT   1
+/* This marks a buffer as device write-only (otherwise device read-only). */
+#define VIRTQ_DESC_F_WRITE     2
+/* This means the buffer contains a list of buffer descriptors. */
+#define VIRTQ_DESC_F_INDIRECT   4
+        /* The flags as indicated above. */
+        le16 flags;
+        /* Next field if flags & NEXT */
+        le16 next;
+};
+\end{lstlisting}
+
+The number of descriptors in the table is defined by the queue size
+for this virtqueue: this is the maximum possible descriptor chain length.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to this structure as vring_desc, and the constants as
+VRING_DESC_F_NEXT, etc, but the layout and values were identical.
+\end{note}
+
+\devicenormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
+read a device-writable buffer (it MAY do so for debugging or diagnostic
+purposes).
+
+\drivernormative{\subsubsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
+Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
+this implies that loops in the descriptor chain are forbidden!
+
+\subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+
+Some devices benefit by concurrently dispatching a large number
+of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this (see \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}). To increase
+ring capacity the driver can store a table of indirect
+descriptors anywhere in memory, and insert a descriptor in main
+virtqueue (with \field{flags}\&VIRTQ_DESC_F_INDIRECT on) that refers to memory buffer
+containing this indirect descriptor table; \field{addr} and \field{len}
+refer to the indirect table address and length in bytes,
+respectively.
+
+The indirect table layout structure looks like this
+(\field{len} is the length of the descriptor that refers to this table,
+which is a variable, so this code won't compile):
+
+\begin{lstlisting}
+struct indirect_descriptor_table {
+        /* The actual descriptors (16 bytes each) */
+        struct virtq_desc desc[len / 16];
+};
+\end{lstlisting}
+
+The first indirect descriptor is located at start of the indirect
+descriptor table (index 0), additional indirect descriptors are
+chained by \field{next}. An indirect descriptor without a valid \field{next}
+(with \field{flags}\&VIRTQ_DESC_F_NEXT off) signals the end of the descriptor.
+A single indirect descriptor
+table can include both device-readable and device-writable descriptors.
+
+\drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
+VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
+set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only
+one table per descriptor).
+
+A driver MUST NOT create a descriptor chain longer than the Queue Size of
+the device.
+
+A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
+in \field{flags}.
+
+\devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
+
+The device MUST handle the case of zero or more normal chained
+descriptors followed by a single descriptor with \field{flags}\&VIRTQ_DESC_F_INDIRECT.
+
+\begin{note}
+While unusual (most implementations either create a chain solely using
+non-indirect descriptors, or use a single indirect element), such a
+layout is valid.
+\end{note}
+
+\subsection{The Virtqueue Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
+
+\begin{lstlisting}
+struct virtq_avail {
+#define VIRTQ_AVAIL_F_NO_INTERRUPT      1
+        le16 flags;
+        le16 idx;
+        le16 ring[ /* Queue Size */ ];
+        le16 used_event; /* Only if VIRTIO_F_EVENT_IDX */
+};
+\end{lstlisting}
+
+The driver uses the available ring to offer buffers to the
+device: each ring entry refers to the head of a descriptor chain.  It is only
+written by the driver and read by the device.
+
+\field{idx} field indicates where the driver would put the next descriptor
+entry in the ring (modulo the queue size). This starts at 0, and increases.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to this structure as vring_avail, and the constant as
+VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
+\end{note}
+
+\subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
+the \field{flags} field in the available ring offers a crude mechanism for the driver to inform
+the device that it doesn't want interrupts when buffers are used.  Otherwise
+\field{used_event} is a more performant alternative where the driver
+specifies how far the device can progress before interrupting.
+
+Neither of these interrupt suppression methods are reliable, as they
+are not synchronized with the device, but they serve as
+useful optimizations.
+
+\drivernormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The driver MUST set \field{flags} to 0 or 1.
+\item The driver MAY set \field{flags} to 1 to advise
+the device that interrupts are not needed.
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The driver MUST set \field{flags} to 0.
+\item The driver MAY use \field{used_event} to advise the device that interrupts are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
+used ring will reach the value \field{used_event} + 1).
+\end{itemize}
+
+The driver MUST handle spurious interrupts from the device.
+
+\devicenormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The device MUST ignore the \field{used_event} value.
+\item After the device writes a descriptor index into the used ring:
+  \begin{itemize}
+  \item If \field{flags} is 1, the device SHOULD NOT send an interrupt.
+  \item If \field{flags} is 0, the device MUST send an interrupt.
+  \end{itemize}
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The device MUST ignore the lower bit of \field{flags}.
+\item After the device writes a descriptor index into the used ring:
+  \begin{itemize}
+  \item If the \field{idx} field in the used ring (which determined
+    where that descriptor index was placed) was equal to
+    \field{used_event}, the device MUST send an interrupt.
+  \item Otherwise the device SHOULD NOT send an interrupt.
+  \end{itemize}
+\end{itemize}
+
+\begin{note}
+For example, if \field{used_event} is 0, then a device using
+  VIRTIO_F_EVENT_IDX would interrupt after the first buffer is
+  used (and again after the 65536th buffer, etc).
+\end{note}
+
+\subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+\begin{lstlisting}
+struct virtq_used {
+#define VIRTQ_USED_F_NO_NOTIFY  1
+        le16 flags;
+        le16 idx;
+        struct virtq_used_elem ring[ /* Queue Size */];
+        le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
+};
+
+/* le32 is used here for ids for padding reasons. */
+struct virtq_used_elem {
+        /* Index of start of used descriptor chain. */
+        le32 id;
+        /* Total length of the descriptor chain which was used (written to) */
+        le32 len;
+};
+\end{lstlisting}
+
+The used ring is where the device returns buffers once it is done with
+them: it is only written to by the device, and read by the driver.
+
+Each entry in the ring is a pair: \field{id} indicates the head entry of the
+descriptor chain describing the buffer (this matches an entry
+placed in the available ring by the guest earlier), and \field{len} the total
+of bytes written into the buffer.
+
+\begin{note}
+\field{len} is particularly useful
+for drivers using untrusted buffers: if a driver does not know exactly
+how much has been written by the device, the driver would have to zero
+the buffer in advance to ensure no data leakage occurs.
+
+For example, a network driver may hand a received buffer directly to
+an unprivileged userspace application.  If the network device has not
+overwritten the bytes which were in that buffer, this could leak the
+contents of freed memory from other processes to the application.
+\end{note}
+
+\field{idx} field indicates where the driver would put the next descriptor
+entry in the ring (modulo the queue size). This starts at 0, and increases.
+
+\begin{note}
+The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
+referred to these structures as vring_used and vring_used_elem, and
+the constant as VRING_USED_F_NO_NOTIFY, but the layout and value were
+identical.
+\end{note}
+
+\subsubsection{Legacy Interface: The Virtqueue Used
+Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues
+/ The Virtqueue Used Ring/ Legacy Interface: The Virtqueue Used
+Ring}
+
+Historically, many drivers ignored the \field{len} value, as a
+result, many devices set \field{len} incorrectly.  Thus, when
+using the legacy interface, it is generally a good idea to ignore
+the \field{len} value in used ring entries if possible.  Specific
+known issues are listed per device type.
+
+\devicenormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+The device MUST set \field{len} prior to updating the used \field{idx}.
+
+The device MUST write at least \field{len} bytes to descriptor,
+beginning at the first device-writable buffer,
+prior to updating the used \field{idx}.
+
+The device MAY write more than \field{len} bytes to descriptor.
+
+\begin{note}
+There are potential error cases where a device might not know what
+parts of the buffers have been written.  This is why \field{len} is
+permitted to be an underestimate: that's preferable to the driver believing
+that uninitialized memory has been overwritten when it has not.
+\end{note}
+
+\drivernormative{\subsubsection}{The Virtqueue Used Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+The driver MUST NOT make assumptions about data in device-writable buffers
+beyond the first \field{len} bytes, and SHOULD ignore this data.
+
+\subsection{Virtqueue Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+
+The device can suppress notifications in a manner analogous to the way
+drivers can suppress interrupts as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+The device manipulates \field{flags} or \field{avail_event} in the used ring the
+same way the driver manipulates \field{flags} or \field{used_event} in the available ring.
+
+\drivernormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+
+The driver MUST initialize \field{flags} in the used ring to 0 when
+allocating the used ring.
+
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The driver MUST ignore the \field{avail_event} value.
+\item After the driver writes a descriptor index into the available ring:
+  \begin{itemize}
+        \item If \field{flags} is 1, the driver SHOULD NOT send a notification.
+        \item If \field{flags} is 0, the driver MUST send a notification.
+  \end{itemize}
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The driver MUST ignore the lower bit of \field{flags}.
+\item After the driver writes a descriptor index into the available ring:
+  \begin{itemize}
+        \item If the \field{idx} field in the available ring (which determined
+          where that descriptor index was placed) was equal to
+          \field{avail_event}, the driver MUST send a notification.
+        \item Otherwise the driver SHOULD NOT send a notification.
+  \end{itemize}
+\end{itemize}
+
+\devicenormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+\begin{itemize}
+\item The device MUST set \field{flags} to 0 or 1.
+\item The device MAY set \field{flags} to 1 to advise
+the driver that notifications are not needed.
+\end{itemize}
+
+Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
+\begin{itemize}
+\item The device MUST set \field{flags} to 0.
+\item The device MAY use \field{avail_event} to advise the driver that notifications are unnecessary until the driver writes entry with an index specified by \field{avail_event} into the available ring (equivalently, until \field{idx} in the
+available ring will reach the value \field{avail_event} + 1).
+\end{itemize}
+
+The device MUST handle spurious notifications from the driver.
+
+\subsection{Helpers for Operating Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Helpers for Operating Virtqueues}
+
+The Linux Kernel Source code contains the definitions above and
+helper routines in a more usable form, in
+include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
+and Red Hat under the (3-clause) BSD license so that it can be
+freely used by all other projects, and is reproduced (with slight
+variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
-- 
MST



---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 04/16] content: move virtqueue operation description

[Michael S. Tsirkin]

virtqueue operation description is specific to the virtqueue
format. Move it out to split-ring.tex and update all
references.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
---
 conformance.tex |   4 +-
 content.tex     | 171 +++-------------------------------------------------
 split-ring.tex  | 181 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 185 insertions(+), 171 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index f59e360..55d17b4 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -40,9 +40,9 @@ A driver MUST conform to the following normative statements:
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization}
-\item \ref{drivernormative:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-\item \ref{drivernormative:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Cleanup}
 \item \ref{drivernormative:Reserved Feature Bits}
 \end{itemize}
diff --git a/content.tex b/content.tex
index 5b4c4e9..3b4579e 100644
--- a/content.tex
+++ b/content.tex
@@ -337,167 +337,14 @@ And Device Operation / Device Initialization / Set DRIVER-OK}.
 
 \section{Device Operation}\label{sec:General Initialization And Device Operation / Device Operation}
 
-There are two parts to device operation: supplying new buffers to
-the device, and processing used buffers from the device.
-
-\begin{note} As an
-example, the simplest virtio network device has two virtqueues: the
-transmit virtqueue and the receive virtqueue. The driver adds
-outgoing (device-readable) packets to the transmit virtqueue, and then
-frees them after they are used. Similarly, incoming (device-writable)
-buffers are added to the receive virtqueue, and processed after
-they are used.
-\end{note}
-
-\subsection{Supplying Buffers to The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device}
-
-The driver offers buffers to one of the device's virtqueues as follows:
-
-\begin{enumerate}
-\item\label{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Buffers} The driver places the buffer into free descriptor(s) in the
-   descriptor table, chaining as necessary (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}).
-
-\item\label{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Index} The driver places the index of the head of the descriptor chain
-   into the next ring entry of the available ring.
-
-\item Steps \ref{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Buffers} and \ref{itm:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Place Index} MAY be performed repeatedly if batching
-  is possible.
-
-\item The driver performs suitable a memory barrier to ensure the device sees
-  the updated descriptor table and available ring before the next
-  step.
-
-\item The available \field{idx} is increased by the number of
-  descriptor chain heads added to the available ring.
-
-\item The driver performs a suitable memory barrier to ensure that it updates
-  the \field{idx} field before checking for notification suppression.
-
-\item If notifications are not suppressed, the driver notifies the device
-    of the new available buffers.
-\end{enumerate}
-
-Note that the above code does not take precautions against the
-available ring buffer wrapping around: this is not possible since
-the ring buffer is the same size as the descriptor table, so step
-(1) will prevent such a condition.
-
-In addition, the maximum queue size is 32768 (the highest power
-of 2 which fits in 16 bits), so the 16-bit \field{idx} value can always
-distinguish between a full and empty buffer.
+When operating the device, each field in the device configuration
+space can be changed by either the driver or the device.
 
-What follows is the requirements of each stage in more detail.
-
-\subsubsection{Placing Buffers Into The Descriptor Table}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Placing Buffers Into The Descriptor Table}
-
-A buffer consists of zero or more device-readable physically-contiguous
-elements followed by zero or more physically-contiguous
-device-writable elements (each has at least one element). This
-algorithm maps it into the descriptor table to form a descriptor
-chain:
-
-for each buffer element, b:
-
-\begin{enumerate}
-\item Get the next free descriptor table entry, d
-\item Set \field{d.addr} to the physical address of the start of b
-\item Set \field{d.len} to the length of b.
-\item If b is device-writable, set \field{d.flags} to VIRTQ_DESC_F_WRITE,
-    otherwise 0.
-\item If there is a buffer element after this:
-    \begin{enumerate}
-    \item Set \field{d.next} to the index of the next free descriptor
-      element.
-    \item Set the VIRTQ_DESC_F_NEXT bit in \field{d.flags}.
-    \end{enumerate}
-\end{enumerate}
-
-In practice, \field{d.next} is usually used to chain free
-descriptors, and a separate count kept to check there are enough
-free descriptors before beginning the mappings.
-
-\subsubsection{Updating The Available Ring}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating The Available Ring}
-
-The descriptor chain head is the first d in the algorithm
-above, ie. the index of the descriptor table entry referring to the first
-part of the buffer.  A naive driver implementation MAY do the following (with the
-appropriate conversion to-and-from little-endian assumed):
-
-\begin{lstlisting}
-avail->ring[avail->idx % qsz] = head;
-\end{lstlisting}
+Whenever such a configuration change is triggered by the device,
+driver is notified. This makes it possible for drivers to
+cache device configuration, avoiding expensive configuration
+reads unless notified.
 
-However, in general the driver MAY add many descriptor chains before it updates
-\field{idx} (at which point they become visible to the
-device), so it is common to keep a counter of how many the driver has added:
-
-\begin{lstlisting}
-avail->ring[(avail->idx + added++) % qsz] = head;
-\end{lstlisting}
-
-\subsubsection{Updating \field{idx}}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-
-\field{idx} always increments, and wraps naturally at
-65536:
-
-\begin{lstlisting}
-avail->idx += added;
-\end{lstlisting}
-
-Once available \field{idx} is updated by the driver, this exposes the
-descriptor and its contents.  The device MAY
-access the descriptor chains the driver created and the
-memory they refer to immediately.
-
-\drivernormative{\paragraph}{Updating idx}{General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating idx}
-The driver MUST perform a suitable memory barrier before the \field{idx} update, to ensure the
-device sees the most up-to-date copy.
-
-\subsubsection{Notifying The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
-
-The actual method of device notification is bus-specific, but generally
-it can be expensive.  So the device MAY suppress such notifications if it
-doesn't need them, as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}.
-
-The driver has to be careful to expose the new \field{idx}
-value before checking if notifications are suppressed.
-
-\drivernormative{\paragraph}{Notifying The Device}{General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device}
-The driver MUST perform a suitable memory barrier before reading \field{flags} or
-\field{avail_event}, to avoid missing a notification.
-
-\subsection{Receiving Used Buffers From The Device}\label{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}
-
-Once the device has used buffers referred to by a descriptor (read from or written to them, or
-parts of both, depending on the nature of the virtqueue and the
-device), it interrupts the driver as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
-
-\begin{note}
-For optimal performance, a driver MAY disable interrupts while processing
-the used ring, but beware the problem of missing interrupts between
-emptying the ring and reenabling interrupts.  This is usually handled by
-re-checking for more used buffers after interrups are re-enabled:
-
-\begin{lstlisting}
-virtq_disable_interrupts(vq);
-
-for (;;) {
-        if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) {
-                virtq_enable_interrupts(vq);
-                mb();
-
-                if (vq->last_seen_used != le16_to_cpu(virtq->used.idx))
-                        break;
-
-                virtq_disable_interrupts(vq);
-        }
-
-        struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz];
-        process_buffer(e);
-        vq->last_seen_used++;
-}
-\end{lstlisting}
-\end{note}
 
 \subsection{Notification of Device Configuration Changes}\label{sec:General Initialization And Device Operation / Device Operation / Notification of Device Configuration Changes}
 
@@ -3017,9 +2864,7 @@ If VIRTIO_NET_HDR_F_NEEDS_CSUM is not set, the device MUST NOT
 rely on the packet checksum being correct.
 \paragraph{Packet Transmission Interrupt}\label{sec:Device Types / Network Device / Device Operation / Packet Transmission / Packet Transmission Interrupt}
 
-Often a driver will suppress transmission interrupts using the
-VIRTQ_AVAIL_F_NO_INTERRUPT flag
- (see \ref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}~\nameref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device})
+Often a driver will suppress transmission virtqueue interrupts
 and check for used packets in the transmit path of following
 packets.
 
@@ -3079,7 +2924,7 @@ if VIRTIO_NET_F_MRG_RXBUF is not negotiated.}
 
 When a packet is copied into a buffer in the receiveq, the
 optimal path is to disable further interrupts for the receiveq
-(see \ref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}~\nameref{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device}) and process
+and process
 packets until no more are found, then re-enable them.
 
 Processing incoming packets involves:
diff --git a/split-ring.tex b/split-ring.tex
index 418f63d..404660b 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -1,11 +1,12 @@
 \section{Split Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Split Virtqueues}
-The split virtqueue format is the original format used by legacy
-virtio devices.  The split virtqueue format separates the
-virtqueue into several parts, where each part is write-able by
-either the driver or the device, but not both. Multiple
-locations need to be updated when making a buffer available
-and when marking it as used.
+The split virtqueue format was the only format supported
+by the version 1.0 (and earlier) of this standard.
 
+The split virtqueue format separates the virtqueue into several
+parts, where each part is write-able by either the driver or the
+device, but not both. Multiple parts and/or locations within
+a part need to be updated when making a buffer
+available and when marking it as used.
 
 Each queue has a 16-bit queue size
 parameter, which sets the number of entries and implies the total size
@@ -496,3 +497,171 @@ include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM
 and Red Hat under the (3-clause) BSD license so that it can be
 freely used by all other projects, and is reproduced (with slight
 variation) in \ref{sec:virtio-queue.h}~\nameref{sec:virtio-queue.h}.
+
+\subsection{Virtqueue Operation}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Operation}
+
+There are two parts to virtqueue operation: supplying new
+available buffers to the device, and processing used buffers from
+the device.
+
+\begin{note} As an
+example, the simplest virtio network device has two virtqueues: the
+transmit virtqueue and the receive virtqueue. The driver adds
+outgoing (device-readable) packets to the transmit virtqueue, and then
+frees them after they are used. Similarly, incoming (device-writable)
+buffers are added to the receive virtqueue, and processed after
+they are used.
+\end{note}
+
+What follows is the requirements of each of these two parts
+when using the split virtqueue format in more detail.
+
+\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device}
+
+The driver offers buffers to one of the device's virtqueues as follows:
+
+\begin{enumerate}
+\item\label{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Buffers} The driver places the buffer into free descriptor(s) in the
+   descriptor table, chaining as necessary (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}).
+
+\item\label{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Index} The driver places the index of the head of the descriptor chain
+   into the next ring entry of the available ring.
+
+\item Steps \ref{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Buffers} and \ref{itm:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Place Index} MAY be performed repeatedly if batching
+  is possible.
+
+\item The driver performs suitable a memory barrier to ensure the device sees
+  the updated descriptor table and available ring before the next
+  step.
+
+\item The available \field{idx} is increased by the number of
+  descriptor chain heads added to the available ring.
+
+\item The driver performs a suitable memory barrier to ensure that it updates
+  the \field{idx} field before checking for notification suppression.
+
+\item If notifications are not suppressed, the driver notifies the device
+    of the new available buffers.
+\end{enumerate}
+
+Note that the above code does not take precautions against the
+available ring buffer wrapping around: this is not possible since
+the ring buffer is the same size as the descriptor table, so step
+(1) will prevent such a condition.
+
+In addition, the maximum queue size is 32768 (the highest power
+of 2 which fits in 16 bits), so the 16-bit \field{idx} value can always
+distinguish between a full and empty buffer.
+
+What follows is the requirements of each stage in more detail.
+
+\subsubsection{Placing Buffers Into The Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Buffers Into The Descriptor Table}
+
+A buffer consists of zero or more device-readable physically-contiguous
+elements followed by zero or more physically-contiguous
+device-writable elements (each has at least one element). This
+algorithm maps it into the descriptor table to form a descriptor
+chain:
+
+for each buffer element, b:
+
+\begin{enumerate}
+\item Get the next free descriptor table entry, d
+\item Set \field{d.addr} to the physical address of the start of b
+\item Set \field{d.len} to the length of b.
+\item If b is device-writable, set \field{d.flags} to VIRTQ_DESC_F_WRITE,
+    otherwise 0.
+\item If there is a buffer element after this:
+    \begin{enumerate}
+    \item Set \field{d.next} to the index of the next free descriptor
+      element.
+    \item Set the VIRTQ_DESC_F_NEXT bit in \field{d.flags}.
+    \end{enumerate}
+\end{enumerate}
+
+In practice, \field{d.next} is usually used to chain free
+descriptors, and a separate count kept to check there are enough
+free descriptors before beginning the mappings.
+
+\subsubsection{Updating The Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating The Available Ring}
+
+The descriptor chain head is the first d in the algorithm
+above, ie. the index of the descriptor table entry referring to the first
+part of the buffer.  A naive driver implementation MAY do the following (with the
+appropriate conversion to-and-from little-endian assumed):
+
+\begin{lstlisting}
+avail->ring[avail->idx % qsz] = head;
+\end{lstlisting}
+
+However, in general the driver MAY add many descriptor chains before it updates
+\field{idx} (at which point they become visible to the
+device), so it is common to keep a counter of how many the driver has added:
+
+\begin{lstlisting}
+avail->ring[(avail->idx + added++) % qsz] = head;
+\end{lstlisting}
+
+\subsubsection{Updating \field{idx}}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+
+\field{idx} always increments, and wraps naturally at
+65536:
+
+\begin{lstlisting}
+avail->idx += added;
+\end{lstlisting}
+
+Once available \field{idx} is updated by the driver, this exposes the
+descriptor and its contents.  The device MAY
+access the descriptor chains the driver created and the
+memory they refer to immediately.
+
+\drivernormative{\paragraph}{Updating idx}{Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
+The driver MUST perform a suitable memory barrier before the \field{idx} update, to ensure the
+device sees the most up-to-date copy.
+
+\subsubsection{Notifying The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+
+The actual method of device notification is bus-specific, but generally
+it can be expensive.  So the device MAY suppress such notifications if it
+doesn't need them, as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}.
+
+The driver has to be careful to expose the new \field{idx}
+value before checking if notifications are suppressed.
+
+\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+The driver MUST perform a suitable memory barrier before reading \field{flags} or
+\field{avail_event}, to avoid missing a notification.
+
+\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Receiving Used Buffers From The Device}
+
+Once the device has used buffers referred to by a descriptor (read from or written to them, or
+parts of both, depending on the nature of the virtqueue and the
+device), it interrupts the driver as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+
+\begin{note}
+For optimal performance, a driver MAY disable interrupts while processing
+the used ring, but beware the problem of missing interrupts between
+emptying the ring and reenabling interrupts.  This is usually handled by
+re-checking for more used buffers after interrups are re-enabled:
+
+\begin{lstlisting}
+virtq_disable_interrupts(vq);
+
+for (;;) {
+        if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) {
+                virtq_enable_interrupts(vq);
+                mb();
+
+                if (vq->last_seen_used != le16_to_cpu(virtq->used.idx))
+                        break;
+
+                virtq_disable_interrupts(vq);
+        }
+
+        struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz];
+        process_buffer(e);
+        vq->last_seen_used++;
+}
+\end{lstlisting}
+\end{note}
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 05/16] content: len -> used length, used ring -> vq

[Michael S. Tsirkin]

Document buffer used len and use that terminology everywhere in the
generic section.

Further, drop the 'used ring' terminology and just say virtqueue.

Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 39 ++++++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 19 deletions(-)

diff --git a/content.tex b/content.tex
index 3b4579e..4350ecf 100644
--- a/content.tex
+++ b/content.tex
@@ -242,7 +242,8 @@ a used buffer to the queue - i.e. lets the driver
 know by marking the buffer as used. Device can then trigger
 a device event - i.e. send an interrupt to the driver.
 
-For queue operation detail, see \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}.
+Device reports the number of bytes it has written to memory for
+each buffer it uses. This is referred to as ``used length''.
 
 \input{split-ring.tex}
 
@@ -1304,7 +1305,7 @@ The driver interrupt handler would typically:
     \begin{itemize}
       \item Read the ISR Status field, which will reset it to zero.
       \item If the lower bit is set:
-        look through the used rings of all virtqueues for the
+        look through all virtqueues for the
         device, to see if any progress has been made by the device
         which requires servicing.
       \item If the second lower bit is set:
@@ -1313,8 +1314,7 @@ The driver interrupt handler would typically:
   \item If MSI-X capability is enabled:
     \begin{itemize}
       \item
-        Look through the used rings of
-        all virtqueues mapped to that MSI-X vector for the
+        Look through all virtqueues mapped to that MSI-X vector for the
         device, to see if any progress has been made by the device
         which requires servicing.
       \item
@@ -1728,8 +1728,7 @@ nor behaviour:
   }
   \hline
   \mmioreg{QueueNum}{Virtual queue size}{0x038}{W}{%
-    Queue size is the number of elements in the queue, therefore size
-    of the descriptor table and both available and used rings.
+    Queue size is the number of elements in the queue.
     Writing to this register notifies the device what size of the
     queue the driver will use. This applies to the queue selected by
     writing to \field{QueueSel}.
@@ -2709,7 +2708,7 @@ when VIRTIO_NET_F_MRG_RXBUF was negotiated; without that feature the
 structure was 2 bytes shorter.
 
 When using the legacy interface, the driver SHOULD ignore the
-\field{len} value in used ring entries for the transmit queues
+used length for the transmit queues
 and the controlq queue.
 \begin{note}
 Historically, some devices put
@@ -2868,8 +2867,8 @@ Often a driver will suppress transmission virtqueue interrupts
 and check for used packets in the transmit path of following
 packets.
 
-The normal behavior in this interrupt handler is to retrieve and
-new descriptors from the used ring and free the corresponding
+The normal behavior in this interrupt handler is to retrieve
+used buffers from the virtqueue and free the corresponding
 headers and packets.
 
 \subsubsection{Setting Up Receive Buffers}\label{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}
@@ -2936,7 +2935,7 @@ Processing incoming packets involves:
   This allows receipt of large packets without having to allocate large
   buffers: a packet that does not fit in a single buffer can flow
   over to the next buffer, and so on. In this case, there will be
-  at least \field{num_buffers} in the used ring, and the device
+  at least \field{num_buffers} used buffers in the virtqueue, and the device
   chains them together to form a single packet in a way similar to
   how it would store it in a single buffer spread over multiple
   descriptors.
@@ -2990,8 +2989,8 @@ MUST use all buffers but the last (i.e. the first $num_buffers -
 supplied by the driver.
 
 The device MUST use all buffers used by a single receive
-packet together, by atomically incrementing \field{idx} in the
-used ring by the \field{num_buffers} value.
+packet together, such that at least \field{num_buffers} are
+observed by driver as used.
 
 If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device MUST set
 \field{flags} to zero and SHOULD supply a fully checksummed
@@ -3378,7 +3377,8 @@ The device MUST queue packets only on any receiveq1 before the
 VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command.
 
 The device MUST NOT queue packets on receive queues greater than
-\field{virtqueue_pairs} once it has placed the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in the used ring.
+\field{virtqueue_pairs} once it has placed the
+VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in a used buffer.
 
 \subparagraph{Legacy Interface: Automatic receive steering in multiqueue mode}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Legacy Interface: Automatic receive steering in multiqueue mode}
 When using the legacy interface, transitional devices and drivers
@@ -3744,7 +3744,7 @@ according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
 When using the legacy interface, transitional drivers
-SHOULD ignore the \field{len} value in used ring entries.
+SHOULD ignore the used length values.
 \begin{note}
 Historically, some devices put the total descriptor length,
 or the total length of device-writable buffers there,
@@ -4081,7 +4081,7 @@ according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
 When using the legacy interface, the driver SHOULD ignore the
-\field{len} value in used ring entries for the transmit queues
+used length values for the transmit queues
 and the control transmitq.
 \begin{note}
 Historically, some devices put the total descriptor length there,
@@ -4337,7 +4337,8 @@ and acknowledging the deflate request.
 \paragraph{Legacy Interface: Device Operation}\label{sec:Device
 Types / Memory Balloon Device / Device Operation / Legacy
 Interface: Device Operation}
-When using the legacy interface, the driver SHOULD ignore the \field{len} value in used ring entries.
+When using the legacy interface, the driver SHOULD ignore the
+used length values.
 \begin{note}
 Historically, some devices put the total descriptor length there,
 even though no data was actually written.
@@ -4370,8 +4371,7 @@ and notifies the device. A request for memory statistics proceeds
 as follows:
 
 \begin{enumerate}
-\item The device pushes the buffer onto the used ring and sends an
-  interrupt.
+\item The device uses the buffer and sends an interrupt.
 
 \item The driver pops the used buffer and discards it.
 
@@ -4632,7 +4632,8 @@ queue and the event queue.
 \paragraph{Legacy Interface: Device Operation}\label{sec:Device
 Types / SCSI Host Device / Device Operation / Legacy
 Interface: Device Operation}
-When using the legacy interface, the driver SHOULD ignore the \field{len} value in used ring entries.
+When using the legacy interface, the driver SHOULD ignore the
+used length values.
 \begin{note}
 Historically, devices put the total descriptor length,
 or the total length of device-writable buffers there,
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 06/16] content: generalize transport ring part naming

[Michael S. Tsirkin]

Replace descriptor table/available ring/used ring
with descriptor area/driver area/device area
in all transports.

Document what's in which area.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
---
 content.tex    | 61 ++++++++++++++++++++++++++++++++++++++--------------------
 split-ring.tex |  6 +++---
 2 files changed, 43 insertions(+), 24 deletions(-)

diff --git a/content.tex b/content.tex
index 4350ecf..9fc9673 100644
--- a/content.tex
+++ b/content.tex
@@ -245,6 +245,24 @@ a device event - i.e. send an interrupt to the driver.
 Device reports the number of bytes it has written to memory for
 each buffer it uses. This is referred to as ``used length''.
 
+Each virtqueue can consist of up to 3 parts:
+\begin{itemize}
+\item Descriptor Area - used for describing buffers
+\item Driver Area - extra data supplied by driver to the device
+\item Device Area - extra data supplied by device to driver
+\end{itemize}
+
+\begin{note}
+Note that previous versions of this spec used different names for
+these parts (following \ref{sec:Basic Facilities of a Virtio Device / Split Virtqueues}):
+\begin{itemize}
+\item Descriptor Table - for the Descriptor Area
+\item Available Ring - for the Driver Area
+\item Used Ring - for the Device Area
+\end{itemize}
+
+\end{note}
+
 \input{split-ring.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
@@ -667,8 +685,8 @@ struct virtio_pci_common_cfg {
         le16 queue_enable;              /* read-write */
         le16 queue_notify_off;          /* read-only for driver */
         le64 queue_desc;                /* read-write */
-        le64 queue_avail;               /* read-write */
-        le64 queue_used;                /* read-write */
+        le64 queue_driver;              /* read-write */
+        le64 queue_device;              /* read-write */
 };
 \end{lstlisting}
 
@@ -728,13 +746,13 @@ struct virtio_pci_common_cfg {
         \end{note}
 
 \item[\field{queue_desc}]
-        The driver writes the physical address of Descriptor Table here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
+        The driver writes the physical address of Descriptor Area here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
 
-\item[\field{queue_avail}]
-        The driver writes the physical address of Available Ring here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
+\item[\field{queue_driver}]
+        The driver writes the physical address of Driver Area here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
 
-\item[\field{queue_used}]
-        The driver writes the physical address of Used Ring here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
+\item[\field{queue_device}]
+        The driver writes the physical address of Device Area here.  See section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1496,24 +1514,24 @@ All register values are organized as Little Endian.
     See also p. \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}~\nameref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}.
   }
   \hline 
-  \mmiodreg{QueueDescLow}{QueueDescHigh}{Virtual queue's Descriptor Table 64 bit long physical address}{0x080}{0x084}{W}{%
+  \mmiodreg{QueueDescLow}{QueueDescHigh}{Virtual queue's Descriptor Area 64 bit long physical address}{0x080}{0x084}{W}{%
     Writing to these two registers (lower 32 bits of the address
     to \field{QueueDescLow}, higher 32 bits to \field{QueueDescHigh}) notifies
-    the device about location of the Descriptor Table of the queue
+    the device about location of the Descriptor Area of the queue
     selected by writing to \field{QueueSel} register.
   }
   \hline 
-  \mmiodreg{QueueAvailLow}{QueueAvailHigh}{Virtual queue's Available Ring 64 bit long physical address}{0x090}{0x094}{W}{%
+  \mmiodreg{QueueDriverLow}{QueueDriverHigh}{Virtual queue's Driver Area 64 bit long physical address}{0x090}{0x094}{W}{%
     Writing to these two registers (lower 32 bits of the address
     to \field{QueueAvailLow}, higher 32 bits to \field{QueueAvailHigh}) notifies
-    the device about location of the Available Ring of the queue
+    the device about location of the Driver Area of the queue
     selected by writing to \field{QueueSel}.
   }
   \hline 
-  \mmiodreg{QueueUsedLow}{QueueUsedHigh}{Virtual queue's Used Ring 64 bit long physical address}{0x0a0}{0x0a4}{W}{%
+  \mmiodreg{QueueDeviceLow}{QueueDeviceHigh}{Virtual queue's Device Area 64 bit long physical address}{0x0a0}{0x0a4}{W}{%
     Writing to these two registers (lower 32 bits of the address
     to \field{QueueUsedLow}, higher 32 bits to \field{QueueUsedHigh}) notifies
-    the device about location of the Used Ring of the queue
+    the device about location of the Device Area of the queue
     selected by writing to \field{QueueSel}.
   }
   \hline 
@@ -1631,11 +1649,11 @@ The driver will typically initialize the virtual queue in the following way:
 \item Notify the device about the queue size by writing the size to
    \field{QueueNum}.
 
-\item Write physical addresses of the queue's Descriptor Table,
-   Available Ring and Used Ring to (respectively) the
+\item Write physical addresses of the queue's Descriptor Area,
+   Driver Area and Device Area to (respectively) the
    \field{QueueDescLow}/\field{QueueDescHigh},
-   \field{QueueAvailLow}/\field{QueueAvailHigh} and
-   \field{QueueUsedLow}/\field{QueueUsedHigh} register pairs.
+   \field{QueueDriverLow}/\field{QueueDriverHigh} and
+   \field{QueueDeviceLow}/\field{QueueDeviceHigh} register pairs.
 
 \item Write 0x1 to \field{QueueReady}.
 \end{enumerate}
@@ -2025,13 +2043,14 @@ struct vq_info_block {
         be32 res0;
         be16 index;
         be16 num;
-        be64 avail;
-        be64 used;
+        be64 driver;
+        be64 device;
 };
 \end{lstlisting}
 
-\field{desc}, \field{avail} and \field{used} contain the guest addresses for the descriptor table,
-available ring and used ring for queue \field{index}, respectively. The actual
+\field{desc}, \field{driver} and \field{device} contain the guest
+addresses for the descriptor area,
+available area and used area for queue \field{index}, respectively. The actual
 virtqueue size (number of allocated buffers) is transmitted in \field{num}.
 
 \devicenormative{\paragraph}{Configuring a Virtqueue}{Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue}
diff --git a/split-ring.tex b/split-ring.tex
index 404660b..9601a53 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -15,9 +15,9 @@ of the queue.
 Each virtqueue consists of three parts:
 
 \begin{itemize}
-\item Descriptor Table
-\item Available Ring
-\item Used Ring
+\item Descriptor Table - occupies the Descriptor Area
+\item Available Ring - occupies the Driver Area
+\item Used Ring - occupies the Device Area
 \end{itemize}
 
 where each part is physically-contiguous in guest memory,
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 07/16] content: generalize rest of text

[Michael S. Tsirkin]

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
---
 content.tex | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/content.tex b/content.tex
index 9fc9673..5634c7d 100644
--- a/content.tex
+++ b/content.tex
@@ -1467,8 +1467,7 @@ All register values are organized as Little Endian.
   }
   \hline 
   \mmioreg{QueueNum}{Virtual queue size}{0x038}{W}{%
-    Queue size is the number of elements in the queue, therefore in each
-    of the Descriptor Table, the Available Ring and the Used Ring.
+    Queue size is the number of elements in the queue.
     Writing to this register notifies the device what size of the
     queue the driver will use. This applies to the queue selected by
     writing to \field{QueueSel}.
@@ -1491,9 +1490,9 @@ All register values are organized as Little Endian.
     caused the device interrupt to be asserted.
     The following events are possible:
     \begin{description}
-      \item[Used Ring Update] - bit 0 - the interrupt was asserted
-        because the device has updated the Used
-        Ring in at least one of the active virtual queues.
+      \item[Used Buffer Update] - bit 0 - the interrupt was asserted
+        because the device has used a buffer
+        in at least one of the active virtual queues.
       \item [Configuration Change] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
     \end{description}
@@ -1642,9 +1641,8 @@ The driver will typically initialize the virtual queue in the following way:
    \field{QueueNumMax}. If the returned value is zero (0x0) the
    queue is not available.
 
-\item Allocate and zero the queue pages, making sure the memory
-   is physically contiguous. It is recommended to align the
-   Used Ring to an optimal boundary (usually the page size).
+\item Allocate and zero the queue memory, making sure the memory
+   is physically contiguous.
 
 \item Notify the device about the queue size by writing the size to
    \field{QueueNum}.
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 08/16] split-ring: generalize text

[Michael S. Tsirkin]

Update generic text to talk about available/used buffers, not rings.
Move some split-ring specific text to the correct section.

Update conformance section with link to the new conformance clause.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 conformance.tex |  1 +
 content.tex     | 10 ++++------
 split-ring.tex  |  4 ++++
 3 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 55d17b4..e4efe33 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -38,6 +38,7 @@ A driver MUST conform to the following normative statements:
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
diff --git a/content.tex b/content.tex
index 5634c7d..e1e30a0 100644
--- a/content.tex
+++ b/content.tex
@@ -381,12 +381,10 @@ of a device are live once the device has been reset.
 
 \drivernormative{\subsection}{Device Cleanup}{General Initialization And Device Operation / Device Cleanup}
 
-A driver MUST NOT alter descriptor table entries which have been
-exposed in the available ring (and not marked consumed by the device
-in the used ring) of a live virtqueue.
-
-A driver MUST NOT decrement the available \field{idx} on a live virtqueue (ie.
-there is no way to ``unexpose'' buffers).
+A driver MUST NOT alter virtqueue entries for exposed buffers -
+i.e. buffers which have been
+made available to the device (and not been used by the device)
+of a live virtqueue.
 
 Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
 
diff --git a/split-ring.tex b/split-ring.tex
index 9601a53..a594d41 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -296,6 +296,10 @@ referred to this structure as vring_avail, and the constant as
 VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
 \end{note}
 
+\drivernormative{\subsubsection}{The Virtqueue Available Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
+A driver MUST NOT decrement the available \field{idx} on a virtqueue (ie.
+there is no way to ``unexpose'' buffers).
+
 \subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
 
 If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 09/16] split-ring: typo: aligment

[Michael S. Tsirkin]

---
 split-ring.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/split-ring.tex b/split-ring.tex
index a594d41..87ecee2 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -23,7 +23,7 @@ Each virtqueue consists of three parts:
 where each part is physically-contiguous in guest memory,
 and has different alignment requirements.
 
-The memory aligment and size requirements, in bytes, of each part of the
+The memory alignment and size requirements, in bytes, of each part of the
 virtqueue are summarized in the following table:
 
 \begin{tabular}{|l|l|l|}
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 10/16] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

Performance analysis of this is in my kvm forum 2016 presentation.  The
idea is to have a r/w descriptor in a ring structure, replacing the used
and available ring, index and descriptor buffer.

This is also easier for devices to implement than the 1.0 layout.
Several more enhancements will be necessary to actually make this
efficient for devices to use.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
---
 content.tex     |  28 ++-
 packed-ring.tex | 680 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 705 insertions(+), 3 deletions(-)
 create mode 100644 packed-ring.tex

diff --git a/content.tex b/content.tex
index e1e30a0..73f40b7 100644
--- a/content.tex
+++ b/content.tex
@@ -263,8 +263,20 @@ these parts (following \ref{sec:Basic Facilities of a Virtio Device / Split Virt
 
 \end{note}
 
+Two formats are supported: Split Virtqueues (see \ref{sec:Basic
+Facilities of a Virtio Device / Split
+Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
+Split Virtqueues}) and Packed Virtqueues (see \ref{sec:Basic
+Facilities of a Virtio Device / Packed
+Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device /
+Packed Virtqueues}).
+
+Every driver and device supports either the Packed or the Split
+Virtqueue format, or both.
+
 \input{split-ring.tex}
 
+\input{packed-ring.tex}
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -5215,10 +5227,15 @@ Currently these device-independent feature bits defined:
 \begin{description}
   \item[VIRTIO_F_RING_INDIRECT_DESC (28)] Negotiating this feature indicates
   that the driver can use descriptors with the VIRTQ_DESC_F_INDIRECT
-  flag set, as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}.
-
+  flag set, as described in \ref{sec:Basic Facilities of a Virtio
+Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
+Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device /
+Virtqueues / The Virtqueue Descriptor Table / Indirect
+Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}~\nameref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}.
   \item[VIRTIO_F_RING_EVENT_IDX(29)] This feature enables the \field{used_event}
-  and the \field{avail_event} fields as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression} and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}.
+  and the \field{avail_event} fields as described in
+\ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}, \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring} and \ref{sec:Packed Virtqueues / Driver and Device Event Suppression}.
+
 
   \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
     specification, giving a simple way to detect legacy devices or drivers.
@@ -5228,6 +5245,9 @@ Currently these device-independent feature bits defined:
   addresses in memory.  If this feature bit is set to 0, then the device emits
   physical addresses which are not translated further, even though an IOMMU
   may be present.
+  \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
+  support for the packed virtqueue layout as described in
+  \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5241,6 +5261,8 @@ passed to the device into physical addresses in memory.  If
 VIRTIO_F_IOMMU_PLATFORM is not offered, then a driver MUST pass only physical
 addresses to the device.
 
+A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
diff --git a/packed-ring.tex b/packed-ring.tex
new file mode 100644
index 0000000..2943df0
--- /dev/null
+++ b/packed-ring.tex
@@ -0,0 +1,680 @@
+\section{Packed Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}
+
+Packed virtqueues is an alternative compact virtqueue layout using
+read-write memory, that is memory that is both read and written
+by both host and guest.
+
+Use of packed virtqueues is negotiated by the VIRTIO_F_RING_PACKED
+feature bit.
+
+Packed virtqueues support up to $2^{15}$ entries each.
+
+With current transports, virtqueues are located in guest memory
+allocated by driver.
+Each packed virtqueue consists of three parts:
+
+\begin{itemize}
+\item Descriptor Ring - occupies the Descriptor Area
+\item Driver Event Suppression - occupies the Driver Area
+\item Device Event Suppression - occupies the Device Area
+\end{itemize}
+
+Where Descriptor Ring in turn consists of descriptors,
+and where each descriptor can contain the following parts:
+
+\begin{itemize}
+\item Buffer ID
+\item Element Address
+\item Element Length
+\item Flags
+\end{itemize}
+
+A buffer consists of zero or more device-readable physically-contiguous
+elements followed by zero or more physically-contiguous
+device-writable elements (each buffer has at least one element).
+
+When the driver wants to send such a buffer to the device, it
+writes at least one available descriptor describing elements of
+the buffer into the Descriptor Ring.  The descriptor(s) are
+associated with a buffer by means of a Buffer ID stored within
+the descriptor.
+
+Driver then notifies the device. When the device has finished
+processing the buffer, it writes a used device descriptor
+including the Buffer ID into the Descriptor Ring (overwriting a
+driver descriptor previously made available), and sends an
+interrupt.
+
+Descriptor Ring is used in a circular manner: driver writes
+descriptors into the ring in order. After reaching end of ring,
+the next descriptor is placed at head of the ring.  Once ring is
+full of driver descriptors, driver stops sending new requests and
+waits for device to start processing descriptors and to write out
+some used descriptors before making new driver descriptors
+available.
+
+Similarly, device reads descriptors from the ring in order and
+detects that a driver descriptor has been made available.  As
+processing of descriptors is completed used descriptors are
+written by the device back into the ring.
+
+Note: after reading driver descriptors and starting their
+processing in order, device might complete their processing out
+of order.  Used device descriptors are written in the order
+in which their processing is complete.
+
+Device Event Suppression data structure is write-only by the
+device. It includes information for reducing the number of
+device events - i.e. driver notifications to device.
+
+Driver Event Suppression data structure is read-only by the
+device. It includes information for reducing the number of
+driver events - i.e. device interrupts to driver.
+
+\subsection{Driver and Device Ring Wrap Counters}
+\label{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}
+Each of the driver and the device are expected to maintain,
+internally, a single-bit ring wrap counter initialized to 1.
+
+The counter maintained by the driver is called the Driver
+Ring Wrap Counter. Driver changes the value of this counter
+each time it makes available the
+last descriptor in the ring (after making the last descriptor
+available).
+
+The counter maintained by the device is called the Device Ring Wrap
+Counter.  Device changes the value of this counter
+each time it uses the last descriptor in
+the ring (after marking the last descriptor used).
+
+It is easy to see that the Driver Ring Wrap Counter in the driver matches
+the Device Ring Wrap Counter in the device when both are processing the same
+descriptor, or when all available descriptors have been used.
+
+To mark a descriptor as available and used, both driver and
+device use the following two flags:
+\begin{lstlisting}
+#define VIRTQ_DESC_F_AVAIL     (1 << 7)
+#define VIRTQ_DESC_F_USED      (1 << 15)
+\end{lstlisting}
+
+To mark a descriptor as available, driver sets the
+VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Driver
+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_USED bit to match the
+\emph{inverse} value (i.e. to not match the internal Driver Ring
+Wrap Counter).
+
+To mark a descriptor as used, device sets the
+VIRTQ_DESC_F_USED bit in Flags to match the internal Device
+Ring Wrap Counter.  It also sets the VIRTQ_DESC_F_AVAIL bit to match the
+\emph{same} value.
+
+Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different
+for an available descriptor and equal for a used descriptor.
+
+Note that this observation is mostly useful for sanity-checking
+as these are necessary but not sufficient conditions - for
+example, all descriptors are zero-initialized. To detect used and
+available descriptors it is possible for drivers and devices to
+keep track of the last observed value of
+VIRTQ_DESC_F_USED/VIRTQ_DESC_F_AVAIL.  Other techniques to detect
+VIRTQ_DESC_F_AVAIL/VIRTQ_DESC_F_USED bit changes might also be
+possible.
+
+\subsection{Polling of available and used descriptors}
+\label{sec:Packed Virtqueues / Polling of available and used descriptors}
+
+Writes of device and driver descriptors can generally be
+reordered, but each side (driver and device) are only required to
+poll (or test) a single location in memory: next device descriptor after
+the one they processed previously, in circular order.
+
+Sometimes device needs to only write out a single used descriptor
+after processing a batch of multiple available descriptors.  As
+described in more detail below, this can happen when using
+descriptor chaining or with in-order
+use of descriptors.  In this case, device writes out a used
+descriptor with buffer id of the last descriptor in the group.
+After processing the used descriptor, both device and driver then
+skip forward in the ring the number of the remaining descriptors
+in the group until processing (reading for the driver and writing
+for the device) the next used descriptor.
+
+\subsection{Write Flag}
+\label{sec:Packed Virtqueues / Write Flag}
+
+In an available descriptor, VIRTQ_DESC_F_WRITE bit within Flags
+is used to mark a descriptor as corresponding to a write-only or
+read-only element of a buffer.
+
+\begin{lstlisting}
+/* This marks a descriptor as device write-only (otherwise device read-only). */
+#define VIRTQ_DESC_F_WRITE     2
+\end{lstlisting}
+
+In a used descriptor, this bit is used to specify whether any
+data has been written by the device into any parts of the buffer.
+
+
+\subsection{Element Address and Length}
+\label{sec:Packed Virtqueues / Element Address and Length}
+
+In an available descriptor, Element Address corresponds to the
+physical address of the buffer element. The length of the element assumed
+to be physically contigious is stored in Element Length.
+
+In a used descriptor, Element Address is unused. Element Length
+specifies the length of the buffer that has been initialized
+(written to) by the device.
+
+Element length is reserved for used descriptors without the
+VIRTQ_DESC_F_WRITE flag, and is ignored by drivers.
+
+\subsection{Scatter-Gather Support}
+\label{sec:Packed Virtqueues / Scatter-Gather Support}
+
+Some drivers need an ability to supply a list of multiple buffer
+elements (also known as a scatter/gather list) with a request.
+Two features support this: descriptor chaining and indirect descriptors.
+
+If neither feature is in use by the driver, each buffer is
+physically-contigious, either read-only or write-only and is
+described completely by a single descriptor.
+
+While unusual (most implementations either create all lists
+solely using non-indirect descriptors, or always use a single
+indirect element), if both features have been negotiated, mixing
+direct and direct descriptors in a ring is valid, as long as each
+list only contains descriptors of a given type.
+
+Scatter/gather lists only apply to available descriptors. A
+single used descriptor corresponds to the whole list.
+
+The device limits the number of descriptors in a list through a
+transport-specific and/or device-specific value. If not limited,
+the maximum number of descriptors in a list is the virt queue
+size.
+
+\subsection{Next Flag: Descriptor Chaining}
+\label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
+
+The packed ring format allows driver to supply
+a scatter/gather list to the device
+by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT in
+Flags for all but the last available descriptor.
+
+\begin{lstlisting}
+/* This marks a buffer as continuing. */
+#define VIRTQ_DESC_F_NEXT   1
+\end{lstlisting}
+
+Buffer ID is included in the last descriptor in the list.
+
+The driver always makes the first descriptor in the list
+available after the rest of the list has been written out into
+the ring. This guarantees that the device will never observe a
+partial scatter/gather list in the ring.
+
+Note: all flags, including VIRTQ_DESC_F_AVAIL, VIRTQ_DESC_F_USED,
+VIRTQ_DESC_F_WRITE must be set/cleared correctly in all
+descriptors in the list, not just the first one.
+
+Device only writes out a single used descriptor for the whole
+list. It then skips forward according to the number of
+descriptors in the list. Driver needs to keep track of the size
+of the list corresponding to each buffer ID, to be able to skip
+to where the next used descriptor is written by the device.
+
+For example, if descriptors are used in the same order in which
+they are made available, this will result in the used descriptor
+overwriting the first available descriptor in the list, the used
+descriptor for the next list overwriting the first available
+descriptor in the next list, etc.
+
+VIRTQ_DESC_F_NEXT is reserved in used descriptors, and
+should be ignored by drivers.
+
+\subsection{Indirect Flag: Scatter-Gather Support}
+\label{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}
+
+Some devices benefit by concurrently dispatching a large number
+of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this. To increase
+ring capacity the driver can store a (read-only by the device) table of indirect
+descriptors anywhere in memory, and insert a descriptor in main
+virtqueue (with \field{Flags} bit VIRTQ_DESC_F_INDIRECT on) that refers to
+a buffer element
+containing this indirect descriptor table; \field{addr} and \field{len}
+refer to the indirect table address and length in bytes,
+respectively.
+\begin{lstlisting}
+/* This means the element contains a table of descriptors. */
+#define VIRTQ_DESC_F_INDIRECT   4
+\end{lstlisting}
+
+The indirect table layout structure looks like this
+(\field{len} is the Buffer Length of the descriptor that refers to this table,
+which is a variable):
+
+\begin{lstlisting}
+struct indirect_descriptor_table {
+        /* The actual descriptor structures (struct virtq_desc each) */
+        struct virtq_desc desc[len / sizeof(struct virtq_desc)];
+};
+\end{lstlisting}
+
+The first descriptor is located at start of the indirect
+descriptor table, additional indirect descriptors come
+immediately afterwards. \field{Flags} bit VIRTQ_DESC_F_WRITE is the
+only valid flag for descriptors in the indirect table. Others
+are reserved and are ignored by the device.
+Buffer ID is also reserved and is ignored by the device.
+
+In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
+is reserved and is ignored by the device.
+
+\subsection{Multi-buffer requests}
+\label{sec:Packed Virtqueues / Multi-buffer requests}
+Some devices combine multiple buffers as part of processing of a
+single request.  These devices always mark the descriptor
+corresponding to the first buffer in the request used after the
+rest of the descriptors (corresponding to rest of the buffers) in
+the request - which follow the first descriptor in ring order -
+has been marked used and written out into the ring.  This
+guarantees that the driver will never observe a partial request
+in the ring.
+
+\subsection{Driver and Device Event Suppression}
+\label{sec:Packed Virtqueues / Driver and Device Event Suppression}
+In many systems driver and device notifications involve
+significant overhead. To mitigate this overhead,
+each virtqueue includes two identical structures used for
+controlling notifications between device and driver.
+
+Driver Event Suppression structure is read-only by the
+device and controls the events sent by the device
+to the driver (e.g. interrupts).
+
+Device Event Suppression structure is read-only by
+the driver and controls the events sent by the driver
+to the device (e.g. IO).
+
+Each of these Event Suppression structures controls
+both Descriptor Ring events and structure events, and
+each includes the following fields:
+
+\begin{description}
+\item [Descriptor Ring Change Event Flags] Takes values:
+\begin{lstlisting}
+/* Enable events */
+#define RING_EVENT_FLAGS_ENABLE 0x0
+/* Disable events */
+#define RING_EVENT_FLAGS_DISABLE 0x1
+/*
+ * Enable events for a specific descriptor
+ * (as specified by Descriptor Ring Change Event Offset/Wrap Counter).
+ * Only valid if VIRTIO_F_RING_EVENT_IDX has been negotiated.
+ */
+#define RING_EVENT_FLAGS_DESC 0x2
+/* The value 0x3 is reserved */
+\end{lstlisting}
+\item [Descriptor Ring Change Event Offset] If Event Flags set to descriptor
+specific event: offset within the ring (in units of descriptor
+size). Event will only trigger when this descriptor is
+made available/used respectively.
+\item [Descriptor Ring Change Event Wrap Counter] If Event Flags set to descriptor
+specific event: offset within the ring (in units of descriptor
+size). Event will only trigger when Ring Wrap Counter
+matches this value and a descriptor is
+made available/used respectively.
+\end{description}
+
+After writing out some descriptors, both device and driver
+are expected to consult the relevant structure to find out
+whether interrupt/notification should be sent.
+
+\subsubsection{Structure Size and Alignment}
+\label{sec:Packed Virtqueues / Structure Size and Alignment}
+
+Each part of the virtqueue is physically-contiguous in guest memory,
+and has different alignment requirements.
+
+The memory alignment and size requirements, in bytes, of each part of the
+virtqueue are summarized in the following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Virtqueue Part    & Alignment & Size \\
+\hline \hline
+Descriptor Ring  & 16        & $16 * $(Queue Size) \\
+\hline
+Device Event Suppression    & 4         & 4 \\
+ \hline
+Driver Event Suppression         & 4         & 4 \\
+ \hline
+\end{tabular}
+
+The Alignment column gives the minimum alignment for each part
+of the virtqueue.
+
+The Size column gives the total number of bytes for each
+part of the virtqueue.
+
+Queue Size corresponds to the maximum number of descriptors in the
+virtqueue\footnote{For example, if Queue Size is 4 then at most 4 buffers
+can be queued at any given time.}.  Queue Size value does not
+have to be a power of 2 unless enforced by the transport.
+
+\drivernormative{\subsection}{Virtqueues}{Basic Facilities of a
+Virtio Device / Packed Virtqueues}
+The driver MUST ensure that the physical address of the first byte
+of each virtqueue part is a multiple of the specified alignment value
+in the above table.
+
+\devicenormative{\subsection}{Virtqueues}{Basic Facilities of a
+Virtio Device / Packed Virtqueues}
+The device MUST start processing driver descriptors in the order
+in which they appear in the ring.
+The device MUST start writing device descriptors into the ring in
+the order in which they complete.
+Device MAY reorder descriptor writes once they are started.
+
+\subsection{The Virtqueue Descriptor Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue
+Descriptor Format}
+
+The available descriptor refers to the buffers the driver is sending
+to the device. \field{addr} is a physical address, and the
+descriptor is identified with a buffer using the \field{id} field.
+
+\begin{lstlisting}
+struct virtq_desc {
+        /* Buffer Address. */
+        le64 addr;
+        /* Buffer Length. */
+        le32 len;
+        /* Buffer ID. */
+        le16 id;
+        /* The flags depending on descriptor type. */
+        le16 flags;
+};
+\end{lstlisting}
+
+The descriptor ring is zero-initialized.
+
+\subsection{Event Suppression Structure Format}\label{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event Suppression Structure
+Format}
+
+The following structure is used to reduce the number of
+notifications sent between driver and device.
+
+\begin{lstlisting}
+le16 desc_event_off : 15, /* Descriptor Event Offset */
+     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
+le16 desc_event_flags : 2, /* Descriptor Event Flags */
+     reserved : 14; /* Reserved, set to 0 */
+\end{lstlisting}
+
+\devicenormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table}
+A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT
+read a device-writable buffer.
+A device MUST NOT use a descriptor unless it observes
+VIRTQ_DESC_F_AVAIL bit in its \field{flags} being changed
+(e.g. as compared to the initial zero value).
+A device MUST NOT change a descriptor after changing it's
+VIRTQ_DESC_F_USED bit in its \field{flags}.
+
+\drivernormative{\subsection}{The Virtqueue Descriptor Table}{Basic Facilities of a Virtio Device / PAcked Virtqueues / The Virtqueue Descriptor Table}
+A driver MUST NOT change a descriptor unless it observes
+VIRTQ_DESC_F_USED bit in its \field{flags} being changed.
+A driver MUST NOT change a descriptor after changing
+VIRTQ_DESC_F_AVAIL bit in its \field{flags}.
+When notifying the device, driver MUST set
+\field{next_off} and
+\field{next_wrap} to match the next descriptor
+not yet made available to the device.
+A driver MAY send multiple notifications without making
+any new descriptors available to the device.
+
+\drivernormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
+Virtio Device / Packed Virtqueues / Scatter-Gather Support}
+A driver MUST NOT create a descriptor list longer than allowed
+by the device.
+
+A driver MUST NOT create a descriptor list longer than the Queue
+Size.
+
+This implies that loops in the descriptor list are forbidden!
+
+The driver MUST place any device-writable descriptor elements after
+any device-readable descriptor elements.
+
+A driver MUST NOT depend on the device to use more descriptors
+to be able to write out all descriptors in a list. A driver
+MUST make sure there's enough space in the ring
+for the whole list before making the first descriptor in the list
+available to the device.
+
+A driver MUST NOT make the first descriptor in the list available
+before all subsequent descriptors comprising the list are made
+available.
+
+\devicenormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
+Virtio Device / Packed Virtqueues / Scatter-Gather Support}
+The device MUST use descriptors in a list chained by the
+VIRTQ_DESC_F_NEXT flag in the same order that they
+were made available by the driver.
+
+The device MAY limit the number of buffers it will allow in a
+list.
+
+\drivernormative{\subsection}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
+The driver MUST NOT set the DESC_F_INDIRECT flag unless the
+VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
+set any flags except DESC_F_WRITE within an indirect descriptor.
+
+A driver MUST NOT create a descriptor chain longer than allowed
+by the device.
+
+A driver MUST NOT write direct descriptors with
+DESC_F_INDIRECT set in a scatter-gather list linked by
+VIRTQ_DESC_F_NEXT.
+\field{flags}.
+
+\subsection{Virtqueue Operation}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Virtqueue Operation}
+
+There are two parts to virtqueue operation: supplying new
+available buffers to the device, and processing used buffers from
+the device.
+
+What follows is the requirements of each of these two parts
+when using the packed virtqueue format in more detail.
+
+\subsection{Supplying Buffers to The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device}
+
+The driver offers buffers to one of the device's virtqueues as follows:
+
+\begin{enumerate}
+\item The driver places the buffer into free descriptor in the Descriptor Ring.
+
+\item The driver performs a suitable memory barrier to ensure that it updates
+  the descriptor(s) before checking for notification suppression.
+
+\item If notifications are not suppressed, the driver notifies the device
+    of the new available buffers.
+\end{enumerate}
+
+What follows is the requirements of each stage in more detail.
+
+\subsubsection{Placing Available Buffers Into The Descriptor Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Placing Available Buffers Into The Descriptor Ring}
+
+For each buffer element, b:
+
+\begin{enumerate}
+\item Get the next descriptor table entry, d
+\item Get the next free buffer id value
+\item Set \field{d.addr} to the physical address of the start of b
+\item Set \field{d.len} to the length of b.
+\item Set \field{d.id} to the buffer id
+\item Calculate the flags as follows:
+\begin{enumerate}
+\item If b is device-writable, set the VIRTQ_DESC_F_WRITE bit to 1, otherwise 0
+\item Set VIRTQ_DESC_F_AVAIL bit to the current value of the Driver Ring Wrap Counter
+\item Set VIRTQ_DESC_F_USED bit to inverse value
+\end{enumerate}
+\item Perform a memory barrier to ensure that the descriptor has
+      been initialized
+\item Set \field{d.flags} to the calculated flags value
+\item If d is the last descriptor in the ring, toggle the
+      Driver Ring Wrap Counter
+\item Otherwise, increment d to point at the next descriptor
+\end{enumerate}
+
+This makes a single descriptor buffer available. However, in
+general the driver MAY make use of a batch of descriptors as part
+of a single request. In that case, it defers updating
+the descriptor flags for the first descriptor
+(and the previous memory barrier) until after the rest of
+the descriptors have been initialized.
+
+Once the descriptor \field{flags} is updated by the driver, this exposes the
+descriptor and its contents.  The device MAY
+access the descriptor and any following descriptors the driver created and the
+memory they refer to immediately.
+
+\drivernormative{\paragraph}{Updating flags}{Basic Facilities of
+a Virtio Device / Packed Virtqueues / Supplying Buffers to The
+Device / Updating flags}
+The driver MUST perform a suitable memory barrier before the
+\field{flags} update, to ensure the
+device sees the most up-to-date copy.
+
+\subsubsection{Notifying The Device}\label{sec:Basic Facilities
+of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+
+The actual method of device notification is bus-specific, but generally
+it can be expensive.  So the device MAY suppress such notifications if it
+doesn't need them, using the Driver Event Suppression structure
+as detailed in section \ref{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event
+Suppression Structure Format}.
+
+The driver has to be careful to expose the new \field{flags}
+value before checking if notifications are suppressed.
+
+\subsubsection{Implementation Example}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Implementation Example}
+
+Below is an example driver code. It does not attempt to reduce
+the number of device interrupts, neither does it support
+the VIRTIO_F_RING_EVENT_IDX feature.
+
+\begin{lstlisting}
+/* Note: vq->avail_wrap_count is initialized to 1 */
+/* Note: vq->ids is an array same size as the ring */
+
+first = vq->next_avail;
+id = alloc_id(vq);
+
+for (each buffer element b) {
+        vq->ids[vq->next_avail] = -1;
+        vq->desc[vq->next_avail].address = get_addr(b);
+        vq->desc[vq->next_avail].len = get_len(b);
+
+        avail = vq->avail_wrap_count ? VIRTQ_DESC_F_AVAIL : 0;
+        used = !vq->avail_wrap_count ? VIRTQ_DESC_F_USED : 0;
+        f = get_flags(b) | avail | used;
+	if (b is not the last buffer element) {
+		f |= VIRTQ_DESC_F_NEXT;
+	}
+
+        /* Don't mark the 1st descriptor available until all of them are ready. */
+        if (vq->next_avail == first) {
+                flags = f;
+        } else {
+                vq->desc[vq->next_avail].flags = f;
+        }
+
+        last = vq->next_avail;
+
+        vq->next_avail++;
+
+        if (vq->next_avail >= vq->size) {
+                vq->next_avail = 0;
+                vq->avail_wrap_count \^= 1;
+        }
+
+
+}
+/* ID included in the last descriptor in the list */
+vq->ids[last] = vq->desc[last].id = id;
+write_memory_barrier();
+vq->desc[first].flags = flags;
+
+memory_barrier();
+
+if (vq->device_event.flags != RING_EVENT_FLAGS_DISABLE) {
+        notify_device(vq);
+}
+
+\end{lstlisting}
+
+
+\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+The driver MUST perform a suitable memory barrier before reading
+the Driver Event Suppression structure, to avoid missing a notification.
+
+\subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Receiving Used Buffers From The Device}
+
+Once the device has used buffers referred to by a descriptor (read from or written to them, or
+parts of both, depending on the nature of the virtqueue and the
+device), it interrupts the driver
+as detailed in section \ref{sec:Basic
+Facilities of a Virtio Device / Packed Virtqueues / Event
+Suppression Structure Format}.
+
+\begin{note}
+For optimal performance, a driver MAY disable interrupts while processing
+the used buffers, but beware the problem of missing interrupts between
+emptying the ring and reenabling interrupts.  This is usually handled by
+re-checking for more used buffers after interrups are re-enabled:
+\end{note}
+
+\begin{lstlisting}
+/* Note: vq->used_wrap_count is initialized to 1 */
+
+vq->driver_event.flags = RING_EVENT_FLAGS_DISABLE;
+
+for (;;) {
+        struct virtq_desc *d = vq->desc[vq->next_used];
+
+        flags = d->flags;
+        bool used = flags & VIRTQ_DESC_F_USED;
+
+        if (used != vq->used_wrap_count) {
+                vq->driver_event.flags = RING_EVENT_FLAGS_ENABLE;
+                memory_barrier();
+
+                flags = d->flags;
+                bool used = flags & VIRTQ_DESC_F_USED;
+                if (used != vq->used_wrap_count) {
+                        break;
+                }
+
+                vq->driver_event.flags = RING_EVENT_FLAGS_DISABLE;
+        }
+
+        read_memory_barrier();
+
+	/* skip descriptors until we find the correct ID */
+        do {
+		found = vq->ids[vq->next_used] == d->id;
+                vq->next_used++;
+                if (vq->next_used >= vq->size) {
+                        vq->next_used = 0;
+                        vq->used_wrap_count \^= 1;
+                }
+        } while (!found);
+
+        process_buffer(d);
+}
+\end{lstlisting}
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 11/16] content: in-order buffer use

[Michael S. Tsirkin]

Using descriptors in-order is sometimes beneficial.  Add an option for
that - per-format detail allowing more optimizations will be added by
follow-up patches.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 content.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/content.tex b/content.tex
index 73f40b7..c57a918 100644
--- a/content.tex
+++ b/content.tex
@@ -245,6 +245,15 @@ a device event - i.e. send an interrupt to the driver.
 Device reports the number of bytes it has written to memory for
 each buffer it uses. This is referred to as ``used length''.
 
+Device is not generally required to use buffers in
+the same order in which they have been made available
+by the driver.
+
+Some devices always use descriptors in the same order in which
+they have been made available. These devices can offer the
+VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+might allow optimizations or simplify driver and/or device code.
+
 Each virtqueue can consist of up to 3 parts:
 \begin{itemize}
 \item Descriptor Area - used for describing buffers
@@ -5248,6 +5257,9 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
   support for the packed virtqueue layout as described in
   \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
+  \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
+  that all buffers are used by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5273,6 +5285,9 @@ translates bus addresses from the device into physical addresses in memory.
 A device MAY fail to operate further if VIRTIO_F_IOMMU_PLATFORM is not
 accepted.
 
+If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
+buffers in the same order in which they have been available.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 12/16] packed-ring: add in order support

[Michael S. Tsirkin]

Support in-order requests for packed rings.
This allows selective write-out of used descriptors.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 packed-ring.tex | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/packed-ring.tex b/packed-ring.tex
index 2943df0..12bab67 100644
--- a/packed-ring.tex
+++ b/packed-ring.tex
@@ -272,6 +272,30 @@ Buffer ID is also reserved and is ignored by the device.
 In Descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE
 is reserved and is ignored by the device.
 
+\subsection{In-order use of descriptors}
+\label{sec:Packed Virtqueues / In-order use of descriptors}
+
+Some devices always use descriptors in the same order in which
+they have been made available. These devices can offer the
+VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch.
+
+Device then skips forward in the ring according to the size of
+the batch. Driver needs to look up the used Buffer ID and
+calculate the batch size to be able to advance to where the next
+used descriptor will be written by the device.
+
+This will result in the used descriptor overwriting the first
+available descriptor in the batch, the used descriptor for the
+next batch overwriting the first available descriptor in the next
+batch, etc.
+
+The skipped buffers (for which no used descriptor was written)
+are assumed to have been used (read or written) by the
+device completely.
+
 \subsection{Multi-buffer requests}
 \label{sec:Packed Virtqueues / Multi-buffer requests}
 Some devices combine multiple buffers as part of processing of a
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 13/16] split-ring: in order feature

[Michael S. Tsirkin]

For a split ring, require that drivers use descriptors in order too.
This allows devices to skip reading the available ring.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 split-ring.tex | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/split-ring.tex b/split-ring.tex
index 87ecee2..df278fe 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -203,6 +203,10 @@ struct virtq_desc {
 The number of descriptors in the table is defined by the queue size
 for this virtqueue: this is the maximum possible descriptor chain length.
 
+If VIRTIO_F_IN_ORDER has been negotiated, driver uses
+descriptors in ring order: starting from offset 0 in the table,
+and wrapping around at the end of the table.
+
 \begin{note}
 The legacy \hyperref[intro:Virtio PCI Draft]{[Virtio PCI Draft]}
 referred to this structure as vring_desc, and the constants as
@@ -218,6 +222,12 @@ purposes).
 Drivers MUST NOT add a descriptor chain over than $2^{32}$ bytes long in total;
 this implies that loops in the descriptor chain are forbidden!
 
+If VIRTIO_F_IN_ORDER has been negotiated, and when making a
+descriptor with VRING_DESC_F_NEXT set in \field{flags} at offset
+$x$ in the table available to the device, driver MUST set
+\field{next} to $0$ for the last descriptor in the table
+(where $x = queue\_size - 1$) and to $x + 1$ for the rest of the descriptors.
+
 \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
 
 Some devices benefit by concurrently dispatching a large number
@@ -247,6 +257,10 @@ chained by \field{next}. An indirect descriptor without a valid \field{next}
 A single indirect descriptor
 table can include both device-readable and device-writable descriptors.
 
+If VIRTIO_F_IN_ORDER has been negotiated, indirect descriptors
+use sequential indices, in-order: index 0 followed by index 1
+followed by index 2, etc.
+
 \drivernormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
 The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the
 VIRTIO_F_INDIRECT_DESC feature was negotiated.   The driver MUST NOT
@@ -259,6 +273,10 @@ the device.
 A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
 in \field{flags}.
 
+If VIRTIO_F_IN_ORDER has been negotiated, indirect descriptors
+MUST appear sequentially, with \field{next} taking the value
+of 1 for the 1st descriptor, 2 for the 2nd one, etc.
+
 \devicenormative{\paragraph}{Indirect Descriptors}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
 The device MUST ignore the write-only flag (\field{flags}\&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Michael S. Tsirkin]

Motivation for the new feature is included in the text.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex      | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
 introduction.tex |   4 +-
 notifications.c  |   3 ++
 3 files changed, 140 insertions(+), 8 deletions(-)
 create mode 100644 notifications.c

diff --git a/content.tex b/content.tex
index c57a918..4261913 100644
--- a/content.tex
+++ b/content.tex
@@ -283,9 +283,77 @@ Packed Virtqueues}).
 Every driver and device supports either the Packed or the Split
 Virtqueue format, or both.
 
+\subsection{Driver notifications} \label{sec:Virtqueues / Driver notifications}
+Driver is sometimes required to notify the device after
+making changes to the virtqueue.
+
+When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
+this notification involves sending the
+virtqueue number to the device (depending on the transport).
+
+However, some devices benefit from ability to find out the number of
+available descriptors in the ring, and whether to send
+interrupts to drivers without accessing virtqueue in memory:
+for efficiency or as a debugging aid.
+
+To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA
+has been negotiated, driver notifications to the device include
+the following information:
+
+\begin{description}
+\item [VQ number]
+\item [Offset]
+      Within the ring where the next available ring entry
+      will be written.
+      Without VIRTIO_F_RING_PACKED this refers to the
+      15 least significant bits of the available index.
+      With VIRTIO_F_RING_PACKED this refers to the offset
+      (in units of descritor entries)
+      within the descriptor ring where the next available
+      descriptor will be written.
+\item [Wrap Counter]
+      With VIRTIO_F_RING_PACKED this is the wrap counter
+      referring to the next available descriptor.
+      Without VIRTIO_F_RING_PACKED this is the most significant bit
+      of the available index.
+\end{description}
+
+Note that driver can trigger multiple notifications even without
+making any more changes to the ring. When VIRTIO_F_NOTIFICATION_DATA
+has been negotiated, these notifications would then have
+identical \field{Offset} and \field{Wrap Counter} values.
+
 \input{split-ring.tex}
 
 \input{packed-ring.tex}
+
+\subsubsection{Driver notifications}
+
+\label{sec:Packed Virtqueues / Driver notifications}
+Whenever not suppressed by Device Event Suppression,
+driver is required to notify the device after
+making changes to the virtqueue.
+
+Some devices benefit from ability to find out the number of
+available descriptors in the ring, and whether to send
+interrupts to drivers without accessing virtqueue in memory:
+for efficiency or as a debugging aid.
+
+To help with these optimizations, driver notifications
+to the device include the following information:
+
+\begin{itemize}
+\item VQ number
+\item Offset (in units of descriptor size) within the ring
+      where the next available descriptor will be written
+\item Wrap Counter referring to the next available
+      descriptor
+\end{itemize}
+
+Note that driver can trigger multiple notifications even without
+making any more changes to the ring. These would then have
+identical \field{Offset} and \field{Wrap Counter} values.
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -862,7 +930,9 @@ the same Queue Notify address for all queues.
 \devicenormative{\paragraph}{Notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 The device MUST present at least one notification capability.
 
-The \field{cap.offset} MUST be 2-byte aligned.  
+For devices not offering VIRTIO_F_NOTIFICATION_DATA:
+
+The \field{cap.offset} MUST be 2-byte aligned.
 
 The device MUST either present \field{notify_off_multiplier} as an even power of 2,
 or present \field{notify_off_multiplier} as 0.
@@ -876,6 +946,23 @@ For all queues, the value \field{cap.length} presented by the device MUST satisf
 cap.length >= queue_notify_off * notify_off_multiplier + 2
 \end{lstlisting}
 
+For devices offering VIRTIO_F_NOTIFICATION_DATA:
+
+The device MUST either present \field{notify_off_multiplier} as a
+number that is a power of 2 that is also a multiple 4,
+or present \field{notify_off_multiplier} as 0.
+
+The \field{cap.offset} MUST be 4-byte aligned.
+
+The value \field{cap.length} presented by the device MUST be at least 4
+and MUST be large enough to support queue notification offsets
+for all supported queues in all possible configurations.
+
+For all queues, the value \field{cap.length} presented by the device MUST satisfy:
+\begin{lstlisting}
+cap.length >= queue_notify_off * notify_off_multiplier + 4
+\end{lstlisting}
+
 \subsubsection{ISR status capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
 
 The VIRTIO_PCI_CAP_ISR_CFG capability
@@ -1268,8 +1355,25 @@ separate cache lines.
 
 \subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notifying The Device}
 
-The driver notifies the device by writing the 16-bit virtqueue index
-of this virtqueue to the Queue Notify address.  See \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} for how to calculate this address.
+When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
+the driver notifies the device by writing the 16-bit virtqueue index
+of this virtqueue (in little-endian byte order format)
+to the Queue Notify address.
+
+When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
+the driver notifies the device by writing the following
+32-bit value to the Queue Notify address:
+\begin{lstlisting}
+le32 vqn : 16,
+     next_off : 15,
+     next_wrap : 1;
+\end{lstlisting}
+
+See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
+for the definition of the components.
+
+See \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} for how to calculate the
+Queue Notify address.
 
 \subsubsection{Virtqueue Interrupts From The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device}
 
@@ -1500,8 +1604,19 @@ All register values are organized as Little Endian.
   }
   \hline 
   \mmioreg{QueueNotify}{Queue notifier}{0x050}{W}{%
-    Writing a queue index to this register notifies the device that
-    there are new buffers to process in the queue.
+    Writing a value this register notifies the device that
+    there are new buffers to process in a queue.
+
+    When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
+    the value written is the queue index.
+
+    When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
+    the value has the following format:
+
+    \lstinputlisting{notifications.c}
+
+    See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
+    for the definition of the components.
   }
   \hline 
   \mmioreg{InterruptStatus}{Interrupt status}{0x60}{R}{%
@@ -2340,12 +2455,22 @@ GPR  &   Input Value     & Output Value \\
 \hline
   2   &  Subchannel ID    & Host Cookie  \\
 \hline
-  3   & Virtqueue number  &              \\
+  3   & Notification data &              \\
 \hline
   4   &   Host Cookie     &              \\
 \hline
 \end{tabular}
 
+When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
+the \field{Notification data} includes the Virtqueue number.
+
+When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
+the value has the following format:
+\lstinputlisting{notifications.c}
+
+See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
+for the definition of the components.
+
 \devicenormative{\paragraph}{Guest->Host Notification}{Virtio Transport Options / Virtio over channel I/O / Device Operation / Guest->Host Notification}
 The device MUST ignore bits 0-31 (counting from the left) of GPR2.
 This aligns passing the subchannel ID with the way it is passed
@@ -5260,6 +5385,10 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_NOTIFICATION_DATA(36)] This feature indicates
+  that drivers pass extra data (besides identifying the Virtqueue)
+  in their device notifications.
+  See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
diff --git a/introduction.tex b/introduction.tex
index 3cb7a70..d0b770e 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -163,8 +163,8 @@ from the least significant to the most significant bit.
 
 For example:
 \begin{lstlisting}
-be16 A : 15;
-be16 B : 1;
+be16 A : 15,
+     B : 1;
 \end{lstlisting}
 documents the value A stored in the low 15 bit of a 16 bit
 integer and the value B stored in the high bit of the 16 bit
diff --git a/notifications.c b/notifications.c
new file mode 100644
index 0000000..2ae96d4
--- /dev/null
+++ b/notifications.c
@@ -0,0 +1,3 @@
+u32 vqn : 16,
+    next_off : 15,
+    next_wrap : 1;
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH v9 15/16] makediff: update to show diff from master

[Michael S. Tsirkin]

Down the road, when we are close to releasing
v1.1 we will want to show diff from cs04. But for now,
it's handy to generate the diff from master,
this way each new feature is redlined separately.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 makediff.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/makediff.sh b/makediff.sh
index 1dd75d4..2ef92fc 100755
--- a/makediff.sh
+++ b/makediff.sh
@@ -9,7 +9,7 @@ export DATESTR=${DATESTR:-`cat REVISION-DATE`}
 MAIN=$1
 PATH=.:${PATH}
 cur="$PWD"
-oldrev=`git rev-list -1 origin/tags/v1.0-cs03`
+oldrev=`git rev-list -1 origin/master`
 newrev=`git rev-list -1 HEAD`
 rm -fr old new
 git clone $PWD old
@@ -19,7 +19,6 @@ while read -r rev; do
 	echo "Applying $rev"
 	git cherry-pick `git rev-list -1 -F --grep "$rev" $newrev` || exit 1
 done << 'EOF'
-headerfile: rename virtio_ring to virtio queue
 EOF
 
 #mv specvars.tex specvars-orig.tex
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH dont commit v9 16/16] REVISION: set to 1.1 packed wd09

[Michael S. Tsirkin]

This is just a technicality to generate PDF as a working draft.
My submit scripts insist on including it but feel free ignore,
there is no need to commit this part.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 REVISION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/REVISION b/REVISION
index 6a9f46a..c1f9103 100644
--- a/REVISION
+++ b/REVISION
@@ -1 +1 @@
-virtio-v1.0-cs04
+virtio-v1.1-wd09
-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 09/16] split-ring: typo: aligment

[Cornelia Huck]

On Thu, 1 Mar 2018 01:31:34 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> ---
>  split-ring.tex | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/split-ring.tex b/split-ring.tex
> index a594d41..87ecee2 100644
> --- a/split-ring.tex
> +++ b/split-ring.tex
> @@ -23,7 +23,7 @@ Each virtqueue consists of three parts:
>  where each part is physically-contiguous in guest memory,
>  and has different alignment requirements.
>  
> -The memory aligment and size requirements, in bytes, of each part of the
> +The memory alignment and size requirements, in bytes, of each part of the
>  virtqueue are summarized in the following table:
>  
>  \begin{tabular}{|l|l|l|}

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 08/16] split-ring: generalize text

[Cornelia Huck]

On Thu, 1 Mar 2018 01:31:33 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Update generic text to talk about available/used buffers, not rings.
> Move some split-ring specific text to the correct section.
> 
> Update conformance section with link to the new conformance clause.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  conformance.tex |  1 +
>  content.tex     | 10 ++++------
>  split-ring.tex  |  4 ++++
>  3 files changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/conformance.tex b/conformance.tex
> index 55d17b4..e4efe33 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -38,6 +38,7 @@ A driver MUST conform to the following normative statements:
>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
> +\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
> diff --git a/content.tex b/content.tex
> index 5634c7d..e1e30a0 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -381,12 +381,10 @@ of a device are live once the device has been reset.
>  
>  \drivernormative{\subsection}{Device Cleanup}{General Initialization And Device Operation / Device Cleanup}
>  
> -A driver MUST NOT alter descriptor table entries which have been
> -exposed in the available ring (and not marked consumed by the device
> -in the used ring) of a live virtqueue.
> -
> -A driver MUST NOT decrement the available \field{idx} on a live virtqueue (ie.
> -there is no way to ``unexpose'' buffers).
> +A driver MUST NOT alter virtqueue entries for exposed buffers -
> +i.e. buffers which have been
> +made available to the device (and not been used by the device)
> +of a live virtqueue.
>  
>  Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
>  
> diff --git a/split-ring.tex b/split-ring.tex
> index 9601a53..a594d41 100644
> --- a/split-ring.tex
> +++ b/split-ring.tex
> @@ -296,6 +296,10 @@ referred to this structure as vring_avail, and the constant as
>  VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
>  \end{note}
>  
> +\drivernormative{\subsubsection}{The Virtqueue Available Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
> +A driver MUST NOT decrement the available \field{idx} on a virtqueue (ie.

s/ie./i.e./

as you're touching it anyway.

> +there is no way to ``unexpose'' buffers).
> +
>  \subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
>  
>  If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Cornelia Huck]

On Thu, 1 Mar 2018 01:31:37 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Motivation for the new feature is included in the text.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex      | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  introduction.tex |   4 +-
>  notifications.c  |   3 ++
>  3 files changed, 140 insertions(+), 8 deletions(-)
>  create mode 100644 notifications.c
> 
> diff --git a/content.tex b/content.tex
> index c57a918..4261913 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -283,9 +283,77 @@ Packed Virtqueues}).
>  Every driver and device supports either the Packed or the Split
>  Virtqueue format, or both.
>  
> +\subsection{Driver notifications} \label{sec:Virtqueues / Driver notifications}
> +Driver is sometimes required to notify the device after
> +making changes to the virtqueue.
> +
> +When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> +this notification involves sending the
> +virtqueue number to the device (depending on the transport).
> +
> +However, some devices benefit from ability to find out the number of
> +available descriptors in the ring, and whether to send
> +interrupts to drivers without accessing virtqueue in memory:
> +for efficiency or as a debugging aid.
> +
> +To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA
> +has been negotiated, driver notifications to the device include
> +the following information:
> +
> +\begin{description}
> +\item [VQ number]
> +\item [Offset]
> +      Within the ring where the next available ring entry
> +      will be written.
> +      Without VIRTIO_F_RING_PACKED this refers to the
> +      15 least significant bits of the available index.
> +      With VIRTIO_F_RING_PACKED this refers to the offset
> +      (in units of descritor entries)
> +      within the descriptor ring where the next available
> +      descriptor will be written.
> +\item [Wrap Counter]
> +      With VIRTIO_F_RING_PACKED this is the wrap counter
> +      referring to the next available descriptor.
> +      Without VIRTIO_F_RING_PACKED this is the most significant bit
> +      of the available index.
> +\end{description}
> +
> +Note that driver can trigger multiple notifications even without
> +making any more changes to the ring. When VIRTIO_F_NOTIFICATION_DATA
> +has been negotiated, these notifications would then have
> +identical \field{Offset} and \field{Wrap Counter} values.
> +
>  \input{split-ring.tex}
>  
>  \input{packed-ring.tex}
> +
> +\subsubsection{Driver notifications}
> +
> +\label{sec:Packed Virtqueues / Driver notifications}
> +Whenever not suppressed by Device Event Suppression,
> +driver is required to notify the device after
> +making changes to the virtqueue.
> +
> +Some devices benefit from ability to find out the number of
> +available descriptors in the ring, and whether to send
> +interrupts to drivers without accessing virtqueue in memory:
> +for efficiency or as a debugging aid.
> +
> +To help with these optimizations, driver notifications
> +to the device include the following information:
> +
> +\begin{itemize}
> +\item VQ number
> +\item Offset (in units of descriptor size) within the ring
> +      where the next available descriptor will be written
> +\item Wrap Counter referring to the next available
> +      descriptor
> +\end{itemize}
> +
> +Note that driver can trigger multiple notifications even without
> +making any more changes to the ring. These would then have
> +identical \field{Offset} and \field{Wrap Counter} values.

Isn't that duplicating the information for the generic case?

(And if you wanted to specify something specific for the packed case,
shouldn't it go into packed-ring.tex?)

> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
>  We start with an overview of device initialization, then expand on the

(...)

> +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> +the driver notifies the device by writing the following
> +32-bit value to the Queue Notify address:
> +\begin{lstlisting}
> +le32 vqn : 16,
> +     next_off : 15,
> +     next_wrap : 1;

Don't we want to write this as

le32 vqn : 16;
le32 next_off :15;
le32 next_wrap : 1;

?

> +\end{lstlisting}
> +
> +See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
> +for the definition of the components.
> +
> +See \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} for how to calculate the
> +Queue Notify address.
>  
>  \subsubsection{Virtqueue Interrupts From The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device}
>  
> @@ -1500,8 +1604,19 @@ All register values are organized as Little Endian.
>    }
>    \hline 
>    \mmioreg{QueueNotify}{Queue notifier}{0x050}{W}{%
> -    Writing a queue index to this register notifies the device that
> -    there are new buffers to process in the queue.
> +    Writing a value this register notifies the device that
> +    there are new buffers to process in a queue.
> +
> +    When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> +    the value written is the queue index.
> +
> +    When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> +    the value has the following format:
> +
> +    \lstinputlisting{notifications.c}

Doesn't mmio require this to be le explicitly?

> +
> +    See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
> +    for the definition of the components.
>    }
>    \hline 
>    \mmioreg{InterruptStatus}{Interrupt status}{0x60}{R}{%
> @@ -2340,12 +2455,22 @@ GPR  &   Input Value     & Output Value \\
>  \hline
>    2   &  Subchannel ID    & Host Cookie  \\
>  \hline
> -  3   & Virtqueue number  &              \\
> +  3   & Notification data &              \\
>  \hline
>    4   &   Host Cookie     &              \\
>  \hline
>  \end{tabular}
>  
> +When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> +the \field{Notification data} includes the Virtqueue number.
> +
> +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> +the value has the following format:
> +\lstinputlisting{notifications.c}

And we probably want to make this be explicitly.

> +
> +See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
> +for the definition of the components.
> +
>  \devicenormative{\paragraph}{Guest->Host Notification}{Virtio Transport Options / Virtio over channel I/O / Device Operation / Guest->Host Notification}
>  The device MUST ignore bits 0-31 (counting from the left) of GPR2.
>  This aligns passing the subchannel ID with the way it is passed
> @@ -5260,6 +5385,10 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
>    \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
>    that all buffers are used by the device in the same
>    order in which they have been made available.
> +  \item[VIRTIO_F_NOTIFICATION_DATA(36)] This feature indicates
> +  that drivers pass extra data (besides identifying the Virtqueue)
> +  in their device notifications.
> +  See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
>  \end{description}
>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> diff --git a/introduction.tex b/introduction.tex
> index 3cb7a70..d0b770e 100644
> --- a/introduction.tex
> +++ b/introduction.tex
> @@ -163,8 +163,8 @@ from the least significant to the most significant bit.
>  
>  For example:
>  \begin{lstlisting}
> -be16 A : 15;
> -be16 B : 1;
> +be16 A : 15,
> +     B : 1;

Why are you dropping the second be16?

>  \end{lstlisting}
>  documents the value A stored in the low 15 bit of a 16 bit
>  integer and the value B stored in the high bit of the 16 bit
> diff --git a/notifications.c b/notifications.c
> new file mode 100644
> index 0000000..2ae96d4
> --- /dev/null
> +++ b/notifications.c
> @@ -0,0 +1,3 @@
> +u32 vqn : 16,
> +    next_off : 15,
> +    next_wrap : 1;

I'm wondering how useful the u32 notation is here.


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 10/16] packed virtqueues: more efficient virtqueue layout

[Michael S. Tsirkin]

On Thu, Mar 01, 2018 at 01:31:34AM +0200, Michael S. Tsirkin wrote:
> Performance analysis of this is in my kvm forum 2016 presentation.  The
> idea is to have a r/w descriptor in a ring structure, replacing the used
> and available ring, index and descriptor buffer.
> 
> This is also easier for devices to implement than the 1.0 layout.
> Several more enhancements will be necessary to actually make this
> efficient for devices to use.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> Acked-by: Cornelia Huck <cohuck@redhat.com>

OK so it's been almost a week and no more comments, but I went
over it myself again and found a bug in the pseudocode. It did
not handle out of order used buffers correctly.  The right thing
to do is to record the number of descriptors - not the id - in a
driver-specific data structure.

As it's just a pseudocode change, I assume it's not a reason for
more delay.  Will include in the final version which I plan to
start voting on early next week.


diff --git a/packed-ring.tex b/packed-ring.tex
index 12bab67..99912c3 100644
--- a/packed-ring.tex
+++ b/packed-ring.tex
@@ -594,12 +594,15 @@ the VIRTIO_F_RING_EVENT_IDX feature.
 
 \begin{lstlisting}
 /* Note: vq->avail_wrap_count is initialized to 1 */
-/* Note: vq->ids is an array same size as the ring */
+/* Note: vq->sgs is an array same size as the ring */
 
-first = vq->next_avail;
 id = alloc_id(vq);
 
+first = vq->next_avail;
+sgs = 0;
 for (each buffer element b) {
+        sgs++;
+
         vq->ids[vq->next_avail] = -1;
         vq->desc[vq->next_avail].address = get_addr(b);
         vq->desc[vq->next_avail].len = get_len(b);
@@ -607,9 +610,9 @@ for (each buffer element b) {
         avail = vq->avail_wrap_count ? VIRTQ_DESC_F_AVAIL : 0;
         used = !vq->avail_wrap_count ? VIRTQ_DESC_F_USED : 0;
         f = get_flags(b) | avail | used;
-	if (b is not the last buffer element) {
-		f |= VIRTQ_DESC_F_NEXT;
-	}
+        if (b is not the last buffer element) {
+                f |= VIRTQ_DESC_F_NEXT;
+        }
 
         /* Don't mark the 1st descriptor available until all of them are ready. */
         if (vq->next_avail == first) {
@@ -626,11 +629,10 @@ for (each buffer element b) {
                 vq->next_avail = 0;
                 vq->avail_wrap_count \^= 1;
         }
-
-
 }
+vq->sgs[id] = sgs;
 /* ID included in the last descriptor in the list */
-vq->ids[last] = vq->desc[last].id = id;
+vq->desc[last].id = id;
 write_memory_barrier();
 vq->desc[first].flags = flags;
 
@@ -689,15 +691,17 @@ for (;;) {
 
         read_memory_barrier();
 
-	/* skip descriptors until we find the correct ID */
-        do {
-		found = vq->ids[vq->next_used] == d->id;
-                vq->next_used++;
-                if (vq->next_used >= vq->size) {
-                        vq->next_used = 0;
-                        vq->used_wrap_count \^= 1;
-                }
-        } while (!found);
+        /* skip descriptors until the next buffer */
+        id = d->id;
+        assert(id < vq->size);
+        sgs = vq->sgs[id];
+        vq->next_used += sgs;
+        if (vq->next_used >= vq->size) {
+                vq->next_used -= vq->size;
+                vq->used_wrap_count \^= 1;
+        }
+
+        free_id(vq, id);
 
         process_buffer(d);
 }

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Michael S. Tsirkin]

On Wed, Mar 07, 2018 at 12:11:58PM +0100, Cornelia Huck wrote:
> On Thu, 1 Mar 2018 01:31:37 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > Motivation for the new feature is included in the text.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex      | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> >  introduction.tex |   4 +-
> >  notifications.c  |   3 ++
> >  3 files changed, 140 insertions(+), 8 deletions(-)
> >  create mode 100644 notifications.c
> > 
> > diff --git a/content.tex b/content.tex
> > index c57a918..4261913 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -283,9 +283,77 @@ Packed Virtqueues}).
> >  Every driver and device supports either the Packed or the Split
> >  Virtqueue format, or both.
> >  
> > +\subsection{Driver notifications} \label{sec:Virtqueues / Driver notifications}
> > +Driver is sometimes required to notify the device after
> > +making changes to the virtqueue.
> > +
> > +When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> > +this notification involves sending the
> > +virtqueue number to the device (depending on the transport).
> > +
> > +However, some devices benefit from ability to find out the number of
> > +available descriptors in the ring, and whether to send
> > +interrupts to drivers without accessing virtqueue in memory:
> > +for efficiency or as a debugging aid.
> > +
> > +To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA
> > +has been negotiated, driver notifications to the device include
> > +the following information:
> > +
> > +\begin{description}
> > +\item [VQ number]
> > +\item [Offset]
> > +      Within the ring where the next available ring entry
> > +      will be written.
> > +      Without VIRTIO_F_RING_PACKED this refers to the
> > +      15 least significant bits of the available index.
> > +      With VIRTIO_F_RING_PACKED this refers to the offset
> > +      (in units of descritor entries)
> > +      within the descriptor ring where the next available
> > +      descriptor will be written.
> > +\item [Wrap Counter]
> > +      With VIRTIO_F_RING_PACKED this is the wrap counter
> > +      referring to the next available descriptor.
> > +      Without VIRTIO_F_RING_PACKED this is the most significant bit
> > +      of the available index.
> > +\end{description}
> > +
> > +Note that driver can trigger multiple notifications even without
> > +making any more changes to the ring. When VIRTIO_F_NOTIFICATION_DATA
> > +has been negotiated, these notifications would then have
> > +identical \field{Offset} and \field{Wrap Counter} values.
> > +
> >  \input{split-ring.tex}
> >  
> >  \input{packed-ring.tex}
> > +
> > +\subsubsection{Driver notifications}
> > +
> > +\label{sec:Packed Virtqueues / Driver notifications}
> > +Whenever not suppressed by Device Event Suppression,
> > +driver is required to notify the device after
> > +making changes to the virtqueue.
> > +
> > +Some devices benefit from ability to find out the number of
> > +available descriptors in the ring, and whether to send
> > +interrupts to drivers without accessing virtqueue in memory:
> > +for efficiency or as a debugging aid.
> > +
> > +To help with these optimizations, driver notifications
> > +to the device include the following information:
> > +
> > +\begin{itemize}
> > +\item VQ number
> > +\item Offset (in units of descriptor size) within the ring
> > +      where the next available descriptor will be written
> > +\item Wrap Counter referring to the next available
> > +      descriptor
> > +\end{itemize}
> > +
> > +Note that driver can trigger multiple notifications even without
> > +making any more changes to the ring. These would then have
> > +identical \field{Offset} and \field{Wrap Counter} values.
> 
> Isn't that duplicating the information for the generic case?
> 
> (And if you wanted to specify something specific for the packed case,
> shouldn't it go into packed-ring.tex?)

Right. Cut-n-pasted twice. Will drop.


> > +
> >  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> >  
> >  We start with an overview of device initialization, then expand on the
> 
> (...)
> 
> > +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > +the driver notifies the device by writing the following
> > +32-bit value to the Queue Notify address:
> > +\begin{lstlisting}
> > +le32 vqn : 16,
> > +     next_off : 15,
> > +     next_wrap : 1;
> 
> Don't we want to write this as
> 
> le32 vqn : 16;
> le32 next_off :15;
> le32 next_wrap : 1;
> 
> ?

Same thing in C, but would be more confusing IMHO since it will be up to
the reader to figure out which fields comprise the 32 bit integer.


> > +\end{lstlisting}
> > +
> > +See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
> > +for the definition of the components.
> > +
> > +See \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} for how to calculate the
> > +Queue Notify address.
> >  
> >  \subsubsection{Virtqueue Interrupts From The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device}
> >  
> > @@ -1500,8 +1604,19 @@ All register values are organized as Little Endian.
> >    }
> >    \hline 
> >    \mmioreg{QueueNotify}{Queue notifier}{0x050}{W}{%
> > -    Writing a queue index to this register notifies the device that
> > -    there are new buffers to process in the queue.
> > +    Writing a value this register notifies the device that
> > +    there are new buffers to process in a queue.
> > +
> > +    When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> > +    the value written is the queue index.
> > +
> > +    When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > +    the value has the following format:
> > +
> > +    \lstinputlisting{notifications.c}
> 
> Doesn't mmio require this to be le explicitly?

Oops you are right. The Latex I guess I will have to share with PCI instead.

> > +
> > +    See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
> > +    for the definition of the components.
> >    }
> >    \hline 
> >    \mmioreg{InterruptStatus}{Interrupt status}{0x60}{R}{%
> > @@ -2340,12 +2455,22 @@ GPR  &   Input Value     & Output Value \\
> >  \hline
> >    2   &  Subchannel ID    & Host Cookie  \\
> >  \hline
> > -  3   & Virtqueue number  &              \\
> > +  3   & Notification data &              \\
> >  \hline
> >    4   &   Host Cookie     &              \\
> >  \hline
> >  \end{tabular}
> >  
> > +When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> > +the \field{Notification data} includes the Virtqueue number.
> > +
> > +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > +the value has the following format:
> > +\lstinputlisting{notifications.c}
> 
> And we probably want to make this be explicitly.

Are you sure?
I looked at s390 code and it just uses VQ index in native
endian-ness, so I kept this consistent.
No skin off my nose but pls let me know.

> > +
> > +See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}
> > +for the definition of the components.
> > +
> >  \devicenormative{\paragraph}{Guest->Host Notification}{Virtio Transport Options / Virtio over channel I/O / Device Operation / Guest->Host Notification}
> >  The device MUST ignore bits 0-31 (counting from the left) of GPR2.
> >  This aligns passing the subchannel ID with the way it is passed
> > @@ -5260,6 +5385,10 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
> >    \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
> >    that all buffers are used by the device in the same
> >    order in which they have been made available.
> > +  \item[VIRTIO_F_NOTIFICATION_DATA(36)] This feature indicates
> > +  that drivers pass extra data (besides identifying the Virtqueue)
> > +  in their device notifications.
> > +  See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
> >  \end{description}
> >  
> >  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> > diff --git a/introduction.tex b/introduction.tex
> > index 3cb7a70..d0b770e 100644
> > --- a/introduction.tex
> > +++ b/introduction.tex
> > @@ -163,8 +163,8 @@ from the least significant to the most significant bit.
> >  
> >  For example:
> >  \begin{lstlisting}
> > -be16 A : 15;
> > -be16 B : 1;
> > +be16 A : 15,
> > +     B : 1;
> 
> Why are you dropping the second be16?

To make it clearer these two fields are part of the same
integer, without need to count bits.

> >  \end{lstlisting}
> >  documents the value A stored in the low 15 bit of a 16 bit
> >  integer and the value B stored in the high bit of the 16 bit
> > diff --git a/notifications.c b/notifications.c
> > new file mode 100644
> > index 0000000..2ae96d4
> > --- /dev/null
> > +++ b/notifications.c
> > @@ -0,0 +1,3 @@
> > +u32 vqn : 16,
> > +    next_off : 15,
> > +    next_wrap : 1;
> 
> I'm wondering how useful the u32 notation is here.

It says vqn in low 16 bits of a 32 bit counter, so e.g.
on LE system byte 0 and on BE system byte 3.
Compare to 

be32 vqn : 16,
     next_off : 15,
     next_wrap : 1;

where we say it's in low 16 bits of a 32 bit BE integer,
so byte 3.

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Cornelia Huck]

On Wed, 7 Mar 2018 16:09:33 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Mar 07, 2018 at 12:11:58PM +0100, Cornelia Huck wrote:
> > On Thu, 1 Mar 2018 01:31:37 +0200
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:

> > > +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > > +the driver notifies the device by writing the following
> > > +32-bit value to the Queue Notify address:
> > > +\begin{lstlisting}
> > > +le32 vqn : 16,
> > > +     next_off : 15,
> > > +     next_wrap : 1;  
> > 
> > Don't we want to write this as
> > 
> > le32 vqn : 16;
> > le32 next_off :15;
> > le32 next_wrap : 1;
> > 
> > ?  
> 
> Same thing in C, but would be more confusing IMHO since it will be up to
> the reader to figure out which fields comprise the 32 bit integer.

It looked weird to me. Other opinions?

> > > @@ -2340,12 +2455,22 @@ GPR  &   Input Value     & Output Value \\
> > >  \hline
> > >    2   &  Subchannel ID    & Host Cookie  \\
> > >  \hline
> > > -  3   & Virtqueue number  &              \\
> > > +  3   & Notification data &              \\
> > >  \hline
> > >    4   &   Host Cookie     &              \\
> > >  \hline
> > >  \end{tabular}
> > >  
> > > +When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> > > +the \field{Notification data} includes the Virtqueue number.
> > > +
> > > +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > > +the value has the following format:
> > > +\lstinputlisting{notifications.c}  
> > 
> > And we probably want to make this be explicitly.  
> 
> Are you sure?
> I looked at s390 code and it just uses VQ index in native
> endian-ness, so I kept this consistent.

Native endian is always be on s390. The registers for the hypercall are
therefore always be.

> > > diff --git a/notifications.c b/notifications.c
> > > new file mode 100644
> > > index 0000000..2ae96d4
> > > --- /dev/null
> > > +++ b/notifications.c
> > > @@ -0,0 +1,3 @@
> > > +u32 vqn : 16,
> > > +    next_off : 15,
> > > +    next_wrap : 1;  
> > 
> > I'm wondering how useful the u32 notation is here.  
> 
> It says vqn in low 16 bits of a 32 bit counter, so e.g.
> on LE system byte 0 and on BE system byte 3.
> Compare to 
> 
> be32 vqn : 16,
>      next_off : 15,
>      next_wrap : 1;
> 
> where we say it's in low 16 bits of a 32 bit BE integer,
> so byte 3.

I was referring to the endianness: AFAICS, pci and mmio want le, ccw
wants be. If we want to be explicit, notifications.c does not make much
sense IMO.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Michael S. Tsirkin]

On Wed, Mar 07, 2018 at 03:49:41PM +0100, Cornelia Huck wrote:
> On Wed, 7 Mar 2018 16:09:33 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Wed, Mar 07, 2018 at 12:11:58PM +0100, Cornelia Huck wrote:
> > > On Thu, 1 Mar 2018 01:31:37 +0200
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > > > +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > > > +the driver notifies the device by writing the following
> > > > +32-bit value to the Queue Notify address:
> > > > +\begin{lstlisting}
> > > > +le32 vqn : 16,
> > > > +     next_off : 15,
> > > > +     next_wrap : 1;  
> > > 
> > > Don't we want to write this as
> > > 
> > > le32 vqn : 16;
> > > le32 next_off :15;
> > > le32 next_wrap : 1;
> > > 
> > > ?  
> > 
> > Same thing in C, but would be more confusing IMHO since it will be up to
> > the reader to figure out which fields comprise the 32 bit integer.
> 
> It looked weird to me. Other opinions?
> 
> > > > @@ -2340,12 +2455,22 @@ GPR  &   Input Value     & Output Value \\
> > > >  \hline
> > > >    2   &  Subchannel ID    & Host Cookie  \\
> > > >  \hline
> > > > -  3   & Virtqueue number  &              \\
> > > > +  3   & Notification data &              \\
> > > >  \hline
> > > >    4   &   Host Cookie     &              \\
> > > >  \hline
> > > >  \end{tabular}
> > > >  
> > > > +When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> > > > +the \field{Notification data} includes the Virtqueue number.
> > > > +
> > > > +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > > > +the value has the following format:
> > > > +\lstinputlisting{notifications.c}  
> > > 
> > > And we probably want to make this be explicitly.  
> > 
> > Are you sure?
> > I looked at s390 code and it just uses VQ index in native
> > endian-ness, so I kept this consistent.
> 
> Native endian is always be on s390. The registers for the hypercall are
> therefore always be.

Oh I thought there's ppc LE now.

> > > > diff --git a/notifications.c b/notifications.c
> > > > new file mode 100644
> > > > index 0000000..2ae96d4
> > > > --- /dev/null
> > > > +++ b/notifications.c
> > > > @@ -0,0 +1,3 @@
> > > > +u32 vqn : 16,
> > > > +    next_off : 15,
> > > > +    next_wrap : 1;  
> > > 
> > > I'm wondering how useful the u32 notation is here.  
> > 
> > It says vqn in low 16 bits of a 32 bit counter, so e.g.
> > on LE system byte 0 and on BE system byte 3.
> > Compare to 
> > 
> > be32 vqn : 16,
> >      next_off : 15,
> >      next_wrap : 1;
> > 
> > where we say it's in low 16 bits of a 32 bit BE integer,
> > so byte 3.
> 
> I was referring to the endianness: AFAICS, pci and mmio want le, ccw
> wants be. If we want to be explicit, notifications.c does not make much
> sense IMO.

Will split up.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Cornelia Huck]

On Wed, 7 Mar 2018 17:10:21 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Mar 07, 2018 at 03:49:41PM +0100, Cornelia Huck wrote:
> > On Wed, 7 Mar 2018 16:09:33 +0200
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Wed, Mar 07, 2018 at 12:11:58PM +0100, Cornelia Huck wrote:  
> > > > On Thu, 1 Mar 2018 01:31:37 +0200
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:  

> > > > > @@ -2340,12 +2455,22 @@ GPR  &   Input Value     & Output Value \\
> > > > >  \hline
> > > > >    2   &  Subchannel ID    & Host Cookie  \\
> > > > >  \hline
> > > > > -  3   & Virtqueue number  &              \\
> > > > > +  3   & Notification data &              \\
> > > > >  \hline
> > > > >    4   &   Host Cookie     &              \\
> > > > >  \hline
> > > > >  \end{tabular}
> > > > >  
> > > > > +When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> > > > > +the \field{Notification data} includes the Virtqueue number.
> > > > > +
> > > > > +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > > > > +the value has the following format:
> > > > > +\lstinputlisting{notifications.c}    
> > > > 
> > > > And we probably want to make this be explicitly.    
> > > 
> > > Are you sure?
> > > I looked at s390 code and it just uses VQ index in native
> > > endian-ness, so I kept this consistent.  
> > 
> > Native endian is always be on s390. The registers for the hypercall are
> > therefore always be.  
> 
> Oh I thought there's ppc LE now.

ppc is not s390 :)

I highly doubt that there will be non-be s390 ever (too much legacy
software).

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Halil Pasic]

On 03/07/2018 03:49 PM, Cornelia Huck wrote:
>>>> +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
>>>> +the driver notifies the device by writing the following
>>>> +32-bit value to the Queue Notify address:
>>>> +\begin{lstlisting}
>>>> +le32 vqn : 16,
>>>> +     next_off : 15,
>>>> +     next_wrap : 1;  
>>> Don't we want to write this as
>>>
>>> le32 vqn : 16;
>>> le32 next_off :15;
>>> le32 next_wrap : 1;
>>>
>>> ?  
>> Same thing in C, but would be more confusing IMHO since it will be up to
>> the reader to figure out which fields comprise the 32 bit integer.
> It looked weird to me. Other opinions?
> 

Regarding the c11 standard the two are equivalent. Thus it does not
matter to me which notation is used. AFAIK bit-fields are only defined
in the context of structs (and/or unions), so I assumed that. Putting a
struct around it would be much better IMHO.

I don't agree with Michael's argument about 'which fields comprise the
32 bit integer', as IMHO it does not make sense in terms of c11.

Consider 

struct A {
	uint32_t a:30, b:1, c:2:, d:8;
};

I think, in this particular case the notation ain't very helpful in
figuring out what comprises what. For that reason, if I really need
to choose, I would side with Connie on this one.

But there is another, more significant problem IMHO. The guarantees
provided by the C language (c11) regarding the resulting memory layout
are not sufficient to reason about it like Michael's comment and
the bit's of the draft imply. To know the memory layout we need the
ABI specification for the given platform on top of the C standard.

So if the bit fields are about in memory layout, I find the stuff
problematic. If however we use bit-fields only to define how arithmetic
works, then we are fine.

And a side note: A stronger/better reference to the C language could
benefit clarity. Especially if we start relying on the less trivial
properties of the C language.

Hope this helps!

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Cornelia Huck]

On Wed, 7 Mar 2018 17:05:24 +0100
Halil Pasic <pasic@linux.vnet.ibm.com> wrote:

> On 03/07/2018 03:49 PM, Cornelia Huck wrote:
> >>>> +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> >>>> +the driver notifies the device by writing the following
> >>>> +32-bit value to the Queue Notify address:
> >>>> +\begin{lstlisting}
> >>>> +le32 vqn : 16,
> >>>> +     next_off : 15,
> >>>> +     next_wrap : 1;    
> >>> Don't we want to write this as
> >>>
> >>> le32 vqn : 16;
> >>> le32 next_off :15;
> >>> le32 next_wrap : 1;
> >>>
> >>> ?    
> >> Same thing in C, but would be more confusing IMHO since it will be up to
> >> the reader to figure out which fields comprise the 32 bit integer.  
> > It looked weird to me. Other opinions?
> >   
> 
> Regarding the c11 standard the two are equivalent. Thus it does not
> matter to me which notation is used. AFAIK bit-fields are only defined
> in the context of structs (and/or unions), so I assumed that. Putting a
> struct around it would be much better IMHO.
> 
> I don't agree with Michael's argument about 'which fields comprise the
> 32 bit integer', as IMHO it does not make sense in terms of c11.
> 
> Consider 
> 
> struct A {
> 	uint32_t a:30, b:1, c:2:, d:8;
> };
> 
> I think, in this particular case the notation ain't very helpful in
> figuring out what comprises what. For that reason, if I really need
> to choose, I would side with Connie on this one.
> 
> But there is another, more significant problem IMHO. The guarantees
> provided by the C language (c11) regarding the resulting memory layout
> are not sufficient to reason about it like Michael's comment and
> the bit's of the draft imply. To know the memory layout we need the
> ABI specification for the given platform on top of the C standard.
> 
> So if the bit fields are about in memory layout, I find the stuff
> problematic. If however we use bit-fields only to define how arithmetic
> works, then we are fine.

I'm not sure we should go down the C standard rabbit hole. People have
gotten lost in there.

If the clarification of what we mean by this notation (patch 1 + the
update sent later) is not enough, I'd rather prefer us to add a
clarifying sentence/diagram/... there. I was mainly bothered by the
change to the definition in this patch...

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Michael S. Tsirkin]

On Wed, Mar 07, 2018 at 05:05:24PM +0100, Halil Pasic wrote:
> 
> 
> On 03/07/2018 03:49 PM, Cornelia Huck wrote:
> >>>> +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> >>>> +the driver notifies the device by writing the following
> >>>> +32-bit value to the Queue Notify address:
> >>>> +\begin{lstlisting}
> >>>> +le32 vqn : 16,
> >>>> +     next_off : 15,
> >>>> +     next_wrap : 1;  
> >>> Don't we want to write this as
> >>>
> >>> le32 vqn : 16;
> >>> le32 next_off :15;
> >>> le32 next_wrap : 1;
> >>>
> >>> ?  
> >> Same thing in C, but would be more confusing IMHO since it will be up to
> >> the reader to figure out which fields comprise the 32 bit integer.
> > It looked weird to me. Other opinions?
> > 
> 
> Regarding the c11 standard the two are equivalent. Thus it does not
> matter to me which notation is used. AFAIK bit-fields are only defined
> in the context of structs (and/or unions), so I assumed that. Putting a
> struct around it would be much better IMHO.

The point in not writing a struct around it is to
make sure people do not think it's portable C.

> I don't agree with Michael's argument about 'which fields comprise the
> 32 bit integer', as IMHO it does not make sense in terms of c11.
> 
> Consider 
> 
> struct A {
> 	uint32_t a:30, b:1, c:2:, d:8;
> };
> 
> I think, in this particular case the notation ain't very helpful in
> figuring out what comprises what. For that reason, if I really need
> to choose, I would side with Connie on this one.
> 
> But there is another, more significant problem IMHO. The guarantees
> provided by the C language (c11) regarding the resulting memory layout
> are not sufficient to reason about it like Michael's comment and
> the bit's of the draft imply. To know the memory layout we need the
> ABI specification for the given platform on top of the C standard.
> 
> So if the bit fields are about in memory layout, I find the stuff
> problematic. If however we use bit-fields only to define how arithmetic
> works, then we are fine.

Well, it's not C really. It's just a C-like notation.

+When documenting sub-byte data fields, C-like bitfield notation
+is used. Fields within an integer are always listed in order,
+from the least significant to the most significant bit.
+
+For example:
+\begin{lstlisting}
+be16 A : 15;
+be16 B : 1;
+\end{lstlisting}
+documents the value A stored in the low 15 bit of a 16 bit
+integer and the value B stored in the high bit of the 16 bit
+integer, the integer in turn using the big-endian byte order.
+
+Note that this notation typically matches the way bitfields are
+packed by C compilers on little-endian architectures but not the
+way bitfields are packed by C compilers on big-endian
+architectures.




> And a side note: A stronger/better reference to the C language could
> benefit clarity. Especially if we start relying on the less trivial
> properties of the C language.
> 
> Hope this helps!
> 
> Regards,
> Halil

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Michael S. Tsirkin]

On Wed, Mar 07, 2018 at 05:14:27PM +0100, Cornelia Huck wrote:
> On Wed, 7 Mar 2018 17:05:24 +0100
> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
> 
> > On 03/07/2018 03:49 PM, Cornelia Huck wrote:
> > >>>> +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
> > >>>> +the driver notifies the device by writing the following
> > >>>> +32-bit value to the Queue Notify address:
> > >>>> +\begin{lstlisting}
> > >>>> +le32 vqn : 16,
> > >>>> +     next_off : 15,
> > >>>> +     next_wrap : 1;    
> > >>> Don't we want to write this as
> > >>>
> > >>> le32 vqn : 16;
> > >>> le32 next_off :15;
> > >>> le32 next_wrap : 1;
> > >>>
> > >>> ?    
> > >> Same thing in C, but would be more confusing IMHO since it will be up to
> > >> the reader to figure out which fields comprise the 32 bit integer.  
> > > It looked weird to me. Other opinions?
> > >   
> > 
> > Regarding the c11 standard the two are equivalent. Thus it does not
> > matter to me which notation is used. AFAIK bit-fields are only defined
> > in the context of structs (and/or unions), so I assumed that. Putting a
> > struct around it would be much better IMHO.
> > 
> > I don't agree with Michael's argument about 'which fields comprise the
> > 32 bit integer', as IMHO it does not make sense in terms of c11.
> > 
> > Consider 
> > 
> > struct A {
> > 	uint32_t a:30, b:1, c:2:, d:8;
> > };
> > 
> > I think, in this particular case the notation ain't very helpful in
> > figuring out what comprises what. For that reason, if I really need
> > to choose, I would side with Connie on this one.
> > 
> > But there is another, more significant problem IMHO. The guarantees
> > provided by the C language (c11) regarding the resulting memory layout
> > are not sufficient to reason about it like Michael's comment and
> > the bit's of the draft imply. To know the memory layout we need the
> > ABI specification for the given platform on top of the C standard.
> > 
> > So if the bit fields are about in memory layout, I find the stuff
> > problematic. If however we use bit-fields only to define how arithmetic
> > works, then we are fine.
> 
> I'm not sure we should go down the C standard rabbit hole. People have
> gotten lost in there.

+1
In particular virtio already uses C-like syntax for structure
without regard to what the C standard says. It's just pseudo-code,
nothing to be hang about.

> If the clarification of what we mean by this notation (patch 1 + the
> update sent later) is not enough,

Halil, can you pls say whether it's enough?

> I'd rather prefer us to add a
> clarifying sentence/diagram/... there. I was mainly bothered by the
> change to the definition in this patch...

Here's what we are trying to say (for example):

	+the value A stored in the low 15 bit of a 16 bit
	+integer and the value B stored in the high bit of the 16 bit
	+integer, the integer in turn using the big-endian byte order.

Thus we can write:

be16 A : 15,
     B : 1;

which maps to:

be -> big endian
16 -> 16 bit integer

A : 15 - low 15 bits
B : 1 - following 1 bit

Why not repeat be16 twice? Well first of all why repeat information
twice? Second this notation lets us list many integer fields
as we might have in the structure:

be16 A : 15,
     B : 1;
be16 C;

And it also ties to the existing notation for full integers.

Any suggestions on how to do it better?
Please provide an example based on the above.

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Halil Pasic]

On 03/07/2018 08:53 PM, Michael S. Tsirkin wrote:
> On Wed, Mar 07, 2018 at 05:14:27PM +0100, Cornelia Huck wrote:
>> On Wed, 7 Mar 2018 17:05:24 +0100
>> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
>>
>>> On 03/07/2018 03:49 PM, Cornelia Huck wrote:
>>>>>>> +When VIRTIO_F_NOTIFICATION_DATA has been negotiated,
>>>>>>> +the driver notifies the device by writing the following
>>>>>>> +32-bit value to the Queue Notify address:
>>>>>>> +\begin{lstlisting}
>>>>>>> +le32 vqn : 16,
>>>>>>> +     next_off : 15,
>>>>>>> +     next_wrap : 1;    
>>>>>> Don't we want to write this as
>>>>>>
>>>>>> le32 vqn : 16;
>>>>>> le32 next_off :15;
>>>>>> le32 next_wrap : 1;
>>>>>>
>>>>>> ?    
>>>>> Same thing in C, but would be more confusing IMHO since it will be up to
>>>>> the reader to figure out which fields comprise the 32 bit integer.  
>>>> It looked weird to me. Other opinions?
>>>>   
>>>

[..]

>>
>> I'm not sure we should go down the C standard rabbit hole. People have
>> gotten lost in there.
> 
> +1
> In particular virtio already uses C-like syntax for structure
> without regard to what the C standard says. It's just pseudo-code,
> nothing to be hang about.

I'm not the one who started talking abut C (I mean the sentence
'Same thing in C...'). Sadly I did get confused into thinking this
is a syntactic construct borrowed form C along with it's semantic.

I've realized my mistake later.

In that light Connie's proposal does not make sense to me, because
with the semantic we defined in 1.4 Structure Specifications what
Connie wrote is something completely different: The whole thing gets
12 byte instead of 4 (assuming that leftovers are permitted and kind
of reserved for future use -- it ain't really specified if be16 a:16:, b:15;
is legit nor what it means).

> 
>> If the clarification of what we mean by this notation (patch 1 + the
>> update sent later) is not enough,
> 
> Halil, can you pls say whether it's enough?
> 

I think the notation is confusing. It looks like C, it compiles
when put in a struct, but has different semantic. You have a note
for that in 1.4 Structure Specifications, and that's great, but looking
at the construct where it's used, it is too easy to jump on the C train.

I think I would prefer a notation less similar to C bit-fields
to avoid confusion.

I think the definition in  '1.4 Structure Specifications' is
kind of OK but I can't say if it is waterproof or not. One thing is
sure however: the definition lacks a formal definition of the (phrase) syntax
and solely relies on the example for that purpose. So without the example,
there is no definition.

Another thing is the semantic of the fields. I guess those are to be
considered unsigned integers of the specified width with the next in
significance relationship on the bits preserved.

>> I'd rather prefer us to add a
>> clarifying sentence/diagram/... there. I was mainly bothered by the
>> change to the definition in this patch...
> 
> Here's what we are trying to say (for example):
> 
> 	+the value A stored in the low 15 bit of a 16 bit
> 	+integer and the value B stored in the high bit of the 16 bit
> 	+integer, the integer in turn using the big-endian byte order.
> 
> Thus we can write:
> 
> be16 A : 15,
>      B : 1;
> 
> which maps to:
> 
> be -> big endian
> 16 -> 16 bit integer
> 
> A : 15 - low 15 bits
> B : 1 - following 1 bit
> 
> Why not repeat be16 twice? Well first of all why repeat information
> twice? Second this notation lets us list many integer fields
> as we might have in the structure:
> 
> be16 A : 15,
>      B : 1;
> be16 C;
> 

Yeah, I understood this in a meanwhile.

> And it also ties to the existing notation for full integers.
> 
> Any suggestions on how to do it better?

I will have to think about this. Especially if we all agree that
the current notation is not good enough.

One stray idea was something like

be32 (A:16;B:15;C:1);

With 
* A occupying bits 0-15
* B occupying bits 16-30
* C occupying bit 30

And bit n of B (n \in [0..15] being the n-16-th bit of the be32
subdivided into fields.

The idea behind () is that it ain't unusual for tuples, and
also the most common grouping semantic is fitting in a sense
that all the fields are together the be32. The separation by
semicolon is to make it obvious that this has nothing to do
with C and that it's not intended to be implemented with C
bit-fields.

> Please provide an example based on the above.
> 


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Michael S. Tsirkin]

On Thu, Mar 08, 2018 at 02:03:31PM +0100, Halil Pasic wrote:
> One stray idea was something like
> 
> be32 (A:16;B:15;C:1);
> 
> With 
> * A occupying bits 0-15
> * B occupying bits 16-30
> * C occupying bit 30
> 
> And bit n of B (n \in [0..15] being the n-16-th bit of the be32
> subdivided into fields.
> 
> The idea behind () is that it ain't unusual for tuples, and
> also the most common grouping semantic is fitting in a sense
> that all the fields are together the be32. The separation by
> semicolon is to make it obvious that this has nothing to do
> with C and that it's not intended to be implemented with C
> bit-fields.

OK let's look at a real life example:

struct desc_event {
	le16 (
	     desc_event_off : 15; /* Descriptor Event Offset */
	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
	);
	le16 (
	      desc_event_flags : 2; /* Descriptor Event Flags */
	      reserved : 14; /* Reserved, set to 0 */
	);
};

As an option (2), I suggest curly brackets which look a bit more
consistent:

struct desc_event {
	le16 {
	     desc_event_off : 15; /* Descriptor Event Offset */
	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
	};
	le16 {
	      desc_event_flags : 2; /* Descriptor Event Flags */
	      reserved : 14; /* Reserved, set to 0 */
	};
};

Cornelia, Halil - any preferences? Ack on one of the above two?

introduction text accordingly (using curly braces, will adopt
accordingly):

When documenting sub-byte data fields, C-like bitfield notation is used.
Fields within an integer are always listed in order, with their lengths,
from the least significant to the most significant bit. The fields
are considered unsigned integers of the specified width with the next in
significance relationship on the bits preserved.

For example:
\begin{lstlisting}
struct S {
	be16 {
	     A : 15;
	     B : 1;
	};
	be16 C;
}
\end{lstlisting}
documents the value A stored in the low 15 bit of a 16 bit
integer and the value B stored in the high bit of the 16 bit
integer, the integer in turn using the big-endian byte order
and being stored at the beginning of the structure S,
and being followed immediately by an unsigned integer C
stored at offset of 2 bytes (16 bits) from the beginning of
the structure.

Note that this notation somewhat resembles the C bitfield syntax but
should not be naively converted to a bitfield notation for portable
code: it matches the way bitfields are packed by C compilers on
little-endian architectures but not the way bitfields are packed by C
compilers on big-endian architectures.

Assuming that CPU_TO_BE16 converts a 16-bit integer from a native
CPU to the big-endian byte order, the following is the equivalent
portable C code to generate a value in this format:
\begin{lstlisting}
CPU_TO_BE16(B << 15 | A)
\end{lstlisting}


-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Cornelia Huck]

On Thu, 8 Mar 2018 18:19:39 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Thu, Mar 08, 2018 at 02:03:31PM +0100, Halil Pasic wrote:
> > One stray idea was something like
> > 
> > be32 (A:16;B:15;C:1);
> > 
> > With 
> > * A occupying bits 0-15
> > * B occupying bits 16-30
> > * C occupying bit 30
> > 
> > And bit n of B (n \in [0..15] being the n-16-th bit of the be32
> > subdivided into fields.
> > 
> > The idea behind () is that it ain't unusual for tuples, and
> > also the most common grouping semantic is fitting in a sense
> > that all the fields are together the be32. The separation by
> > semicolon is to make it obvious that this has nothing to do
> > with C and that it's not intended to be implemented with C
> > bit-fields.  
> 
> OK let's look at a real life example:
> 
> struct desc_event {
> 	le16 (
> 	     desc_event_off : 15; /* Descriptor Event Offset */
> 	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
> 	);
> 	le16 (
> 	      desc_event_flags : 2; /* Descriptor Event Flags */
> 	      reserved : 14; /* Reserved, set to 0 */
> 	);
> };
> 
> As an option (2), I suggest curly brackets which look a bit more
> consistent:
> 
> struct desc_event {
> 	le16 {
> 	     desc_event_off : 15; /* Descriptor Event Offset */
> 	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
> 	};
> 	le16 {
> 	      desc_event_flags : 2; /* Descriptor Event Flags */
> 	      reserved : 14; /* Reserved, set to 0 */
> 	};
> };
> 
> Cornelia, Halil - any preferences? Ack on one of the above two?

OK, both make it more obvious what is going on there. I think I prefer
option 2 (curly braces).

> 
> introduction text accordingly (using curly braces, will adopt
> accordingly):
> 
> When documenting sub-byte data fields, C-like bitfield notation is used.
> Fields within an integer are always listed in order, with their lengths,
> from the least significant to the most significant bit. The fields
> are considered unsigned integers of the specified width with the next in
> significance relationship on the bits preserved.
> 
> For example:
> \begin{lstlisting}
> struct S {
> 	be16 {
> 	     A : 15;
> 	     B : 1;
> 	};
> 	be16 C;
> }
> \end{lstlisting}
> documents the value A stored in the low 15 bit of a 16 bit
> integer and the value B stored in the high bit of the 16 bit
> integer, the integer in turn using the big-endian byte order
> and being stored at the beginning of the structure S,
> and being followed immediately by an unsigned integer C

add "in big-endian byte order"?

> stored at offset of 2 bytes (16 bits) from the beginning of
> the structure.
> 
> Note that this notation somewhat resembles the C bitfield syntax but
> should not be naively converted to a bitfield notation for portable
> code: it matches the way bitfields are packed by C compilers on
> little-endian architectures but not the way bitfields are packed by C
> compilers on big-endian architectures.
> 
> Assuming that CPU_TO_BE16 converts a 16-bit integer from a native
> CPU to the big-endian byte order, the following is the equivalent
> portable C code to generate a value in this format:

"(the first value in the example above)"?

> \begin{lstlisting}
> CPU_TO_BE16(B << 15 | A)
> \end{lstlisting}

Looks sane.

[I'll not be around for further discussions in the next two weeks, but
I'd vote on the issue on OASIS, should you decide to put up a vote.]

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Halil Pasic]

On 03/08/2018 05:19 PM, Michael S. Tsirkin wrote:
> On Thu, Mar 08, 2018 at 02:03:31PM +0100, Halil Pasic wrote:
>> One stray idea was something like
>>
>> be32 (A:16;B:15;C:1);
>>
>> With 
>> * A occupying bits 0-15
>> * B occupying bits 16-30
>> * C occupying bit 30
>>
>> And bit n of B (n \in [0..15] being the n-16-th bit of the be32
>> subdivided into fields.
>>
>> The idea behind () is that it ain't unusual for tuples, and
>> also the most common grouping semantic is fitting in a sense
>> that all the fields are together the be32. The separation by
>> semicolon is to make it obvious that this has nothing to do
>> with C and that it's not intended to be implemented with C
>> bit-fields.
> 
> OK let's look at a real life example:
> 
> struct desc_event {
> 	le16 (
> 	     desc_event_off : 15; /* Descriptor Event Offset */
> 	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
> 	);
> 	le16 (
> 	      desc_event_flags : 2; /* Descriptor Event Flags */
> 	      reserved : 14; /* Reserved, set to 0 */
> 	);
> };
> 
> As an option (2), I suggest curly brackets which look a bit more
> consistent:
> 
> struct desc_event {
> 	le16 {
> 	     desc_event_off : 15; /* Descriptor Event Offset */
> 	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
> 	};> 	le16 {
> 	      desc_event_flags : 2; /* Descriptor Event Flags */
> 	      reserved : 14; /* Reserved, set to 0 */
> 	};
> };
> 
> Cornelia, Halil - any preferences? Ack on one of the above two?
> 

I'm fine with the curly braces as well. Important for me
is, that we are different enough from C. What you propose here
is already good enough for me, but you will find a couple of
ideas below, which could (IMHO) make it even better.

> introduction text accordingly (using curly braces, will adopt
> accordingly):
> 
> When documenting sub-byte data fields, C-like bitfield notation is used.

How about something like:

Some of the fields to be defined in this specification don't
start or don't end on byte boundary. Such fields are  called bit-fields.
A set of bit-fileds is always defined sub-division of an integer typed field.

What I don't like the original sentence is:
* sub-byte: for me it sounds like smaller than a byte, but A is obvoiusly larger
than a byte
* C-like bitfield notation: We just intentionally moved away from
the C bit-field syntax. The term bitfield ain't defined in the context of this
specification -- I can't tell if it's necessary to.



> Fields within an integer are always listed in order, with their lengths,
> from the least significant to the most significant bit. The fields
> are considered unsigned integers of the specified width with the next in
> significance relationship on the bits preserved.
> 
> For example:
> \begin{lstlisting}
> struct S {
> 	be16 {
> 	     A : 15;
> 	     B : 1;
> 	};

I think it may be beneficial to have a name for the complete be16.
Real life example:

le32 {
	vqn : 16;
	next_off : 15;
	next_wrap : 1;
} notification_data;



> 	be16 C;
> }

We could make the example look like this

struct S {
	be16 {
	     A : 15;
	     B : 1;
	} x;
	be16 y;
}

> \end{lstlisting}
> documents the value A stored in the low 15 bit of a 16 bit
> integer and the value B stored in the high bit of the 16 bit
s/of a 16 bit integer/\field{x}/

> integer, the integer in turn using the big-endian byte order
> and being stored at the beginning of the structure S,
> and being followed immediately by an unsigned integer C

s/C/\field{y}/

> stored at offset of 2 bytes (16 bits) from the beginning of
> the structure.
> 
> Note that this notation somewhat resembles the C bitfield syntax but
> should not be naively converted to a bitfield notation for portable
> code: it matches the way bitfields are packed by C compilers on
> little-endian architectures but not the way bitfields are packed by C
> compilers on big-endian architectures.
> 
> Assuming that CPU_TO_BE16 converts a 16-bit integer from a native
> CPU to the big-endian byte order, the following is the equivalent
> portable C code to generate a value in this format:

s/in this format/to be stored into \filed{x}.

> \begin{lstlisting}
> CPU_TO_BE16(B << 15 | A)
> \end{lstlisting}
> 
> 

Thanks for your patience!

Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] Re: [PATCH v9 14/16] VIRTIO_F_NOTIFICATION_DATA: extra data to devices

[Michael S. Tsirkin]

On Fri, Mar 09, 2018 at 01:25:11PM +0100, Halil Pasic wrote:
> 
> 
> On 03/08/2018 05:19 PM, Michael S. Tsirkin wrote:
> > On Thu, Mar 08, 2018 at 02:03:31PM +0100, Halil Pasic wrote:
> >> One stray idea was something like
> >>
> >> be32 (A:16;B:15;C:1);
> >>
> >> With 
> >> * A occupying bits 0-15
> >> * B occupying bits 16-30
> >> * C occupying bit 30
> >>
> >> And bit n of B (n \in [0..15] being the n-16-th bit of the be32
> >> subdivided into fields.
> >>
> >> The idea behind () is that it ain't unusual for tuples, and
> >> also the most common grouping semantic is fitting in a sense
> >> that all the fields are together the be32. The separation by
> >> semicolon is to make it obvious that this has nothing to do
> >> with C and that it's not intended to be implemented with C
> >> bit-fields.
> > 
> > OK let's look at a real life example:
> > 
> > struct desc_event {
> > 	le16 (
> > 	     desc_event_off : 15; /* Descriptor Event Offset */
> > 	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
> > 	);
> > 	le16 (
> > 	      desc_event_flags : 2; /* Descriptor Event Flags */
> > 	      reserved : 14; /* Reserved, set to 0 */
> > 	);
> > };
> > 
> > As an option (2), I suggest curly brackets which look a bit more
> > consistent:
> > 
> > struct desc_event {
> > 	le16 {
> > 	     desc_event_off : 15; /* Descriptor Event Offset */
> > 	     desc_event_wrap : 1; /* Descriptor Event Wrap Counter */
> > 	};> 	le16 {
> > 	      desc_event_flags : 2; /* Descriptor Event Flags */
> > 	      reserved : 14; /* Reserved, set to 0 */
> > 	};
> > };
> > 
> > Cornelia, Halil - any preferences? Ack on one of the above two?
> > 
> 
> I'm fine with the curly braces as well. Important for me
> is, that we are different enough from C. What you propose here
> is already good enough for me, but you will find a couple of
> ideas below, which could (IMHO) make it even better.
> 
> > introduction text accordingly (using curly braces, will adopt
> > accordingly):
> > 
> > When documenting sub-byte data fields, C-like bitfield notation is used.
> 
> How about something like:
> 
> Some of the fields to be defined in this specification don't
> start or don't end on byte boundary. Such fields are  called bit-fields.
> A set of bit-fileds is always defined sub-division of an integer typed field.
> 
> What I don't like the original sentence is:
> * sub-byte: for me it sounds like smaller than a byte, but A is obvoiusly larger
> than a byte
> * C-like bitfield notation: We just intentionally moved away from
> the C bit-field syntax. The term bitfield ain't defined in the context of this
> specification -- I can't tell if it's necessary to.
> 

I like this. Will use.

> 
> > Fields within an integer are always listed in order, with their lengths,
> > from the least significant to the most significant bit. The fields
> > are considered unsigned integers of the specified width with the next in
> > significance relationship on the bits preserved.
> > 
> > For example:
> > \begin{lstlisting}
> > struct S {
> > 	be16 {
> > 	     A : 15;
> > 	     B : 1;
> > 	};
> 
> I think it may be beneficial to have a name for the complete be16.
> Real life example:
> 
> le32 {
> 	vqn : 16;
> 	next_off : 15;
> 	next_wrap : 1;
> } notification_data;

I kind of dislike this. To me "data" adds nothing useful.
"notification"?


> 
> 
> > 	be16 C;
> > }
> 
> We could make the example look like this
> 
> struct S {
> 	be16 {
> 	     A : 15;
> 	     B : 1;
> 	} x;
> 	be16 y;
> }
> > \end{lstlisting}
> > documents the value A stored in the low 15 bit of a 16 bit
> > integer and the value B stored in the high bit of the 16 bit
> s/of a 16 bit integer/\field{x}/
> 
> > integer, the integer in turn using the big-endian byte order
> > and being stored at the beginning of the structure S,
> > and being followed immediately by an unsigned integer C
> 
> s/C/\field{y}/
> 
> > stored at offset of 2 bytes (16 bits) from the beginning of
> > the structure.
> > 
> > Note that this notation somewhat resembles the C bitfield syntax but
> > should not be naively converted to a bitfield notation for portable
> > code: it matches the way bitfields are packed by C compilers on
> > little-endian architectures but not the way bitfields are packed by C
> > compilers on big-endian architectures.
> > 
> > Assuming that CPU_TO_BE16 converts a 16-bit integer from a native
> > CPU to the big-endian byte order, the following is the equivalent
> > portable C code to generate a value in this format:
> 
> s/in this format/to be stored into \filed{x}.
> 
> > \begin{lstlisting}
> > CPU_TO_BE16(B << 15 | A)
> > \end{lstlisting}
> > 
> > 
> 
> Thanks for your patience!
> 
> Halil

Sounds good, thanks for the suggestions!

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 00/16] packed ring layout spec

[Michael S. Tsirkin]

On Thu, Mar 01, 2018 at 01:31:18AM +0200, Michael S. Tsirkin wrote:
> This addresses comments on v8.
> 
> Thanks a lot to all reviewers of v8 - I hope we are
> finally there or almost there.

FYI the last round of comments was mostly about formatting,
so I plan to post v10 and start voting immediately.
The one exception is VIRTIO_F_NOTIFICATION_DATA where
there were comments about endian-ness on s390.

I will post that too but we'll wait until Cornelia
acks before we vote. That might take a couple more weeks.

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [PATCH v9 08/16] split-ring: generalize text

[Michael S. Tsirkin]

On Wed, Mar 07, 2018 at 12:00:48PM +0100, Cornelia Huck wrote:
> On Thu, 1 Mar 2018 01:31:33 +0200
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > Update generic text to talk about available/used buffers, not rings.
> > Move some split-ring specific text to the correct section.
> > 
> > Update conformance section with link to the new conformance clause.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  conformance.tex |  1 +
> >  content.tex     | 10 ++++------
> >  split-ring.tex  |  4 ++++
> >  3 files changed, 9 insertions(+), 6 deletions(-)
> > 
> > diff --git a/conformance.tex b/conformance.tex
> > index 55d17b4..e4efe33 100644
> > --- a/conformance.tex
> > +++ b/conformance.tex
> > @@ -38,6 +38,7 @@ A driver MUST conform to the following normative statements:
> >  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
> >  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
> >  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
> > +\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
> >  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
> >  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
> >  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
> > diff --git a/content.tex b/content.tex
> > index 5634c7d..e1e30a0 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -381,12 +381,10 @@ of a device are live once the device has been reset.
> >  
> >  \drivernormative{\subsection}{Device Cleanup}{General Initialization And Device Operation / Device Cleanup}
> >  
> > -A driver MUST NOT alter descriptor table entries which have been
> > -exposed in the available ring (and not marked consumed by the device
> > -in the used ring) of a live virtqueue.
> > -
> > -A driver MUST NOT decrement the available \field{idx} on a live virtqueue (ie.
> > -there is no way to ``unexpose'' buffers).
> > +A driver MUST NOT alter virtqueue entries for exposed buffers -
> > +i.e. buffers which have been
> > +made available to the device (and not been used by the device)
> > +of a live virtqueue.
> >  
> >  Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
> >  
> > diff --git a/split-ring.tex b/split-ring.tex
> > index 9601a53..a594d41 100644
> > --- a/split-ring.tex
> > +++ b/split-ring.tex
> > @@ -296,6 +296,10 @@ referred to this structure as vring_avail, and the constant as
> >  VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
> >  \end{note}
> >  
> > +\drivernormative{\subsubsection}{The Virtqueue Available Ring}{Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
> > +A driver MUST NOT decrement the available \field{idx} on a virtqueue (ie.
> 
> s/ie./i.e./
> 
> as you're touching it anyway.

I forgot to address this. Sorry. Let's do it as a patch on top.

> 
> > +there is no way to ``unexpose'' buffers).
> > +
> >  \subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
> >  
> >  If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php