[virtio] [PATCH 0/6] rework notifications terminology

[virtio] [PATCH 0/6] rework notifications terminology

[Halil Pasic]

Abstract 
--------
The terminology around notifications is currently somewhat
fragmented. Unify notifications terminology using available and used
buffer notification consistently for the virtqueue notifications.
Consistent use of the term configuration change notification is of
concern.

Changelog
---------

RFC --> v1:
* Devices and the transports PCI and MMIO were omitted.
Now the rework should be complete.
* Addressed review comments and minor changes for the first
three patches. See individual changelogs. 

Halil Pasic (6):
  notifications: unify notifications wording in core
  notifications: notifications as basic virtio facility
  ccw: map common notifications terminology to ccw
  pci: map common notifications terminology to PCI
  mmio: map common notifications terminology to MMIO
  notifications: update notifications terminology for devices

 cl-os.tex       |    2 +-
 conformance.tex |   10 ++--
 content.tex     |  163 ++++++++++++++++++++++++++++++++++++++++++-------------
 packed-ring.tex |   59 +++++++++++---------
 split-ring.tex  |   72 ++++++++++++++----------
 5 files changed, 205 insertions(+), 101 deletions(-)


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH 1/6] notifications: unify notifications wording in core

[Halil Pasic]

Let us unify the wording when talking about notifications. This change
establishes the terms available buffer notification for what was usually
simply called notification or virtqueue notification in v1.0 and used
buffer notification for what was usually called interrupt.

The term configuration change notification in kept where called so and
consolidated where it's called configuration change interrupt or
similar.

The changes done here are limited to the core part, and don't
conceptually involve neither the transports nor the devices (references
are updated though). Future changes should address these parts.

Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
---
Changelog:

RFC -> v1:
* virtq_disable_notifications() -> virtq_disable_used_buffer_notifications() (thanks Stefan)
* Changed the legacy features part, as it was forgotten.
---
 cl-os.tex       |    2 +-
 conformance.tex |    8 +++---
 content.tex     |   26 +++++++++++--------
 packed-ring.tex |   59 +++++++++++++++++++++++++--------------------
 split-ring.tex  |   72 ++++++++++++++++++++++++++++++++-----------------------
 5 files changed, 95 insertions(+), 72 deletions(-)

diff --git a/cl-os.tex b/cl-os.tex
index da78bf8..fc12cdd 100644
--- a/cl-os.tex
+++ b/cl-os.tex
@@ -2,7 +2,7 @@
 trivial typo
 
 See
-\ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+\ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
 } \\
 \hline
 541 & 11 Oct 2015 & Paolo Bonzini & {virtio-blk: fix typo
diff --git a/conformance.tex b/conformance.tex
index e4efe33..de33396 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -37,10 +37,10 @@ A driver MUST conform to the following normative statements:
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Message Framing}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Available Buffer Notification Suppression}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Updating idx}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Virtqueues / Supplying Buffers to The Device / Notifying The Device}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization}
@@ -158,9 +158,9 @@ A device MUST conform to the following normative statements:
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Message Framing}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}
-\item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+\item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
-\item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Available Buffer Notification Suppression}
 \item \ref{devicenormative:Reserved Feature Bits}
 \end{itemize}
 
diff --git a/content.tex b/content.tex
index 7a92cb1..2e103cf 100644
--- a/content.tex
+++ b/content.tex
@@ -72,7 +72,8 @@ recover by issuing a reset.
 \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field}
 The device MUST initialize \field{device status} to 0 upon reset.
 
-The device MUST NOT consume buffers or notify the driver before DRIVER_OK.
+The device MUST NOT consume buffers or send any used buffer
+notifications to the driver before DRIVER_OK.
 
 \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state
 that a reset is needed.  If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device
@@ -235,12 +236,13 @@ transmit and one for receive.}.
 Driver makes requests available to device by adding
 an available buffer to the queue - i.e. adding a buffer
 describing the request to a virtqueue, and optionally triggering
-a driver event - i.e. sending a notification to the device.
+a driver event - i.e. sending an available buffer notification
+to the device.
 
 Device executes the requests and - when complete - adds
 a used buffer to the queue - i.e. lets the driver
 know by marking the buffer as used. Device can then trigger
-a device event - i.e. send an interrupt to the driver.
+a device event - i.e. send an used buffer notification to the driver.
 
 Device reports the number of bytes it has written to memory for
 each buffer it uses. This is referred to as ``used length''.
@@ -330,7 +332,8 @@ set the FAILED status bit to indicate that it has given up on the
 device (it can reset the device later to restart if desired).  The
 driver MUST NOT continue initialization in that case.
 
-The driver MUST NOT notify the device before setting DRIVER_OK.
+The driver MUST NOT send any buffer available notifications to
+the device before setting DRIVER_OK.
 
 \subsection{Legacy Interface: Device Initialization}\label{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
 Legacy devices did not support the FEATURES_OK status bit, and thus did
@@ -388,10 +391,11 @@ reads unless notified.
 
 \subsection{Notification of Device Configuration Changes}\label{sec:General Initialization And Device Operation / Device Operation / Notification of Device Configuration Changes}
 
-For devices where the device-specific configuration information can be changed, an
-interrupt is delivered when a device-specific configuration change occurs.
+For devices where the device-specific configuration information can be
+changed, a notification is delivered when a device-specific configuration
+change occurs.
 
-In addition, this interrupt is triggered by the device setting
+In addition, this notification is triggered by the device setting
 DEVICE_NEEDS_RESET (see \ref{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}).
 
 \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Device Cleanup}
@@ -5331,7 +5335,7 @@ Virtqueues / The Virtqueue Descriptor Table / Indirect
 Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}~\nameref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Support}.
   \item[VIRTIO_F_RING_EVENT_IDX(29)] This feature enables the \field{used_event}
   and the \field{avail_event} fields as described in
-\ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}, \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring} and \ref{sec:Packed Virtqueues / Driver and Device Event Suppression}.
+\ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring} and \ref{sec:Packed Virtqueues / Driver and Device Event Suppression}.
 
 
   \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
@@ -5382,16 +5386,16 @@ Transitional devices MAY offer the following:
 \begin{description}
 \item[VIRTIO_F_NOTIFY_ON_EMPTY (24)] If this feature
   has been negotiated by driver, the device MUST issue
-  an interrupt if the device runs
+  an used buffer notification if the device runs
   out of available descriptors on a virtqueue, even though
-  interrupts are suppressed using the VIRTQ_AVAIL_F_NO_INTERRUPT
+  notifications are suppressed using the VIRTQ_AVAIL_F_NO_INTERRUPT
   flag or the \field{used_event} field.
 \begin{note}
   An example of a driver using this feature is the legacy
   networking driver: it doesn't need to know every time a packet
   is transmitted, but it does need to free the transmitted
   packets a finite time after they are transmitted. It can avoid
-  using a timer if the device interrupts it when all the packets
+  using a timer if the device notifies it when all the packets
   are transmitted.
 \end{note}
 \end{description}
diff --git a/packed-ring.tex b/packed-ring.tex
index 2ef9559..95a9a35 100644
--- a/packed-ring.tex
+++ b/packed-ring.tex
@@ -43,7 +43,7 @@ The driver then notifies the device. When the device has finished
 processing the buffer, it writes a used device descriptor
 including the Buffer ID into the Descriptor Ring (overwriting a
 driver descriptor previously made available), and sends an
-interrupt.
+used event notification.
 
 The Descriptor Ring is used in a circular manner: the driver writes
 descriptors into the ring in order. After reaching the end of the ring,
@@ -65,11 +65,12 @@ in which their processing is complete.
 
 The Device Event Suppression data structure is write-only by the
 device. It includes information for reducing the number of
-device events - i.e. driver notifications to device.
+device events - i.e. sending an available buffer notification
+to the device.
 
 The Driver Event Suppression data structure is read-only by the
 device. It includes information for reducing the number of
-driver events - i.e. device interrupts to driver.
+driver events - i.e. send an used buffer notification to the driver.
 
 \subsection{Driver and Device Ring Wrap Counters}
 \label{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}
@@ -309,22 +310,20 @@ in the ring.
 
 \subsection{Driver and Device Event Suppression}
 \label{sec:Packed Virtqueues / Driver and Device Event Suppression}
-In many systems driver and device notifications involve
+In many systems used and available buffer  notifications involve
 significant overhead. To mitigate this overhead,
 each virtqueue includes two identical structures used for
 controlling notifications between the device and the driver.
 
 The Driver Event Suppression structure is read-only by the
-device and controls the events sent by the device
-to the driver (e.g. interrupts).
+device and controls the used buffer notifications (sent by the device
+to the driver).
 
 The Device Event Suppression structure is read-only by
-the driver and controls the events sent by the driver
-to the device (e.g. IO).
+the driver and controls the available buffer notifications (sent by the
+driver to the device).
 
-Each of these Event Suppression structures controls
-both Descriptor Ring events and structure events, and
-each includes the following fields:
+Each of these Event Suppression includes the following fields:
 
 \begin{description}
 \item [Descriptor Ring Change Event Flags] Takes values:
@@ -352,9 +351,9 @@ matches this value and a descriptor is
 made available/used respectively.
 \end{description}
 
-After writing out some descriptors, both the device and the driver
+After writing out some descriptors,the device/driver
 are expected to consult the relevant structure to find out
-whether an interrupt/notification should be sent.
+whether an used/available buffer notification should be sent.
 
 \subsubsection{Structure Size and Alignment}
 \label{sec:Packed Virtqueues / Structure Size and Alignment}
@@ -463,7 +462,7 @@ When notifying the device, driver MUST set
 \field{next_off} and
 \field{next_wrap} to match the next descriptor
 not yet made available to the device.
-A driver MAY send multiple notifications without making
+A driver MAY send multiple available buffer notifications without making
 any new descriptors available to the device.
 
 \drivernormative{\subsection}{Scatter-Gather Support}{Basic Facilities of a
@@ -579,13 +578,14 @@ The driver MUST perform a suitable memory barrier before the
 \field{flags} update, to ensure the
 device sees the most up-to-date copy.
 
-\subsubsection{Notifying The Device}\label{sec:Basic Facilities
-of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+\subsubsection{Sending Available Buffer Notifications}\label{sec:Basic Facilities
+of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device
+/ Sending Available Buffer Notifications}
 
 The actual method of device notification is bus-specific, but generally
 it can be expensive.  So the device MAY suppress such notifications if it
-doesn't need them, using the Driver Event Suppression structure
-as detailed in section \ref{sec:Basic
+doesn't need them, using the Event Suppression structure comprising the
+Device Area as detailed in section \ref{sec:Basic
 Facilities of a Virtio Device / Packed Virtqueues / Event
 Suppression Structure Format}.
 
@@ -595,7 +595,7 @@ value before checking if notifications are suppressed.
 \subsubsection{Implementation Example}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Implementation Example}
 
 Below is a driver code example. It does not attempt to reduce
-the number of device interrupts, neither does it support
+the number of available buffer notifications, neither does it support
 the VIRTIO_F_RING_EVENT_IDX feature.
 
 \begin{lstlisting}
@@ -651,24 +651,31 @@ if (vq->device_event.flags != RING_EVENT_FLAGS_DISABLE) {
 \end{lstlisting}
 
 
-\drivernormative{\paragraph}{Notifying The Device}{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying Buffers to The Device / Notifying The Device}
+\drivernormative{\paragraph}{Sending Available Buffer Notifications}
+{Basic Facilities of a Virtio Device / Packed Virtqueues / Supplying
+Buffers to The Device / Sending Available Buffer Notifications}
 The driver MUST perform a suitable memory barrier before reading
-the Driver Event Suppression structure, to avoid missing a notification.
+the Event Suppression structure occupying the Device Area. Failing
+to do so could result in mandatory available buffer notifications
+not being sent.
 
 \subsection{Receiving Used Buffers From The Device}\label{sec:Basic Facilities of a Virtio Device / Packed Virtqueues / Receiving Used Buffers From The Device}
 
 Once the device has used buffers referred to by a descriptor (read from or written to them, or
 parts of both, depending on the nature of the virtqueue and the
-device), it interrupts the driver
+device), it sends an used buffer notification to the driver
 as detailed in section \ref{sec:Basic
 Facilities of a Virtio Device / Packed Virtqueues / Event
 Suppression Structure Format}.
 
 \begin{note}
-For optimal performance, a driver MAY disable interrupts while processing
-the used buffers, but beware the problem of missing interrupts between
-emptying the ring and reenabling interrupts.  This is usually handled by
-re-checking for more used buffers after interrups are re-enabled:
+
+For optimal performance, a driver MAY disable used buffer notifications
+while processing the used buffers, but beware the problem of missing
+notifications between emptying the ring and reenabling used buffer
+notifications.  This is usually handled by re-checking for more used
+buffers after notifications are re-enabled:
+
 \end{note}
 
 \begin{lstlisting}
diff --git a/split-ring.tex b/split-ring.tex
index df278fe..d2f9a07 100644
--- a/split-ring.tex
+++ b/split-ring.tex
@@ -54,7 +54,8 @@ When the driver wants to send a buffer to the device, it fills in
 a slot in the descriptor table (or chains several together), and
 writes the descriptor index into the available ring.  It then
 notifies the device. When the device has finished a buffer, it
-writes the descriptor index into the used ring, and sends an interrupt.
+writes the descriptor index into the used ring, and sends an
+used buffer notification.
 
 \drivernormative{\subsection}{Virtqueues}{Basic Facilities of a Virtio Device / Virtqueues}
 The driver MUST ensure that the physical address of the first byte
@@ -318,44 +319,47 @@ VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.
 A driver MUST NOT decrement the available \field{idx} on a virtqueue (ie.
 there is no way to ``unexpose'' buffers).
 
-\subsection{Virtqueue Interrupt Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+\subsection{Used Buffer Notification Suppression}\label{sec:Basic
+Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}
 
 If the VIRTIO_F_EVENT_IDX feature bit is not negotiated,
 the \field{flags} field in the available ring offers a crude mechanism for the driver to inform
-the device that it doesn't want interrupts when buffers are used.  Otherwise
+the device that it doesn't want notifications when buffers are used.  Otherwise
 \field{used_event} is a more performant alternative where the driver
-specifies how far the device can progress before interrupting.
+specifies how far the device can progress before a notification is
+required.
 
-Neither of these interrupt suppression methods are reliable, as they
+Neither of these notification suppression methods are reliable, as they
 are not synchronized with the device, but they serve as
 useful optimizations.
 
-\drivernormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+\drivernormative{\subsubsection}{Used Buffer Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}
 If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
 \begin{itemize}
 \item The driver MUST set \field{flags} to 0 or 1.
 \item The driver MAY set \field{flags} to 1 to advise
-the device that interrupts are not needed.
+the device that notifications are not needed.
 \end{itemize}
 
 Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
 \begin{itemize}
 \item The driver MUST set \field{flags} to 0.
-\item The driver MAY use \field{used_event} to advise the device that interrupts are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
+\item The driver MAY use \field{used_event} to advise the device that
+notifications are unnecessary until the device writes entry with an index specified by \field{used_event} into the used ring (equivalently, until \field{idx} in the
 used ring will reach the value \field{used_event} + 1).
 \end{itemize}
 
-The driver MUST handle spurious interrupts from the device.
+The driver MUST handle spurious notifications from the device.
 
-\devicenormative{\subsubsection}{Virtqueue Interrupt Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}
+\devicenormative{\subsubsection}{Used Buffer Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}
 
 If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
 \begin{itemize}
 \item The device MUST ignore the \field{used_event} value.
 \item After the device writes a descriptor index into the used ring:
   \begin{itemize}
-  \item If \field{flags} is 1, the device SHOULD NOT send an interrupt.
-  \item If \field{flags} is 0, the device MUST send an interrupt.
+  \item If \field{flags} is 1, the device SHOULD NOT send a notification.
+  \item If \field{flags} is 0, the device MUST send a notification.
   \end{itemize}
 \end{itemize}
 
@@ -366,14 +370,16 @@ Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
   \begin{itemize}
   \item If the \field{idx} field in the used ring (which determined
     where that descriptor index was placed) was equal to
-    \field{used_event}, the device MUST send an interrupt.
-  \item Otherwise the device SHOULD NOT send an interrupt.
+    \field{used_event}, the device MUST send a notification.
+  \item Otherwise the device SHOULD NOT send a notification.
   \end{itemize}
 \end{itemize}
 
 \begin{note}
 For example, if \field{used_event} is 0, then a device using
-  VIRTIO_F_EVENT_IDX would interrupt after the first buffer is
+
+  VIRTIO_F_EVENT_IDX would send an used buffer notification
+  to the driver after the first buffer is
   used (and again after the 65536th buffer, etc).
 \end{note}
 
@@ -460,14 +466,16 @@ that uninitialized memory has been overwritten when it has not.
 The driver MUST NOT make assumptions about data in device-writable buffers
 beyond the first \field{len} bytes, and SHOULD ignore this data.
 
-\subsection{Virtqueue Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\subsection{Available Buffer Notification Suppression}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
 
-The device can suppress notifications in a manner analogous to the way
-drivers can suppress interrupts as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+The device can suppress available buffer notifications in a manner
+analogous to the way drivers can suppress used buffer notifications as
+detailed in section \ref{sec:Basic Facilities of a Virtio Device /
+Virtqueues / Used Buffer Notification Suppression}.
 The device manipulates \field{flags} or \field{avail_event} in the used ring the
 same way the driver manipulates \field{flags} or \field{used_event} in the available ring.
 
-\drivernormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\drivernormative{\subsubsection}{Available Buffer Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Available Buffer Notification Suppression}
 
 The driver MUST initialize \field{flags} in the used ring to 0 when
 allocating the used ring.
@@ -494,7 +502,7 @@ Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
   \end{itemize}
 \end{itemize}
 
-\devicenormative{\subsubsection}{Virtqueue Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Notification Suppression}
+\devicenormative{\subsubsection}{Available Buffer Notification Suppression}{Basic Facilities of a Virtio Device / Virtqueues / Available Buffer Notification Suppression}
 If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
 \begin{itemize}
 \item The device MUST set \field{flags} to 0 or 1.
@@ -562,8 +570,8 @@ The driver offers buffers to one of the device's virtqueues as follows:
 \item The driver performs a suitable memory barrier to ensure that it updates
   the \field{idx} field before checking for notification suppression.
 
-\item If notifications are not suppressed, the driver notifies the device
-    of the new available buffers.
+\item The driver sends an available buffers notification to the device if
+    such notifications are not suppressed.
 \end{enumerate}
 
 Note that the above code does not take precautions against the
@@ -659,26 +667,30 @@ The driver MUST perform a suitable memory barrier before reading \field{flags} o
 
 Once the device has used buffers referred to by a descriptor (read from or written to them, or
 parts of both, depending on the nature of the virtqueue and the
-device), it interrupts the driver as detailed in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Interrupt Suppression}.
+device), it sends an used buffer notification to the driver as detailed
+in section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
+Used Buffer Notification Suppression}.
 
 \begin{note}
-For optimal performance, a driver MAY disable interrupts while processing
-the used ring, but beware the problem of missing interrupts between
-emptying the ring and reenabling interrupts.  This is usually handled by
-re-checking for more used buffers after interrups are re-enabled:
+
+For optimal performance, a driver MAY disable used buffer notifications
+while processing the used ring, but beware the problem of missing
+notifications between emptying the ring and reenabling notifications.  This
+is usually handled by re-checking for more used buffers after
+notifications are re-enabled:
 
 \begin{lstlisting}
-virtq_disable_interrupts(vq);
+virtq_disable_used_buffer_notifications(vq);
 
 for (;;) {
         if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) {
-                virtq_enable_interrupts(vq);
+                virtq_enable_used_buffer_notifications(vq);
                 mb();
 
                 if (vq->last_seen_used != le16_to_cpu(virtq->used.idx))
                         break;
 
-                virtq_disable_interrupts(vq);
+                virtq_disable_used_buffer_notifications(vq);
         }
 
         struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz];
-- 
1.7.1


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH 2/6] notifications: notifications as basic virtio facility

[Halil Pasic]

Let's introduce notifications as basic virtio facility early on. This
shall not only increase the cohesion between core and transport
description by having a well-defined  place where notifications are
introduced, but also give us the opportunity to explain some
discrepancies.

Namely notifications sent by the device to the driver were often called
interrupts prior to v1.1. Getting completely rid of that terminology is
however not viable in case of some names.

Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
---
Changelog:
RFC -> v1:
* implemented Connies suggestions (thanks Connie)
---
 content.tex |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/content.tex b/content.tex
index 2e103cf..d7e2b18 100644
--- a/content.tex
+++ b/content.tex
@@ -8,6 +8,7 @@ device consists of the following parts:
 \begin{itemize}
 \item Device status field
 \item Feature bits
+\item Notifications
 \item Device Configuration space
 \item One or more virtqueues
 \end{itemize}
@@ -150,6 +151,41 @@ requirements documented within these legacy interface sections.
 Specification text within these sections generally does not apply
 to non-transitional devices.
 
+\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
+/ Notifications}
+
+The notion of sending a notification (driver to device or device
+to driver) plays an important role in this specification. The
+modus operandi of the notifications is transport specific.
+
+There are three types of notifications: 
+\begin{itemize}
+\item device configuration space notification
+\item available buffers notification
+\item used buffer notifications. 
+\end{itemize}
+
+Device configuration space notifications and used buffer notifications
+are sent by the device, the recipient is the driver. A device
+configuration space notification indicates that a configuration space
+has changed; a used buffer notification indicates that a buffer may
+have been made used on the virtqueue designated by the notification.
+
+Available buffer notifications are sent by the driver, the recipient is
+the device. This type of notification indicates that a buffer may have
+been made available on the virtqueue designated by the notification.
+
+The semantic, the transport-specific implementations and other
+important aspects of the different notifications are specified in detail
+in the following chapters
+
+Most transports implement notifications sent by the device to the
+driver using interrupts. Therefore, in previous versions of this
+specification, these notifications were often called interrupts.
+Some names defined in this specification still retain this interrupt
+terminology. Occasionally, the term event is used to refer to
+a notification or a receipt of a notification.
+
 \section{Device Configuration Space}\label{sec:Basic Facilities of a Virtio Device / Device Configuration Space}
 
 Device configuration space is generally used for rarely-changing or
-- 
1.7.1


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH 3/6] ccw: map common notifications terminology to ccw

[Halil Pasic]

The various notifications are introduced and specified in the common
(i.e. transport agnostic) portion of this specification. How
notifications are realised for a given transport is something each
transport has to specify.

Let's make the relationship between the virtio-ccw terms and the common
terms more obvious.

Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
---
Changelog:
* implemented Connie's suggestions (typos, grammar)
* avoided 'realized using virtual interrupts'
---
 content.tex |   42 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 42 insertions(+), 0 deletions(-)

diff --git a/content.tex b/content.tex
index d7e2b18..e5b2499 100644
--- a/content.tex
+++ b/content.tex
@@ -1943,6 +1943,18 @@ Bytes & Description & Contents \\
 \hline
 \end{tabular}
 
+A virtio-ccw proxy device facilitates:
+\begin{itemize} 
+\item Discovery and attachment of  virtio devices (as described above).
+\item Initialization of virtqueues and transport specific facilities (using
+      virtio-specific channel commands).
+\item Notifications (via hypercall and a combination of I/O interrupts
+      and indicator bits).
+\end{itemize} 
+
+\subsubsection{Channel Commands for Virtio}\label{sec:Virtio Transport Options / Virtio
+over channel I/O / Basic Concepts/ Channel Commands for Virtio}
+
 In addition to the basic channel commands, virtio-ccw defines a
 set of channel commands related to configuration and operation of
 virtio:
@@ -1963,6 +1975,36 @@ virtio:
 #define CCW_CMD_READ_STATUS 0x72
 \end{lstlisting}
 
+\subsubsection{Notifications}\label{sec:Virtio Transport Options / Virtio
+over channel I/O / Basic Concepts/ Notifications}
+
+Available buffer notifications are realized as a hypercall. No additional
+setup by the driver is needed. The operation of available buffer
+notifications is described in section \ref{sec:Virtio Transport Options /
+Virtio over channel I/O / Device Operation / Guest->Host Notification}.
+
+Used buffer notifications are realized either as so-called classic or
+adapter I/O interrupts depending on a transport level negotiation. The
+initialization is described in sections \ref{sec:Virtio Transport Options
+/ Virtio over channel I/O / Device Initialization / Setting Up Indicators
+/ Setting Up Classic Queue Indicators} and \ref{sec:Virtio Transport
+Options / Virtio over channel I/O / Device Initialization / Setting Up
+Indicators / Setting Up Two-Stage Queue Indicators} respectively.  The
+operation of each flavor is described in sections \ref{sec:Virtio
+Transport Options / Virtio over channel I/O / Device Operation /
+Host->Guest Notification / Notification via Classic I/O Interrupts} and
+\ref{sec:Virtio Transport Options / Virtio over channel I/O / Device
+Operation / Host->Guest Notification / Notification via Adapter I/O
+Interrupts} respectively. 
+
+Configuration change notifications are done using so-called classic I/O
+interrupts. The initialization is described in section \ref{sec:Virtio
+Transport Options / Virtio over channel I/O / Device Initialization /
+Setting Up Indicators / Setting Up Configuration Change Indicators} and
+the operation in section \ref{sec:Virtio Transport Options / Virtio over
+channel I/O / Device Operation / Host->Guest Notification / Notification
+via Classic I/O Interrupts}.
+
 \devicenormative{\subsubsection}{Basic Concepts}{Virtio Transport Options / Virtio over channel I/O / Basic Concepts}
 
 The virtio-ccw device acts like a normal channel device, as specified
-- 
1.7.1


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH 4/6] pci: map common notifications terminology to PCI

[Halil Pasic]

The various notifications are introduced and specified in the common
(i.e. transport agnostic) portion of this specification. How
notifications are realised for a given transport is something each
transport has to specify.

Let's make the relationship between the PCI terms and the common
terms more obvious.

Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
---
 conformance.tex |    2 +-
 content.tex     |   13 +++++++------
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index de33396..29969f1 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -180,7 +180,7 @@ A PCI device MUST conform to the following normative statements:
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Non-transitional Device With Legacy Driver}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
-\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Used Buffer Notifications}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes}
 \end{itemize}
 
diff --git a/content.tex b/content.tex
index e5b2499..ffd0c94 100644
--- a/content.tex
+++ b/content.tex
@@ -941,7 +941,7 @@ causes the device to de-assert the interrupt.
 In this way, driver read of ISR status causes the device to de-assert
 an interrupt.
 
-See sections \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device} and \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes} for how this is used.
+See sections \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Used Buffer Notifications} and \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes} for how this is used.
 
 \devicenormative{\paragraph}{ISR status capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
 
@@ -1306,14 +1306,15 @@ enough to ensure that the separate parts of the virtqueue are on
 separate cache lines.
 }.  There was no mechanism to negotiate the queue size.
 
-\subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notifying The Device}
+\subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}
 
-The driver notifies the device by writing the 16-bit virtqueue index
+The driver sends an available buffer notification the device by writing
+the 16-bit virtqueue index
 of this virtqueue to the Queue Notify address.  See \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} for how to calculate this address.
 
-\subsubsection{Virtqueue Interrupts From The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device}
+\subsubsection{Used Buffer Notifications}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Used Buffer Notifications}
 
-If an interrupt is necessary for a virtqueue, the device would typically act as follows:
+If an used buffer notification is necessary for a virtqueue, the device would typically act as follows:
 
 \begin{itemize}
   \item If MSI-X capability is disabled:
@@ -1332,7 +1333,7 @@ If an interrupt is necessary for a virtqueue, the device would typically act as
     \end{enumerate}
 \end{itemize}
 
-\devicenormative{\paragraph}{Virtqueue Interrupts From The Device}{Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device}
+\devicenormative{\paragraph}{Used Buffer Notifications}{Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Used Buffer Notifications}
 
 If MSI-X capability is enabled and \field{queue_msix_vector} is
 NO_VECTOR for a virtqueue, the device MUST NOT deliver an interrupt
-- 
1.7.1


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH 5/6] mmio: map common notifications terminology to MMIO

[Halil Pasic]

The various notifications are introduced and specified in the common
(i.e. transport agnostic) portion of this specification. How
notifications are realised for a given transport is something each
transport has to specify.

Let's make the relationship between the virtio over MIIO terms and the
common terms more obvious.

Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
---
 content.tex |   26 ++++++++++++++------------
 1 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/content.tex b/content.tex
index ffd0c94..ed34495 100644
--- a/content.tex
+++ b/content.tex
@@ -1550,10 +1550,10 @@ All register values are organized as Little Endian.
     caused the device interrupt to be asserted.
     The following events are possible:
     \begin{description}
-      \item[Used Buffer Update] - bit 0 - the interrupt was asserted
+      \item[Used Buffer Notification] - bit 0 - the interrupt was asserted
         because the device has used a buffer
         in at least one of the active virtual queues.
-      \item [Configuration Change] - bit 1 - the interrupt was
+      \item [Configuration Change Notification] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
     \end{description}
   }
@@ -1716,26 +1716,28 @@ The driver will typically initialize the virtual queue in the following way:
 \item Write 0x1 to \field{QueueReady}.
 \end{enumerate}
 
-\subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifying The Device}
+\subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Available Buffer Notifications}
 
-The driver notifies the device about new buffers being available in
-a queue by writing the index of the updated queue to \field{QueueNotify}.
+The driver sends an available buffer notification to the device a by
+writing the index of the queue to \field{QueueNotify} to be notified.
 
 \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 
 The memory mapped virtio device is using a single, dedicated
 interrupt signal, which is asserted when at least one of the
 bits described in the description of \field{InterruptStatus}
-is set. This is how the device notifies the
-driver about a new used buffer being available in the queue
-or about a change in the device configuration.
+is set. This is how the device sends an used buffer notification
+or a configuration change notifaciton to the device.
 
 \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 After receiving an interrupt, the driver MUST read
-\field{InterruptStatus} to check what caused the interrupt
-(see the register description). After the interrupt is handled,
-the driver MUST acknowledge it by writing a bit mask
-corresponding to the handled events to the InterruptACK register.
+
+\field{InterruptStatus} to check what caused the interrupt (see the
+register description).  The used buffer notification bit being set,
+SHOULD be interpreted as a used buffer notification for each active
+virtqueue.  After the interrupt is handled, the driver MUST acknowledge
+it by writing a bit mask corresponding to the handled events to the
+InterruptACK register.
 
 \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface}
 
-- 
1.7.1


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] [PATCH 6/6] notifications: update notifications terminology for devices

[Halil Pasic]

The specifications of some virtio device types are still using the old
terminology for used buffer notifications and configuration change
notifications calling these interrupts.

Let us fix that.

Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
---
 content.tex |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/content.tex b/content.tex
index ed34495..83bcd3d 100644
--- a/content.tex
+++ b/content.tex
@@ -3044,9 +3044,9 @@ if VIRTIO_NET_F_MRG_RXBUF is not negotiated.}
 \label{sec:Device Types / Network Device / Device Operation / Processing of Packets}%old label for latexdiff
 
 When a packet is copied into a buffer in the receiveq, the
-optimal path is to disable further interrupts for the receiveq
-and process
-packets until no more are found, then re-enable them.
+optimal path is to disable further used buffer notifications for the
+receiveq and process packets until no more are found, then re-enable
+them.
 
 Processing incoming packets involves:
 
@@ -3886,7 +3886,7 @@ The device SHOULD clear the \field{write_zeroes_may_unmap} field of the
 virtio configuration space if and only if a write zeroes request cannot
 result in deallocating one or more sectors.  The device MAY change the
 content of the field during operation of the device; when this happens,
-the device SHOULD trigger a configuration change interrupt.
+the device SHOULD trigger a configuration change notification.
 
 A write is considered volatile when it is submitted; the contents of
 sectors covered by a volatile write are undefined in persistent device
@@ -4178,8 +4178,8 @@ an appropriate log or output method.
 \begin{enumerate}
 \item For output, a buffer containing the characters is placed in
   the port's transmitq\footnote{Because this is high importance and low bandwidth, the current
-Linux implementation polls for the buffer to be used, rather than
-waiting for an interrupt, simplifying the implementation
+Linux implementation polls for the buffer to become used, rather than
+waiting for an used buffer notifications, simplifying the implementation
 significantly. However, for generic serial ports with the
 O_NONBLOCK flag set, the polling limitation is relaxed and the
 consumed buffers are freed upon the next write or poll call or
@@ -4187,11 +4187,11 @@ when a port is closed or hot-unplugged.
 }.
 
 \item When a buffer is used in the receiveq (signalled by an
-  interrupt), the contents is the input to the port associated
+  used buffer notification), the contents is the input to the port associated
   with the virtqueue for which the notification was received.
 
 \item If the driver negotiated the VIRTIO_CONSOLE_F_SIZE feature, a
-  configuration change interrupt indicates that the updated size can
+  configuration change notification indicates that the updated size can
   be read from the configuration fields.  This size applies to port 0 only.
 
 \item If the driver negotiated the VIRTIO_CONSOLE_F_MULTIPORT
@@ -4434,7 +4434,7 @@ The device initialization process is outlined below:
 \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / Device Operation}
 
 The device is driven either by the receipt of a configuration
-change interrupt, or by changing guest memory needs, such as
+change notification, or by changing guest memory needs, such as
 performing memory compaction or responding to out of memory
 conditions.
 
@@ -4571,7 +4571,7 @@ and notifies the device. A request for memory statistics proceeds
 as follows:
 
 \begin{enumerate}
-\item The device uses the buffer and sends an interrupt.
+\item The device uses the buffer and sends an used buffer notification.
 
 \item The driver pops the used buffer and discards it.
 
-- 
1.7.1


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] [PATCH 1/6] notifications: unify notifications wording in core

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

On Thu, Apr 26, 2018 at 12:59:57PM +0200, Halil Pasic wrote:
> @@ -235,12 +236,13 @@ transmit and one for receive.}.
>  Driver makes requests available to device by adding
>  an available buffer to the queue - i.e. adding a buffer
>  describing the request to a virtqueue, and optionally triggering
> -a driver event - i.e. sending a notification to the device.
> +a driver event - i.e. sending an available buffer notification
> +to the device.
>  
>  Device executes the requests and - when complete - adds
>  a used buffer to the queue - i.e. lets the driver
>  know by marking the buffer as used. Device can then trigger
> -a device event - i.e. send an interrupt to the driver.
> +a device event - i.e. send an used buffer notification to the driver.

I would say "a used buffer notification".  "A" vs "an" depends on the
sound of the initial syllable, not the spelling.  So "a used car" vs "an
upper body".

There are several instances of this in this patch.

>  
>  Device reports the number of bytes it has written to memory for
>  each buffer it uses. This is referred to as ``used length''.
> @@ -330,7 +332,8 @@ set the FAILED status bit to indicate that it has given up on the
>  device (it can reset the device later to restart if desired).  The
>  driver MUST NOT continue initialization in that case.
>  
> -The driver MUST NOT notify the device before setting DRIVER_OK.
> +The driver MUST NOT send any buffer available notifications to
> +the device before setting DRIVER_OK.
>  
>  \subsection{Legacy Interface: Device Initialization}\label{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
>  Legacy devices did not support the FEATURES_OK status bit, and thus did
> @@ -388,10 +391,11 @@ reads unless notified.
>  
>  \subsection{Notification of Device Configuration Changes}\label{sec:General Initialization And Device Operation / Device Operation / Notification of Device Configuration Changes}
>  
> -For devices where the device-specific configuration information can be changed, an
> -interrupt is delivered when a device-specific configuration change occurs.
> +For devices where the device-specific configuration information can be
> +changed, a notification is delivered when a device-specific configuration
> +change occurs.

Unlike used/available buffer notifications, the text here just says
"notification" without explicitly saying "configuration change
notification".  I think it makes the spec slightly clearer (and easier
to search) to name the exact type of notification.

> @@ -309,22 +310,20 @@ in the ring.
>  
>  \subsection{Driver and Device Event Suppression}
>  \label{sec:Packed Virtqueues / Driver and Device Event Suppression}
> -In many systems driver and device notifications involve
> +In many systems used and available buffer  notifications involve

s/  / /

> @@ -352,9 +351,9 @@ matches this value and a descriptor is
>  made available/used respectively.
>  \end{description}
>  
> -After writing out some descriptors, both the device and the driver
> +After writing out some descriptors,the device/driver

s/,/, /

> @@ -562,8 +570,8 @@ The driver offers buffers to one of the device's virtqueues as follows:
>  \item The driver performs a suitable memory barrier to ensure that it updates
>    the \field{idx} field before checking for notification suppression.
>  
> -\item If notifications are not suppressed, the driver notifies the device
> -    of the new available buffers.
> +\item The driver sends an available buffers notification to the device if

s/available buffers notification/available buffer notification/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio] Re: [virtio-dev] [PATCH 2/6] notifications: notifications as basic virtio facility

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3942 bytes --]

On Thu, Apr 26, 2018 at 12:59:58PM +0200, Halil Pasic wrote:
> Let's introduce notifications as basic virtio facility early on. This
> shall not only increase the cohesion between core and transport
> description by having a well-defined  place where notifications are
> introduced, but also give us the opportunity to explain some
> discrepancies.
> 
> Namely notifications sent by the device to the driver were often called
> interrupts prior to v1.1. Getting completely rid of that terminology is
> however not viable in case of some names.
> 
> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> ---
> Changelog:
> RFC -> v1:
> * implemented Connies suggestions (thanks Connie)
> ---
>  content.tex |   36 ++++++++++++++++++++++++++++++++++++
>  1 files changed, 36 insertions(+), 0 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index 2e103cf..d7e2b18 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -8,6 +8,7 @@ device consists of the following parts:
>  \begin{itemize}
>  \item Device status field
>  \item Feature bits
> +\item Notifications
>  \item Device Configuration space
>  \item One or more virtqueues
>  \end{itemize}
> @@ -150,6 +151,41 @@ requirements documented within these legacy interface sections.
>  Specification text within these sections generally does not apply
>  to non-transitional devices.
>  
> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
> +/ Notifications}
> +
> +The notion of sending a notification (driver to device or device
> +to driver) plays an important role in this specification. The
> +modus operandi of the notifications is transport specific.
> +
> +There are three types of notifications: 
> +\begin{itemize}
> +\item device configuration space notification
> +\item available buffers notification
> +\item used buffer notifications. 
> +\end{itemize}

"buffers" vs "buffer"; singular vs plural is likely to be used
inconsistently.  Please choose one (either works).

The previous patch said "the term configuration change notification in
kept where called so and consolidated where it's called configuration
change interrupt or similar", but here you say "device configuration
space notification".  Please choose one name and use it consistently.

> +
> +Device configuration space notifications and used buffer notifications
> +are sent by the device, the recipient is the driver. A device
> +configuration space notification indicates that a configuration space
> +has changed; a used buffer notification indicates that a buffer may
> +have been made used on the virtqueue designated by the notification.
> +
> +Available buffer notifications are sent by the driver, the recipient is
> +the device. This type of notification indicates that a buffer may have
> +been made available on the virtqueue designated by the notification.
> +
> +The semantic, the transport-specific implementations and other
> +important aspects of the different notifications are specified in detail
> +in the following chapters

Missing period.

> +
> +Most transports implement notifications sent by the device to the
> +driver using interrupts. Therefore, in previous versions of this
> +specification, these notifications were often called interrupts.
> +Some names defined in this specification still retain this interrupt
> +terminology. Occasionally, the term event is used to refer to
> +a notification or a receipt of a notification.
> +
>  \section{Device Configuration Space}\label{sec:Basic Facilities of a Virtio Device / Device Configuration Space}
>  
>  Device configuration space is generally used for rarely-changing or
> -- 
> 1.7.1
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio] Re: [virtio-dev] [PATCH 3/6] ccw: map common notifications terminology to ccw

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1273 bytes --]

On Thu, Apr 26, 2018 at 12:59:59PM +0200, Halil Pasic wrote:
> The various notifications are introduced and specified in the common
> (i.e. transport agnostic) portion of this specification. How
> notifications are realised for a given transport is something each
> transport has to specify.
> 
> Let's make the relationship between the virtio-ccw terms and the common
> terms more obvious.
> 
> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> ---
> Changelog:
> * implemented Connie's suggestions (typos, grammar)
> * avoided 'realized using virtual interrupts'
> ---
>  content.tex |   42 ++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 42 insertions(+), 0 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index d7e2b18..e5b2499 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -1943,6 +1943,18 @@ Bytes & Description & Contents \\
>  \hline
>  \end{tabular}
>  
> +A virtio-ccw proxy device facilitates:
> +\begin{itemize} 
> +\item Discovery and attachment of  virtio devices (as described above).

s/  / /

> +\item Initialization of virtqueues and transport specific facilities (using

s/transport specific/transport-specific/

Aside from that:

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio] Re: [virtio-dev] [PATCH 1/6] notifications: unify notifications wording in core

[Halil Pasic]

On 05/14/2018 06:36 PM, Stefan Hajnoczi wrote:
> On Thu, Apr 26, 2018 at 12:59:57PM +0200, Halil Pasic wrote:
>> @@ -235,12 +236,13 @@ transmit and one for receive.}.
>>   Driver makes requests available to device by adding
>>   an available buffer to the queue - i.e. adding a buffer
>>   describing the request to a virtqueue, and optionally triggering
>> -a driver event - i.e. sending a notification to the device.
>> +a driver event - i.e. sending an available buffer notification
>> +to the device.
>>   
>>   Device executes the requests and - when complete - adds
>>   a used buffer to the queue - i.e. lets the driver
>>   know by marking the buffer as used. Device can then trigger
>> -a device event - i.e. send an interrupt to the driver.
>> +a device event - i.e. send an used buffer notification to the driver.
> 
> I would say "a used buffer notification".  "A" vs "an" depends on the
> sound of the initial syllable, not the spelling.  So "a used car" vs "an
> upper body".
> 
> There are several instances of this in this patch.
> 

Right. Will try to hunt all of them down.

>>   
>>   Device reports the number of bytes it has written to memory for
>>   each buffer it uses. This is referred to as ``used length''.
>> @@ -330,7 +332,8 @@ set the FAILED status bit to indicate that it has given up on the
>>   device (it can reset the device later to restart if desired).  The
>>   driver MUST NOT continue initialization in that case.
>>   
>> -The driver MUST NOT notify the device before setting DRIVER_OK.
>> +The driver MUST NOT send any buffer available notifications to
>> +the device before setting DRIVER_OK.
>>   
>>   \subsection{Legacy Interface: Device Initialization}\label{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
>>   Legacy devices did not support the FEATURES_OK status bit, and thus did
>> @@ -388,10 +391,11 @@ reads unless notified.
>>   
>>   \subsection{Notification of Device Configuration Changes}\label{sec:General Initialization And Device Operation / Device Operation / Notification of Device Configuration Changes}
>>   
>> -For devices where the device-specific configuration information can be changed, an
>> -interrupt is delivered when a device-specific configuration change occurs.
>> +For devices where the device-specific configuration information can be
>> +changed, a notification is delivered when a device-specific configuration
>> +change occurs.
> 
> Unlike used/available buffer notifications, the text here just says
> "notification" without explicitly saying "configuration change
> notification".  I think it makes the spec slightly clearer (and easier
> to search) to name the exact type of notification.

I decided to not spell it out for stylistic reasons. If I do we end
up with 3xconfiguration and 3xchange in a single sentence. But I'm also
a fan of searching and finding all instances. And I used to say for
specifications it's consistency over style.

I will change it according to your request.


> 
>> @@ -309,22 +310,20 @@ in the ring.
>>   
>>   \subsection{Driver and Device Event Suppression}
>>   \label{sec:Packed Virtqueues / Driver and Device Event Suppression}
>> -In many systems driver and device notifications involve
>> +In many systems used and available buffer  notifications involve
> 
> s/  / /
> 
>> @@ -352,9 +351,9 @@ matches this value and a descriptor is
>>   made available/used respectively.
>>   \end{description}
>>   
>> -After writing out some descriptors, both the device and the driver
>> +After writing out some descriptors,the device/driver
> 
> s/,/, /
> 
>> @@ -562,8 +570,8 @@ The driver offers buffers to one of the device's virtqueues as follows:
>>   \item The driver performs a suitable memory barrier to ensure that it updates
>>     the \field{idx} field before checking for notification suppression.
>>   
>> -\item If notifications are not suppressed, the driver notifies the device
>> -    of the new available buffers.
>> +\item The driver sends an available buffers notification to the device if
> 
> s/available buffers notification/available buffer notification/
> 

Will apply all of these.

Thank you for your review!

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] [PATCH 4/6] pci: map common notifications terminology to PCI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

On Thu, Apr 26, 2018 at 01:00:00PM +0200, Halil Pasic wrote:
> @@ -1306,14 +1306,15 @@ enough to ensure that the separate parts of the virtqueue are on
>  separate cache lines.
>  }.  There was no mechanism to negotiate the queue size.
>  
> -\subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notifying The Device}
> +\subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}
>  
> -The driver notifies the device by writing the 16-bit virtqueue index
> +The driver sends an available buffer notification the device by writing

s/notification the/notification to the/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [virtio] [PATCH 5/6] mmio: map common notifications terminology to MMIO

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2501 bytes --]

On Thu, Apr 26, 2018 at 01:00:01PM +0200, Halil Pasic wrote:
> @@ -1716,26 +1716,28 @@ The driver will typically initialize the virtual queue in the following way:
>  \item Write 0x1 to \field{QueueReady}.
>  \end{enumerate}
>  
> -\subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifying The Device}
> +\subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Available Buffer Notifications}
>  
> -The driver notifies the device about new buffers being available in
> -a queue by writing the index of the updated queue to \field{QueueNotify}.
> +The driver sends an available buffer notification to the device a by

s/device a by/device by/

> +writing the index of the queue to \field{QueueNotify} to be notified.
>  
>  \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
>  
>  The memory mapped virtio device is using a single, dedicated
>  interrupt signal, which is asserted when at least one of the
>  bits described in the description of \field{InterruptStatus}
> -is set. This is how the device notifies the
> -driver about a new used buffer being available in the queue
> -or about a change in the device configuration.
> +is set. This is how the device sends an used buffer notification
> +or a configuration change notifaciton to the device.

s/notifaciton/notification/

Please run a spell checker over the patches in case I missed other
typos.

>  
>  \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
>  After receiving an interrupt, the driver MUST read
> -\field{InterruptStatus} to check what caused the interrupt
> -(see the register description). After the interrupt is handled,
> -the driver MUST acknowledge it by writing a bit mask
> -corresponding to the handled events to the InterruptACK register.
> +
> +\field{InterruptStatus} to check what caused the interrupt (see the
> +register description).  The used buffer notification bit being set,

The comma is unnecessary here.  The "SHOULD ..." clause is not
independent and needs something directly before it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [virtio] [PATCH 6/6] notifications: update notifications terminology for devices

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

On Thu, Apr 26, 2018 at 01:00:02PM +0200, Halil Pasic wrote:
> @@ -4178,8 +4178,8 @@ an appropriate log or output method.
>  \begin{enumerate}
>  \item For output, a buffer containing the characters is placed in
>    the port's transmitq\footnote{Because this is high importance and low bandwidth, the current
> -Linux implementation polls for the buffer to be used, rather than
> -waiting for an interrupt, simplifying the implementation
> +Linux implementation polls for the buffer to become used, rather than
> +waiting for an used buffer notifications, simplifying the implementation

s/for an used buffer notifications/for a used buffer notification/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio] Re: [virtio-dev] [PATCH 1/6] notifications: unify notifications wording in core

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

On Mon, May 14, 2018 at 09:40:43PM +0200, Halil Pasic wrote:
> On 05/14/2018 06:36 PM, Stefan Hajnoczi wrote:
> > On Thu, Apr 26, 2018 at 12:59:57PM +0200, Halil Pasic wrote:
> > > @@ -235,12 +236,13 @@ transmit and one for receive.}.
> > >   Driver makes requests available to device by adding
> > >   an available buffer to the queue - i.e. adding a buffer
> > >   describing the request to a virtqueue, and optionally triggering
> > > -a driver event - i.e. sending a notification to the device.
> > > +a driver event - i.e. sending an available buffer notification
> > > +to the device.
> > >   Device executes the requests and - when complete - adds
> > >   a used buffer to the queue - i.e. lets the driver
> > >   know by marking the buffer as used. Device can then trigger
> > > -a device event - i.e. send an interrupt to the driver.
> > > +a device event - i.e. send an used buffer notification to the driver.
> > 
> > I would say "a used buffer notification".  "A" vs "an" depends on the
> > sound of the initial syllable, not the spelling.  So "a used car" vs "an
> > upper body".
> > 
> > There are several instances of this in this patch.
> > 
> 
> Right. Will try to hunt all of them down.

I noticed other patches with hunks like this:

 ... an
-interrupt ...
+used buffer notification ...

So there are cases that are harder to find due to line-wrapping.  It
should be easy to find them with a Ctrl+F "an used" search in the HTML
version of the document though, or with a multi-line regular expression.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio] Re: [virtio-dev] [PATCH 2/6] notifications: notifications as basic virtio facility

[Halil Pasic]

On 05/14/2018 07:24 PM, Stefan Hajnoczi wrote:
> On Thu, Apr 26, 2018 at 12:59:58PM +0200, Halil Pasic wrote:
>> Let's introduce notifications as basic virtio facility early on. This
>> shall not only increase the cohesion between core and transport
>> description by having a well-defined  place where notifications are
>> introduced, but also give us the opportunity to explain some
>> discrepancies.
>>
>> Namely notifications sent by the device to the driver were often called
>> interrupts prior to v1.1. Getting completely rid of that terminology is
>> however not viable in case of some names.
>>
>> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
>> ---
>> Changelog:
>> RFC -> v1:
>> * implemented Connies suggestions (thanks Connie)
>> ---
>>   content.tex |   36 ++++++++++++++++++++++++++++++++++++
>>   1 files changed, 36 insertions(+), 0 deletions(-)
>>
>> diff --git a/content.tex b/content.tex
>> index 2e103cf..d7e2b18 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -8,6 +8,7 @@ device consists of the following parts:
>>   \begin{itemize}
>>   \item Device status field
>>   \item Feature bits
>> +\item Notifications
>>   \item Device Configuration space
>>   \item One or more virtqueues
>>   \end{itemize}
>> @@ -150,6 +151,41 @@ requirements documented within these legacy interface sections.
>>   Specification text within these sections generally does not apply
>>   to non-transitional devices.
>>   
>> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
>> +/ Notifications}
>> +
>> +The notion of sending a notification (driver to device or device
>> +to driver) plays an important role in this specification. The
>> +modus operandi of the notifications is transport specific.
>> +
>> +There are three types of notifications:
>> +\begin{itemize}
>> +\item device configuration space notification
>> +\item available buffers notification
>> +\item used buffer notifications.
>> +\end{itemize}
> 
> "buffers" vs "buffer"; singular vs plural is likely to be used
> inconsistently.  Please choose one (either works).

I will go with the singular (as originally intended). The majority
also turned out singular -- no idea how I ended up with plural occasionally.

$ git grep -i -e 'buffer *notif'|wc -l
62
$ git grep -i -e 'buffers *notif'
content.tex:\item available buffers notification
split-ring.tex:\item The driver sends an available buffers notification to the device if



> 
> The previous patch said "the term configuration change notification in
> kept where called so and consolidated where it's called configuration
> change interrupt or similar", but here you say "device configuration
> space notification".  Please choose one name and use it consistently.
> 

My bad. Seems the usage of 'configuration space notification' seems to
be  limited to this section. I will replace it with 'configuration change
notification'.

Many thanks for spotting these!

>> +
>> +Device configuration space notifications and used buffer notifications
>> +are sent by the device, the recipient is the driver. A device
>> +configuration space notification indicates that a configuration space
>> +has changed; a used buffer notification indicates that a buffer may
>> +have been made used on the virtqueue designated by the notification.
>> +
>> +Available buffer notifications are sent by the driver, the recipient is
>> +the device. This type of notification indicates that a buffer may have
>> +been made available on the virtqueue designated by the notification.
>> +
>> +The semantic, the transport-specific implementations and other
>> +important aspects of the different notifications are specified in detail
>> +in the following chapters
> 
> Missing period.
> 

Will fix.

>> +
>> +Most transports implement notifications sent by the device to the
>> +driver using interrupts. Therefore, in previous versions of this
>> +specification, these notifications were often called interrupts.
>> +Some names defined in this specification still retain this interrupt
>> +terminology. Occasionally, the term event is used to refer to
>> +a notification or a receipt of a notification.
>> +
>>   \section{Device Configuration Space}\label{sec:Basic Facilities of a Virtio Device / Device Configuration Space}
>>   
>>   Device configuration space is generally used for rarely-changing or
>> -- 
>> 1.7.1
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio] Re: [virtio-dev] Re: [virtio] [PATCH 5/6] mmio: map common notifications terminology to MMIO

[Halil Pasic]

On 05/15/2018 10:58 AM, Stefan Hajnoczi wrote:
> On Thu, Apr 26, 2018 at 01:00:01PM +0200, Halil Pasic wrote:
>> @@ -1716,26 +1716,28 @@ The driver will typically initialize the virtual queue in the following way:
>>   \item Write 0x1 to \field{QueueReady}.
>>   \end{enumerate}
>>   
>> -\subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifying The Device}
>> +\subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Available Buffer Notifications}
>>   
>> -The driver notifies the device about new buffers being available in
>> -a queue by writing the index of the updated queue to \field{QueueNotify}.
>> +The driver sends an available buffer notification to the device a by
> 
> s/device a by/device by/
> 
>> +writing the index of the queue to \field{QueueNotify} to be notified.
>>   
>>   \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
>>   
>>   The memory mapped virtio device is using a single, dedicated
>>   interrupt signal, which is asserted when at least one of the
>>   bits described in the description of \field{InterruptStatus}
>> -is set. This is how the device notifies the
>> -driver about a new used buffer being available in the queue
>> -or about a change in the device configuration.
>> +is set. This is how the device sends an used buffer notification
>> +or a configuration change notifaciton to the device.
> 
> s/notifaciton/notification/
> 
> Please run a spell checker over the patches in case I missed other
> typos.

I did but I missed this one (and maybe some more). Will try to be
more careful when preparing v2.

> 
>>   
>>   \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
>>   After receiving an interrupt, the driver MUST read
>> -\field{InterruptStatus} to check what caused the interrupt
>> -(see the register description). After the interrupt is handled,
>> -the driver MUST acknowledge it by writing a bit mask
>> -corresponding to the handled events to the InterruptACK register.
>> +
>> +\field{InterruptStatus} to check what caused the interrupt (see the
>> +register description).  The used buffer notification bit being set,
> 
> The comma is unnecessary here.  The "SHOULD ..." clause is not
> independent and needs something directly before it.
> 

Right.

I won't comment back to the rest of your comments separately. I recognize
them as legit and I will address them in v2.

Many thanks!

Regards,
Halil


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php