[virtio-dev] [PATCH v10 00/10] Introduce device group and device management

[virtio-dev] [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

Change log:

since v9:
	addressed comments by Parav, Max, Cornelia, David and Zhu Lingshan:
		added link to errno header from Linux
		rename _MEM to _MEMBER
		admin vq num is zero based
		clarify who sends commands where
		minor english tweaks
		clarify command length
		specify interaction with sriov capability
		correct commit log - NumVFs can be 0

	i could not decide what should happen when VFs are
	disabled. for now did not specify.

since v8:
	addressed comments by Cornelia - as we agreed on list
	
since v7:
	make high level error codes match linux, with virtio specific codes
		in a separate field
	renamed _ACCEPT to _USE since that's what it does
	clarified forward compatibility and non pci transports
	support multiple admin vqs
	conformance statements
	lots of changes all over the place to I changed author from Max
	to myself. Don't need to take credit but also don't want
	to blame Max for my mistakes.

since v6:

	- removed some extentions intended for future use.
	  We'll do them when we get there.

	- brought back command list query from v5 in a simplified form -
	  it's here to address the case where a single parent
	  can address multiple groups, such as PF addressing
	  transport vq and sriov vfs.

	- attempt to make terminology more formal.
	In particular a term for whoever controls the group.
 	I am still going back
	and forth between "parent" and "owner" - owner might
	be better after all since it will work if we ever
	have a self group. For now it's parent.

TODO (maybe?) - probably ok to defer until this part is upstream:

	Add "all members" member id.

	Add commands for MSI, feature discovery.

	Add commands for transport vq.


My intent is to try and support both SR-IOV and SIOV
usecases with the same structure and maybe even the same
VQ.

For example, it might make sense to split creating/destroying
SIOV devices from the transport passing data from the guest - the
driver would then not negotiate VIRTIO_F_SR_IOV (which
then means auto-provisioning).

This is out of RFC, since we have two commands which are useful
to discover supported group types (ATM can be none or SR-IOV).


Michael S. Tsirkin (10):
  virtio: document forward compatibility guarantees
  admin: introduce device group and related concepts
  admin: introduce group administration commands
  admin: introduce virtio admin virtqueues
  pci: add admin vq registers to virtio over pci
  mmio: document ADMIN_VQ as reserved
  ccw: document ADMIN_VQ as reserved
  admin: command list discovery
  admin: conformance clauses
  ccw: document more reserved features

 admin.tex        | 540 +++++++++++++++++++++++++++++++++++++++++++++++
 content.tex      | 112 +++++++++-
 introduction.tex |   3 +
 3 files changed, 653 insertions(+), 2 deletions(-)
 create mode 100644 admin.tex

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Michael S. Tsirkin]

Feature negotiation forms the basis of forward compatibility
guarantees of virtio but has never been properly documented.
Do it now.

Suggested-by: Halil Pasic <pasic@linux.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/content.tex b/content.tex
index 0e474dd..0c2d917 100644
--- a/content.tex
+++ b/content.tex
@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 In particular, new fields in the device configuration space are
 indicated by offering a new feature bit.
 
+To keep the feature negotiation mechanism extensible, it is important
+that devices \em{do not} offer any feature bits that they would not be
+able to handle if the driver accepted them (even though drivers are not
+supposed to accept them in the first place even if offered, according to
+this version of the specification.) Likewise, it is important that
+drivers \em{do not} accept feature bits they do not know how to handle
+(even though devices are not supposed to offer them in the first place,
+according to this version of the specification.) The preferred way for
+handling reserved and unexpected features is that the driver ignores
+them.
+
+In particular, this is
+especially important for features limited to specific transports,
+as enabling these for more transports in future versions of the
+specification is highly likely to require changing the behaviour
+from drivers and devices.  Drivers and devices supporting
+multiple transports need to carefully maintain per-transport
+lists of allowed features.
+
 \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
 The driver MUST NOT accept a feature which the device did not offer,
 and MUST NOT accept a feature which requires another feature which was
 not accepted.
 
+The driver MUST validate the feature bits offered by the device.
+The driver MUST ignore and MUST NOT accept any feature bit that is
+\begin{itemize}
+\item not described in this specification,
+\item marked as reserved,
+\item not supported for the specific transport,
+\item not defined for the device type.
+\end{itemize}
+
 The driver SHOULD go into backwards compatibility mode
 if the device does not offer a feature it understands, otherwise MUST
 set the FAILED \field{device status} bit and cease initialization.
 
+By contrast, the driver MUST NOT fail solely because a feature
+it does not understand has been offered by the device.
+
 \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
 The device MUST NOT offer a feature which requires another feature
 which was not offered.  The device SHOULD accept any valid subset
 of features the driver accepts, otherwise it MUST fail to set the
 FEATURES_OK \field{device status} bit when the driver writes it.
 
+The device MUST NOT offer feature bits corresponding to features
+it would not support if accepted by the driver (even if the
+driver is prohibited from accepting the feature bits by the
+specification); for the sake of clarity, this refers to feature
+bits not described in this specification, reserved feature bits
+and feature bits reserved or not supported for the specific
+transport or the specific device type, but this does not preclude
+devices written to a future version of this specification from
+offering such feature bits should such a specification have a
+provision for devices to support the corresponding features.
+
 If a device has successfully negotiated a set of features
 at least once (by accepting the FEATURES_OK \field{device
 status} bit during device initialization), then it SHOULD
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

Each device group has a type. For now, define one initial group:

SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
PCI SR-IOV physical function (PF). This group may contain zero or more
virtio devices according to NumVFs configured.

Each device within a group has a unique identifier. This identifier
is the group member identifier.

Note: one can argue both ways whether the new device group handling
functionality (this and following patches) is closer
to a new device type or a new transport type.

However, I expect that we will add more features in the near future. To
facilitate this as much as possible of the text is located in the new
admin chapter.

I did my best to minimize transport-specific text.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex |  2 ++
 2 files changed, 51 insertions(+)
 create mode 100644 admin.tex

diff --git a/admin.tex b/admin.tex
new file mode 100644
index 0000000..3dc47be
--- /dev/null
+++ b/admin.tex
@@ -0,0 +1,49 @@
+\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
+
+It is occasionally useful to have a device control a group of
+other devices. Terminology used in such cases:
+
+\begin{description}
+\item[Device group]
+        or just group, includes zero or more devices.
+\item[Owner device]
+        or owner, the device controlling the group.
+\item[Member device]
+        a device within a group. The owner device itself is not
+	a member of the group.
+\item[Member identifier]
+        each member has this identifier, unique within the group
+	and used to address it through the owner device.
+\item[Group type identifier]
+	specifies what kind of member devices there are in a
+	group, how is the member identifier is interpreted
+	and what kind of control the owner has.
+	A given owner can control a single group of a given type,
+	thus the type and the owner together identify the group.
+	\footnote{Even though some group types only support
+			specific transports, group type identifiers
+			are global rather than transport-specific -
+			we don't expect a flood of new group types.}
+\end{description}
+
+The following group types, and their identifiers, are currently specified):
+\begin{description}
+\item[SR-IOV group type (0x1)]
+This device group has a PCI Single Root I/O Virtualization
+(SR-IOV) physical function (PF) device as the owner and includes
+all its SR-IOV virtual functions (VFs) as members (see
+\hyperref[intro:PCIe]{[PCIe]}).
+
+The PF device itself is not a member of the group.
+
+The group type identifier for this group is 0x1.
+
+A member identifier for this group can have a value from 0x1 to 0xFFFF
+and equals the SR-IOV VF number of the member device (see
+\hyperref[intro:PCIe]{[PCIe]}).
+
+Both owner and member devices for this group type use the Virtio
+PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
+\end{description}
+
+
diff --git a/content.tex b/content.tex
index 0c2d917..ffe45c4 100644
--- a/content.tex
+++ b/content.tex
@@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\input{admin.tex}
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

This introduces a general structure for group administration commands,
used to control device groups through their owner.

Following patches will introduce specific commands and an interface for
submitting these commands to the owner.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
 introduction.tex |   3 ++
 2 files changed, 111 insertions(+)

diff --git a/admin.tex b/admin.tex
index 3dc47be..7e28b77 100644
--- a/admin.tex
+++ b/admin.tex
@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
 PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
 \end{description}
 
+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
 
+The driver sends group administration commands to the owner device of
+a group to control member devices of the group.
+This mechanism can
+be used, for example, to configure a member device before it is
+initialized by its driver.
+\footnote{The term "administration" is intended to be interpreted
+widely to include any kind of control. See specific commands
+for detail.}
+
+All the group administration commands are of the following form:
+
+\begin{lstlisting}
+struct virtio_admin_cmd {
+        /* Device-readable part */
+        le16 opcode;
+        /*
+         * 1 - SR-IOV
+         * 2 - 65535 reserved
+         */
+        le16 group_type;
+        /* unused, reserved for future extensions */
+        u8 reserved1[12];
+        le64 group_member_id;
+        u8 command_specific_data[];
+
+        /* Device-writable part */
+        le16 status;
+        le16 status_qualifier;
+        /* unused, reserved for future extensions */
+        u8 reserved2[4];
+        u8 command_specific_result[];
+};
+\end{lstlisting}
+
+For all commands, \field{opcode}, \field{group_type} and if
+necessary \field{group_member_id} and \field{command_specific_data} are
+set by the driver, and the owner device sets \field{status} and if
+needed \field{status_qualifier} and
+\field{command_specific_result}.
+
+Generally, any unused device-readable fields are set to zero by the driver
+and ignored by the device.  Any unused device-writeable fields are set to zero
+by the device and ignored by the driver.
+
+\field{opcode} specifies the command. The valid
+values for \field{opcode} can be found in the following table:
+
+\begin{tabular}{|l|l|}
+\hline
+opcode & Name & Command Description \\
+\hline \hline
+0x0000 - 0x7FFF & - &  Group administration commands    \\
+\hline
+0x8000 - 0xFFFF & - & Reserved    \\
+\hline
+\end{tabular}
+
+The \field{group_type} specifies the group type identifier.
+The \field{group_member_id} specifies the member identifier within the group.
+See section \ref{sec:Introduction / Terminology / Device group}
+for the definition of the group type identifier and group member
+identifier.
+
+The following table describes possible \field{status} values;
+to simplify common implementations, they are intentionally
+matching common \hyperref[intro:errno]{Linux error names and numbers}:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Status (decimal) & Name & Description \\
+\hline \hline
+00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
+\hline
+22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
+\hline
+other   & -    & group administration command error  \\
+\hline
+\end{tabular}
+
+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
+is reserved and set to zero by the device.
+
+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
+the following table describes possible \field{status_qialifier} values:
+\begin{tabular}{|l|l|l|}
+\hline
+Status & Name & Description \\
+\hline \hline
+0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
+\hline
+0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
+\hline
+0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
+\hline
+0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
+\hline
+0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or invalid \field{group_member_id} \\
+\hline
+0x05-0xFFFF   & -    & reserved for future use \\
+\hline
+\end{tabular}
+
+Each command uses a different \field{command_specific_data} and
+\field{command_specific_result} structures and the length of
+\field{command_specific_data} and \field{command_specific_result}
+depends on these structures and is described separately or is
+implicit in the structure description.
diff --git a/introduction.tex b/introduction.tex
index 287c5fc..0d849a9 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -68,6 +68,9 @@ \section{Normative References}\label{sec:Normative References}
 	\phantomsection\label{intro:FUSE}\textbf{[FUSE]} &
 	Linux FUSE interface,
 	\newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h}\\
+	\phantomsection\label{intro:errno}\textbf{[errno]} &
+	Linux error names and numbers,
+	\newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/uapi/asm-generic/errno-base.h}\\
         \phantomsection\label{intro:eMMC}\textbf{[eMMC]} &
         eMMC Electrical Standard (5.1), JESD84-B51,
         \newline\url{http://www.jedec.org/sites/default/files/docs/JESD84-B51.pdf}\\
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

The admin virtqueues will be the first interface to issue admin commands.

Currently virtio specification defines control virtqueue to manipulate
features and configuration of the device it operates on. However,
control virtqueue commands are device type specific, which makes it very
difficult to extend for device agnostic commands.

To support this requirement in a more generic way, this patch introduces
a new admin virtqueue interface.

We also support more than one admin virtqueue, for QoS and
scalability requirements.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex |  7 +++--
 2 files changed, 79 insertions(+), 2 deletions(-)

diff --git a/admin.tex b/admin.tex
index 7e28b77..3ffac2e 100644
--- a/admin.tex
+++ b/admin.tex
@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \field{command_specific_data} and \field{command_specific_result}
 depends on these structures and is described separately or is
 implicit in the structure description.
+
+\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
+
+An administration virtqueue of an owner device is used to submit
+group administration commands. An owner device can have more
+than one administration virtqueue.
+
+If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
+or more adminstration virtqueues. The number and locations of the
+administration virtqueues are exposed by the owner device in a transport
+specific manner.
+
+The driver submits commands by queueing them to an arbitrary
+administration virtqueue, and they are processed by the
+device in the order in which they are queued. It is the
+responsibility of the driver to ensure ordering for commands
+placed on different administration virtqueues, because they will
+be executed with no order constraints.
+
+Administration virtqueues are used as follows:
+\begin{itemize}
+\item The driver submits the command using the \field{struct virtio_admin_cmd}
+structure using a buffer consisting of two parts: a device-readable one followed by a
+device-writable one.
+\item the device-readable part includes fields from \field{opcode}
+through \field{command_specific_data}.
+\item the device-writeable buffer includes fields from \field{status}
+through \field{command_specific_result} inclusive.
+\end{itemize}
+
+For each command, this specification describes a distinct
+format structure used for \field{command_specific_data} and
+\field{command_specific_result}, the length of these fields
+depends on the command.
+
+However, to ensure forward compatibility
+\begin{itemize}
+\item drivers are allowed to submit buffers that are longer
+than what the device expects
+(that is, longer than the length of
+\field{opcode} through \field{command_specific_data}).
+This allows the driver to maintain
+a single format structure even if some structure fields are
+unused by the device.
+\item drivers are allowed to submit buffers that are shorter
+than what the device expects
+(that is, shorter than the length of \field{status} through
+\field{command_specific_result}). This allows the device to maintain
+a single format structure even if some structure fields are
+unused by the driver.
+\end{itemize}
+
+The device compares the length of each part (device-readable and
+device-writeable) of the buffer as submitted by driver to what it
+expects and then silently truncates the structures to either the
+length submitted by the driver, or the length described in this
+specification, whichever is shorter.  The device silently ignores
+any data falling outside the shorter of the two lengths. Any
+missing fields are interpreted as set to zero.
+
+Similarly, the driver compares the used buffer length
+of the buffer to what it expects and then silently
+truncates the structure to the used buffer length.
+The driver silently ignores any data falling outside
+the used buffer length reported by the device.  Any missing
+fields are interpreted as set to zero.
+
+This simplifies driver and device implementations since the
+driver/device can simply maintain a single large structure (such
+as a C structure) for a command and its result. As new versions
+of the specification are designed, new fields can be added to the
+tail of a structure, with the driver/device using the full
+structure without concern for versioning.
+>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
diff --git a/content.tex b/content.tex
index ffe45c4..c8647c9 100644
--- a/content.tex
+++ b/content.tex
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 40] Feature bits reserved for extensions to the queue and
+\item[24 to 41] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver can reset a queue individually.
   See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
 
+  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
+  administration virtqueues.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[Michael S. Tsirkin]

Add new registers to the PCI common configuration structure.

These registers will be used for querying the indices of the admin
virtqueues of the owner device. To configure, reset or enable the admin
virtqueues, the driver should follow existing queue configuration/setup
sequence.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/content.tex b/content.tex
index c8647c9..1076125 100644
--- a/content.tex
+++ b/content.tex
@@ -946,6 +946,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
         le16 queue_reset;               /* read-write */
+
+        /* About the administration virtqueue. */
+        le16 admin_queue_index;         /* read-only for driver */
+        le16 admin_queue_num;         /* read-only for driver */
 };
 \end{lstlisting}
 
@@ -1031,6 +1035,20 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         This field exists only if VIRTIO_F_RING_RESET has been
         negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+\item[\field{admin_queue_index}]
+        The device uses this to report the index of the first administration virtqueue.
+        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
+\item[\field{admin_queue_num}]
+	The device uses this to report the number of the
+	supported administration virtqueues.
+	This is a zero based value.  Virtqueues with index
+	between \field{admin_queue_index} and (\field{admin_queue_index} +
+	\field{admin_queue_num}) inclusive serve as administration
+	virtqueues.
+	Thus the number of administration virtqueues equals
+	(\field{admin_queue_num} + 1).
+	This field is valid only if VIRTIO_F_ADMIN_VQ has been
+	negotiated.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1117,6 +1135,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 were used before the queue reset.
 (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver
+configures any administration virtqueues, the driver MUST
+configure the administration virtqueues using the index
+in the range \field{admin_queue_index} to
+\field{admin_queue_index} + \field{admin_queue_num} inclusive.
+The driver MAY configure less administration virtqueues than
+supported by the device.
+
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
@@ -7684,6 +7710,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
   \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
   administration virtqueues.
+  At the moment this feature is only supported for devices using
+  \ref{sec:Virtio Transport Options / Virtio Over PCI
+	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
+	  as the transport and is reserved for future use for
+	  devices using other transports (see
+	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
+	and
+	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
+	handling features reserved for future use.
 
 \end{description}
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

Adding relevant registers needs more work and it's not
clear what the use-case will be as currently only
the PCI transport is supported. But let's keep the
door open on this.
We already say it's reserved in a central place, but it
does not hurt to remind implementers to mask it.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/content.tex b/content.tex
index 1076125..6c3ee52 100644
--- a/content.tex
+++ b/content.tex
@@ -2363,6 +2363,18 @@ \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over M
 
 Notification mechanisms did not change.
 
+\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio Over MMIO / Features reserved for future use}
+
+At this time, devices and drivers utilizing Virtio Over MMIO
+do not support the following features:
+\begin{itemize}
+
+\item VIRTIO_F_ADMIN_VQ
+
+\end{itemize}
+
+These features are reserved for future use.
+
 \section{Virtio Over Channel I/O}\label{sec:Virtio Transport Options / Virtio Over Channel I/O}
 
 S/390 based virtual machines support neither PCI nor MMIO, so a
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 07/10] ccw: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

Adding relevant registers needs more work and it's not
clear what the use-case will be as currently only
the PCI transport is supported. But let's keep the
door open on this.
We already say it's reserved in a central place, but it
does not hurt to remind implementers to mask it.

Note: there are more features to add to this list.
Will be done later with a patch on top.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/content.tex b/content.tex
index 6c3ee52..b414133 100644
--- a/content.tex
+++ b/content.tex
@@ -2977,6 +2977,18 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
 MAY also choose to verify reset completion by reading \field{device status} via
 CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
 
+\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio over channel I/O / Features reserved for future use}
+
+At this time, devices and drivers utilizing Virtio over channel I/O
+do not support the following features:
+\begin{itemize}
+
+\item VIRTIO_F_ADMIN_VQ
+
+\end{itemize}
+
+These features are reserved for future use.
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 08/10] admin: command list discovery

[Michael S. Tsirkin]

Add commands to find out which commands does each group support,
as well as enable their use by driver.
This will be especially useful once we have multiple group types.

An alternative is per-type VQs. This is possible but will
require more per-transport work. Discovery through the vq
helps keep things contained.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 96 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 3ffac2e..1172054 100644
--- a/admin.tex
+++ b/admin.tex
@@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline
 opcode & Name & Command Description \\
 \hline \hline
-0x0000 - 0x7FFF & - &  Group administration commands    \\
+0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
+0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
+0x0002 - 0x7FFF & - &  Group administration commands    \\
 \hline
 0x8000 - 0xFFFF & - & Reserved    \\
 \hline
@@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 depends on these structures and is described separately or is
 implicit in the structure description.
 
+Before sending any administration commands to the device, the driver
+needs to communicate to the device which commands it is going to
+use. Initially (after reset), only two commands are assumed to be used:
+VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
+
+Before sending any other commands for any member of a specific group to
+the device, the driver queries the supported commands via
+VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
+VIRTIO_ADMIN_CMD_LIST_USE.
+
+Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
+VIRTIO_ADMIN_CMD_LIST_USE
+both use the following structure describing the
+command opcodes:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_list {
+       /* Indicates which of the below fields were returned
+       le64 device_admin_cmds[];
+};
+\end{lstlisting}
+
+This structure is an array of 64 bit values in little-endian byte
+order, in which a bit is set if the specific command opcode
+is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value
+in this array corresponding to opcodes 0 to 63,
+\field{device_admin_cmds[1]} is the second 64-bit value
+corresponding to opcodes 64 to 127, etc.
+For example, the array of size 2 including
+the values 0x3 in \field{device_admin_cmds[0]}
+and 0x1 in \field{device_admin_cmds[1]} indicates that only
+opcodes 0, 1 and 64 are supported.
+The length of the array depends on the supported opcodes - it is
+large enough to include bits set for all supported opcodes,
+that is the length can be calculated by starting with the largest
+supported opcode adding one, dividing by 64 and rounding up.
+In other words, for
+VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the
+length of \field{command_specific_result} and
+\field{command_specific_data} respectively will be
+$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division
+with round up and \field{max_cmd} is the largest available command opcode.
+
+The array is also allowed to be larger and to additionally
+include an arbitrary number of all-zero entries.
+
+Accordingly, bits 0 and 1 corresponding to opcode 0
+(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
+(VIRTIO_ADMIN_CMD_LIST_USE) are
+always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
+
+For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
+The \field{group_member_id} is unused. It is set to zero by driver.
+This command has no command specific data.
+The device, upon success, returns a result in
+\field{command_specific_result} in the format
+\field{struct virtio_admin_cmd_list} describing the
+list of administration commands supported for the given group.
+
+
+For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
+is set to 0x1.
+The \field{group_member_id} is unused. It is set to zero by driver.
+The \field{command_specific_data} is in the format
+\field{struct virtio_admin_cmd_list} describing the
+list of administration commands used by the driver.
+This command has no command specific result.
+
+The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
+query the list of commands valid for this group and before sending
+any commands for any member of a group.
+
+The driver then enables use of some of the opcodes by sending to
+the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
+of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
+both understood and used by the driver.
+
+If the device supports the command list used by the driver, the
+device completes the command with status VIRTIO_ADMIN_STATUS_OK.
+If the device does not support the command list, the device
+completes the command with status
+VIRTIO_ADMIN_STATUS_INVALID_FIELD.
+
+Note: drivers are assumed not to set bits in device_admin_cmds
+if they are not familiar with how the command opcode
+is used, since devices could have dependencies between
+command opcodes.
+
+It is assumed that all members in a group support and are used
+with the same list of commands. However, for owner devices
+supporting multiple group types, the list of supported commands
+might differ between different group types.
+
 \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
 
 An administration virtqueue of an owner device is used to submit
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

Add conformance clauses for admin commands and admin virtqueues.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 215 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 1172054..6c4f79c 100644
--- a/admin.tex
+++ b/admin.tex
@@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 supporting multiple group types, the list of supported commands
 might differ between different group types.
 
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+
+The device MUST validate \field{opcode}, \field{group_type} and
+\field{group_member_id}, and if any of these has an invalid or
+unsupported value, set \field{status} to
+VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
+accordingly:
+\begin{itemize}
+\item if \field{group_type} is invalid, \field{status_qualifier}
+	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
+\item otherwise, if \field{opcode} is invalid,
+	\field{status_qualifier} MUST be set to
+	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
+\item otherwise, if \field{group_member_id} is used by the
+	specific command and is invalid, \field{status_qualifier} MUST be
+	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER.
+\end{itemize}
+
+If a command completes successfully, the device MUST set
+\field{status} to VIRTIO_ADMIN_STATUS_OK.
+
+If a command fails, the device MUST set
+\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
+
+If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
+device state MUST NOT change, that is the command MUST NOT have
+any side effects on the device, in particular the device MUST not
+enter an error state as a result of this command.
+
+If a command fails, the device state generally SHOULD NOT change,
+as far as possible.
+
+The device MAY enforce additional restrictions and dependencies on
+opcodes used by the driver and MAY fail the command
+VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
+and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
+if the list of commands used violate internal device dependencies.
+
+If the device supports multiple group types, commands for each group
+type MUST operate independently of each other, in particular,
+the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
+for different group types.
+
+After reset, if the device supports a given group type
+and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
+the device MUST assume
+that the list of legal commands used by the driver consists of
+the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
+
+After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
+the device MUST set the list of legal commands used by the driver
+to the one supplied in \field{command_specific_data}.
+
+The device MUST set the list of legal commands used by the driver
+to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
+
+The device MUST validate commands against the list used by
+the driver and MUST fail any commands not in the list with
+\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
+and \field{status_qualifier} set to
+VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
+
+The list of supported commands MUST NOT shrink (but MAY expand):
+after reporting a given command as supported through
+VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
+as unsupported.  Further, after a given set of commands has been
+used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
+device or system reset the device SHOULD complete successfully
+any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
+list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
+after a device or system reset, the device MUST not fail it
+solely because of the command list used.  Failure to do so would
+interfere with resuming from suspend and error recovery.
+
+When processing a command with the SR-IOV group type,
+if the device does not have an SR-IOV Extended Capability or
+if \field{VF Enable} is clear
+then the device MUST fail all commands with
+\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
+\field{status_qualifier} set to
+VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
+otherwise, if \field{group_member_id} is not
+between $1$ and \field{NumVFs} inclusive,
+the device MUST fail all commands with
+\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
+\field{status_qualifier} set to
+VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER;
+\field{NumVFs}, \field{VF Migration Capable}  and
+\field{VF Enable} refer to registers within the SR-IOV Extended
+Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+
+The driver MAY discover whether device supports a specific group type
+by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
+\field{group_type}.
+
+The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
+and wait for it to be completed with status
+VIRTIO_ADMIN_STATUS_OK before issuing any commands
+(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
+and VIRTIO_ADMIN_CMD_LIST_USE).
+
+The driver SHOULD NOT set bits in device_admin_cmds
+if it is not familiar with how the command opcode
+is used, as dependencies between command opcodes might exist.
+
+The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
+the use of any commands not previously reported as
+supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
+
+The driver MUST NOT use any commands for a given group type
+before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
+list of command opcodes and group type.
+
+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
+VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
+with respective bits cleared in \field{command_specific_data}.
+
+The driver MUST handle a command error with a reserved \field{status}
+value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
+(except possibly for different error reporting/diagnostic messages).
+
+The driver MUST handle a command error with a reserved
+\field{status_qualifier} value in the same way as
+\field{status_qualifier} set to
+VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
+different error reporting/diagnostic messages).
+
+When sending commands with the SR-IOV group type,
+the driver specify a value for \field{group_member_id}
+between $1$ and \field{NumVFs} inclusive,
+the driver MUST also make sure that as long as any such command
+is outstanding, \field{VF Migration Capable} is clear and
+\field{VF Enable} is set;
+\field{NumVFs}, \field{VF Migration Capable}  and
+\field{VF Enable} refer to registers within the SR-IOV Extended
+Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
+
 \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
 
 An administration virtqueue of an owner device is used to submit
@@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
 of the specification are designed, new fields can be added to the
 tail of a structure, with the driver/device using the full
 structure without concern for versioning.
->>>>>>> 0edc690... admin: introduce virtio admin virtqueues
+
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
+
+The device MUST support device-readable and device-writeable buffers
+shorter than described in this specification, by
+\begin{enumerate}
+\item acting as if any data that would be read outside the
+device-readable buffers is set to zero, and
+\item discarding data that would be written outside the
+specified device-writeable buffers.
+\end{enumerate}
+
+The device MUST support device-readable and device-writeable buffers
+longer than described in this specification, by
+\begin{enumerate}
+\item ignoring any data in device-readable buffers outside
+the expected length, and
+\item only writing the expected structure to the device-writeable
+buffers, ignoring any extra buffers, and reporting the
+actual length of data written, in bytes,
+as buffer used length.
+\end{enumerate}
+
+The device SHOULD initialize the device-writeable buffer
+up to the length of the structure described by this specification or
+the length of the buffer supplied by the driver (even if the buffer is
+all set to zero), whichever is shorter.
+
+The device MUST NOT fail a command solely because the buffers
+provided are shorter or longer than described in this
+specification.
+
+The device MUST initialize the device-writeable part of
+\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
+size.
+
+The device MUST initialize \field{status} in \field{struct
+virtio_admin_cmd}.
+
+The device MUST process commands on a given administration virtqueue
+in the order in which they are queued.
+
+If multiple administration virtqueues have been configured,
+device MAY process commands on distinct virtqueues with
+no order constraints.
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
+
+The driver MAY supply device-readable or device-writeable parts
+of \field{struct virtio_admin_cmd} that are longer than described in
+this specification.
+
+The driver SHOULD supply device-readable part of
+\field{struct virtio_admin_cmd} that is at least as
+large as the structure described by this specification
+(even if the structure is all set to zero).
+
+The driver MUST supply both device-readable or device-writeable parts
+of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
+in length.
+
+The device MUST supply both device-readable or device-writeable parts
+of \field{struct virtio_admin_cmd} that are larger than zero in
+length. However, \field{command_specific_data} and
+\field{command_specific_result} MAY be zero in length, unless
+specified otherwise for the command.
+
+The driver MUST NOT assume that the device will initialize the whole
+device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
+the driver MUST act as if the structure
+outside the part of the buffer used by the device
+is set to zero.
+
+If multiple administration virtqueues have been configured,
+the driver MUST ensure ordering for commands
+placed on different administration virtqueues.
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v10 10/10] ccw: document more reserved features

[Michael S. Tsirkin]

vq reset and shared memory are unsupported, too.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/content.tex b/content.tex
index b414133..efcd8bc 100644
--- a/content.tex
+++ b/content.tex
@@ -2984,6 +2984,8 @@ \subsection{Features reserved for future use}\label{sec:Virtio Transport Options
 \begin{itemize}
 
 \item VIRTIO_F_ADMIN_VQ
+\item VIRTIO_F_RING_RESET
+\item Shared memory regions including VIRTIO_PMEM_F_SHMEM_REGION
 
 \end{itemize}
 
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 00/10] Introduce device group and device management

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 8:05 AM
> Change log:
> 
> since v9:
This version is supposed to be v11 from last v10 [1].
https://lists.oasis-open.org/archives/virtio-dev/202302/msg00174.html

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 00/10] Introduce device group and device management

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 8:05 AM
 
> My intent is to try and support both SR-IOV and SIOV usecases with the same
> structure and maybe even the same VQ.
> 
> For example, it might make sense to split creating/destroying SIOV devices
> from the transport passing data from the guest - the driver would then not
> negotiate VIRTIO_F_SR_IOV (which then means auto-provisioning).
> 
> This is out of RFC, since we have two commands which are useful to discover
> supported group types (ATM can be none or SR-IOV).

I guess you missed expanding the cover letter for our discussion on [1] to list use cases a little more tangible.
So let us have them here.
Some I can think of are good starting points are:

1. virtio VF features query and configuration space provisioning
2. virtio VF resource (queues, msix vectors count) provisioning

I plan to post the follow-up patch for #1 as the actual use of AQ.

[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00246.html

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 8:05 AM
> 
> Feature negotiation forms the basis of forward compatibility guarantees of
> virtio but has never been properly documented.
> Do it now.
> 
> Suggested-by: Halil Pasic <pasic@linux.ibm.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---

It may be little painful, but in 10 patch series, it is worth having per patch change log.
Because I do not know my reviewed by tag in [1] was dropped because of a big change, if so what was it or it was simply missed out.

[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00237.html

>  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 0e474dd..0c2d917 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> Virtio Device / Feature B  In particular, new fields in the device configuration
> space are  indicated by offering a new feature bit.
> 
> +To keep the feature negotiation mechanism extensible, it is important
> +that devices \em{do not} offer any feature bits that they would not be
> +able to handle if the driver accepted them (even though drivers are not
> +supposed to accept them in the first place even if offered, according
> +to this version of the specification.) Likewise, it is important that
> +drivers \em{do not} accept feature bits they do not know how to handle
> +(even though devices are not supposed to offer them in the first place,
> +according to this version of the specification.) The preferred way for
> +handling reserved and unexpected features is that the driver ignores
> +them.
> +
> +In particular, this is
> +especially important for features limited to specific transports, as
> +enabling these for more transports in future versions of the
> +specification is highly likely to require changing the behaviour from
> +drivers and devices.  Drivers and devices supporting multiple
> +transports need to carefully maintain per-transport lists of allowed
> +features.
> +
>  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
> Feature Bits}  The driver MUST NOT accept a feature which the device did not
> offer,  and MUST NOT accept a feature which requires another feature which
> was  not accepted.
> 
> +The driver MUST validate the feature bits offered by the device.
> +The driver MUST ignore and MUST NOT accept any feature bit that is
> +\begin{itemize} \item not described in this specification, \item marked
> +as reserved, \item not supported for the specific transport, \item not
> +defined for the device type.
> +\end{itemize}
> +
>  The driver SHOULD go into backwards compatibility mode  if the device does
The grammar tool is suggesting s/backwards/backward.

> not offer a feature it understands, otherwise MUST  set the FAILED \field{device
> status} bit and cease initialization.
> 
> +By contrast, the driver MUST NOT fail solely because a feature it does
> +not understand has been offered by the device.
> +
Above line can be written without introducing "has been" given we have only two entities (driver, dev) to care about.

By contrast, the driver MUST NOT fail solely because it does not understand the feature offered by the device.

>  \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
> Feature Bits}  The device MUST NOT offer a feature which requires another
> feature  which was not offered.  The device SHOULD accept any valid subset  of
Extra white space between feature and which, subset and of.

> features the driver accepts, otherwise it MUST fail to set the  FEATURES_OK
> \field{device status} bit when the driver writes it.
> 
Extra white space between the FEATURE_OK

> +The device MUST NOT offer feature bits corresponding to features it
> +would not support if accepted by the driver (even if the driver is
> +prohibited from accepting the feature bits by the specification); for
> +the sake of clarity, this refers to feature bits not described in this
> +specification, reserved feature bits and feature bits reserved or not
> +supported for the specific transport or the specific device type, but
> +this does not preclude devices written to a future version of this
> +specification from offering such feature bits should such a
> +specification have a provision for devices to support the corresponding
> +features.
> +
>  If a device has successfully negotiated a set of features  at least once (by
> accepting the FEATURES_OK \field{device  status} bit during device
> initialization), then it SHOULD
> --
> MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 02/10] admin: introduce device group and related concepts

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 8:05 AM
> 
> Each device group has a type. For now, define one initial group:
> 
> SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given PCI SR-IOV physical
> function (PF). This group may contain zero or more virtio devices according to
> NumVFs configured.
> 
> Each device within a group has a unique identifier. This identifier is the group
> member identifier.
> 
> Note: one can argue both ways whether the new device group handling
> functionality (this and following patches) is closer to a new device type or a
> new transport type.
> 
> However, I expect that we will add more features in the near future. To
> facilitate this as much as possible of the text is located in the new admin
> chapter.
> 
> I did my best to minimize transport-specific text.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex |  2 ++
>  2 files changed, 51 insertions(+)
>  create mode 100644 admin.tex
> 
You missed addressing the comments of [1].
Can you please fix them?

[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00235.html

> diff --git a/admin.tex b/admin.tex
> new file mode 100644
> index 0000000..3dc47be
> --- /dev/null
> +++ b/admin.tex
> @@ -0,0 +1,49 @@
> +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device /
> +Device groups}
> +
> +It is occasionally useful to have a device control a group of other
> +devices. Terminology used in such cases:
> +
> +\begin{description}
> +\item[Device group]
> +        or just group, includes zero or more devices.
> +\item[Owner device]
> +        or owner, the device controlling the group.
> +\item[Member device]
> +        a device within a group. The owner device itself is not
> +	a member of the group.
> +\item[Member identifier]
> +        each member has this identifier, unique within the group
> +	and used to address it through the owner device.
> +\item[Group type identifier]
> +	specifies what kind of member devices there are in a
> +	group, how is the member identifier is interpreted
> +	and what kind of control the owner has.
> +	A given owner can control a single group of a given type,
> +	thus the type and the owner together identify the group.
> +	\footnote{Even though some group types only support
> +			specific transports, group type identifiers
> +			are global rather than transport-specific -
> +			we don't expect a flood of new group types.}
> \end{description}
> +
> +The following group types, and their identifiers, are currently specified):
> +\begin{description}
> +\item[SR-IOV group type (0x1)]
> +This device group has a PCI Single Root I/O Virtualization
> +(SR-IOV) physical function (PF) device as the owner and includes all
> +its SR-IOV virtual functions (VFs) as members (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +The PF device itself is not a member of the group.
> +
> +The group type identifier for this group is 0x1.
> +
> +A member identifier for this group can have a value from 0x1 to 0xFFFF
> +and equals the SR-IOV VF number of the member device (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +Both owner and member devices for this group type use the Virtio PCI
> +transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> +\end{description}
> +
> +
> diff --git a/content.tex b/content.tex
> index 0c2d917..ffe45c4 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of
> a Virtio Device / Expo  types. It is RECOMMENDED that devices generate
> version 4  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> 
> +\input{admin.tex}
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General
> Initialization And Device Operation}
> 
>  We start with an overview of device initialization, then expand on the
> --
> MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 8:05 AM

> +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> +is reserved and set to zero by the device.
> +
s/status_qialifier/status_qualifier
Missed from v10 of Feb.

> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> +describes possible \field{status_qialifier} values:
s/status_qialifier/status_qualifier

Can you please add other useful error codes in addition to the EINVAL?
Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.

And with that, VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER moves under ENODEV error code.

> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status & Name & Description \\
> +\hline \hline
> +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command
> error: no additional information  \\
> +\hline
> +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or
> invalid \field{opcode}  \\
> +\hline
> +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or
> invalid field within \field{command_specific_data}  \\
> +\hline
> +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or
> invalid \field{group_type} \\
> +\hline
> +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or
> invalid \field{group_member_id} \\
> +\hline
> +0x05-0xFFFF   & -    & reserved for future use \\
> +\hline
> +\end{tabular}


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 8:05 AM

> +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues

I requested to fix this in v10 of Feb, it was in some other patch.
This hunk shifted here.

> +  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device
> + exposes one or more  administration virtqueues.
> +
Please remove extra white space between more and administration.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 8:05 AM
> 
> Adding relevant registers needs more work and it's not clear what the use-case
> will be as currently only the PCI transport is supported. But let's keep the door
> open on this.
> We already say it's reserved in a central place, but it does not hurt to remind
> implementers to mask it.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

Did you miss reviewed-by from [1] or this is an old series reposted?
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html

I will wait for your response before reviewing more, as I am confused now between missing items of v10 of Feb and March.

Can you please have the per-patch change log? I learnt and practice that from netdev community.
It has been very useful to have effective reviews (especially when series is long).

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 02/10] admin: introduce device group and related concepts

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 161 bytes --]

On Thu, Mar 02, 2023 at 08:04:57AM -0500, Michael S. Tsirkin wrote:
> +The following group types, and their identifiers, are currently specified):

spurious ')'

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6108 bytes --]

On Thu, Mar 02, 2023 at 08:05:02AM -0500, Michael S. Tsirkin wrote:
> This introduces a general structure for group administration commands,
> used to control device groups through their owner.
> 
> Following patches will introduce specific commands and an interface for
> submitting these commands to the owner.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
>  introduction.tex |   3 ++
>  2 files changed, 111 insertions(+)
> 
> diff --git a/admin.tex b/admin.tex
> index 3dc47be..7e28b77 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
>  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>  \end{description}
>  
> +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>  
> +The driver sends group administration commands to the owner device of

I notice that the terminology is simply "the driver". "Owner driver"
and "group member driver" might be clearer because there will be two
(possibly different) drivers involved.

> +a group to control member devices of the group.
> +This mechanism can
> +be used, for example, to configure a member device before it is
> +initialized by its driver.
> +\footnote{The term "administration" is intended to be interpreted
> +widely to include any kind of control. See specific commands
> +for detail.}
> +
> +All the group administration commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        le16 opcode;
> +        /*
> +         * 1 - SR-IOV
> +         * 2 - 65535 reserved
> +         */
> +        le16 group_type;
> +        /* unused, reserved for future extensions */
> +        u8 reserved1[12];
> +        le64 group_member_id;
> +        u8 command_specific_data[];
> +
> +        /* Device-writable part */
> +        le16 status;
> +        le16 status_qualifier;
> +        /* unused, reserved for future extensions */
> +        u8 reserved2[4];
> +        u8 command_specific_result[];
> +};
> +\end{lstlisting}
> +
> +For all commands, \field{opcode}, \field{group_type} and if
> +necessary \field{group_member_id} and \field{command_specific_data} are
> +set by the driver, and the owner device sets \field{status} and if
> +needed \field{status_qualifier} and
> +\field{command_specific_result}.
> +
> +Generally, any unused device-readable fields are set to zero by the driver
> +and ignored by the device.  Any unused device-writeable fields are set to zero
> +by the device and ignored by the driver.
> +
> +\field{opcode} specifies the command. The valid
> +values for \field{opcode} can be found in the following table:
> +
> +\begin{tabular}{|l|l|}
> +\hline
> +opcode & Name & Command Description \\
> +\hline \hline
> +0x0000 - 0x7FFF & - &  Group administration commands    \\
> +\hline
> +0x8000 - 0xFFFF & - & Reserved    \\
> +\hline
> +\end{tabular}

I thought all commands are "group administration commands" but this
table makes it look like they are just a subset (0x0000 - 0x7FFF) of
group administration commands, which is a paradox.

> +
> +The \field{group_type} specifies the group type identifier.
> +The \field{group_member_id} specifies the member identifier within the group.
> +See section \ref{sec:Introduction / Terminology / Device group}
> +for the definition of the group type identifier and group member
> +identifier.
> +
> +The following table describes possible \field{status} values;
> +to simplify common implementations, they are intentionally
> +matching common \hyperref[intro:errno]{Linux error names and numbers}:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status (decimal) & Name & Description \\
> +\hline \hline
> +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> +\hline
> +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> +\hline
> +other   & -    & group administration command error  \\
> +\hline
> +\end{tabular}
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}

s/qialifier/qualifier/

> +is reserved and set to zero by the device.
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
> +the following table describes possible \field{status_qialifier} values:

s/qialifier/qualifier/

> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status & Name & Description \\
> +\hline \hline
> +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
> +\hline
> +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
> +\hline
> +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
> +\hline
> +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
> +\hline
> +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or invalid \field{group_member_id} \\
> +\hline
> +0x05-0xFFFF   & -    & reserved for future use \\
> +\hline
> +\end{tabular}
> +
> +Each command uses a different \field{command_specific_data} and
> +\field{command_specific_result} structures and the length of
> +\field{command_specific_data} and \field{command_specific_result}
> +depends on these structures and is described separately or is
> +implicit in the structure description.

On more thing:

Does the owner device see commands in order but may complete them in any
order?

I think this information might be useful just to make it clear that
driver authors shouldn't make assumptions about ordering and completion
order, e.g. pipelining a bunch of dependent commands doesn't work
because the first command is not necessarily completed before the second
command is started.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

On Thu, Mar 02, 2023 at 06:40:29PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> 
> > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > +is reserved and set to zero by the device.
> > +
> s/status_qialifier/status_qualifier
> Missed from v10 of Feb.
> 
> > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> > +describes possible \field{status_qialifier} values:
> s/status_qialifier/status_qualifier
> 
> Can you please add other useful error codes in addition to the EINVAL?
> Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.

Please define a unique constant for each error condition that can occur
instead of sharing catch-all errno constants between multiple error
conditions. If a driver wants to squash them together into an errno,
that's fine, but I think doing this at the hardware interface level is
just propagating the mistakes of errnos.

Only status_qualifier is needed and the vague status field can be
dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and
ENODEV is useful. They have no meaning to the driver, only the
status_qualifier really indicates what is going on.

I'm sure you guys have discussed this previously, but please provide
rationale in the spec because it looks weird to someone with fresh eyes.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 7139 bytes --]

On Thu, Mar 02, 2023 at 08:05:06AM -0500, Michael S. Tsirkin wrote:
> The admin virtqueues will be the first interface to issue admin commands.
> 
> Currently virtio specification defines control virtqueue to manipulate
> features and configuration of the device it operates on. However,
> control virtqueue commands are device type specific, which makes it very
> difficult to extend for device agnostic commands.
> 
> To support this requirement in a more generic way, this patch introduces
> a new admin virtqueue interface.

Is this referring to the existing virtio-net, virtio-scsi, etc control
virtqueues?

I see the admin virtqueue as the virtqueue equivalent to transport
feature bits. The admin queue does nothing device type-specific (net,
scsi, etc) and instead focusses on transport and core virtio tasks.

Keeping the device-specific virtqueue separate from the admin virtqueue
is simpler and has fewer potential problems. I don't think creating
common infrastructure for device-specific control virtqueues across
device types worthwhile or within the scope of this patch series.

> We also support more than one admin virtqueue, for QoS and
> scalability requirements.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex |  7 +++--
>  2 files changed, 79 insertions(+), 2 deletions(-)
> 
> diff --git a/admin.tex b/admin.tex
> index 7e28b77..3ffac2e 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{command_specific_data} and \field{command_specific_result}
>  depends on these structures and is described separately or is
>  implicit in the structure description.
> +
> +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> +
> +An administration virtqueue of an owner device is used to submit
> +group administration commands. An owner device can have more
> +than one administration virtqueue.
> +
> +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
> +or more adminstration virtqueues. The number and locations of the
> +administration virtqueues are exposed by the owner device in a transport
> +specific manner.
> +
> +The driver submits commands by queueing them to an arbitrary
> +administration virtqueue, and they are processed by the
> +device in the order in which they are queued. It is the
> +responsibility of the driver to ensure ordering for commands
> +placed on different administration virtqueues, because they will
> +be executed with no order constraints.

Does "they are processed by the device in the order in which they are
queued" mean only 1 admin command can be running at any given time and
therefore they will complete in order? This would allow pipelining
dependent commands but prevent long-running commands because the
virtqueue is blocked while executing a command.

> +
> +Administration virtqueues are used as follows:
> +\begin{itemize}
> +\item The driver submits the command using the \field{struct virtio_admin_cmd}
> +structure using a buffer consisting of two parts: a device-readable one followed by a
> +device-writable one.
> +\item the device-readable part includes fields from \field{opcode}
> +through \field{command_specific_data}.
> +\item the device-writeable buffer includes fields from \field{status}
> +through \field{command_specific_result} inclusive.
> +\end{itemize}
> +
> +For each command, this specification describes a distinct
> +format structure used for \field{command_specific_data} and
> +\field{command_specific_result}, the length of these fields
> +depends on the command.
> +
> +However, to ensure forward compatibility
> +\begin{itemize}
> +\item drivers are allowed to submit buffers that are longer
> +than what the device expects
> +(that is, longer than the length of
> +\field{opcode} through \field{command_specific_data}).
> +This allows the driver to maintain
> +a single format structure even if some structure fields are
> +unused by the device.
> +\item drivers are allowed to submit buffers that are shorter
> +than what the device expects
> +(that is, shorter than the length of \field{status} through
> +\field{command_specific_result}). This allows the device to maintain
> +a single format structure even if some structure fields are
> +unused by the driver.
> +\end{itemize}
> +
> +The device compares the length of each part (device-readable and
> +device-writeable) of the buffer as submitted by driver to what it
> +expects and then silently truncates the structures to either the
> +length submitted by the driver, or the length described in this
> +specification, whichever is shorter.  The device silently ignores
> +any data falling outside the shorter of the two lengths. Any
> +missing fields are interpreted as set to zero.
> +
> +Similarly, the driver compares the used buffer length
> +of the buffer to what it expects and then silently
> +truncates the structure to the used buffer length.
> +The driver silently ignores any data falling outside
> +the used buffer length reported by the device.  Any missing
> +fields are interpreted as set to zero.
> +
> +This simplifies driver and device implementations since the
> +driver/device can simply maintain a single large structure (such
> +as a C structure) for a command and its result. As new versions
> +of the specification are designed, new fields can be added to the
> +tail of a structure, with the driver/device using the full
> +structure without concern for versioning.
> +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues

A merge conflict line crept into the patch?

> diff --git a/content.tex b/content.tex
> index ffe45c4..c8647c9 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>  \begin{description}
>  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
>  
> -\item[24 to 40] Feature bits reserved for extensions to the queue and
> +\item[24 to 41] Feature bits reserved for extensions to the queue and
>    feature negotiation mechanisms
>  
> -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
> +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
>  \end{description}
>  
>  \begin{note}
> @@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>    that the driver can reset a queue individually.
>    See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>  
> +  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
> +  administration virtqueues.
> +
>  \end{description}
>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> -- 
> MST
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4443 bytes --]

On Thu, Mar 02, 2023 at 08:05:10AM -0500, Michael S. Tsirkin wrote:
> Add new registers to the PCI common configuration structure.
> 
> These registers will be used for querying the indices of the admin
> virtqueues of the owner device. To configure, reset or enable the admin
> virtqueues, the driver should follow existing queue configuration/setup
> sequence.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 35 +++++++++++++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index c8647c9..1076125 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -946,6 +946,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>          le64 queue_device;              /* read-write */
>          le16 queue_notify_data;         /* read-only for driver */
>          le16 queue_reset;               /* read-write */
> +
> +        /* About the administration virtqueue. */
> +        le16 admin_queue_index;         /* read-only for driver */
> +        le16 admin_queue_num;         /* read-only for driver */
>  };
>  \end{lstlisting}
>  
> @@ -1031,6 +1035,20 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>          This field exists only if VIRTIO_F_RING_RESET has been
>          negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>  
> +\item[\field{admin_queue_index}]
> +        The device uses this to report the index of the first administration virtqueue.
> +        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
> +\item[\field{admin_queue_num}]
> +	The device uses this to report the number of the
> +	supported administration virtqueues.
> +	This is a zero based value.  Virtqueues with index
> +	between \field{admin_queue_index} and (\field{admin_queue_index} +
> +	\field{admin_queue_num}) inclusive serve as administration
> +	virtqueues.
> +	Thus the number of administration virtqueues equals
> +	(\field{admin_queue_num} + 1).

Why is this value zero-based? virtio_pci_common_cfg's num_queues field
is le16 and not zero-based. It is not possible to have
admin_queue_index=0 admin_queue_num=0xffff because num_queues=0x10000
cannot be represented.

I suggest making this field behave the same as virtio_pci_common_cfg's
num_queues.

> +	This field is valid only if VIRTIO_F_ADMIN_VQ has been
> +	negotiated.
>  \end{description}
>  
>  \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
> @@ -1117,6 +1135,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>  were used before the queue reset.
>  (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>  
> +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver
> +configures any administration virtqueues, the driver MUST
> +configure the administration virtqueues using the index
> +in the range \field{admin_queue_index} to
> +\field{admin_queue_index} + \field{admin_queue_num} inclusive.
> +The driver MAY configure less administration virtqueues than

s/less/fewer/

> +supported by the device.
> +
>  \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
>  
>  The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
> @@ -7684,6 +7710,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  
>    \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
>    administration virtqueues.
> +  At the moment this feature is only supported for devices using
> +  \ref{sec:Virtio Transport Options / Virtio Over PCI
> +	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
> +	  as the transport and is reserved for future use for
> +	  devices using other transports (see
> +	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
> +	and
> +	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> +	handling features reserved for future use.
>  
>  \end{description}
>  
> -- 
> MST
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 557 bytes --]

On Thu, Mar 02, 2023 at 08:05:14AM -0500, Michael S. Tsirkin wrote:
> Adding relevant registers needs more work and it's not
> clear what the use-case will be as currently only
> the PCI transport is supported. But let's keep the
> door open on this.
> We already say it's reserved in a central place, but it
> does not hurt to remind implementers to mask it.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 12 ++++++++++++
>  1 file changed, 12 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 07/10] ccw: document ADMIN_VQ as reserved

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

On Thu, Mar 02, 2023 at 08:05:18AM -0500, Michael S. Tsirkin wrote:
> Adding relevant registers needs more work and it's not
> clear what the use-case will be as currently only
> the PCI transport is supported. But let's keep the
> door open on this.
> We already say it's reserved in a central place, but it
> does not hurt to remind implementers to mask it.
> 
> Note: there are more features to add to this list.
> Will be done later with a patch on top.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 12 ++++++++++++
>  1 file changed, 12 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 08/10] admin: command list discovery

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6643 bytes --]

On Thu, Mar 02, 2023 at 08:05:22AM -0500, Michael S. Tsirkin wrote:
> Add commands to find out which commands does each group support,
> as well as enable their use by driver.
> This will be especially useful once we have multiple group types.
> 
> An alternative is per-type VQs. This is possible but will
> require more per-transport work. Discovery through the vq
> helps keep things contained.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 96 insertions(+), 1 deletion(-)
> 
> diff --git a/admin.tex b/admin.tex
> index 3ffac2e..1172054 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline
>  opcode & Name & Command Description \\
>  \hline \hline
> -0x0000 - 0x7FFF & - &  Group administration commands    \\
> +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> +0x0002 - 0x7FFF & - &  Group administration commands    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved    \\
>  \hline
> @@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  depends on these structures and is described separately or is
>  implicit in the structure description.
>  
> +Before sending any administration commands to the device, the driver
> +needs to communicate to the device which commands it is going to
> +use. Initially (after reset), only two commands are assumed to be used:
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Before sending any other commands for any member of a specific group to
> +the device, the driver queries the supported commands via
> +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via

"will use" probably means "is capable of using" rather than "is sure to
use"? It might be worth tweaking the language here to make that clear.

> +VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE
> +both use the following structure describing the
> +command opcodes:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_list {
> +       /* Indicates which of the below fields were returned
> +       le64 device_admin_cmds[];
> +};
> +\end{lstlisting}
> +
> +This structure is an array of 64 bit values in little-endian byte
> +order, in which a bit is set if the specific command opcode
> +is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value
> +in this array corresponding to opcodes 0 to 63,
> +\field{device_admin_cmds[1]} is the second 64-bit value
> +corresponding to opcodes 64 to 127, etc.
> +For example, the array of size 2 including
> +the values 0x3 in \field{device_admin_cmds[0]}
> +and 0x1 in \field{device_admin_cmds[1]} indicates that only
> +opcodes 0, 1 and 64 are supported.
> +The length of the array depends on the supported opcodes - it is
> +large enough to include bits set for all supported opcodes,
> +that is the length can be calculated by starting with the largest
> +supported opcode adding one, dividing by 64 and rounding up.
> +In other words, for
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the
> +length of \field{command_specific_result} and
> +\field{command_specific_data} respectively will be
> +$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division
> +with round up and \field{max_cmd} is the largest available command opcode.
> +
> +The array is also allowed to be larger and to additionally
> +include an arbitrary number of all-zero entries.
> +
> +Accordingly, bits 0 and 1 corresponding to opcode 0
> +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
> +(VIRTIO_ADMIN_CMD_LIST_USE) are
> +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +This command has no command specific data.
> +The device, upon success, returns a result in
> +\field{command_specific_result} in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands supported for the given group.
> +
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
> +is set to 0x1.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +The \field{command_specific_data} is in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands used by the driver.
> +This command has no command specific result.
> +
> +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> +query the list of commands valid for this group and before sending
> +any commands for any member of a group.
> +
> +The driver then enables use of some of the opcodes by sending to
> +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
> +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
> +both understood and used by the driver.
> +
> +If the device supports the command list used by the driver, the
> +device completes the command with status VIRTIO_ADMIN_STATUS_OK.
> +If the device does not support the command list, the device
> +completes the command with status
> +VIRTIO_ADMIN_STATUS_INVALID_FIELD.
> +
> +Note: drivers are assumed not to set bits in device_admin_cmds
> +if they are not familiar with how the command opcode
> +is used, since devices could have dependencies between
> +command opcodes.
> +
> +It is assumed that all members in a group support and are used
> +with the same list of commands. However, for owner devices
> +supporting multiple group types, the list of supported commands
> +might differ between different group types.

Is the purpose of VIRTIO_ADMIN_CMD_LIST_USE so that new commands can be
added that are mandatory for device operation. Failure to include them
in the VIRTIO_ADMIN_CMD_LIST_USE command terminates device
initialization with an error?

Are commands evolved by defining flag/version bits in the command
specific data struct? For example, if a new field needs to be added to a
command and a 0 value carries a meaning (so it cannot simply be added to
the end of the existing struct with backwards compatibility).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 10/10] ccw: document more reserved features

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

On Thu, Mar 02, 2023 at 08:05:30AM -0500, Michael S. Tsirkin wrote:
> vq reset and shared memory are unsupported, too.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 2 ++
>  1 file changed, 2 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 01:38:34PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> > Change log:
> > 
> > since v9:
> This version is supposed to be v11 from last v10 [1].
> https://lists.oasis-open.org/archives/virtio-dev/202302/msg00174.html

Good catch. Sorry.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 06:39:29PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> > 
> > Feature negotiation forms the basis of forward compatibility guarantees of
> > virtio but has never been properly documented.
> > Do it now.
> > 
> > Suggested-by: Halil Pasic <pasic@linux.ibm.com>
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> 
> It may be little painful, but in 10 patch series, it is worth having per patch change log.
> Because I do not know my reviewed by tag in [1] was dropped because of a big change, if so what was it or it was simply missed out.
> 
> [1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00237.html

I just lost it, no change.

> >  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 42 insertions(+)
> > 
> > diff --git a/content.tex b/content.tex
> > index 0e474dd..0c2d917 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> > Virtio Device / Feature B  In particular, new fields in the device configuration
> > space are  indicated by offering a new feature bit.
> > 
> > +To keep the feature negotiation mechanism extensible, it is important
> > +that devices \em{do not} offer any feature bits that they would not be
> > +able to handle if the driver accepted them (even though drivers are not
> > +supposed to accept them in the first place even if offered, according
> > +to this version of the specification.) Likewise, it is important that
> > +drivers \em{do not} accept feature bits they do not know how to handle
> > +(even though devices are not supposed to offer them in the first place,
> > +according to this version of the specification.) The preferred way for
> > +handling reserved and unexpected features is that the driver ignores
> > +them.
> > +
> > +In particular, this is
> > +especially important for features limited to specific transports, as
> > +enabling these for more transports in future versions of the
> > +specification is highly likely to require changing the behaviour from
> > +drivers and devices.  Drivers and devices supporting multiple
> > +transports need to carefully maintain per-transport lists of allowed
> > +features.
> > +
> >  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
> > Feature Bits}  The driver MUST NOT accept a feature which the device did not
> > offer,  and MUST NOT accept a feature which requires another feature which
> > was  not accepted.
> > 
> > +The driver MUST validate the feature bits offered by the device.
> > +The driver MUST ignore and MUST NOT accept any feature bit that is
> > +\begin{itemize} \item not described in this specification, \item marked
> > +as reserved, \item not supported for the specific transport, \item not
> > +defined for the device type.
> > +\end{itemize}
> > +
> >  The driver SHOULD go into backwards compatibility mode  if the device does
> The grammar tool is suggesting s/backwards/backward.
> 
> > not offer a feature it understands, otherwise MUST  set the FAILED \field{device
> > status} bit and cease initialization.
> > 
> > +By contrast, the driver MUST NOT fail solely because a feature it does
> > +not understand has been offered by the device.
> > +
> Above line can be written without introducing "has been" given we have only two entities (driver, dev) to care about.
> 
> By contrast, the driver MUST NOT fail solely because it does not understand the feature offered by the device.
> 
> >  \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
> > Feature Bits}  The device MUST NOT offer a feature which requires another
> > feature  which was not offered.  The device SHOULD accept any valid subset  of
> Extra white space between feature and which, subset and of.
> 
> > features the driver accepts, otherwise it MUST fail to set the  FEATURES_OK
> > \field{device status} bit when the driver writes it.
> > 
> Extra white space between the FEATURE_OK
> 
> > +The device MUST NOT offer feature bits corresponding to features it
> > +would not support if accepted by the driver (even if the driver is
> > +prohibited from accepting the feature bits by the specification); for
> > +the sake of clarity, this refers to feature bits not described in this
> > +specification, reserved feature bits and feature bits reserved or not
> > +supported for the specific transport or the specific device type, but
> > +this does not preclude devices written to a future version of this
> > +specification from offering such feature bits should such a
> > +specification have a provision for devices to support the corresponding
> > +features.
> > +
> >  If a device has successfully negotiated a set of features  at least once (by
> > accepting the FEATURES_OK \field{device  status} bit during device
> > initialization), then it SHOULD
> > --
> > MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 06:39:47PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> > 
> > Each device group has a type. For now, define one initial group:
> > 
> > SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given PCI SR-IOV physical
> > function (PF). This group may contain zero or more virtio devices according to
> > NumVFs configured.
> > 
> > Each device within a group has a unique identifier. This identifier is the group
> > member identifier.
> > 
> > Note: one can argue both ways whether the new device group handling
> > functionality (this and following patches) is closer to a new device type or a
> > new transport type.
> > 
> > However, I expect that we will add more features in the near future. To
> > facilitate this as much as possible of the text is located in the new admin
> > chapter.
> > 
> > I did my best to minimize transport-specific text.
> > 
> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  content.tex |  2 ++
> >  2 files changed, 51 insertions(+)
> >  create mode 100644 admin.tex
> > 
> You missed addressing the comments of [1].
> Can you please fix them?
> 
> [1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00235.html

Hmm oh ok. At least ): one for sure.

> > diff --git a/admin.tex b/admin.tex
> > new file mode 100644
> > index 0000000..3dc47be
> > --- /dev/null
> > +++ b/admin.tex
> > @@ -0,0 +1,49 @@
> > +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device /
> > +Device groups}
> > +
> > +It is occasionally useful to have a device control a group of other
> > +devices. Terminology used in such cases:
> > +
> > +\begin{description}
> > +\item[Device group]
> > +        or just group, includes zero or more devices.
> > +\item[Owner device]
> > +        or owner, the device controlling the group.
> > +\item[Member device]
> > +        a device within a group. The owner device itself is not
> > +	a member of the group.
> > +\item[Member identifier]
> > +        each member has this identifier, unique within the group
> > +	and used to address it through the owner device.
> > +\item[Group type identifier]
> > +	specifies what kind of member devices there are in a
> > +	group, how is the member identifier is interpreted
> > +	and what kind of control the owner has.
> > +	A given owner can control a single group of a given type,
> > +	thus the type and the owner together identify the group.
> > +	\footnote{Even though some group types only support
> > +			specific transports, group type identifiers
> > +			are global rather than transport-specific -
> > +			we don't expect a flood of new group types.}
> > \end{description}
> > +
> > +The following group types, and their identifiers, are currently specified):
> > +\begin{description}
> > +\item[SR-IOV group type (0x1)]
> > +This device group has a PCI Single Root I/O Virtualization
> > +(SR-IOV) physical function (PF) device as the owner and includes all
> > +its SR-IOV virtual functions (VFs) as members (see
> > +\hyperref[intro:PCIe]{[PCIe]}).
> > +
> > +The PF device itself is not a member of the group.
> > +
> > +The group type identifier for this group is 0x1.
> > +
> > +A member identifier for this group can have a value from 0x1 to 0xFFFF
> > +and equals the SR-IOV VF number of the member device (see
> > +\hyperref[intro:PCIe]{[PCIe]}).
> > +
> > +Both owner and member devices for this group type use the Virtio PCI
> > +transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > +\end{description}
> > +
> > +
> > diff --git a/content.tex b/content.tex
> > index 0c2d917..ffe45c4 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of
> > a Virtio Device / Expo  types. It is RECOMMENDED that devices generate
> > version 4  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > 
> > +\input{admin.tex}
> > +
> >  \chapter{General Initialization And Device Operation}\label{sec:General
> > Initialization And Device Operation}
> > 
> >  We start with an overview of device initialization, then expand on the
> > --
> > MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 06:40:29PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> 
> > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > +is reserved and set to zero by the device.
> > +
> s/status_qialifier/status_qualifier
> Missed from v10 of Feb.
> 
> > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> > +describes possible \field{status_qialifier} values:
> s/status_qialifier/status_qualifier
> 
> Can you please add other useful error codes in addition to the EINVAL?
> Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.

We'll add them when there's a use-case.

> And with that, VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER moves under ENODEV error code.


No I think that belongs in in EINVAL - command is malformed.
ENODEV is more "The wrong type of device was given to a function that
expects a particular sort of device."


> > +\begin{tabular}{|l|l|l|}
> > +\hline
> > +Status & Name & Description \\
> > +\hline \hline
> > +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command
> > error: no additional information  \\
> > +\hline
> > +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or
> > invalid \field{opcode}  \\
> > +\hline
> > +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or
> > invalid field within \field{command_specific_data}  \\
> > +\hline
> > +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or
> > invalid \field{group_type} \\
> > +\hline
> > +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or
> > invalid \field{group_member_id} \\
> > +\hline
> > +0x05-0xFFFF   & -    & reserved for future use \\
> > +\hline
> > +\end{tabular}


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 06:40:42PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> 
> > +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
> 
> I requested to fix this in v10 of Feb, it was in some other patch.
> This hunk shifted here.

weird I remember fix it.

> > +  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device
> > + exposes one or more  administration virtqueues.
> > +
> Please remove extra white space between more and administration.

I will but it does not matter at all latex ignores it.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 06:40:55PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> > 
> > Adding relevant registers needs more work and it's not clear what the use-case
> > will be as currently only the PCI transport is supported. But let's keep the door
> > open on this.
> > We already say it's reserved in a central place, but it does not hurt to remind
> > implementers to mask it.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> Did you miss reviewed-by from [1] or this is an old series reposted?
> [1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html
> 
> I will wait for your response before reviewing more, as I am confused now between missing items of v10 of Feb and March.

There was such a huge flamewar around error codes I probably just missed
some useful comments among the noise.

> Can you please have the per-patch change log? I learnt and practice that from netdev community.
> It has been very useful to have effective reviews (especially when series is long).

Can't promise really I will need to look for tooling for that.
Unlikely to happen with this patch.
I'm not asking others to do it either.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 06:40:55PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 2, 2023 8:05 AM
> > 
> > Adding relevant registers needs more work and it's not clear what the use-case
> > will be as currently only the PCI transport is supported. But let's keep the door
> > open on this.
> > We already say it's reserved in a central place, but it does not hurt to remind
> > implementers to mask it.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> Did you miss reviewed-by from [1] or this is an old series reposted?
> [1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html


yes I lost it.

> I will wait for your response before reviewing more, as I am confused now between missing items of v10 of Feb and March.
> 
> Can you please have the per-patch change log? I learnt and practice that from netdev community.
> It has been very useful to have effective reviews (especially when series is long).


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 03:10:11PM -0500, Stefan Hajnoczi wrote:
> On Thu, Mar 02, 2023 at 08:05:02AM -0500, Michael S. Tsirkin wrote:
> > This introduces a general structure for group administration commands,
> > used to control device groups through their owner.
> > 
> > Following patches will introduce specific commands and an interface for
> > submitting these commands to the owner.
> > 
> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
> >  introduction.tex |   3 ++
> >  2 files changed, 111 insertions(+)
> > 
> > diff --git a/admin.tex b/admin.tex
> > index 3dc47be..7e28b77 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
> >  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> >  \end{description}
> >  
> > +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> >  
> > +The driver sends group administration commands to the owner device of
> 
> I notice that the terminology is simply "the driver". "Owner driver"
> and "group member driver" might be clearer because there will be two
> (possibly different) drivers involved.

Hmm I don't really want to repeat owner everywhere.
I will clarify that in this section simple "driver" and "device" are
owner, "member device" and "member driver" is always called explicitly.



> > +a group to control member devices of the group.
> > +This mechanism can
> > +be used, for example, to configure a member device before it is
> > +initialized by its driver.
> > +\footnote{The term "administration" is intended to be interpreted
> > +widely to include any kind of control. See specific commands
> > +for detail.}
> > +
> > +All the group administration commands are of the following form:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd {
> > +        /* Device-readable part */
> > +        le16 opcode;
> > +        /*
> > +         * 1 - SR-IOV
> > +         * 2 - 65535 reserved
> > +         */
> > +        le16 group_type;
> > +        /* unused, reserved for future extensions */
> > +        u8 reserved1[12];
> > +        le64 group_member_id;
> > +        u8 command_specific_data[];
> > +
> > +        /* Device-writable part */
> > +        le16 status;
> > +        le16 status_qualifier;
> > +        /* unused, reserved for future extensions */
> > +        u8 reserved2[4];
> > +        u8 command_specific_result[];
> > +};
> > +\end{lstlisting}
> > +
> > +For all commands, \field{opcode}, \field{group_type} and if
> > +necessary \field{group_member_id} and \field{command_specific_data} are
> > +set by the driver, and the owner device sets \field{status} and if
> > +needed \field{status_qualifier} and
> > +\field{command_specific_result}.
> > +
> > +Generally, any unused device-readable fields are set to zero by the driver
> > +and ignored by the device.  Any unused device-writeable fields are set to zero
> > +by the device and ignored by the driver.
> > +
> > +\field{opcode} specifies the command. The valid
> > +values for \field{opcode} can be found in the following table:
> > +
> > +\begin{tabular}{|l|l|}
> > +\hline
> > +opcode & Name & Command Description \\
> > +\hline \hline
> > +0x0000 - 0x7FFF & - &  Group administration commands    \\
> > +\hline
> > +0x8000 - 0xFFFF & - & Reserved    \\
> > +\hline
> > +\end{tabular}
> 
> I thought all commands are "group administration commands" but this
> table makes it look like they are just a subset (0x0000 - 0x7FFF) of
> group administration commands, which is a paradox.

Well the rest are reserved, maybe we will have more command types who
knows. No?


> > +
> > +The \field{group_type} specifies the group type identifier.
> > +The \field{group_member_id} specifies the member identifier within the group.
> > +See section \ref{sec:Introduction / Terminology / Device group}
> > +for the definition of the group type identifier and group member
> > +identifier.
> > +
> > +The following table describes possible \field{status} values;
> > +to simplify common implementations, they are intentionally
> > +matching common \hyperref[intro:errno]{Linux error names and numbers}:
> > +
> > +\begin{tabular}{|l|l|l|}
> > +\hline
> > +Status (decimal) & Name & Description \\
> > +\hline \hline
> > +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> > +\hline
> > +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> > +\hline
> > +other   & -    & group administration command error  \\
> > +\hline
> > +\end{tabular}
> > +
> > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> 
> s/qialifier/qualifier/
> 
> > +is reserved and set to zero by the device.
> > +
> > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
> > +the following table describes possible \field{status_qialifier} values:
> 
> s/qialifier/qualifier/
> 
> > +\begin{tabular}{|l|l|l|}
> > +\hline
> > +Status & Name & Description \\
> > +\hline \hline
> > +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
> > +\hline
> > +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
> > +\hline
> > +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
> > +\hline
> > +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
> > +\hline
> > +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or invalid \field{group_member_id} \\
> > +\hline
> > +0x05-0xFFFF   & -    & reserved for future use \\
> > +\hline
> > +\end{tabular}
> > +
> > +Each command uses a different \field{command_specific_data} and
> > +\field{command_specific_result} structures and the length of
> > +\field{command_specific_data} and \field{command_specific_result}
> > +depends on these structures and is described separately or is
> > +implicit in the structure description.
> 
> On more thing:
> 
> Does the owner device see commands in order but may complete them in any
> order?
> 
> I think this information might be useful just to make it clear that
> driver authors shouldn't make assumptions about ordering and completion
> order, e.g. pipelining a bunch of dependent commands doesn't work
> because the first command is not necessarily completed before the second
> command is started.

I think this is discussed when we discuss sending commands through vq.
in this section it's just about commands generally whatever the way
to send them to device. no?


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 03:19:12PM -0500, Stefan Hajnoczi wrote:
> On Thu, Mar 02, 2023 at 06:40:29PM +0000, Parav Pandit wrote:
> > 
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, March 2, 2023 8:05 AM
> > 
> > > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > > +is reserved and set to zero by the device.
> > > +
> > s/status_qialifier/status_qualifier
> > Missed from v10 of Feb.
> > 
> > > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> > > +describes possible \field{status_qialifier} values:
> > s/status_qialifier/status_qualifier
> > 
> > Can you please add other useful error codes in addition to the EINVAL?
> > Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
> 
> Please define a unique constant for each error condition that can occur
> instead of sharing catch-all errno constants between multiple error
> conditions. If a driver wants to squash them together into an errno,
> that's fine, but I think doing this at the hardware interface level is
> just propagating the mistakes of errnos.
> 
> Only status_qualifier is needed and the vague status field can be
> dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and
> ENODEV is useful. They have no meaning to the driver, only the
> status_qualifier really indicates what is going on.

At a high level at the moment we have only two cases:
- ok
- invalid input supplied by driver

maybe we will have more reasons for a failure - remains to
be seen.





> 
> I'm sure you guys have discussed this previously, but please provide
> rationale in the spec because it looks weird to someone with fresh eyes.
> 
> Stefan

Really most drivers just want to propagate errno to userspace.
All the detailed reporting is for sure well intentional but
in the end it is at best printed into log - end to end
people just end up with a switch statement
converting these to errno codes.
So we are passing them from device and this way there will be
some uniformity.


-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 03:40:07PM -0500, Stefan Hajnoczi wrote:
> On Thu, Mar 02, 2023 at 08:05:06AM -0500, Michael S. Tsirkin wrote:
> > The admin virtqueues will be the first interface to issue admin commands.
> > 
> > Currently virtio specification defines control virtqueue to manipulate
> > features and configuration of the device it operates on. However,
> > control virtqueue commands are device type specific, which makes it very
> > difficult to extend for device agnostic commands.
> > 
> > To support this requirement in a more generic way, this patch introduces
> > a new admin virtqueue interface.
> 
> Is this referring to the existing virtio-net, virtio-scsi, etc control
> virtqueues?
> 
> I see the admin virtqueue as the virtqueue equivalent to transport
> feature bits. The admin queue does nothing device type-specific (net,
> scsi, etc) and instead focusses on transport and core virtio tasks.
> 
> Keeping the device-specific virtqueue separate from the admin virtqueue
> is simpler and has fewer potential problems. I don't think creating
> common infrastructure for device-specific control virtqueues across
> device types worthwhile or within the scope of this patch series.

yes this commit log is outdated. referred to original
proposal by Max which also planned to replace cvq.
will update.


> > We also support more than one admin virtqueue, for QoS and
> > scalability requirements.
> > 
> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  content.tex |  7 +++--
> >  2 files changed, 79 insertions(+), 2 deletions(-)
> > 
> > diff --git a/admin.tex b/admin.tex
> > index 7e28b77..3ffac2e 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \field{command_specific_data} and \field{command_specific_result}
> >  depends on these structures and is described separately or is
> >  implicit in the structure description.
> > +
> > +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > +
> > +An administration virtqueue of an owner device is used to submit
> > +group administration commands. An owner device can have more
> > +than one administration virtqueue.
> > +
> > +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
> > +or more adminstration virtqueues. The number and locations of the
> > +administration virtqueues are exposed by the owner device in a transport
> > +specific manner.
> > +
> > +The driver submits commands by queueing them to an arbitrary
> > +administration virtqueue, and they are processed by the
> > +device in the order in which they are queued. It is the
> > +responsibility of the driver to ensure ordering for commands
> > +placed on different administration virtqueues, because they will
> > +be executed with no order constraints.
> 
> Does "they are processed by the device in the order in which they are
> queued" mean only 1 admin command can be running at any given time and
> therefore they will complete in order? This would allow pipelining
> dependent commands but prevent long-running commands because the
> virtqueue is blocked while executing a command.

we have multiple vqs for that.

> > +
> > +Administration virtqueues are used as follows:
> > +\begin{itemize}
> > +\item The driver submits the command using the \field{struct virtio_admin_cmd}
> > +structure using a buffer consisting of two parts: a device-readable one followed by a
> > +device-writable one.
> > +\item the device-readable part includes fields from \field{opcode}
> > +through \field{command_specific_data}.
> > +\item the device-writeable buffer includes fields from \field{status}
> > +through \field{command_specific_result} inclusive.
> > +\end{itemize}
> > +
> > +For each command, this specification describes a distinct
> > +format structure used for \field{command_specific_data} and
> > +\field{command_specific_result}, the length of these fields
> > +depends on the command.
> > +
> > +However, to ensure forward compatibility
> > +\begin{itemize}
> > +\item drivers are allowed to submit buffers that are longer
> > +than what the device expects
> > +(that is, longer than the length of
> > +\field{opcode} through \field{command_specific_data}).
> > +This allows the driver to maintain
> > +a single format structure even if some structure fields are
> > +unused by the device.
> > +\item drivers are allowed to submit buffers that are shorter
> > +than what the device expects
> > +(that is, shorter than the length of \field{status} through
> > +\field{command_specific_result}). This allows the device to maintain
> > +a single format structure even if some structure fields are
> > +unused by the driver.
> > +\end{itemize}
> > +
> > +The device compares the length of each part (device-readable and
> > +device-writeable) of the buffer as submitted by driver to what it
> > +expects and then silently truncates the structures to either the
> > +length submitted by the driver, or the length described in this
> > +specification, whichever is shorter.  The device silently ignores
> > +any data falling outside the shorter of the two lengths. Any
> > +missing fields are interpreted as set to zero.
> > +
> > +Similarly, the driver compares the used buffer length
> > +of the buffer to what it expects and then silently
> > +truncates the structure to the used buffer length.
> > +The driver silently ignores any data falling outside
> > +the used buffer length reported by the device.  Any missing
> > +fields are interpreted as set to zero.
> > +
> > +This simplifies driver and device implementations since the
> > +driver/device can simply maintain a single large structure (such
> > +as a C structure) for a command and its result. As new versions
> > +of the specification are designed, new fields can be added to the
> > +tail of a structure, with the driver/device using the full
> > +structure without concern for versioning.
> > +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
> 
> A merge conflict line crept into the patch?

yes. I fixed it I thought but somehow it's still there :(

> > diff --git a/content.tex b/content.tex
> > index ffe45c4..c8647c9 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
> >  \begin{description}
> >  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> >  
> > -\item[24 to 40] Feature bits reserved for extensions to the queue and
> > +\item[24 to 41] Feature bits reserved for extensions to the queue and
> >    feature negotiation mechanisms
> >  
> > -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
> > +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> >  \end{description}
> >  
> >  \begin{note}
> > @@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >    that the driver can reset a queue individually.
> >    See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
> >  
> > +  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
> > +  administration virtqueues.
> > +
> >  \end{description}
> >  
> >  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> > -- 
> > MST
> > 



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 06:40:55PM +0000, Parav Pandit wrote:
> Did you miss reviewed-by from [1] or this is an old series reposted?
> [1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html

As a general rule, we don't strictly need to track reviewed by since
there's a ballot (and presumably people review before voting).

People also tack on Signed-off-by: (and I do it too) but
as long as we don't document what it means it's kind of vague,
and the process of subscribing to the mailing list is
a kind of replacement.

If you think everyone needs to follow practices like netdev does,
we really need something written up, and agree on it.

E.g.  I work on the linux kernel too, so I can copy practices from
there, but even linux isn't uniform.

And I wonder whether it's worth it - it definitely makes
contributing to Linux harder, and even within Linux it pushes
contributors away. At least for Linux tracking history
in a precise way is extremely important since it's helpful
with stability. Spec is very different.

Until we have a good contribution documentation I think we should not
ask people to follow a pseudo linux work flow with requests like "please
split this patchset up" or "track changes across patch versions"
simply because there's no good docs to teach people what exactly is
the best practice.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 8046 bytes --]

On Thu, Mar 02, 2023 at 06:57:24PM -0500, Michael S. Tsirkin wrote:
> On Thu, Mar 02, 2023 at 03:10:11PM -0500, Stefan Hajnoczi wrote:
> > On Thu, Mar 02, 2023 at 08:05:02AM -0500, Michael S. Tsirkin wrote:
> > > This introduces a general structure for group administration commands,
> > > used to control device groups through their owner.
> > > 
> > > Following patches will introduce specific commands and an interface for
> > > submitting these commands to the owner.
> > > 
> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > ---
> > >  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  introduction.tex |   3 ++
> > >  2 files changed, 111 insertions(+)
> > > 
> > > diff --git a/admin.tex b/admin.tex
> > > index 3dc47be..7e28b77 100644
> > > --- a/admin.tex
> > > +++ b/admin.tex
> > > @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
> > >  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > >  \end{description}
> > >  
> > > +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > >  
> > > +The driver sends group administration commands to the owner device of
> > 
> > I notice that the terminology is simply "the driver". "Owner driver"
> > and "group member driver" might be clearer because there will be two
> > (possibly different) drivers involved.
> 
> Hmm I don't really want to repeat owner everywhere.
> I will clarify that in this section simple "driver" and "device" are
> owner, "member device" and "member driver" is always called explicitly.

Sounds good.

> > > +a group to control member devices of the group.
> > > +This mechanism can
> > > +be used, for example, to configure a member device before it is
> > > +initialized by its driver.
> > > +\footnote{The term "administration" is intended to be interpreted
> > > +widely to include any kind of control. See specific commands
> > > +for detail.}
> > > +
> > > +All the group administration commands are of the following form:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd {
> > > +        /* Device-readable part */
> > > +        le16 opcode;
> > > +        /*
> > > +         * 1 - SR-IOV
> > > +         * 2 - 65535 reserved
> > > +         */
> > > +        le16 group_type;
> > > +        /* unused, reserved for future extensions */
> > > +        u8 reserved1[12];
> > > +        le64 group_member_id;
> > > +        u8 command_specific_data[];
> > > +
> > > +        /* Device-writable part */
> > > +        le16 status;
> > > +        le16 status_qualifier;
> > > +        /* unused, reserved for future extensions */
> > > +        u8 reserved2[4];
> > > +        u8 command_specific_result[];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +For all commands, \field{opcode}, \field{group_type} and if
> > > +necessary \field{group_member_id} and \field{command_specific_data} are
> > > +set by the driver, and the owner device sets \field{status} and if
> > > +needed \field{status_qualifier} and
> > > +\field{command_specific_result}.
> > > +
> > > +Generally, any unused device-readable fields are set to zero by the driver
> > > +and ignored by the device.  Any unused device-writeable fields are set to zero
> > > +by the device and ignored by the driver.
> > > +
> > > +\field{opcode} specifies the command. The valid
> > > +values for \field{opcode} can be found in the following table:
> > > +
> > > +\begin{tabular}{|l|l|}
> > > +\hline
> > > +opcode & Name & Command Description \\
> > > +\hline \hline
> > > +0x0000 - 0x7FFF & - &  Group administration commands    \\
> > > +\hline
> > > +0x8000 - 0xFFFF & - & Reserved    \\
> > > +\hline
> > > +\end{tabular}
> > 
> > I thought all commands are "group administration commands" but this
> > table makes it look like they are just a subset (0x0000 - 0x7FFF) of
> > group administration commands, which is a paradox.
> 
> Well the rest are reserved, maybe we will have more command types who
> knows. No?

I see. Does that mean the reserved commands don't need to be in the same
format as struct virtio_admin_cmd?

The entire section is called "Group administration commands" but I get
the impression it's talking both about admin virtqueue commands in
general and specifically about group administration commands.

Is it possible to structure this as follows:

  Admin Commands
    ...common stuff...
    Group Administration Commands (0x0000-0x7fff)
      ...specific to group administration commands...
    Reserved (0x8000-0xffff)

> > > +
> > > +The \field{group_type} specifies the group type identifier.
> > > +The \field{group_member_id} specifies the member identifier within the group.
> > > +See section \ref{sec:Introduction / Terminology / Device group}
> > > +for the definition of the group type identifier and group member
> > > +identifier.
> > > +
> > > +The following table describes possible \field{status} values;
> > > +to simplify common implementations, they are intentionally
> > > +matching common \hyperref[intro:errno]{Linux error names and numbers}:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Status (decimal) & Name & Description \\
> > > +\hline \hline
> > > +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> > > +\hline
> > > +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> > > +\hline
> > > +other   & -    & group administration command error  \\
> > > +\hline
> > > +\end{tabular}
> > > +
> > > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > 
> > s/qialifier/qualifier/
> > 
> > > +is reserved and set to zero by the device.
> > > +
> > > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
> > > +the following table describes possible \field{status_qialifier} values:
> > 
> > s/qialifier/qualifier/
> > 
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Status & Name & Description \\
> > > +\hline \hline
> > > +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
> > > +\hline
> > > +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
> > > +\hline
> > > +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
> > > +\hline
> > > +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
> > > +\hline
> > > +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or invalid \field{group_member_id} \\
> > > +\hline
> > > +0x05-0xFFFF   & -    & reserved for future use \\
> > > +\hline
> > > +\end{tabular}
> > > +
> > > +Each command uses a different \field{command_specific_data} and
> > > +\field{command_specific_result} structures and the length of
> > > +\field{command_specific_data} and \field{command_specific_result}
> > > +depends on these structures and is described separately or is
> > > +implicit in the structure description.
> > 
> > On more thing:
> > 
> > Does the owner device see commands in order but may complete them in any
> > order?
> > 
> > I think this information might be useful just to make it clear that
> > driver authors shouldn't make assumptions about ordering and completion
> > order, e.g. pipelining a bunch of dependent commands doesn't work
> > because the first command is not necessarily completed before the second
> > command is started.
> 
> I think this is discussed when we discuss sending commands through vq.
> in this section it's just about commands generally whatever the way
> to send them to device. no?

Okay. I'm reading this patch series linearly and the question popped
into my mind here. Glad it's already covered elsewhere.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]

On Thu, Mar 02, 2023 at 07:01:56PM -0500, Michael S. Tsirkin wrote:
> On Thu, Mar 02, 2023 at 03:19:12PM -0500, Stefan Hajnoczi wrote:
> > On Thu, Mar 02, 2023 at 06:40:29PM +0000, Parav Pandit wrote:
> > > 
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Thursday, March 2, 2023 8:05 AM
> > > 
> > > > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > > > +is reserved and set to zero by the device.
> > > > +
> > > s/status_qialifier/status_qualifier
> > > Missed from v10 of Feb.
> > > 
> > > > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> > > > +describes possible \field{status_qialifier} values:
> > > s/status_qialifier/status_qualifier
> > > 
> > > Can you please add other useful error codes in addition to the EINVAL?
> > > Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
> > 
> > Please define a unique constant for each error condition that can occur
> > instead of sharing catch-all errno constants between multiple error
> > conditions. If a driver wants to squash them together into an errno,
> > that's fine, but I think doing this at the hardware interface level is
> > just propagating the mistakes of errnos.
> > 
> > Only status_qualifier is needed and the vague status field can be
> > dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and
> > ENODEV is useful. They have no meaning to the driver, only the
> > status_qualifier really indicates what is going on.
> 
> At a high level at the moment we have only two cases:
> - ok
> - invalid input supplied by driver
> 
> maybe we will have more reasons for a failure - remains to
> be seen.
> 
> 
> 
> 
> 
> > 
> > I'm sure you guys have discussed this previously, but please provide
> > rationale in the spec because it looks weird to someone with fresh eyes.
> > 
> > Stefan
> 
> Really most drivers just want to propagate errno to userspace.
> All the detailed reporting is for sure well intentional but
> in the end it is at best printed into log - end to end
> people just end up with a switch statement
> converting these to errno codes.
> So we are passing them from device and this way there will be
> some uniformity.

Please clarify the rationale in the spec. I don't agree with it, as
explained in my earlier email, but as long as its documented, people can
try to follow it.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Fri, Mar 03, 2023 at 08:13:37AM -0500, Stefan Hajnoczi wrote:
> On Thu, Mar 02, 2023 at 06:57:24PM -0500, Michael S. Tsirkin wrote:
> > On Thu, Mar 02, 2023 at 03:10:11PM -0500, Stefan Hajnoczi wrote:
> > > On Thu, Mar 02, 2023 at 08:05:02AM -0500, Michael S. Tsirkin wrote:
> > > > This introduces a general structure for group administration commands,
> > > > used to control device groups through their owner.
> > > > 
> > > > Following patches will introduce specific commands and an interface for
> > > > submitting these commands to the owner.
> > > > 
> > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > ---
> > > >  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
> > > >  introduction.tex |   3 ++
> > > >  2 files changed, 111 insertions(+)
> > > > 
> > > > diff --git a/admin.tex b/admin.tex
> > > > index 3dc47be..7e28b77 100644
> > > > --- a/admin.tex
> > > > +++ b/admin.tex
> > > > @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
> > > >  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > > >  \end{description}
> > > >  
> > > > +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > > >  
> > > > +The driver sends group administration commands to the owner device of
> > > 
> > > I notice that the terminology is simply "the driver". "Owner driver"
> > > and "group member driver" might be clearer because there will be two
> > > (possibly different) drivers involved.
> > 
> > Hmm I don't really want to repeat owner everywhere.
> > I will clarify that in this section simple "driver" and "device" are
> > owner, "member device" and "member driver" is always called explicitly.
> 
> Sounds good.
> 
> > > > +a group to control member devices of the group.
> > > > +This mechanism can
> > > > +be used, for example, to configure a member device before it is
> > > > +initialized by its driver.
> > > > +\footnote{The term "administration" is intended to be interpreted
> > > > +widely to include any kind of control. See specific commands
> > > > +for detail.}
> > > > +
> > > > +All the group administration commands are of the following form:
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd {
> > > > +        /* Device-readable part */
> > > > +        le16 opcode;
> > > > +        /*
> > > > +         * 1 - SR-IOV
> > > > +         * 2 - 65535 reserved
> > > > +         */
> > > > +        le16 group_type;
> > > > +        /* unused, reserved for future extensions */
> > > > +        u8 reserved1[12];
> > > > +        le64 group_member_id;
> > > > +        u8 command_specific_data[];
> > > > +
> > > > +        /* Device-writable part */
> > > > +        le16 status;
> > > > +        le16 status_qualifier;
> > > > +        /* unused, reserved for future extensions */
> > > > +        u8 reserved2[4];
> > > > +        u8 command_specific_result[];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +For all commands, \field{opcode}, \field{group_type} and if
> > > > +necessary \field{group_member_id} and \field{command_specific_data} are
> > > > +set by the driver, and the owner device sets \field{status} and if
> > > > +needed \field{status_qualifier} and
> > > > +\field{command_specific_result}.
> > > > +
> > > > +Generally, any unused device-readable fields are set to zero by the driver
> > > > +and ignored by the device.  Any unused device-writeable fields are set to zero
> > > > +by the device and ignored by the driver.
> > > > +
> > > > +\field{opcode} specifies the command. The valid
> > > > +values for \field{opcode} can be found in the following table:
> > > > +
> > > > +\begin{tabular}{|l|l|}
> > > > +\hline
> > > > +opcode & Name & Command Description \\
> > > > +\hline \hline
> > > > +0x0000 - 0x7FFF & - &  Group administration commands    \\
> > > > +\hline
> > > > +0x8000 - 0xFFFF & - & Reserved    \\
> > > > +\hline
> > > > +\end{tabular}
> > > 
> > > I thought all commands are "group administration commands" but this
> > > table makes it look like they are just a subset (0x0000 - 0x7FFF) of
> > > group administration commands, which is a paradox.
> > 
> > Well the rest are reserved, maybe we will have more command types who
> > knows. No?
> 
> I see. Does that mean the reserved commands don't need to be in the same
> format as struct virtio_admin_cmd?

I am not good in predicting future ...

> The entire section is called "Group administration commands" but I get
> the impression it's talking both about admin virtqueue commands in
> general and specifically about group administration commands.
> 
> Is it possible to structure this as follows:
> 
>   Admin Commands
>     ...common stuff...
>     Group Administration Commands (0x0000-0x7fff)
>       ...specific to group administration commands...
>     Reserved (0x8000-0xffff)

What if I just write it like this:


> > > > +\hline \hline
> > > > +0x0000 - 0x7FFF & - &  commands using struct virtio_admin_cmd    \\
> > > > +\hline
> > > > +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\


will this address the comment?


> > > > +
> > > > +The \field{group_type} specifies the group type identifier.
> > > > +The \field{group_member_id} specifies the member identifier within the group.
> > > > +See section \ref{sec:Introduction / Terminology / Device group}
> > > > +for the definition of the group type identifier and group member
> > > > +identifier.
> > > > +
> > > > +The following table describes possible \field{status} values;
> > > > +to simplify common implementations, they are intentionally
> > > > +matching common \hyperref[intro:errno]{Linux error names and numbers}:
> > > > +
> > > > +\begin{tabular}{|l|l|l|}
> > > > +\hline
> > > > +Status (decimal) & Name & Description \\
> > > > +\hline \hline
> > > > +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> > > > +\hline
> > > > +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> > > > +\hline
> > > > +other   & -    & group administration command error  \\
> > > > +\hline
> > > > +\end{tabular}
> > > > +
> > > > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > > 
> > > s/qialifier/qualifier/
> > > 
> > > > +is reserved and set to zero by the device.
> > > > +
> > > > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
> > > > +the following table describes possible \field{status_qialifier} values:
> > > 
> > > s/qialifier/qualifier/
> > > 
> > > > +\begin{tabular}{|l|l|l|}
> > > > +\hline
> > > > +Status & Name & Description \\
> > > > +\hline \hline
> > > > +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
> > > > +\hline
> > > > +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
> > > > +\hline
> > > > +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
> > > > +\hline
> > > > +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
> > > > +\hline
> > > > +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or invalid \field{group_member_id} \\
> > > > +\hline
> > > > +0x05-0xFFFF   & -    & reserved for future use \\
> > > > +\hline
> > > > +\end{tabular}
> > > > +
> > > > +Each command uses a different \field{command_specific_data} and
> > > > +\field{command_specific_result} structures and the length of
> > > > +\field{command_specific_data} and \field{command_specific_result}
> > > > +depends on these structures and is described separately or is
> > > > +implicit in the structure description.
> > > 
> > > On more thing:
> > > 
> > > Does the owner device see commands in order but may complete them in any
> > > order?
> > > 
> > > I think this information might be useful just to make it clear that
> > > driver authors shouldn't make assumptions about ordering and completion
> > > order, e.g. pipelining a bunch of dependent commands doesn't work
> > > because the first command is not necessarily completed before the second
> > > command is started.
> > 
> > I think this is discussed when we discuss sending commands through vq.
> > in this section it's just about commands generally whatever the way
> > to send them to device. no?
> 
> Okay. I'm reading this patch series linearly and the question popped
> into my mind here. Glad it's already covered elsewhere.
> 
> Stefan



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Fri, Mar 03, 2023 at 08:17:03AM -0500, Stefan Hajnoczi wrote:
> On Thu, Mar 02, 2023 at 07:01:56PM -0500, Michael S. Tsirkin wrote:
> > On Thu, Mar 02, 2023 at 03:19:12PM -0500, Stefan Hajnoczi wrote:
> > > On Thu, Mar 02, 2023 at 06:40:29PM +0000, Parav Pandit wrote:
> > > > 
> > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > Sent: Thursday, March 2, 2023 8:05 AM
> > > > 
> > > > > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > > > > +is reserved and set to zero by the device.
> > > > > +
> > > > s/status_qialifier/status_qualifier
> > > > Missed from v10 of Feb.
> > > > 
> > > > > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> > > > > +describes possible \field{status_qialifier} values:
> > > > s/status_qialifier/status_qualifier
> > > > 
> > > > Can you please add other useful error codes in addition to the EINVAL?
> > > > Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
> > > 
> > > Please define a unique constant for each error condition that can occur
> > > instead of sharing catch-all errno constants between multiple error
> > > conditions. If a driver wants to squash them together into an errno,
> > > that's fine, but I think doing this at the hardware interface level is
> > > just propagating the mistakes of errnos.
> > > 
> > > Only status_qualifier is needed and the vague status field can be
> > > dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and
> > > ENODEV is useful. They have no meaning to the driver, only the
> > > status_qualifier really indicates what is going on.
> > 
> > At a high level at the moment we have only two cases:
> > - ok
> > - invalid input supplied by driver
> > 
> > maybe we will have more reasons for a failure - remains to
> > be seen.
> > 
> > 
> > 
> > 
> > 
> > > 
> > > I'm sure you guys have discussed this previously, but please provide
> > > rationale in the spec because it looks weird to someone with fresh eyes.
> > > 
> > > Stefan
> > 
> > Really most drivers just want to propagate errno to userspace.
> > All the detailed reporting is for sure well intentional but
> > in the end it is at best printed into log - end to end
> > people just end up with a switch statement
> > converting these to errno codes.
> > So we are passing them from device and this way there will be
> > some uniformity.
> 
> Please clarify the rationale in the spec. I don't agree with it, as
> explained in my earlier email, but as long as its documented, people can
> try to follow it.
> 
> Stefan

It's 2:2 for now, you are with Parav, me and Cornelia like it :)
Or I will try to document it better.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4067 bytes --]

On Thu, Mar 02, 2023 at 07:05:14PM -0500, Michael S. Tsirkin wrote:
> On Thu, Mar 02, 2023 at 03:40:07PM -0500, Stefan Hajnoczi wrote:
> > On Thu, Mar 02, 2023 at 08:05:06AM -0500, Michael S. Tsirkin wrote:
> > > The admin virtqueues will be the first interface to issue admin commands.
> > > 
> > > Currently virtio specification defines control virtqueue to manipulate
> > > features and configuration of the device it operates on. However,
> > > control virtqueue commands are device type specific, which makes it very
> > > difficult to extend for device agnostic commands.
> > > 
> > > To support this requirement in a more generic way, this patch introduces
> > > a new admin virtqueue interface.
> > 
> > Is this referring to the existing virtio-net, virtio-scsi, etc control
> > virtqueues?
> > 
> > I see the admin virtqueue as the virtqueue equivalent to transport
> > feature bits. The admin queue does nothing device type-specific (net,
> > scsi, etc) and instead focusses on transport and core virtio tasks.
> > 
> > Keeping the device-specific virtqueue separate from the admin virtqueue
> > is simpler and has fewer potential problems. I don't think creating
> > common infrastructure for device-specific control virtqueues across
> > device types worthwhile or within the scope of this patch series.
> 
> yes this commit log is outdated. referred to original
> proposal by Max which also planned to replace cvq.
> will update.
> 
> 
> > > We also support more than one admin virtqueue, for QoS and
> > > scalability requirements.
> > > 
> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > ---
> > >  admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >  content.tex |  7 +++--
> > >  2 files changed, 79 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/admin.tex b/admin.tex
> > > index 7e28b77..3ffac2e 100644
> > > --- a/admin.tex
> > > +++ b/admin.tex
> > > @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >  \field{command_specific_data} and \field{command_specific_result}
> > >  depends on these structures and is described separately or is
> > >  implicit in the structure description.
> > > +
> > > +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > +
> > > +An administration virtqueue of an owner device is used to submit
> > > +group administration commands. An owner device can have more
> > > +than one administration virtqueue.
> > > +
> > > +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
> > > +or more adminstration virtqueues. The number and locations of the
> > > +administration virtqueues are exposed by the owner device in a transport
> > > +specific manner.
> > > +
> > > +The driver submits commands by queueing them to an arbitrary
> > > +administration virtqueue, and they are processed by the
> > > +device in the order in which they are queued. It is the
> > > +responsibility of the driver to ensure ordering for commands
> > > +placed on different administration virtqueues, because they will
> > > +be executed with no order constraints.
> > 
> > Does "they are processed by the device in the order in which they are
> > queued" mean only 1 admin command can be running at any given time and
> > therefore they will complete in order? This would allow pipelining
> > dependent commands but prevent long-running commands because the
> > virtqueue is blocked while executing a command.
> 
> we have multiple vqs for that.

This reminds me of the async challenges with QEMU's QMP monitor.

Will it ever be possible to abort an in-flight command? I guess the
abort command would need to be submitted on another virtqueue, but how
do you identify the in-flight command that you wish to cancel?

Please clarify the blocking semantics in the spec because it wasn't
clear to me.

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Fri, Mar 03, 2023 at 08:28:40AM -0500, Stefan Hajnoczi wrote:
> On Thu, Mar 02, 2023 at 07:05:14PM -0500, Michael S. Tsirkin wrote:
> > On Thu, Mar 02, 2023 at 03:40:07PM -0500, Stefan Hajnoczi wrote:
> > > On Thu, Mar 02, 2023 at 08:05:06AM -0500, Michael S. Tsirkin wrote:
> > > > The admin virtqueues will be the first interface to issue admin commands.
> > > > 
> > > > Currently virtio specification defines control virtqueue to manipulate
> > > > features and configuration of the device it operates on. However,
> > > > control virtqueue commands are device type specific, which makes it very
> > > > difficult to extend for device agnostic commands.
> > > > 
> > > > To support this requirement in a more generic way, this patch introduces
> > > > a new admin virtqueue interface.
> > > 
> > > Is this referring to the existing virtio-net, virtio-scsi, etc control
> > > virtqueues?
> > > 
> > > I see the admin virtqueue as the virtqueue equivalent to transport
> > > feature bits. The admin queue does nothing device type-specific (net,
> > > scsi, etc) and instead focusses on transport and core virtio tasks.
> > > 
> > > Keeping the device-specific virtqueue separate from the admin virtqueue
> > > is simpler and has fewer potential problems. I don't think creating
> > > common infrastructure for device-specific control virtqueues across
> > > device types worthwhile or within the scope of this patch series.
> > 
> > yes this commit log is outdated. referred to original
> > proposal by Max which also planned to replace cvq.
> > will update.
> > 
> > 
> > > > We also support more than one admin virtqueue, for QoS and
> > > > scalability requirements.
> > > > 
> > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > ---
> > > >  admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >  content.tex |  7 +++--
> > > >  2 files changed, 79 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/admin.tex b/admin.tex
> > > > index 7e28b77..3ffac2e 100644
> > > > --- a/admin.tex
> > > > +++ b/admin.tex
> > > > @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > >  \field{command_specific_data} and \field{command_specific_result}
> > > >  depends on these structures and is described separately or is
> > > >  implicit in the structure description.
> > > > +
> > > > +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > > +
> > > > +An administration virtqueue of an owner device is used to submit
> > > > +group administration commands. An owner device can have more
> > > > +than one administration virtqueue.
> > > > +
> > > > +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
> > > > +or more adminstration virtqueues. The number and locations of the
> > > > +administration virtqueues are exposed by the owner device in a transport
> > > > +specific manner.
> > > > +
> > > > +The driver submits commands by queueing them to an arbitrary
> > > > +administration virtqueue, and they are processed by the
> > > > +device in the order in which they are queued. It is the
> > > > +responsibility of the driver to ensure ordering for commands
> > > > +placed on different administration virtqueues, because they will
> > > > +be executed with no order constraints.
> > > 
> > > Does "they are processed by the device in the order in which they are
> > > queued" mean only 1 admin command can be running at any given time and
> > > therefore they will complete in order? This would allow pipelining
> > > dependent commands but prevent long-running commands because the
> > > virtqueue is blocked while executing a command.
> > 
> > we have multiple vqs for that.
> 
> This reminds me of the async challenges with QEMU's QMP monitor.
> 
> Will it ever be possible to abort an in-flight command? I guess the
> abort command would need to be submitted on another virtqueue, but how
> do you identify the in-flight command that you wish to cancel?
> 
> Please clarify the blocking semantics in the spec because it wasn't
> clear to me.
> 
> Thanks,
> Stefan

I don't really get what does "blocking" mean. Nothing is required to
block I think.
I guess for abort it is still in order but a different vq
should not be needed.

Maybe we can use ID of buffer to cancel commands?

1. driver submits command as ID = 1
2. driver submits abort as ID 2
3. device sees the abort and cancels ID 2
4. device uses buffer 1
5. device uses buffer 3 - abort is complete

Device gets commands and processes them in order. It's just like scsi:
multiple queues, if you break it you get to keep both pieces.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6006 bytes --]

On Fri, Mar 03, 2023 at 08:37:31AM -0500, Michael S. Tsirkin wrote:
> On Fri, Mar 03, 2023 at 08:28:40AM -0500, Stefan Hajnoczi wrote:
> > On Thu, Mar 02, 2023 at 07:05:14PM -0500, Michael S. Tsirkin wrote:
> > > On Thu, Mar 02, 2023 at 03:40:07PM -0500, Stefan Hajnoczi wrote:
> > > > On Thu, Mar 02, 2023 at 08:05:06AM -0500, Michael S. Tsirkin wrote:
> > > > > The admin virtqueues will be the first interface to issue admin commands.
> > > > > 
> > > > > Currently virtio specification defines control virtqueue to manipulate
> > > > > features and configuration of the device it operates on. However,
> > > > > control virtqueue commands are device type specific, which makes it very
> > > > > difficult to extend for device agnostic commands.
> > > > > 
> > > > > To support this requirement in a more generic way, this patch introduces
> > > > > a new admin virtqueue interface.
> > > > 
> > > > Is this referring to the existing virtio-net, virtio-scsi, etc control
> > > > virtqueues?
> > > > 
> > > > I see the admin virtqueue as the virtqueue equivalent to transport
> > > > feature bits. The admin queue does nothing device type-specific (net,
> > > > scsi, etc) and instead focusses on transport and core virtio tasks.
> > > > 
> > > > Keeping the device-specific virtqueue separate from the admin virtqueue
> > > > is simpler and has fewer potential problems. I don't think creating
> > > > common infrastructure for device-specific control virtqueues across
> > > > device types worthwhile or within the scope of this patch series.
> > > 
> > > yes this commit log is outdated. referred to original
> > > proposal by Max which also planned to replace cvq.
> > > will update.
> > > 
> > > 
> > > > > We also support more than one admin virtqueue, for QoS and
> > > > > scalability requirements.
> > > > > 
> > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > > ---
> > > > >  admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >  content.tex |  7 +++--
> > > > >  2 files changed, 79 insertions(+), 2 deletions(-)
> > > > > 
> > > > > diff --git a/admin.tex b/admin.tex
> > > > > index 7e28b77..3ffac2e 100644
> > > > > --- a/admin.tex
> > > > > +++ b/admin.tex
> > > > > @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > > >  \field{command_specific_data} and \field{command_specific_result}
> > > > >  depends on these structures and is described separately or is
> > > > >  implicit in the structure description.
> > > > > +
> > > > > +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > > > +
> > > > > +An administration virtqueue of an owner device is used to submit
> > > > > +group administration commands. An owner device can have more
> > > > > +than one administration virtqueue.
> > > > > +
> > > > > +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
> > > > > +or more adminstration virtqueues. The number and locations of the
> > > > > +administration virtqueues are exposed by the owner device in a transport
> > > > > +specific manner.
> > > > > +
> > > > > +The driver submits commands by queueing them to an arbitrary
> > > > > +administration virtqueue, and they are processed by the
> > > > > +device in the order in which they are queued. It is the
> > > > > +responsibility of the driver to ensure ordering for commands
> > > > > +placed on different administration virtqueues, because they will
> > > > > +be executed with no order constraints.
> > > > 
> > > > Does "they are processed by the device in the order in which they are
> > > > queued" mean only 1 admin command can be running at any given time and
> > > > therefore they will complete in order? This would allow pipelining
> > > > dependent commands but prevent long-running commands because the
> > > > virtqueue is blocked while executing a command.
> > > 
> > > we have multiple vqs for that.
> > 
> > This reminds me of the async challenges with QEMU's QMP monitor.
> > 
> > Will it ever be possible to abort an in-flight command? I guess the
> > abort command would need to be submitted on another virtqueue, but how
> > do you identify the in-flight command that you wish to cancel?
> > 
> > Please clarify the blocking semantics in the spec because it wasn't
> > clear to me.
> > 
> > Thanks,
> > Stefan
> 
> I don't really get what does "blocking" mean. Nothing is required to
> block I think.
> I guess for abort it is still in order but a different vq
> should not be needed.
> 
> Maybe we can use ID of buffer to cancel commands?
> 
> 1. driver submits command as ID = 1
> 2. driver submits abort as ID 2
> 3. device sees the abort and cancels ID 2
> 4. device uses buffer 1
> 5. device uses buffer 3 - abort is complete
> 
> Device gets commands and processes them in order. It's just like scsi:
> multiple queues, if you break it you get to keep both pieces.

Now I'm confused because your earlier answer that "we have multiple vqs
for that" implied that a command that takes 1 second will stop further
processing of that vq (what I called "blocking") and the driver will
have to resort to another vq if it wishes to run other commands right
away. But your cancel example seems to use a single virtqueue, so that
would mean commands don't block after all.

My question rephrased:
What happens if a command takes 1 second to complete, is the device
allowed to process the next command from the virtqueue during this time,
possibly completing it before the first command?

This requires additional clarification in the spec because "they are
processed by the device in the order in which they are queued" does not
explain whether commands block the virtqueue (in order completion) or
not (out of order completion).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5785 bytes --]

On Fri, Mar 03, 2023 at 08:18:43AM -0500, Michael S. Tsirkin wrote:
> On Fri, Mar 03, 2023 at 08:13:37AM -0500, Stefan Hajnoczi wrote:
> > On Thu, Mar 02, 2023 at 06:57:24PM -0500, Michael S. Tsirkin wrote:
> > > On Thu, Mar 02, 2023 at 03:10:11PM -0500, Stefan Hajnoczi wrote:
> > > > On Thu, Mar 02, 2023 at 08:05:02AM -0500, Michael S. Tsirkin wrote:
> > > > > This introduces a general structure for group administration commands,
> > > > > used to control device groups through their owner.
> > > > > 
> > > > > Following patches will introduce specific commands and an interface for
> > > > > submitting these commands to the owner.
> > > > > 
> > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > > ---
> > > > >  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >  introduction.tex |   3 ++
> > > > >  2 files changed, 111 insertions(+)
> > > > > 
> > > > > diff --git a/admin.tex b/admin.tex
> > > > > index 3dc47be..7e28b77 100644
> > > > > --- a/admin.tex
> > > > > +++ b/admin.tex
> > > > > @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
> > > > >  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > > > >  \end{description}
> > > > >  
> > > > > +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > > > >  
> > > > > +The driver sends group administration commands to the owner device of
> > > > 
> > > > I notice that the terminology is simply "the driver". "Owner driver"
> > > > and "group member driver" might be clearer because there will be two
> > > > (possibly different) drivers involved.
> > > 
> > > Hmm I don't really want to repeat owner everywhere.
> > > I will clarify that in this section simple "driver" and "device" are
> > > owner, "member device" and "member driver" is always called explicitly.
> > 
> > Sounds good.
> > 
> > > > > +a group to control member devices of the group.
> > > > > +This mechanism can
> > > > > +be used, for example, to configure a member device before it is
> > > > > +initialized by its driver.
> > > > > +\footnote{The term "administration" is intended to be interpreted
> > > > > +widely to include any kind of control. See specific commands
> > > > > +for detail.}
> > > > > +
> > > > > +All the group administration commands are of the following form:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_admin_cmd {
> > > > > +        /* Device-readable part */
> > > > > +        le16 opcode;
> > > > > +        /*
> > > > > +         * 1 - SR-IOV
> > > > > +         * 2 - 65535 reserved
> > > > > +         */
> > > > > +        le16 group_type;
> > > > > +        /* unused, reserved for future extensions */
> > > > > +        u8 reserved1[12];
> > > > > +        le64 group_member_id;
> > > > > +        u8 command_specific_data[];
> > > > > +
> > > > > +        /* Device-writable part */
> > > > > +        le16 status;
> > > > > +        le16 status_qualifier;
> > > > > +        /* unused, reserved for future extensions */
> > > > > +        u8 reserved2[4];
> > > > > +        u8 command_specific_result[];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +For all commands, \field{opcode}, \field{group_type} and if
> > > > > +necessary \field{group_member_id} and \field{command_specific_data} are
> > > > > +set by the driver, and the owner device sets \field{status} and if
> > > > > +needed \field{status_qualifier} and
> > > > > +\field{command_specific_result}.
> > > > > +
> > > > > +Generally, any unused device-readable fields are set to zero by the driver
> > > > > +and ignored by the device.  Any unused device-writeable fields are set to zero
> > > > > +by the device and ignored by the driver.
> > > > > +
> > > > > +\field{opcode} specifies the command. The valid
> > > > > +values for \field{opcode} can be found in the following table:
> > > > > +
> > > > > +\begin{tabular}{|l|l|}
> > > > > +\hline
> > > > > +opcode & Name & Command Description \\
> > > > > +\hline \hline
> > > > > +0x0000 - 0x7FFF & - &  Group administration commands    \\
> > > > > +\hline
> > > > > +0x8000 - 0xFFFF & - & Reserved    \\
> > > > > +\hline
> > > > > +\end{tabular}
> > > > 
> > > > I thought all commands are "group administration commands" but this
> > > > table makes it look like they are just a subset (0x0000 - 0x7FFF) of
> > > > group administration commands, which is a paradox.
> > > 
> > > Well the rest are reserved, maybe we will have more command types who
> > > knows. No?
> > 
> > I see. Does that mean the reserved commands don't need to be in the same
> > format as struct virtio_admin_cmd?
> 
> I am not good in predicting future ...
> 
> > The entire section is called "Group administration commands" but I get
> > the impression it's talking both about admin virtqueue commands in
> > general and specifically about group administration commands.
> > 
> > Is it possible to structure this as follows:
> > 
> >   Admin Commands
> >     ...common stuff...
> >     Group Administration Commands (0x0000-0x7fff)
> >       ...specific to group administration commands...
> >     Reserved (0x8000-0xffff)
> 
> What if I just write it like this:
> 
> 
> > > > > +\hline \hline
> > > > > +0x0000 - 0x7FFF & - &  commands using struct virtio_admin_cmd    \\
> > > > > +\hline
> > > > > +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> 
> 
> will this address the comment?

Yes, thanks.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> What happens if a command takes 1 second to complete, is the device
> allowed to process the next command from the virtqueue during this time,
> possibly completing it before the first command?
> 
> This requires additional clarification in the spec because "they are
> processed by the device in the order in which they are queued" does not
> explain whether commands block the virtqueue (in order completion) or
> not (out of order completion).

Oh I begin to see. Hmm how does e.g. virtio scsi handle this?

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1661 bytes --]

On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > What happens if a command takes 1 second to complete, is the device
> > allowed to process the next command from the virtqueue during this time,
> > possibly completing it before the first command?
> > 
> > This requires additional clarification in the spec because "they are
> > processed by the device in the order in which they are queued" does not
> > explain whether commands block the virtqueue (in order completion) or
> > not (out of order completion).
> 
> Oh I begin to see. Hmm how does e.g. virtio scsi handle this?

virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
Several may be processed by the device at the same time.

They rely on multi-queue for abort operations:

In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are
sent on the control virtqueue. The the request identifier namespace is
shared across all virtqueues so it's possible to abort a request that
was submitted to any command virtqueue.

NVMe also follows the same design where abort commands are sent on the
Admin Submission Queue instead of an I/O Submission Queue. It's possible
to identify NVMe requests by <Submission Queue ID, Command Identifier>.

virtio-blk doesn't support aborting requests.

I think the logic behind this design is that if a queue gets stuck
processing long-running requests, then the device should not be forced
to perform lookahead in the queue to find abort commands. A separate
control/admin queue is used for the abort requests.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > What happens if a command takes 1 second to complete, is the device
> > > allowed to process the next command from the virtqueue during this time,
> > > possibly completing it before the first command?
> > > 
> > > This requires additional clarification in the spec because "they are
> > > processed by the device in the order in which they are queued" does not
> > > explain whether commands block the virtqueue (in order completion) or
> > > not (out of order completion).
> > 
> > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> 
> virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> Several may be processed by the device at the same time.

Let's say I submit a write followed by read - is read
guaranteed to return an up to date info?

> They rely on multi-queue for abort operations:
> 
> In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are
> sent on the control virtqueue. The the request identifier namespace is
> shared across all virtqueues so it's possible to abort a request that
> was submitted to any command virtqueue.
> 
> NVMe also follows the same design where abort commands are sent on the
> Admin Submission Queue instead of an I/O Submission Queue. It's possible
> to identify NVMe requests by <Submission Queue ID, Command Identifier>.
> 
> virtio-blk doesn't support aborting requests.
> 
> I think the logic behind this design is that if a queue gets stuck
> processing long-running requests, then the device should not be forced
> to perform lookahead in the queue to find abort commands. A separate
> control/admin queue is used for the abort requests.
> 
> Stefan



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Mon, Mar 06, 2023 at 08:58:12AM CET, jasowang@redhat.com wrote:
>
>在 2023/3/3 08:05, Michael S. Tsirkin 写道:
>> On Thu, Mar 02, 2023 at 03:40:07PM -0500, Stefan Hajnoczi wrote:
>> > On Thu, Mar 02, 2023 at 08:05:06AM -0500, Michael S. Tsirkin wrote:
>> > > The admin virtqueues will be the first interface to issue admin commands.
>> > > 
>> > > Currently virtio specification defines control virtqueue to manipulate
>> > > features and configuration of the device it operates on. However,
>> > > control virtqueue commands are device type specific, which makes it very
>> > > difficult to extend for device agnostic commands.
>> > > 
>> > > To support this requirement in a more generic way, this patch introduces
>> > > a new admin virtqueue interface.
>> > Is this referring to the existing virtio-net, virtio-scsi, etc control
>> > virtqueues?
>> > 
>> > I see the admin virtqueue as the virtqueue equivalent to transport
>> > feature bits. The admin queue does nothing device type-specific (net,
>> > scsi, etc) and instead focusses on transport and core virtio tasks.
>> > 
>> > Keeping the device-specific virtqueue separate from the admin virtqueue
>> > is simpler and has fewer potential problems. I don't think creating
>> > common infrastructure for device-specific control virtqueues across
>> > device types worthwhile or within the scope of this patch series.
>> yes this commit log is outdated. referred to original
>> proposal by Max which also planned to replace cvq.
>
>
>Any advantages of allowing PF to change VF's filters? If you're referring

Which filters are you referring to?


>provisioning, the semantic should be different:
>
>1) using admin virtqueue to provision attributes of mac, #vpqs
>2) once the provisioning is done, we should fail another provisioning request

Wait. Why are you assuming provisioning of any kind? In which patch this
is covered?


>
>But 1) should be different from what is used for ctrl vq which is designed
>for the use of the driver not the management.
>
>Thanks
>
>
>> will update.
>> 
>> 
>> > > We also support more than one admin virtqueue, for QoS and
>> > > scalability requirements.
>> > > 
>> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> > > ---
>> > >   admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >   content.tex |  7 +++--
>> > >   2 files changed, 79 insertions(+), 2 deletions(-)
>> > > 
>> > > diff --git a/admin.tex b/admin.tex
>> > > index 7e28b77..3ffac2e 100644
>> > > --- a/admin.tex
>> > > +++ b/admin.tex
>> > > @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>> > >   \field{command_specific_data} and \field{command_specific_result}
>> > >   depends on these structures and is described separately or is
>> > >   implicit in the structure description.
>> > > +
>> > > +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>> > > +
>> > > +An administration virtqueue of an owner device is used to submit
>> > > +group administration commands. An owner device can have more
>> > > +than one administration virtqueue.
>> > > +
>> > > +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
>> > > +or more adminstration virtqueues. The number and locations of the
>> > > +administration virtqueues are exposed by the owner device in a transport
>> > > +specific manner.
>> > > +
>> > > +The driver submits commands by queueing them to an arbitrary
>> > > +administration virtqueue, and they are processed by the
>> > > +device in the order in which they are queued. It is the
>> > > +responsibility of the driver to ensure ordering for commands
>> > > +placed on different administration virtqueues, because they will
>> > > +be executed with no order constraints.
>> > Does "they are processed by the device in the order in which they are
>> > queued" mean only 1 admin command can be running at any given time and
>> > therefore they will complete in order? This would allow pipelining
>> > dependent commands but prevent long-running commands because the
>> > virtqueue is blocked while executing a command.
>> we have multiple vqs for that.
>> 
>> > > +
>> > > +Administration virtqueues are used as follows:
>> > > +\begin{itemize}
>> > > +\item The driver submits the command using the \field{struct virtio_admin_cmd}
>> > > +structure using a buffer consisting of two parts: a device-readable one followed by a
>> > > +device-writable one.
>> > > +\item the device-readable part includes fields from \field{opcode}
>> > > +through \field{command_specific_data}.
>> > > +\item the device-writeable buffer includes fields from \field{status}
>> > > +through \field{command_specific_result} inclusive.
>> > > +\end{itemize}
>> > > +
>> > > +For each command, this specification describes a distinct
>> > > +format structure used for \field{command_specific_data} and
>> > > +\field{command_specific_result}, the length of these fields
>> > > +depends on the command.
>> > > +
>> > > +However, to ensure forward compatibility
>> > > +\begin{itemize}
>> > > +\item drivers are allowed to submit buffers that are longer
>> > > +than what the device expects
>> > > +(that is, longer than the length of
>> > > +\field{opcode} through \field{command_specific_data}).
>> > > +This allows the driver to maintain
>> > > +a single format structure even if some structure fields are
>> > > +unused by the device.
>> > > +\item drivers are allowed to submit buffers that are shorter
>> > > +than what the device expects
>> > > +(that is, shorter than the length of \field{status} through
>> > > +\field{command_specific_result}). This allows the device to maintain
>> > > +a single format structure even if some structure fields are
>> > > +unused by the driver.
>> > > +\end{itemize}
>> > > +
>> > > +The device compares the length of each part (device-readable and
>> > > +device-writeable) of the buffer as submitted by driver to what it
>> > > +expects and then silently truncates the structures to either the
>> > > +length submitted by the driver, or the length described in this
>> > > +specification, whichever is shorter.  The device silently ignores
>> > > +any data falling outside the shorter of the two lengths. Any
>> > > +missing fields are interpreted as set to zero.
>> > > +
>> > > +Similarly, the driver compares the used buffer length
>> > > +of the buffer to what it expects and then silently
>> > > +truncates the structure to the used buffer length.
>> > > +The driver silently ignores any data falling outside
>> > > +the used buffer length reported by the device.  Any missing
>> > > +fields are interpreted as set to zero.
>> > > +
>> > > +This simplifies driver and device implementations since the
>> > > +driver/device can simply maintain a single large structure (such
>> > > +as a C structure) for a command and its result. As new versions
>> > > +of the specification are designed, new fields can be added to the
>> > > +tail of a structure, with the driver/device using the full
>> > > +structure without concern for versioning.
>> > > +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
>> > A merge conflict line crept into the patch?
>> yes. I fixed it I thought but somehow it's still there :(
>> 
>> > > diff --git a/content.tex b/content.tex
>> > > index ffe45c4..c8647c9 100644
>> > > --- a/content.tex
>> > > +++ b/content.tex
>> > > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>> > >   \begin{description}
>> > >   \item[0 to 23, and 50 to 127] Feature bits for the specific device type
>> > > -\item[24 to 40] Feature bits reserved for extensions to the queue and
>> > > +\item[24 to 41] Feature bits reserved for extensions to the queue and
>> > >     feature negotiation mechanisms
>> > > -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
>> > > +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
>> > >   \end{description}
>> > >   \begin{note}
>> > > @@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>> > >     that the driver can reset a queue individually.
>> > >     See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>> > > +  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
>> > > +  administration virtqueues.
>> > > +
>> > >   \end{description}
>> > >   \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
>> > > -- 
>> > > MST
>> > > 
>> 
>
>
>---------------------------------------------------------------------
>To unsubscribe from this mail list, you must leave the OASIS TC that
>generates this mail.  Follow this link to all your TCs in OASIS at:
>https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4529 bytes --]

On Mon, Mar 06, 2023 at 05:28:03PM +0200, Max Gurtovoy wrote:
> 
> 
> On 06/03/2023 13:20, Stefan Hajnoczi wrote:
> > On Mon, Mar 06, 2023 at 04:00:50PM +0800, Jason Wang wrote:
> > > 
> > > 在 2023/3/6 08:03, Stefan Hajnoczi 写道:
> > > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > > What happens if a command takes 1 second to complete, is the device
> > > > > > allowed to process the next command from the virtqueue during this time,
> > > > > > possibly completing it before the first command?
> > > > > > 
> > > > > > This requires additional clarification in the spec because "they are
> > > > > > processed by the device in the order in which they are queued" does not
> > > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > > not (out of order completion).
> > > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > > Several may be processed by the device at the same time.
> > > > 
> > > > They rely on multi-queue for abort operations:
> > > > 
> > > > In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are
> > > > sent on the control virtqueue. The the request identifier namespace is
> > > > shared across all virtqueues so it's possible to abort a request that
> > > > was submitted to any command virtqueue.
> > > > 
> > > > NVMe also follows the same design where abort commands are sent on the
> > > > Admin Submission Queue instead of an I/O Submission Queue. It's possible
> > > > to identify NVMe requests by <Submission Queue ID, Command Identifier>.
> > > > 
> > > > virtio-blk doesn't support aborting requests.
> > > > 
> > > > I think the logic behind this design is that if a queue gets stuck
> > > > processing long-running requests, then the device should not be forced
> > > > to perform lookahead in the queue to find abort commands. A separate
> > > > control/admin queue is used for the abort requests.
> > > 
> > > 
> > > Or device need mandate some kind of QOS here, e.g a request must be complete
> > > in some time. Otherwise we don't have sufficient reliability for using it as
> > > management task?
> > 
> > Yes, if all commands can be executed in bounded time then a guarantee is
> > possible.
> > 
> > Here is an example where that's hard: imagine a virtio-blk device backed
> > by network storage. When an admin queue command is used to delete a
> > group member, any of the group member's in-flight I/O requests need to
> > be aborted. If the network hangs while the group member is being
> > deleted, then the device can't complete an orderly shutdown of I/O
> > requests in a reasonable time.
> > 
> > That example shows a basic group admin command that I think Michael is
> > about to propose. We can't avoid this problem by not making it a group
> > admin command - it needs to be a group admin command. So I think it's
> > likely that there will be admin commands that take an unbounded amount
> > of time to complete. One way to achieve what you mentioned is timeouts.
> 
> I think that you're getting into device specific implementation details and
> I'm not sure it's necessary.
> 
> I don't think we need to abort admin commands. Admin commands can be
> flushed/aborted during the device reset phase.
> Only IO commands should have the possibility to being aborted as you
> mentioned in NVMe and SCSI (and potentially in virtio-blk).

It's a general design issue that should be clarified now rather than
being left unspecified.

I'm not saying that it must be possible to abort admin commands. There
are other options, like requiring the device itself to fail a command
after a timeout.

Or we could say that admin commands must complete within bounded time,
but I'm not sure that is implementable for some device types like
virtio-blk, virtio-scsi, and virtiofs.

> For your example, stopping a member is possible even it there are some
> errors in the network. You can for example destroy all the connections to
> the remote target and complete all the BIOS with some error.

Forgetting about in-flight requests doesn't necessarily make them go
away. It creates a race between forgotten requests and reconnection. In
the worst case a forgotten write request takes effect after
reconnection, causing data corruption.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Jiri Pirko]

Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:

[...]


>  admin: introduce device group and related concepts
>  admin: introduce group administration commands
>  admin: introduce virtio admin virtqueues

The ordering seems a bit odd to me. I think that at the beginning, the
admin virtqueue should be introduced with generic concepts (including
the generic command structure), only after that the section introducing
device group stuff and related commands.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 02/10] admin: introduce device group and related concepts

[David Edmondson]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> Each device group has a type. For now, define one initial group:

"one initial type"

> SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
> PCI SR-IOV physical function (PF). This group may contain zero or more
> virtio devices according to NumVFs configured.
>
> Each device within a group has a unique identifier. This identifier
> is the group member identifier.
>
> Note: one can argue both ways whether the new device group handling
> functionality (this and following patches) is closer
> to a new device type or a new transport type.
>
> However, I expect that we will add more features in the near future. To
> facilitate this as much as possible of the text is located in the new
> admin chapter.
>
> I did my best to minimize transport-specific text.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex |  2 ++
>  2 files changed, 51 insertions(+)
>  create mode 100644 admin.tex
>
> diff --git a/admin.tex b/admin.tex
> new file mode 100644
> index 0000000..3dc47be
> --- /dev/null
> +++ b/admin.tex
> @@ -0,0 +1,49 @@
> +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
> +
> +It is occasionally useful to have a device control a group of
> +other devices. Terminology used in such cases:
> +
> +\begin{description}
> +\item[Device group]
> +        or just group, includes zero or more devices.
> +\item[Owner device]
> +        or owner, the device controlling the group.
> +\item[Member device]
> +        a device within a group. The owner device itself is not
> +	a member of the group.
> +\item[Member identifier]
> +        each member has this identifier, unique within the group
> +	and used to address it through the owner device.
> +\item[Group type identifier]
> +	specifies what kind of member devices there are in a
> +	group, how is the member identifier is interpreted

s/how is/how/

> +	and what kind of control the owner has.
> +	A given owner can control a single group of a given type,
> +	thus the type and the owner together identify the group.

If a given owner can control groups of different types (which is implied
by the above?), perhaps worth being explicit that this is the case, or
not if that is not the intention.

> +	\footnote{Even though some group types only support
> +			specific transports, group type identifiers
> +			are global rather than transport-specific -
> +			we don't expect a flood of new group types.}
> +\end{description}
> +
> +The following group types, and their identifiers, are currently specified):

Spurious close parenthesis.

> +\begin{description}
> +\item[SR-IOV group type (0x1)]
> +This device group has a PCI Single Root I/O Virtualization
> +(SR-IOV) physical function (PF) device as the owner and includes
> +all its SR-IOV virtual functions (VFs) as members (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +The PF device itself is not a member of the group.
> +
> +The group type identifier for this group is 0x1.
> +
> +A member identifier for this group can have a value from 0x1 to 0xFFFF
> +and equals the SR-IOV VF number of the member device (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +Both owner and member devices for this group type use the Virtio
> +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> +\end{description}
> +
> +
> diff --git a/content.tex b/content.tex
> index 0c2d917..ffe45c4 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  types. It is RECOMMENDED that devices generate version 4
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>  
> +\input{admin.tex}
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
>  We start with an overview of device initialization, then expand on the
> -- 
> MST
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that 
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > What happens if a command takes 1 second to complete, is the device
> > > > > allowed to process the next command from the virtqueue during this time,
> > > > > possibly completing it before the first command?
> > > > > 
> > > > > This requires additional clarification in the spec because "they are
> > > > > processed by the device in the order in which they are queued" does not
> > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > not (out of order completion).
> > > > 
> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > 
> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > Several may be processed by the device at the same time.
> > 
> > Let's say I submit a write followed by read - is read
> > guaranteed to return an up to date info?
> 
> In general, no. The driver must wait for the write completion before
> submitting the read if it wants consistency.
> 
> Stefan

I see.  I think it's a good design to follow then.

I'll just copy
	The driver queues requests to an arbitrary request queue, and
	they are used by the device on that same queue. It is the
	responsibility of the driver to ensure strict request ordering
	for commands placed on different queues, because they will be
	consumed with no order constraints.

replacing "request" with "admin".

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 01:41:30PM +0100, Jiri Pirko wrote:
> Thu, Mar 02, 2023 at 02:05:06PM CET, mst@redhat.com wrote:
> >The admin virtqueues will be the first interface to issue admin commands.
> >
> >Currently virtio specification defines control virtqueue to manipulate
> >features and configuration of the device it operates on. However,
> >control virtqueue commands are device type specific, which makes it very
> >difficult to extend for device agnostic commands.
> >
> >To support this requirement in a more generic way, this patch introduces
> >a new admin virtqueue interface.
> >
> >We also support more than one admin virtqueue, for QoS and
> >scalability requirements.
> >
> >Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> >Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >---
> > admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > content.tex |  7 +++--
> > 2 files changed, 79 insertions(+), 2 deletions(-)
> >
> >diff --git a/admin.tex b/admin.tex
> >index 7e28b77..3ffac2e 100644
> >--- a/admin.tex
> >+++ b/admin.tex
> >@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > \field{command_specific_data} and \field{command_specific_result}
> > depends on these structures and is described separately or is
> > implicit in the structure description.
> >+
> >+\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >+
> >+An administration virtqueue of an owner device is used to submit
> >+group administration commands. An owner device can have more
> 
> I admit I'm confused. You introduce a concept of admin virtqueue, which
> sounds quite generic to me, usable in future by much more things than
> "device groups", correct?
> 
> If yes, here you say "group administration commands" which contradics
> that idea. On multiple places the text this patchset introduces
> this very muych tights to groups. Like in struct virtio_admin_cmd
> which contains fields very specific to groups.
> 
> If no, isn't it a mistake as the admin queue should be here to
> handle more than just group commands?

For now, no.

Passing commands to devices themselves is already covered in spec
reasonably well though not in a generic way.

What we lack is passing commands about one device to another device.
E.g. control VFs through PFs.
This is what groups do.
But if we see more uses we can always add them.


I'd rather avoid being too generic though.




> 
> >+than one administration virtqueue.
> >+
> >+If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
> >+or more adminstration virtqueues. The number and locations of the
> >+administration virtqueues are exposed by the owner device in a transport
> >+specific manner.
> >+
> >+The driver submits commands by queueing them to an arbitrary
> >+administration virtqueue, and they are processed by the
> >+device in the order in which they are queued. It is the
> >+responsibility of the driver to ensure ordering for commands
> >+placed on different administration virtqueues, because they will
> >+be executed with no order constraints.
> >+
> >+Administration virtqueues are used as follows:
> >+\begin{itemize}
> >+\item The driver submits the command using the \field{struct virtio_admin_cmd}
> >+structure using a buffer consisting of two parts: a device-readable one followed by a
> >+device-writable one.
> >+\item the device-readable part includes fields from \field{opcode}
> >+through \field{command_specific_data}.
> >+\item the device-writeable buffer includes fields from \field{status}
> >+through \field{command_specific_result} inclusive.
> >+\end{itemize}
> >+
> >+For each command, this specification describes a distinct
> >+format structure used for \field{command_specific_data} and
> >+\field{command_specific_result}, the length of these fields
> >+depends on the command.
> >+
> >+However, to ensure forward compatibility
> >+\begin{itemize}
> >+\item drivers are allowed to submit buffers that are longer
> >+than what the device expects
> >+(that is, longer than the length of
> >+\field{opcode} through \field{command_specific_data}).
> >+This allows the driver to maintain
> >+a single format structure even if some structure fields are
> >+unused by the device.
> >+\item drivers are allowed to submit buffers that are shorter
> >+than what the device expects
> >+(that is, shorter than the length of \field{status} through
> >+\field{command_specific_result}). This allows the device to maintain
> >+a single format structure even if some structure fields are
> >+unused by the driver.
> >+\end{itemize}
> >+
> >+The device compares the length of each part (device-readable and
> >+device-writeable) of the buffer as submitted by driver to what it
> >+expects and then silently truncates the structures to either the
> >+length submitted by the driver, or the length described in this
> >+specification, whichever is shorter.  The device silently ignores
> >+any data falling outside the shorter of the two lengths. Any
> >+missing fields are interpreted as set to zero.
> >+
> >+Similarly, the driver compares the used buffer length
> >+of the buffer to what it expects and then silently
> >+truncates the structure to the used buffer length.
> >+The driver silently ignores any data falling outside
> >+the used buffer length reported by the device.  Any missing
> >+fields are interpreted as set to zero.
> >+
> >+This simplifies driver and device implementations since the
> >+driver/device can simply maintain a single large structure (such
> >+as a C structure) for a command and its result. As new versions
> >+of the specification are designed, new fields can be added to the
> >+tail of a structure, with the driver/device using the full
> >+structure without concern for versioning.
> >+>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
> 
> Conflict resolution leftover.
> 
> 
> >diff --git a/content.tex b/content.tex
> >index ffe45c4..c8647c9 100644
> >--- a/content.tex
> >+++ b/content.tex
> >@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
> > \begin{description}
> > \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> > 
> >-\item[24 to 40] Feature bits reserved for extensions to the queue and
> >+\item[24 to 41] Feature bits reserved for extensions to the queue and
> >   feature negotiation mechanisms
> > 
> >-\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
> >+\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> > \end{description}
> > 
> > \begin{note}
> >@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >   that the driver can reset a queue individually.
> >   See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
> > 
> >+  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
> >+  administration virtqueues.
> >+
> > \end{description}
> > 
> > \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> >-- 
> >MST
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe from this mail list, you must leave the OASIS TC that 
> >generates this mail.  Follow this link to all your TCs in OASIS at:
> >https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 05:37:09PM +0200, Max Gurtovoy wrote:
> 
> 
> On 06/03/2023 14:41, Jiri Pirko wrote:
> > Thu, Mar 02, 2023 at 02:05:06PM CET, mst@redhat.com wrote:
> > > The admin virtqueues will be the first interface to issue admin commands.
> > > 
> > > Currently virtio specification defines control virtqueue to manipulate
> > > features and configuration of the device it operates on. However,
> > > control virtqueue commands are device type specific, which makes it very
> > > difficult to extend for device agnostic commands.
> > > 
> > > To support this requirement in a more generic way, this patch introduces
> > > a new admin virtqueue interface.
> > > 
> > > We also support more than one admin virtqueue, for QoS and
> > > scalability requirements.
> > > 
> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > ---
> > > admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > content.tex |  7 +++--
> > > 2 files changed, 79 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/admin.tex b/admin.tex
> > > index 7e28b77..3ffac2e 100644
> > > --- a/admin.tex
> > > +++ b/admin.tex
> > > @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > \field{command_specific_data} and \field{command_specific_result}
> > > depends on these structures and is described separately or is
> > > implicit in the structure description.
> > > +
> > > +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > +
> > > +An administration virtqueue of an owner device is used to submit
> > > +group administration commands. An owner device can have more
> > 
> > I admit I'm confused. You introduce a concept of admin virtqueue, which
> > sounds quite generic to me, usable in future by much more things than
> > "device groups", correct?
> > 
> > If yes, here you say "group administration commands" which contradics
> > that idea. On multiple places the text this patchset introduces
> > this very muych tights to groups. Like in struct virtio_admin_cmd
> > which contains fields very specific to groups.
> > 
> > If no, isn't it a mistake as the admin queue should be here to
> > handle more than just group commands?
> 
> This was my original idea to have various different types of admin
> operations allowed by an AQ. Not only group management.
> Somehow this was changed down the road when MST took ownership :)
> Maybe it's fine as an initial start but this shouldn't restrict the general
> idea.
> 
> I had in mind things like general health commands, log commands, fw upgrade
>

I'm not sure what if anything unifies these though.  I looked at
specific proposals on the table and this covered them, and
scope was already too broad and was not converging because of this.
People are not good at predicting the future, so I confidently
predict that we'll make some changes down the road ;)


-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 11:35:15AM +0100, Jiri Pirko wrote:
> Well, lots of these things (if not majority) are customary, not
> really written up anywhere in Linux kernel netdev world. It's most of
> the time just a matter of convenience I would say. For example
> splitted-up patch changelog. I do it, but most of the people don't (even
> maintainers). From time to time during review, I request it as I don't
> want to be lost in the changes. It is nice if the submitter tries to
> make reviewer's life a bit easier :)

True. With all this lore let's just agree netdev is not the easiest
project to get started with, right?

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2173 bytes --]

On Mon, Mar 06, 2023 at 01:37:31PM -0500, Michael S. Tsirkin wrote:
> On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> > On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> > > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> > > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > > What happens if a command takes 1 second to complete, is the device
> > > > > > allowed to process the next command from the virtqueue during this time,
> > > > > > possibly completing it before the first command?
> > > > > > 
> > > > > > This requires additional clarification in the spec because "they are
> > > > > > processed by the device in the order in which they are queued" does not
> > > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > > not (out of order completion).
> > > > > 
> > > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > > 
> > > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > > Several may be processed by the device at the same time.
> > > 
> > > Let's say I submit a write followed by read - is read
> > > guaranteed to return an up to date info?
> > 
> > In general, no. The driver must wait for the write completion before
> > submitting the read if it wants consistency.
> > 
> > Stefan
> 
> I see.  I think it's a good design to follow then.
> 
> I'll just copy
> 	The driver queues requests to an arbitrary request queue, and
> 	they are used by the device on that same queue. It is the
> 	responsibility of the driver to ensure strict request ordering
> 	for commands placed on different queues, because they will be
> 	consumed with no order constraints.
> 
> replacing "request" with "admin".

That sounds like it's only about multi-queue because it says "... for
commands placed on different queues". What I mentioned about a write
followed by a read quest also applies within a single queue.

Can you clarify the semantics in the single queue case?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 03:17:59PM -0500, Stefan Hajnoczi wrote:
> On Mon, Mar 06, 2023 at 01:37:31PM -0500, Michael S. Tsirkin wrote:
> > On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> > > On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> > > > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> > > > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > > > What happens if a command takes 1 second to complete, is the device
> > > > > > > allowed to process the next command from the virtqueue during this time,
> > > > > > > possibly completing it before the first command?
> > > > > > > 
> > > > > > > This requires additional clarification in the spec because "they are
> > > > > > > processed by the device in the order in which they are queued" does not
> > > > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > > > not (out of order completion).
> > > > > > 
> > > > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > > > 
> > > > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > > > Several may be processed by the device at the same time.
> > > > 
> > > > Let's say I submit a write followed by read - is read
> > > > guaranteed to return an up to date info?
> > > 
> > > In general, no. The driver must wait for the write completion before
> > > submitting the read if it wants consistency.
> > > 
> > > Stefan
> > 
> > I see.  I think it's a good design to follow then.
> > 
> > I'll just copy
> > 	The driver queues requests to an arbitrary request queue, and
> > 	they are used by the device on that same queue. It is the
> > 	responsibility of the driver to ensure strict request ordering
> > 	for commands placed on different queues, because they will be
> > 	consumed with no order constraints.
> > 
> > replacing "request" with "admin".
> 
> That sounds like it's only about multi-queue because it says "... for
> commands placed on different queues". What I mentioned about a write
> followed by a read quest also applies within a single queue.

Well that text is from the virtio scsi spec. turning it around does
it need clarification there?

> Can you clarify the semantics in the single queue case?
> 
> Stefan



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 01:53:50PM +0000, David Edmondson wrote:
> "Michael S. Tsirkin" <mst@redhat.com> writes:
> 
> > Feature negotiation forms the basis of forward compatibility
> > guarantees of virtio but has never been properly documented.
> > Do it now.
> >
> > Suggested-by: Halil Pasic <pasic@linux.ibm.com>
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 42 insertions(+)
> >
> > diff --git a/content.tex b/content.tex
> > index 0e474dd..0c2d917 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
> >  In particular, new fields in the device configuration space are
> >  indicated by offering a new feature bit.
> >  
> > +To keep the feature negotiation mechanism extensible, it is important
> > +that devices \em{do not} offer any feature bits that they would not be
> > +able to handle if the driver accepted them (even though drivers are not
> > +supposed to accept them in the first place even if offered, according to
> > +this version of the specification.) 
> 
> I find this (the bit in parenthesis) confusing.
> 
> Why are drivers not supposed to accept features that they have been
> offered, given that they can't know that the device cannot handle the
> feature that it just offered?
> 
> Is this alluding to the later section:
> 
> > feature bits not described in this specification, reserved feature
> > bits and feature bits reserved or not supported for the specific
> > transport or the specific device type
> 
> ?

exactly. how would you put this better? given an example?

> > Likewise, it is important that
> > +drivers \em{do not} accept feature bits they do not know how to handle
> > +(even though devices are not supposed to offer them in the first place,
> > +according to this version of the specification.) The preferred way for
> > +handling reserved and unexpected features is that the driver ignores
> > +them.
> > +
> > +In particular, this is
> > +especially important for features limited to specific transports,
> > +as enabling these for more transports in future versions of the
> > +specification is highly likely to require changing the behaviour
> > +from drivers and devices.
> 
> "require changed behaviour from drivers and devices".
> 
> > Drivers and devices supporting
> > +multiple transports need to carefully maintain per-transport
> > +lists of allowed features.
> > +
> >  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
> >  The driver MUST NOT accept a feature which the device did not offer,
> >  and MUST NOT accept a feature which requires another feature which was
> >  not accepted.
> >  
> > +The driver MUST validate the feature bits offered by the device.
> > +The driver MUST ignore and MUST NOT accept any feature bit that is
> > +\begin{itemize}
> > +\item not described in this specification,
> > +\item marked as reserved,
> > +\item not supported for the specific transport,
> > +\item not defined for the device type.
> > +\end{itemize}
> > +
> >  The driver SHOULD go into backwards compatibility mode
> >  if the device does not offer a feature it understands, otherwise MUST
> >  set the FAILED \field{device status} bit and cease initialization.
> >  
> > +By contrast, the driver MUST NOT fail solely because a feature
> > +it does not understand has been offered by the device.
> > +
> >  \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
> >  The device MUST NOT offer a feature which requires another feature
> >  which was not offered.  The device SHOULD accept any valid subset
> >  of features the driver accepts, otherwise it MUST fail to set the
> >  FEATURES_OK \field{device status} bit when the driver writes it.
> >  
> > +The device MUST NOT offer feature bits corresponding to features
> > +it would not support if accepted by the driver (even if the
> > +driver is prohibited from accepting the feature bits by the
> > +specification); for the sake of clarity, this refers to feature
> > +bits not described in this specification, reserved feature bits
> > +and feature bits reserved or not supported for the specific
> > +transport or the specific device type, but this does not preclude
> > +devices written to a future version of this specification from
> > +offering such feature bits should such a specification have a
> > +provision for devices to support the corresponding features.
> > +
> >  If a device has successfully negotiated a set of features
> >  at least once (by accepting the FEATURES_OK \field{device
> >  status} bit during device initialization), then it SHOULD
> > -- 
> > MST
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe from this mail list, you must leave the OASIS TC that 
> > generates this mail.  Follow this link to all your TCs in OASIS at:
> > https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 05:40:04PM +0100, Jiri Pirko wrote:
> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
> 
> [...]
> 
> 
> >  admin: introduce device group and related concepts
> >  admin: introduce group administration commands
> >  admin: introduce virtio admin virtqueues
> 
> The ordering seems a bit odd to me. I think that at the beginning, the
> admin virtqueue should be introduced with generic concepts (including
> the generic command structure), only after that the section introducing
> device group stuff and related commands.

That's intentional.

Jason is working on additional ways to pass commands to
devices, using memory mapped registers.

Thus admin commands stand by themselves, and the queue is
just a way to issue them.



> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 05:14:09PM +0200, Max Gurtovoy wrote:
> > > This would allow pipelining
> > > dependent commands but prevent long-running commands because the
> > > virtqueue is blocked while executing a command.
> > 
> > we have multiple vqs for that.
> 
> why do we need multiple admin vqs ?

You are literally replying to a discussion with a use-case.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 01:29:30PM +0100, Jiri Pirko wrote:
> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
> 
> [...]
> 
> >
> >TODO (maybe?) - probably ok to defer until this part is upstream:
> >
> >	Add "all members" member id.
> >
> >	Add commands for MSI, feature discovery.
> >
> >	Add commands for transport vq.
> >
> >
> >My intent is to try and support both SR-IOV and SIOV
> >usecases with the same structure and maybe even the same
> >VQ.
> 
> Sorry to be late to the party, I'm trying to catch up. Is it common to
> have cover letter for features this brief? I mean, from the cover
> letter, I'm totally unable to understand what you are introducing here.
> 
> Could you elaborate about what you are aiming to achive with this?
> Could you shed some usecases perhaps?
> 
> I have to be missing something obvious, but I don't get why any notion
> of SR-IOV could be beneficial for virtio.
> 

Good point, I'll add a bit of motivation.

For SR-IOV, it is not unusual for PFs to excercise control over VFs.
There is interest in the community to include an interface to allow this
in the virtio spec, when the PF is a virtio device.  This is what this
patch does.




> >
> >For example, it might make sense to split creating/destroying
> >SIOV devices from the transport passing data from the guest - the
> >driver would then not negotiate VIRTIO_F_SR_IOV (which
> >then means auto-provisioning).
> >
> >This is out of RFC, since we have two commands which are useful
> >to discover supported group types (ATM can be none or SR-IOV).
> >
> >
> >Michael S. Tsirkin (10):
> >  virtio: document forward compatibility guarantees
> >  admin: introduce device group and related concepts
> >  admin: introduce group administration commands
> >  admin: introduce virtio admin virtqueues
> >  pci: add admin vq registers to virtio over pci
> >  mmio: document ADMIN_VQ as reserved
> >  ccw: document ADMIN_VQ as reserved
> >  admin: command list discovery
> >  admin: conformance clauses
> >  ccw: document more reserved features
> >
> > admin.tex        | 540 +++++++++++++++++++++++++++++++++++++++++++++++
> > content.tex      | 112 +++++++++-
> > introduction.tex |   3 +
> > 3 files changed, 653 insertions(+), 2 deletions(-)
> > create mode 100644 admin.tex
> >
> >-- 
> >MST
> >
> >
> >This publicly archived list offers a means to provide input to the
> >OASIS Virtual I/O Device (VIRTIO) TC.
> >
> >In order to verify user consent to the Feedback License terms and
> >to minimize spam in the list archive, subscription is required
> >before posting.
> >
> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >List help: virtio-comment-help@lists.oasis-open.org
> >List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >Committee: https://www.oasis-open.org/committees/virtio/
> >Join OASIS: https://www.oasis-open.org/join/
> >
> 
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Jiri Pirko]

Mon, Mar 06, 2023 at 11:48:32PM CET, mst@redhat.com wrote:
>On Mon, Mar 06, 2023 at 05:40:04PM +0100, Jiri Pirko wrote:
>> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
>> 
>> [...]
>> 
>> 
>> >  admin: introduce device group and related concepts
>> >  admin: introduce group administration commands
>> >  admin: introduce virtio admin virtqueues
>> 
>> The ordering seems a bit odd to me. I think that at the beginning, the
>> admin virtqueue should be introduced with generic concepts (including
>> the generic command structure), only after that the section introducing
>> device group stuff and related commands.
>
>That's intentional.
>
>Jason is working on additional ways to pass commands to
>devices, using memory mapped registers.
>
>Thus admin commands stand by themselves, and the queue is
>just a way to issue them.

Okay, in that case, woudnlt it make sense if the generic CMD text
(including the generic command structure) would go first, then the
"group" related cmds?


>
>
>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>> 
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>> 
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Jiri Pirko]

Mon, Mar 06, 2023 at 11:54:45PM CET, mst@redhat.com wrote:
>On Mon, Mar 06, 2023 at 01:29:30PM +0100, Jiri Pirko wrote:
>> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
>> 
>> [...]
>> 
>> >
>> >TODO (maybe?) - probably ok to defer until this part is upstream:
>> >
>> >	Add "all members" member id.
>> >
>> >	Add commands for MSI, feature discovery.
>> >
>> >	Add commands for transport vq.
>> >
>> >
>> >My intent is to try and support both SR-IOV and SIOV
>> >usecases with the same structure and maybe even the same
>> >VQ.
>> 
>> Sorry to be late to the party, I'm trying to catch up. Is it common to
>> have cover letter for features this brief? I mean, from the cover
>> letter, I'm totally unable to understand what you are introducing here.
>> 
>> Could you elaborate about what you are aiming to achive with this?
>> Could you shed some usecases perhaps?
>> 
>> I have to be missing something obvious, but I don't get why any notion
>> of SR-IOV could be beneficial for virtio.
>> 
>
>Good point, I'll add a bit of motivation.
>
>For SR-IOV, it is not unusual for PFs to excercise control over VFs.

I understand the concepts of SR-IOV. Yet I fail to see the need of such
concept in virtio world. SR-IOV is very specific solution for PCI
functions instantiation, and I believe that it is already considered
quite limiting in many aspects. Does not make sense to me to introduce
it for virtio. But again, I may be missing something crucial, I just
would like to see the motivation, needs, usecases for this crystal
clear, which is opposite to the current cover letter I'm afraid :/


>There is interest in the community to include an interface to allow this
>in the virtio spec, when the PF is a virtio device.  This is what this
>patch does.

Yeah, but why? As I asked before, what are the usecases? The fact there
is interest in the community does not mean it makes sense to have it :)


>
>
>
>
>> >
>> >For example, it might make sense to split creating/destroying
>> >SIOV devices from the transport passing data from the guest - the
>> >driver would then not negotiate VIRTIO_F_SR_IOV (which
>> >then means auto-provisioning).
>> >
>> >This is out of RFC, since we have two commands which are useful
>> >to discover supported group types (ATM can be none or SR-IOV).
>> >
>> >
>> >Michael S. Tsirkin (10):
>> >  virtio: document forward compatibility guarantees
>> >  admin: introduce device group and related concepts
>> >  admin: introduce group administration commands
>> >  admin: introduce virtio admin virtqueues
>> >  pci: add admin vq registers to virtio over pci
>> >  mmio: document ADMIN_VQ as reserved
>> >  ccw: document ADMIN_VQ as reserved
>> >  admin: command list discovery
>> >  admin: conformance clauses
>> >  ccw: document more reserved features
>> >
>> > admin.tex        | 540 +++++++++++++++++++++++++++++++++++++++++++++++
>> > content.tex      | 112 +++++++++-
>> > introduction.tex |   3 +
>> > 3 files changed, 653 insertions(+), 2 deletions(-)
>> > create mode 100644 admin.tex
>> >
>> >-- 
>> >MST
>> >
>> >
>> >This publicly archived list offers a means to provide input to the
>> >OASIS Virtual I/O Device (VIRTIO) TC.
>> >
>> >In order to verify user consent to the Feedback License terms and
>> >to minimize spam in the list archive, subscription is required
>> >before posting.
>> >
>> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> >List help: virtio-comment-help@lists.oasis-open.org
>> >List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> >Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> >List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> >Committee: https://www.oasis-open.org/committees/virtio/
>> >Join OASIS: https://www.oasis-open.org/join/
>> >
>> 
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>> 
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>> 
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Mon, Mar 06, 2023 at 07:40:38PM CET, mst@redhat.com wrote:
>On Mon, Mar 06, 2023 at 01:41:30PM +0100, Jiri Pirko wrote:
>> Thu, Mar 02, 2023 at 02:05:06PM CET, mst@redhat.com wrote:
>> >The admin virtqueues will be the first interface to issue admin commands.
>> >
>> >Currently virtio specification defines control virtqueue to manipulate
>> >features and configuration of the device it operates on. However,
>> >control virtqueue commands are device type specific, which makes it very
>> >difficult to extend for device agnostic commands.
>> >
>> >To support this requirement in a more generic way, this patch introduces
>> >a new admin virtqueue interface.
>> >
>> >We also support more than one admin virtqueue, for QoS and
>> >scalability requirements.
>> >
>> >Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> >Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> >---
>> > admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > content.tex |  7 +++--
>> > 2 files changed, 79 insertions(+), 2 deletions(-)
>> >
>> >diff --git a/admin.tex b/admin.tex
>> >index 7e28b77..3ffac2e 100644
>> >--- a/admin.tex
>> >+++ b/admin.tex
>> >@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>> > \field{command_specific_data} and \field{command_specific_result}
>> > depends on these structures and is described separately or is
>> > implicit in the structure description.
>> >+
>> >+\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>> >+
>> >+An administration virtqueue of an owner device is used to submit
>> >+group administration commands. An owner device can have more
>> 
>> I admit I'm confused. You introduce a concept of admin virtqueue, which
>> sounds quite generic to me, usable in future by much more things than
>> "device groups", correct?
>> 
>> If yes, here you say "group administration commands" which contradics
>> that idea. On multiple places the text this patchset introduces
>> this very muych tights to groups. Like in struct virtio_admin_cmd
>> which contains fields very specific to groups.
>> 
>> If no, isn't it a mistake as the admin queue should be here to
>> handle more than just group commands?
>
>For now, no.

Hmm, if not for now, the future exension would not be so simple, I fear.


>
>Passing commands to devices themselves is already covered in spec
>reasonably well though not in a generic way.

You mean using the control queue, correct?

From one of the patch description of this patchset I understand that you
cannot use control queue for this because control queue is
device-specific, yet group control is device-agnostic.

My undestanding therefore was, that the admin queue you are introducing
serves as a generic carrier for device-agnostic commands, in parallel
for having control queue serving as a generic carrier of device-specific
commands. If this is not the case, I think it would be nice to describe
the exact monivation and scope of admin queue.


>
>What we lack is passing commands about one device to another device.
>E.g. control VFs through PFs.

Could you provide examples of such commands please?


>This is what groups do.
>But if we see more uses we can always add them.
>
>
>I'd rather avoid being too generic though.

In that case, why not to avoid using generic terms and stay
"group-centric"? What I mean is:
"Administration Virtqueues" -> "Group Administration Virtqueues"
"struct virtio_admin_cmd" -> "struct virtio_group_admin_cmd"

Etc. Helps to avoid confusion.


>
>
>
>
>> 
>> >+than one administration virtqueue.


[...]



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Jiri Pirko]

Mon, Mar 06, 2023 at 07:46:09PM CET, mst@redhat.com wrote:
>On Mon, Mar 06, 2023 at 11:35:15AM +0100, Jiri Pirko wrote:
>> Well, lots of these things (if not majority) are customary, not
>> really written up anywhere in Linux kernel netdev world. It's most of
>> the time just a matter of convenience I would say. For example
>> splitted-up patch changelog. I do it, but most of the people don't (even
>> maintainers). From time to time during review, I request it as I don't
>> want to be lost in the changes. It is nice if the submitter tries to
>> make reviewer's life a bit easier :)
>
>True. With all this lore let's just agree netdev is not the easiest
>project to get started with, right?

No clue, I'm soaked in for too long already :)


>
>-- 
>MST
>
>
>This publicly archived list offers a means to provide input to the
>OASIS Virtual I/O Device (VIRTIO) TC.
>
>In order to verify user consent to the Feedback License terms and
>to minimize spam in the list archive, subscription is required
>before posting.
>
>Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>List help: virtio-comment-help@lists.oasis-open.org
>List archive: https://lists.oasis-open.org/archives/virtio-comment/
>Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>Committee: https://www.oasis-open.org/committees/virtio/
>Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Mon, Mar 06, 2023 at 04:14:09PM CET, mgurtovoy@nvidia.com wrote:
>
>
>On 03/03/2023 2:05, Michael S. Tsirkin wrote:
>> On Thu, Mar 02, 2023 at 03:40:07PM -0500, Stefan Hajnoczi wrote:
>> > On Thu, Mar 02, 2023 at 08:05:06AM -0500, Michael S. Tsirkin wrote:
>> > > The admin virtqueues will be the first interface to issue admin commands.
>> > > 
>> > > Currently virtio specification defines control virtqueue to manipulate
>> > > features and configuration of the device it operates on. However,
>> > > control virtqueue commands are device type specific, which makes it very
>> > > difficult to extend for device agnostic commands.
>> > > 
>> > > To support this requirement in a more generic way, this patch introduces
>> > > a new admin virtqueue interface.
>> > 
>> > Is this referring to the existing virtio-net, virtio-scsi, etc control
>> > virtqueues?
>> > 
>> > I see the admin virtqueue as the virtqueue equivalent to transport
>> > feature bits. The admin queue does nothing device type-specific (net,
>> > scsi, etc) and instead focusses on transport and core virtio tasks.
>> > 
>> > Keeping the device-specific virtqueue separate from the admin virtqueue
>> > is simpler and has fewer potential problems. I don't think creating
>> > common infrastructure for device-specific control virtqueues across
>> > device types worthwhile or within the scope of this patch series.
>> 
>> yes this commit log is outdated. referred to original
>> proposal by Max which also planned to replace cvq.
>> will update.
>> 
>> 
>> > > We also support more than one admin virtqueue, for QoS and
>> > > scalability requirements.
>> > > 
>> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> > > ---
>> > >   admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >   content.tex |  7 +++--
>> > >   2 files changed, 79 insertions(+), 2 deletions(-)
>> > > 
>> > > diff --git a/admin.tex b/admin.tex
>> > > index 7e28b77..3ffac2e 100644
>> > > --- a/admin.tex
>> > > +++ b/admin.tex
>> > > @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>> > >   \field{command_specific_data} and \field{command_specific_result}
>> > >   depends on these structures and is described separately or is
>> > >   implicit in the structure description.
>> > > +
>> > > +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>> > > +
>> > > +An administration virtqueue of an owner device is used to submit
>> > > +group administration commands. An owner device can have more
>> > > +than one administration virtqueue.
>> > > +
>> > > +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
>> > > +or more adminstration virtqueues. The number and locations of the
>> > > +administration virtqueues are exposed by the owner device in a transport
>> > > +specific manner.
>> > > +
>> > > +The driver submits commands by queueing them to an arbitrary
>> > > +administration virtqueue, and they are processed by the
>> > > +device in the order in which they are queued. It is the
>> > > +responsibility of the driver to ensure ordering for commands
>> > > +placed on different administration virtqueues, because they will
>> > > +be executed with no order constraints.
>> > 
>> > Does "they are processed by the device in the order in which they are
>> > queued" mean only 1 admin command can be running at any given time and
>> > therefore they will complete in order? This would allow pipelining
>> > dependent commands but prevent long-running commands because the
>> > virtqueue is blocked while executing a command.
>> 
>> we have multiple vqs for that.
>
>why do we need multiple admin vqs ?
>This is not performance oriented queue.
>I don't think we need that.
>I introduced F_AQ_IN_ORDER feature bit to distinguish the in_order logic
>(same as we have for data queues).
>Please re-think using this instead of complicating ourselves to multi AQ
>specification.

Well, out of order processing is parallel to multiple queues isn't it?
The only way to achieve that would require to have the same about of
queues as possible parallel out of order commands, which does not
make sense, does it?


>
>We have a single ctrl vq so no need to introduce multiple admin vqs...
>
>> 
>> > > +
>> > > +Administration virtqueues are used as follows:
>> > > +\begin{itemize}
>> > > +\item The driver submits the command using the \field{struct virtio_admin_cmd}
>> > > +structure using a buffer consisting of two parts: a device-readable one followed by a
>> > > +device-writable one.
>> > > +\item the device-readable part includes fields from \field{opcode}
>> > > +through \field{command_specific_data}.
>> > > +\item the device-writeable buffer includes fields from \field{status}
>> > > +through \field{command_specific_result} inclusive.
>> > > +\end{itemize}
>> > > +
>> > > +For each command, this specification describes a distinct
>> > > +format structure used for \field{command_specific_data} and
>> > > +\field{command_specific_result}, the length of these fields
>> > > +depends on the command.
>> > > +
>> > > +However, to ensure forward compatibility
>> > > +\begin{itemize}
>> > > +\item drivers are allowed to submit buffers that are longer
>> > > +than what the device expects
>> > > +(that is, longer than the length of
>> > > +\field{opcode} through \field{command_specific_data}).
>> > > +This allows the driver to maintain
>> > > +a single format structure even if some structure fields are
>> > > +unused by the device.
>> > > +\item drivers are allowed to submit buffers that are shorter
>> > > +than what the device expects
>> > > +(that is, shorter than the length of \field{status} through
>> > > +\field{command_specific_result}). This allows the device to maintain
>> > > +a single format structure even if some structure fields are
>> > > +unused by the driver.
>> > > +\end{itemize}
>> > > +
>> > > +The device compares the length of each part (device-readable and
>> > > +device-writeable) of the buffer as submitted by driver to what it
>> > > +expects and then silently truncates the structures to either the
>> > > +length submitted by the driver, or the length described in this
>> > > +specification, whichever is shorter.  The device silently ignores
>> > > +any data falling outside the shorter of the two lengths. Any
>> > > +missing fields are interpreted as set to zero.
>> > > +
>> > > +Similarly, the driver compares the used buffer length
>> > > +of the buffer to what it expects and then silently
>> > > +truncates the structure to the used buffer length.
>> > > +The driver silently ignores any data falling outside
>> > > +the used buffer length reported by the device.  Any missing
>> > > +fields are interpreted as set to zero.
>> > > +
>> > > +This simplifies driver and device implementations since the
>> > > +driver/device can simply maintain a single large structure (such
>> > > +as a C structure) for a command and its result. As new versions
>> > > +of the specification are designed, new fields can be added to the
>> > > +tail of a structure, with the driver/device using the full
>> > > +structure without concern for versioning.
>> > > +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
>> > 
>> > A merge conflict line crept into the patch?
>> 
>> yes. I fixed it I thought but somehow it's still there :(
>> 
>> > > diff --git a/content.tex b/content.tex
>> > > index ffe45c4..c8647c9 100644
>> > > --- a/content.tex
>> > > +++ b/content.tex
>> > > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>> > >   \begin{description}
>> > >   \item[0 to 23, and 50 to 127] Feature bits for the specific device type
>> > > -\item[24 to 40] Feature bits reserved for extensions to the queue and
>> > > +\item[24 to 41] Feature bits reserved for extensions to the queue and
>> > >     feature negotiation mechanisms
>> > > -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
>> > > +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
>> > >   \end{description}
>> > >   \begin{note}
>> > > @@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>> > >     that the driver can reset a queue individually.
>> > >     See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>> > > +  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
>> > > +  administration virtqueues.
>> > > +
>> > >   \end{description}
>> > >   \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
>> > > -- 
>> > > MST
>> > > 
>> 
>> 
>
>This publicly archived list offers a means to provide input to the
>OASIS Virtual I/O Device (VIRTIO) TC.
>
>In order to verify user consent to the Feedback License terms and
>to minimize spam in the list archive, subscription is required
>before posting.
>
>Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>List help: virtio-comment-help@lists.oasis-open.org
>List archive: https://lists.oasis-open.org/archives/virtio-comment/
>Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>Committee: https://www.oasis-open.org/committees/virtio/
>Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
>On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
>> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
>> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
>> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
>> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
>> > > > > What happens if a command takes 1 second to complete, is the device
>> > > > > allowed to process the next command from the virtqueue during this time,
>> > > > > possibly completing it before the first command?
>> > > > > 
>> > > > > This requires additional clarification in the spec because "they are
>> > > > > processed by the device in the order in which they are queued" does not
>> > > > > explain whether commands block the virtqueue (in order completion) or
>> > > > > not (out of order completion).
>> > > > 
>> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
>> > > 
>> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
>> > > Several may be processed by the device at the same time.
>> > 
>> > Let's say I submit a write followed by read - is read
>> > guaranteed to return an up to date info?
>> 
>> In general, no. The driver must wait for the write completion before
>> submitting the read if it wants consistency.
>> 
>> Stefan
>
>I see.  I think it's a good design to follow then.

Hmm, is it suitable to have this approach for configuration interface?
Storage device is a different beast, having parallel reads and writes
makes complete sense for performance.

->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c rep

There is no dependency, even between writes.

But in case of configuration, does not make any sense to me.
Why is it needed? To pass the burden of consistency of
configuration to driver sounds odd at least.

I sense there is no concete idea about what the "admin virtqueue" should
serve for exactly.



>
>I'll just copy
>	The driver queues requests to an arbitrary request queue, and
>	they are used by the device on that same queue. It is the
>	responsibility of the driver to ensure strict request ordering
>	for commands placed on different queues, because they will be
>	consumed with no order constraints.
>
>replacing "request" with "admin".
>
>-- 
>MST
>
>
>This publicly archived list offers a means to provide input to the
>OASIS Virtual I/O Device (VIRTIO) TC.
>
>In order to verify user consent to the Feedback License terms and
>to minimize spam in the list archive, subscription is required
>before posting.
>
>Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>List help: virtio-comment-help@lists.oasis-open.org
>List archive: https://lists.oasis-open.org/archives/virtio-comment/
>Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>Committee: https://www.oasis-open.org/committees/virtio/
>Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[David Edmondson]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> On Mon, Mar 06, 2023 at 01:53:50PM +0000, David Edmondson wrote:
>> "Michael S. Tsirkin" <mst@redhat.com> writes:
>> 
>> > Feature negotiation forms the basis of forward compatibility
>> > guarantees of virtio but has never been properly documented.
>> > Do it now.
>> >
>> > Suggested-by: Halil Pasic <pasic@linux.ibm.com>
>> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> > ---
>> >  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
>> >  1 file changed, 42 insertions(+)
>> >
>> > diff --git a/content.tex b/content.tex
>> > index 0e474dd..0c2d917 100644
>> > --- a/content.tex
>> > +++ b/content.tex
>> > @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>> >  In particular, new fields in the device configuration space are
>> >  indicated by offering a new feature bit.
>> >  
>> > +To keep the feature negotiation mechanism extensible, it is important
>> > +that devices \em{do not} offer any feature bits that they would not be
>> > +able to handle if the driver accepted them (even though drivers are not
>> > +supposed to accept them in the first place even if offered, according to
>> > +this version of the specification.) 
>> 
>> I find this (the bit in parenthesis) confusing.
>> 
>> Why are drivers not supposed to accept features that they have been
>> offered, given that they can't know that the device cannot handle the
>> feature that it just offered?
>> 
>> Is this alluding to the later section:
>> 
>> > feature bits not described in this specification, reserved feature
>> > bits and feature bits reserved or not supported for the specific
>> > transport or the specific device type
>> 
>> ?
>
> exactly. how would you put this better? given an example?

Perhaps it would be enough to say:

> (even though drivers are not supposed to accept unrecognised features in
> the first place even if offered, according to the specification)

"Unrecognised" is intended as a shorthand for the whole "not described,
reserved, ...". Maybe "unrecognised or reserved"?

>> > Likewise, it is important that
>> > +drivers \em{do not} accept feature bits they do not know how to handle
>> > +(even though devices are not supposed to offer them in the first place,
>> > +according to this version of the specification.) The preferred way for
>> > +handling reserved and unexpected features is that the driver ignores
>> > +them.
>> > +
>> > +In particular, this is
>> > +especially important for features limited to specific transports,
>> > +as enabling these for more transports in future versions of the
>> > +specification is highly likely to require changing the behaviour
>> > +from drivers and devices.
>> 
>> "require changed behaviour from drivers and devices".
>> 
>> > Drivers and devices supporting
>> > +multiple transports need to carefully maintain per-transport
>> > +lists of allowed features.
>> > +
>> >  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
>> >  The driver MUST NOT accept a feature which the device did not offer,
>> >  and MUST NOT accept a feature which requires another feature which was
>> >  not accepted.
>> >  
>> > +The driver MUST validate the feature bits offered by the device.
>> > +The driver MUST ignore and MUST NOT accept any feature bit that is
>> > +\begin{itemize}
>> > +\item not described in this specification,
>> > +\item marked as reserved,
>> > +\item not supported for the specific transport,
>> > +\item not defined for the device type.
>> > +\end{itemize}
>> > +
>> >  The driver SHOULD go into backwards compatibility mode
>> >  if the device does not offer a feature it understands, otherwise MUST
>> >  set the FAILED \field{device status} bit and cease initialization.
>> >  
>> > +By contrast, the driver MUST NOT fail solely because a feature
>> > +it does not understand has been offered by the device.
>> > +
>> >  \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
>> >  The device MUST NOT offer a feature which requires another feature
>> >  which was not offered.  The device SHOULD accept any valid subset
>> >  of features the driver accepts, otherwise it MUST fail to set the
>> >  FEATURES_OK \field{device status} bit when the driver writes it.
>> >  
>> > +The device MUST NOT offer feature bits corresponding to features
>> > +it would not support if accepted by the driver (even if the
>> > +driver is prohibited from accepting the feature bits by the
>> > +specification); for the sake of clarity, this refers to feature
>> > +bits not described in this specification, reserved feature bits
>> > +and feature bits reserved or not supported for the specific
>> > +transport or the specific device type, but this does not preclude
>> > +devices written to a future version of this specification from
>> > +offering such feature bits should such a specification have a
>> > +provision for devices to support the corresponding features.
>> > +
>> >  If a device has successfully negotiated a set of features
>> >  at least once (by accepting the FEATURES_OK \field{device
>> >  status} bit during device initialization), then it SHOULD
>> > -- 
>> > MST
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe from this mail list, you must leave the OASIS TC that 
>> > generates this mail.  Follow this link to all your TCs in OASIS at:
>> > https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
-- 
He caught a fleeting glimpse of a man, moving uphill pursued by a bus.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 03/10] admin: introduce group administration commands

[David Edmondson]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> This introduces a general structure for group administration commands,
> used to control device groups through their owner.
>
> Following patches will introduce specific commands and an interface for
> submitting these commands to the owner.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
>  introduction.tex |   3 ++
>  2 files changed, 111 insertions(+)
>
> diff --git a/admin.tex b/admin.tex
> index 3dc47be..7e28b77 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
>  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>  \end{description}
>  
> +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>  
> +The driver sends group administration commands to the owner device of
> +a group to control member devices of the group.
> +This mechanism can
> +be used, for example, to configure a member device before it is
> +initialized by its driver.
> +\footnote{The term "administration" is intended to be interpreted
> +widely to include any kind of control. See specific commands
> +for detail.}
> +
> +All the group administration commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        le16 opcode;
> +        /*
> +         * 1 - SR-IOV
> +         * 2 - 65535 reserved

As before, this formatting is potentially confusing. There should be a
difference between "value X has meaning Y" and "a range from X to Y".

> +         */
> +        le16 group_type;
> +        /* unused, reserved for future extensions */
> +        u8 reserved1[12];
> +        le64 group_member_id;
> +        u8 command_specific_data[];
> +
> +        /* Device-writable part */
> +        le16 status;
> +        le16 status_qualifier;
> +        /* unused, reserved for future extensions */
> +        u8 reserved2[4];
> +        u8 command_specific_result[];
> +};
> +\end{lstlisting}
> +
> +For all commands, \field{opcode}, \field{group_type} and if
> +necessary \field{group_member_id} and \field{command_specific_data} are
> +set by the driver, and the owner device sets \field{status} and if
> +needed \field{status_qualifier} and
> +\field{command_specific_result}.
> +
> +Generally, any unused device-readable fields are set to zero by the driver
> +and ignored by the device.  Any unused device-writeable fields are set to zero
> +by the device and ignored by the driver.
> +
> +\field{opcode} specifies the command. The valid
> +values for \field{opcode} can be found in the following table:
> +
> +\begin{tabular}{|l|l|}
> +\hline
> +opcode & Name & Command Description \\
> +\hline \hline
> +0x0000 - 0x7FFF & - &  Group administration commands    \\
> +\hline
> +0x8000 - 0xFFFF & - & Reserved    \\
> +\hline
> +\end{tabular}
> +
> +The \field{group_type} specifies the group type identifier.
> +The \field{group_member_id} specifies the member identifier within the group.
> +See section \ref{sec:Introduction / Terminology / Device group}
> +for the definition of the group type identifier and group member
> +identifier.
> +
> +The following table describes possible \field{status} values;
> +to simplify common implementations, they are intentionally
> +matching common \hyperref[intro:errno]{Linux error names and numbers}:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status (decimal) & Name & Description \\
> +\hline \hline
> +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> +\hline
> +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> +\hline
> +other   & -    & group administration command error  \\
> +\hline
> +\end{tabular}
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}

"status_qualifier"

> +is reserved and set to zero by the device.
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
> +the following table describes possible \field{status_qialifier} values:
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status & Name & Description \\
> +\hline \hline
> +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
> +\hline
> +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
> +\hline
> +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
> +\hline
> +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
> +\hline
> +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER    & unsupported or invalid \field{group_member_id} \\
> +\hline
> +0x05-0xFFFF   & -    & reserved for future use \\
> +\hline
> +\end{tabular}
> +
> +Each command uses a different \field{command_specific_data} and
> +\field{command_specific_result} structures and the length of
> +\field{command_specific_data} and \field{command_specific_result}
> +depends on these structures and is described separately or is
> +implicit in the structure description.

Would it make sense to add some advisory text about alignment of the
fields after command_specific_data? It would be annoying if status,
status_qualifier, etc. were poorly aligned for a common hardware
implementation.

> diff --git a/introduction.tex b/introduction.tex
> index 287c5fc..0d849a9 100644
> --- a/introduction.tex
> +++ b/introduction.tex
> @@ -68,6 +68,9 @@ \section{Normative References}\label{sec:Normative References}
>  	\phantomsection\label{intro:FUSE}\textbf{[FUSE]} &
>  	Linux FUSE interface,
>  	\newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h}\\
> +	\phantomsection\label{intro:errno}\textbf{[errno]} &
> +	Linux error names and numbers,
> +	\newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/uapi/asm-generic/errno-base.h}\\
>          \phantomsection\label{intro:eMMC}\textbf{[eMMC]} &
>          eMMC Electrical Standard (5.1), JESD84-B51,
>          \newline\url{http://www.jedec.org/sites/default/files/docs/JESD84-B51.pdf}\\
> -- 
> MST
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that 
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
-- 
There is one bird in my house.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[David Edmondson]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> The admin virtqueues will be the first interface to issue admin commands.

"first interface used" ?

> Currently virtio specification defines control virtqueue to manipulate

"Currently the virtio specification defines the control virtqueue"

> features and configuration of the device it operates on. However,
> control virtqueue commands are device type specific, which makes it very
> difficult to extend for device agnostic commands.
>
> To support this requirement in a more generic way, this patch introduces
> a new admin virtqueue interface.
>
> We also support more than one admin virtqueue, for QoS and
> scalability requirements.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex   | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex |  7 +++--
>  2 files changed, 79 insertions(+), 2 deletions(-)
>
> diff --git a/admin.tex b/admin.tex
> index 7e28b77..3ffac2e 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{command_specific_data} and \field{command_specific_result}
>  depends on these structures and is described separately or is
>  implicit in the structure description.
> +
> +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> +
> +An administration virtqueue of an owner device is used to submit
> +group administration commands. An owner device can have more
> +than one administration virtqueue.
> +
> +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
> +or more adminstration virtqueues. The number and locations of the
> +administration virtqueues are exposed by the owner device in a transport
> +specific manner.
> +
> +The driver submits commands by queueing them to an arbitrary
> +administration virtqueue, and they are processed by the
> +device in the order in which they are queued. It is the
> +responsibility of the driver to ensure ordering for commands
> +placed on different administration virtqueues, because they will
> +be executed with no order constraints.
> +
> +Administration virtqueues are used as follows:
> +\begin{itemize}
> +\item The driver submits the command using the \field{struct virtio_admin_cmd}
> +structure using a buffer consisting of two parts: a device-readable one followed by a
> +device-writable one.
> +\item the device-readable part includes fields from \field{opcode}
> +through \field{command_specific_data}.
> +\item the device-writeable buffer includes fields from \field{status}
> +through \field{command_specific_result} inclusive.
> +\end{itemize}
> +
> +For each command, this specification describes a distinct
> +format structure used for \field{command_specific_data} and
> +\field{command_specific_result}, the length of these fields
> +depends on the command.
> +
> +However, to ensure forward compatibility
> +\begin{itemize}
> +\item drivers are allowed to submit buffers that are longer
> +than what the device expects

s/what//

> +(that is, longer than the length of
> +\field{opcode} through \field{command_specific_data}).
> +This allows the driver to maintain
> +a single format structure even if some structure fields are
> +unused by the device.
> +\item drivers are allowed to submit buffers that are shorter
> +than what the device expects
> +(that is, shorter than the length of \field{status} through
> +\field{command_specific_result}). This allows the device to maintain
> +a single format structure even if some structure fields are
> +unused by the driver.
> +\end{itemize}

Given that the previous patch declared this as a single data structure
(containing both command and result), with no length indicators for the
command specific data or result, how will a device know how much space
the driver has allocated for the command_specific_data or
command_specific result?

> +The device compares the length of each part (device-readable and
> +device-writeable) of the buffer as submitted by driver to what it
> +expects and then silently truncates the structures to either the
> +length submitted by the driver, or the length described in this
> +specification, whichever is shorter.  The device silently ignores
> +any data falling outside the shorter of the two lengths. Any
> +missing fields are interpreted as set to zero.
> +
> +Similarly, the driver compares the used buffer length
> +of the buffer to what it expects and then silently
> +truncates the structure to the used buffer length.
> +The driver silently ignores any data falling outside
> +the used buffer length reported by the device.  Any missing
> +fields are interpreted as set to zero.
> +
> +This simplifies driver and device implementations since the
> +driver/device can simply maintain a single large structure (such
> +as a C structure) for a command and its result. As new versions
> +of the specification are designed, new fields can be added to the
> +tail of a structure, with the driver/device using the full
> +structure without concern for versioning.
> +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues

Merge detritus.

> diff --git a/content.tex b/content.tex
> index ffe45c4..c8647c9 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>  \begin{description}
>  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
>  
> -\item[24 to 40] Feature bits reserved for extensions to the queue and
> +\item[24 to 41] Feature bits reserved for extensions to the queue and
>    feature negotiation mechanisms
>  
> -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
> +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
>  \end{description}
>  
>  \begin{note}
> @@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>    that the driver can reset a queue individually.
>    See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>  
> +  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
> +  administration virtqueues.
> +
>  \end{description}
>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> -- 
> MST
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that 
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
-- 
And removed his hat, in respect of her presence.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery

[David Edmondson]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> Add commands to find out which commands does each group support,
> as well as enable their use by driver.
> This will be especially useful once we have multiple group types.
>
> An alternative is per-type VQs. This is possible but will
> require more per-transport work. Discovery through the vq
> helps keep things contained.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 96 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 3ffac2e..1172054 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline
>  opcode & Name & Command Description \\
>  \hline \hline
> -0x0000 - 0x7FFF & - &  Group administration commands    \\
> +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> +0x0002 - 0x7FFF & - &  Group administration commands    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved    \\
>  \hline
> @@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  depends on these structures and is described separately or is
>  implicit in the structure description.
>  
> +Before sending any administration commands to the device, the driver
> +needs to communicate to the device which commands it is going to
> +use. Initially (after reset), only two commands are assumed to be used:
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Before sending any other commands for any member of a specific group to
> +the device, the driver queries the supported commands via
> +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
> +VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE
> +both use the following structure describing the
> +command opcodes:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_list {
> +       /* Indicates which of the below fields were returned
> +       le64 device_admin_cmds[];
> +};
> +\end{lstlisting}
> +
> +This structure is an array of 64 bit values in little-endian byte
> +order, in which a bit is set if the specific command opcode
> +is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value

64-bit.

> +in this array corresponding to opcodes 0 to 63,
> +\field{device_admin_cmds[1]} is the second 64-bit value
> +corresponding to opcodes 64 to 127, etc.
> +For example, the array of size 2 including
> +the values 0x3 in \field{device_admin_cmds[0]}
> +and 0x1 in \field{device_admin_cmds[1]} indicates that only
> +opcodes 0, 1 and 64 are supported.
> +The length of the array depends on the supported opcodes - it is
> +large enough to include bits set for all supported opcodes,
> +that is the length can be calculated by starting with the largest
> +supported opcode adding one, dividing by 64 and rounding up.
> +In other words, for
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the
> +length of \field{command_specific_result} and
> +\field{command_specific_data} respectively will be
> +$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division
> +with round up and \field{max_cmd} is the largest available command opcode.
> +
> +The array is also allowed to be larger and to additionally
> +include an arbitrary number of all-zero entries.
> +
> +Accordingly, bits 0 and 1 corresponding to opcode 0
> +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
> +(VIRTIO_ADMIN_CMD_LIST_USE) are
> +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +This command has no command specific data.
> +The device, upon success, returns a result in
> +\field{command_specific_result} in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands supported for the given group.
> +
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
> +is set to 0x1.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +The \field{command_specific_data} is in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands used by the driver.
> +This command has no command specific result.
> +
> +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> +query the list of commands valid for this group and before sending
> +any commands for any member of a group.
> +
> +The driver then enables use of some of the opcodes by sending to
> +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
> +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
> +both understood and used by the driver.
> +
> +If the device supports the command list used by the driver, the
> +device completes the command with status VIRTIO_ADMIN_STATUS_OK.
> +If the device does not support the command list, the device
> +completes the command with status
> +VIRTIO_ADMIN_STATUS_INVALID_FIELD.
> +
> +Note: drivers are assumed not to set bits in device_admin_cmds
> +if they are not familiar with how the command opcode
> +is used, since devices could have dependencies between
> +command opcodes.
> +
> +It is assumed that all members in a group support and are used
> +with the same list of commands. However, for owner devices
> +supporting multiple group types, the list of supported commands
> +might differ between different group types.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>  
>  An administration virtqueue of an owner device is used to submit
> -- 
> MST
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
-- 
If I could buy my reasoning, I'd pay to lose.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[David Edmondson]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> Add conformance clauses for admin commands and admin virtqueues.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 215 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 1172054..6c4f79c 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  supporting multiple group types, the list of supported commands
>  might differ between different group types.
>  
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The device MUST validate \field{opcode}, \field{group_type} and
> +\field{group_member_id}, and if any of these has an invalid or
> +unsupported value, set \field{status} to
> +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
> +accordingly:
> +\begin{itemize}
> +\item if \field{group_type} is invalid, \field{status_qualifier}
> +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> +\item otherwise, if \field{opcode} is invalid,
> +	\field{status_qualifier} MUST be set to
> +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
> +\item otherwise, if \field{group_member_id} is used by the
> +	specific command and is invalid, \field{status_qualifier} MUST be
> +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER.
> +\end{itemize}
> +
> +If a command completes successfully, the device MUST set
> +\field{status} to VIRTIO_ADMIN_STATUS_OK.
> +
> +If a command fails, the device MUST set
> +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
> +
> +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
> +device state MUST NOT change, that is the command MUST NOT have
> +any side effects on the device, in particular the device MUST not

MUST NOT

> +enter an error state as a result of this command.
> +
> +If a command fails, the device state generally SHOULD NOT change,
> +as far as possible.
> +
> +The device MAY enforce additional restrictions and dependencies on
> +opcodes used by the driver and MAY fail the command
> +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> +if the list of commands used violate internal device dependencies.
> +
> +If the device supports multiple group types, commands for each group
> +type MUST operate independently of each other, in particular,
> +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
> +for different group types.
> +
> +After reset, if the device supports a given group type
> +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
> +the device MUST assume
> +that the list of legal commands used by the driver consists of
> +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
> +the device MUST set the list of legal commands used by the driver
> +to the one supplied in \field{command_specific_data}.
> +
> +The device MUST set the list of legal commands used by the driver
> +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.

Are these last two paragraphs not saying the same thing?

> +The device MUST validate commands against the list used by
> +the driver and MUST fail any commands not in the list with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
> +
> +The list of supported commands MUST NOT shrink (but MAY expand):
> +after reporting a given command as supported through
> +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> +as unsupported.  Further, after a given set of commands has been
> +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
> +device or system reset the device SHOULD complete successfully
> +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
> +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
> +after a device or system reset, the device MUST not fail it
> +solely because of the command list used.  Failure to do so would
> +interfere with resuming from suspend and error recovery.
> +
> +When processing a command with the SR-IOV group type,
> +if the device does not have an SR-IOV Extended Capability or
> +if \field{VF Enable} is clear
> +then the device MUST fail all commands with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> +otherwise, if \field{group_member_id} is not
> +between $1$ and \field{NumVFs} inclusive,
> +the device MUST fail all commands with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER;
> +\field{NumVFs}, \field{VF Migration Capable}  and
> +\field{VF Enable} refer to registers within the SR-IOV Extended
> +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The driver MAY discover whether device supports a specific group type
> +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
> +\field{group_type}.
> +
> +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
> +and wait for it to be completed with status
> +VIRTIO_ADMIN_STATUS_OK before issuing any commands
> +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
> +and VIRTIO_ADMIN_CMD_LIST_USE).
> +
> +The driver SHOULD NOT set bits in device_admin_cmds
> +if it is not familiar with how the command opcode
> +is used, as dependencies between command opcodes might exist.
> +
> +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
> +the use of any commands not previously reported as
> +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +The driver MUST NOT use any commands for a given group type
> +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
> +list of command opcodes and group type.
> +
> +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> +with respective bits cleared in \field{command_specific_data}.

This runs contrary to the assertion "The list of supported commands MUST
NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
commands initially available.

> +The driver MUST handle a command error with a reserved \field{status}
> +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +(except possibly for different error reporting/diagnostic messages).
> +
> +The driver MUST handle a command error with a reserved
> +\field{status_qualifier} value in the same way as
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
> +different error reporting/diagnostic messages).
> +
> +When sending commands with the SR-IOV group type,
> +the driver specify a value for \field{group_member_id}
> +between $1$ and \field{NumVFs} inclusive,
> +the driver MUST also make sure that as long as any such command
> +is outstanding, \field{VF Migration Capable} is clear and
> +\field{VF Enable} is set;
> +\field{NumVFs}, \field{VF Migration Capable}  and
> +\field{VF Enable} refer to registers within the SR-IOV Extended
> +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>  
>  An administration virtqueue of an owner device is used to submit
> @@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
>  of the specification are designed, new fields can be added to the
>  tail of a structure, with the driver/device using the full
>  structure without concern for versioning.
> ->>>>>>> 0edc690... admin: introduce virtio admin virtqueues

Oh, here is where it got removed :-)

> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The device MUST support device-readable and device-writeable buffers
> +shorter than described in this specification, by
> +\begin{enumerate}
> +\item acting as if any data that would be read outside the
> +device-readable buffers is set to zero, and
> +\item discarding data that would be written outside the
> +specified device-writeable buffers.
> +\end{enumerate}
> +
> +The device MUST support device-readable and device-writeable buffers
> +longer than described in this specification, by
> +\begin{enumerate}
> +\item ignoring any data in device-readable buffers outside
> +the expected length, and
> +\item only writing the expected structure to the device-writeable
> +buffers, ignoring any extra buffers, and reporting the
> +actual length of data written, in bytes,
> +as buffer used length.
> +\end{enumerate}
> +
> +The device SHOULD initialize the device-writeable buffer
> +up to the length of the structure described by this specification or
> +the length of the buffer supplied by the driver (even if the buffer is
> +all set to zero), whichever is shorter.
> +
> +The device MUST NOT fail a command solely because the buffers
> +provided are shorter or longer than described in this
> +specification.
> +
> +The device MUST initialize the device-writeable part of
> +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
> +size.
> +
> +The device MUST initialize \field{status} in \field{struct
> +virtio_admin_cmd}.
> +
> +The device MUST process commands on a given administration virtqueue
> +in the order in which they are queued.
> +
> +If multiple administration virtqueues have been configured,
> +device MAY process commands on distinct virtqueues with
> +no order constraints.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The driver MAY supply device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are longer than described in
> +this specification.
> +
> +The driver SHOULD supply device-readable part of
> +\field{struct virtio_admin_cmd} that is at least as
> +large as the structure described by this specification
> +(even if the structure is all set to zero).
> +
> +The driver MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
> +in length.

This addresses my question on a previous patch.

> +The device MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are larger than zero in
> +length. However, \field{command_specific_data} and
> +\field{command_specific_result} MAY be zero in length, unless
> +specified otherwise for the command.
> +
> +The driver MUST NOT assume that the device will initialize the whole
> +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
> +the driver MUST act as if the structure
> +outside the part of the buffer used by the device
> +is set to zero.
> +
> +If multiple administration virtqueues have been configured,
> +the driver MUST ensure ordering for commands
> +placed on different administration virtqueues.
> -- 
> MST
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
-- 
Music has magic, it's good clear syncopation.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Jiri Pirko]

Fri, Mar 03, 2023 at 09:23:14PM CET, stefanha@redhat.com wrote:
>On Fri, Mar 03, 2023 at 08:18:43AM -0500, Michael S. Tsirkin wrote:
>> On Fri, Mar 03, 2023 at 08:13:37AM -0500, Stefan Hajnoczi wrote:
>> > On Thu, Mar 02, 2023 at 06:57:24PM -0500, Michael S. Tsirkin wrote:
>> > > On Thu, Mar 02, 2023 at 03:10:11PM -0500, Stefan Hajnoczi wrote:
>> > > > On Thu, Mar 02, 2023 at 08:05:02AM -0500, Michael S. Tsirkin wrote:
>> > > > > This introduces a general structure for group administration commands,
>> > > > > used to control device groups through their owner.
>> > > > > 
>> > > > > Following patches will introduce specific commands and an interface for
>> > > > > submitting these commands to the owner.
>> > > > > 
>> > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> > > > > ---
>> > > > >  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >  introduction.tex |   3 ++
>> > > > >  2 files changed, 111 insertions(+)
>> > > > > 
>> > > > > diff --git a/admin.tex b/admin.tex
>> > > > > index 3dc47be..7e28b77 100644
>> > > > > --- a/admin.tex
>> > > > > +++ b/admin.tex
>> > > > > @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
>> > > > >  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>> > > > >  \end{description}
>> > > > >  
>> > > > > +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>> > > > >  
>> > > > > +The driver sends group administration commands to the owner device of
>> > > > 
>> > > > I notice that the terminology is simply "the driver". "Owner driver"
>> > > > and "group member driver" might be clearer because there will be two
>> > > > (possibly different) drivers involved.
>> > > 
>> > > Hmm I don't really want to repeat owner everywhere.
>> > > I will clarify that in this section simple "driver" and "device" are
>> > > owner, "member device" and "member driver" is always called explicitly.
>> > 
>> > Sounds good.
>> > 
>> > > > > +a group to control member devices of the group.
>> > > > > +This mechanism can
>> > > > > +be used, for example, to configure a member device before it is
>> > > > > +initialized by its driver.
>> > > > > +\footnote{The term "administration" is intended to be interpreted
>> > > > > +widely to include any kind of control. See specific commands
>> > > > > +for detail.}
>> > > > > +
>> > > > > +All the group administration commands are of the following form:
>> > > > > +
>> > > > > +\begin{lstlisting}
>> > > > > +struct virtio_admin_cmd {
>> > > > > +        /* Device-readable part */
>> > > > > +        le16 opcode;
>> > > > > +        /*
>> > > > > +         * 1 - SR-IOV
>> > > > > +         * 2 - 65535 reserved
>> > > > > +         */
>> > > > > +        le16 group_type;
>> > > > > +        /* unused, reserved for future extensions */
>> > > > > +        u8 reserved1[12];
>> > > > > +        le64 group_member_id;
>> > > > > +        u8 command_specific_data[];
>> > > > > +
>> > > > > +        /* Device-writable part */
>> > > > > +        le16 status;
>> > > > > +        le16 status_qualifier;
>> > > > > +        /* unused, reserved for future extensions */
>> > > > > +        u8 reserved2[4];
>> > > > > +        u8 command_specific_result[];
>> > > > > +};
>> > > > > +\end{lstlisting}
>> > > > > +
>> > > > > +For all commands, \field{opcode}, \field{group_type} and if
>> > > > > +necessary \field{group_member_id} and \field{command_specific_data} are
>> > > > > +set by the driver, and the owner device sets \field{status} and if
>> > > > > +needed \field{status_qualifier} and
>> > > > > +\field{command_specific_result}.
>> > > > > +
>> > > > > +Generally, any unused device-readable fields are set to zero by the driver
>> > > > > +and ignored by the device.  Any unused device-writeable fields are set to zero
>> > > > > +by the device and ignored by the driver.
>> > > > > +
>> > > > > +\field{opcode} specifies the command. The valid
>> > > > > +values for \field{opcode} can be found in the following table:
>> > > > > +
>> > > > > +\begin{tabular}{|l|l|}
>> > > > > +\hline
>> > > > > +opcode & Name & Command Description \\
>> > > > > +\hline \hline
>> > > > > +0x0000 - 0x7FFF & - &  Group administration commands    \\
>> > > > > +\hline
>> > > > > +0x8000 - 0xFFFF & - & Reserved    \\
>> > > > > +\hline
>> > > > > +\end{tabular}
>> > > > 
>> > > > I thought all commands are "group administration commands" but this
>> > > > table makes it look like they are just a subset (0x0000 - 0x7FFF) of
>> > > > group administration commands, which is a paradox.
>> > > 
>> > > Well the rest are reserved, maybe we will have more command types who
>> > > knows. No?
>> > 
>> > I see. Does that mean the reserved commands don't need to be in the same
>> > format as struct virtio_admin_cmd?
>> 
>> I am not good in predicting future ...
>> 
>> > The entire section is called "Group administration commands" but I get
>> > the impression it's talking both about admin virtqueue commands in
>> > general and specifically about group administration commands.
>> > 
>> > Is it possible to structure this as follows:
>> > 
>> >   Admin Commands
>> >     ...common stuff...
>> >     Group Administration Commands (0x0000-0x7fff)
>> >       ...specific to group administration commands...
>> >     Reserved (0x8000-0xffff)
>> 
>> What if I just write it like this:
>> 
>> 
>> > > > > +\hline \hline
>> > > > > +0x0000 - 0x7FFF & - &  commands using struct virtio_admin_cmd    \\
>> > > > > +\hline
>> > > > > +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>> 
>> 
>> will this address the comment?
>
>Yes, thanks.

What is the reason for having this table? I fail to see it.


>
>Stefan



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3343 bytes --]

On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
> Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
> >On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> >> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> >> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> >> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> >> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> >> > > > > What happens if a command takes 1 second to complete, is the device
> >> > > > > allowed to process the next command from the virtqueue during this time,
> >> > > > > possibly completing it before the first command?
> >> > > > > 
> >> > > > > This requires additional clarification in the spec because "they are
> >> > > > > processed by the device in the order in which they are queued" does not
> >> > > > > explain whether commands block the virtqueue (in order completion) or
> >> > > > > not (out of order completion).
> >> > > > 
> >> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> >> > > 
> >> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> >> > > Several may be processed by the device at the same time.
> >> > 
> >> > Let's say I submit a write followed by read - is read
> >> > guaranteed to return an up to date info?
> >> 
> >> In general, no. The driver must wait for the write completion before
> >> submitting the read if it wants consistency.
> >> 
> >> Stefan
> >
> >I see.  I think it's a good design to follow then.
> 
> Hmm, is it suitable to have this approach for configuration interface?
> Storage device is a different beast, having parallel reads and writes
> makes complete sense for performance.
> 
> ->read a req
> ->read b req
> ->read c req
> <-read a rep
> <-read b rep
> <-read c rep
> 
> There is no dependency, even between writes.
> 
> But in case of configuration, does not make any sense to me.
> Why is it needed? To pass the burden of consistency of
> configuration to driver sounds odd at least.
> 
> I sense there is no concete idea about what the "admin virtqueue" should
> serve for exactly.

It's useful for long-running commands because they prevent other
commands from executing.

An example I've given is that deleting a group member might require
waiting for the group member's I/O activity to finish. If that I/O
activity cannot be cancelled instantaneously, then it could take an
unbounded amount of time to delete the group member. The device would be
unable to process futher admin commands.

Group member creation might have similar issues if it involves acquiring
remote resources (e.g. connecting to a Ceph cluster or allocating ports
on a distributed network switch). It can be impossible to defer resource
acquisition/initialization because because VIRTIO devices must be
available as soon as the driver can see them (i.e. how do populate
Configuration Space fields if you don't have the details of the resource
yet?).

So I have raised two questions:

1. What are the admin queue command completion semantics: in-order or
   out-of-order command completion?

2. Will there be long-running commands and how will we deal with them
   when they hang?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Tue, Mar 07, 2023 at 03:39:11PM CET, stefanha@redhat.com wrote:
>On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
>> Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
>> >On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
>> >> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
>> >> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
>> >> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
>> >> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
>> >> > > > > What happens if a command takes 1 second to complete, is the device
>> >> > > > > allowed to process the next command from the virtqueue during this time,
>> >> > > > > possibly completing it before the first command?
>> >> > > > > 
>> >> > > > > This requires additional clarification in the spec because "they are
>> >> > > > > processed by the device in the order in which they are queued" does not
>> >> > > > > explain whether commands block the virtqueue (in order completion) or
>> >> > > > > not (out of order completion).
>> >> > > > 
>> >> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
>> >> > > 
>> >> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
>> >> > > Several may be processed by the device at the same time.
>> >> > 
>> >> > Let's say I submit a write followed by read - is read
>> >> > guaranteed to return an up to date info?
>> >> 
>> >> In general, no. The driver must wait for the write completion before
>> >> submitting the read if it wants consistency.
>> >> 
>> >> Stefan
>> >
>> >I see.  I think it's a good design to follow then.
>> 
>> Hmm, is it suitable to have this approach for configuration interface?
>> Storage device is a different beast, having parallel reads and writes
>> makes complete sense for performance.
>> 
>> ->read a req
>> ->read b req
>> ->read c req
>> <-read a rep
>> <-read b rep
>> <-read c rep
>> 
>> There is no dependency, even between writes.
>> 
>> But in case of configuration, does not make any sense to me.
>> Why is it needed? To pass the burden of consistency of
>> configuration to driver sounds odd at least.
>> 
>> I sense there is no concete idea about what the "admin virtqueue" should
>> serve for exactly.
>
>It's useful for long-running commands because they prevent other
>commands from executing.
>
>An example I've given is that deleting a group member might require
>waiting for the group member's I/O activity to finish. If that I/O
>activity cannot be cancelled instantaneously, then it could take an
>unbounded amount of time to delete the group member. The device would be
>unable to process futher admin commands.

I see. Then I believe that the device should handle the dependencies.
Example 1:
-> REQ cmd to create group member A
-> REQ cmd to create group member B
<- REP cmd to create group member A
<- REP cmd to create group member B

The device according to internal implementation can either serialize the
2 group member creations or do it in parallel, if it supports it.

Example 2:
-> REQ cmd to create group member A
-> REQ cmd config group member A
<- REP cmd to create group member A
<- REP cmd config group member A

Here the serialization is necessary and the device is the one to take
care of it.

Makes sense?

>
>Group member creation might have similar issues if it involves acquiring
>remote resources (e.g. connecting to a Ceph cluster or allocating ports
>on a distributed network switch). It can be impossible to defer resource

Sidetrack: this is really fuzzy to me, how the new member is going to be
plugged into backend (network). Over the time, we learned that the
creation of device from the other side (switch side) makes more sense.
That is why I asked for motivation to introduce this infra.


>acquisition/initialization because because VIRTIO devices must be
>available as soon as the driver can see them (i.e. how do populate
>Configuration Space fields if you don't have the details of the resource
>yet?).
>
>So I have raised two questions:
>
>1. What are the admin queue command completion semantics: in-order or
>   out-of-order command completion?

I would add "dependencies/serialization" here.


>
>2. Will there be long-running commands and how will we deal with them
>   when they hang?

Yeah, sounds legit to define it in spec.

>
>Stefan



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
> I sense there is no concete idea about what the "admin virtqueue" should
> serve for exactly.

Because the virtqueue is just a virtqueue - a way to pass around
commands.  Admin commands are used whenever one needs to talk to one
device about another one.
-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Tue, Mar 07, 2023 at 08:36:41AM +0100, Jiri Pirko wrote:
> Hmm, if not for now, the future exension would not be so simple, I fear.

Without knowing what it is I can't say.

> 
> >
> >Passing commands to devices themselves is already covered in spec
> >reasonably well though not in a generic way.
> 
> You mean using the control queue, correct?

Depends on the device type. network devices have a control queue, yes.

> >From one of the patch description of this patchset I understand that you
> cannot use control queue for this because control queue is
> device-specific, yet group control is device-agnostic.
> 
> My undestanding therefore was, that the admin queue you are introducing
> serves as a generic carrier for device-agnostic commands, in parallel
> for having control queue serving as a generic carrier of device-specific
> commands. If this is not the case, I think it would be nice to describe
> the exact monivation and scope of admin queue.

Nope unfortunately.  This queue is just a carrier for admin commands.
admin commands are commands that talk to one device about other
devices. There's clearly no mechanism in the spec to do that,
so we plug this hole.



> 
> >
> >What we lack is passing commands about one device to another device.
> >E.g. control VFs through PFs.
> 
> Could you provide examples of such commands please?

For example a common feature is to program a vlan and have device
put a given VF inside this vlan.

In a virtualization scenario host controls this vlan programming giving
the network a measure of protection from VFs.  If a VF is passed through
to a VM, IOMMU limits VFs to only access guest memory so host has to do
this programming through a PF.



> 
> >This is what groups do.
> >But if we see more uses we can always add them.
> >
> >
> >I'd rather avoid being too generic though.
> 
> In that case, why not to avoid using generic terms and stay
> "group-centric"? What I mean is:
> "Administration Virtqueues" -> "Group Administration Virtqueues"
> "struct virtio_admin_cmd" -> "struct virtio_group_admin_cmd"
> 
> Etc. Helps to avoid confusion.

Sure, I tried to do that but missed some opportunities.
Will address.

> 
> >
> >
> >
> >
> >> 
> >> >+than one administration virtqueue.
> 
> 
> [...]
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

On Tue, Mar 07, 2023 at 08:17:03AM +0100, Jiri Pirko wrote:
> Mon, Mar 06, 2023 at 11:48:32PM CET, mst@redhat.com wrote:
> >On Mon, Mar 06, 2023 at 05:40:04PM +0100, Jiri Pirko wrote:
> >> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
> >> 
> >> [...]
> >> 
> >> 
> >> >  admin: introduce device group and related concepts
> >> >  admin: introduce group administration commands
> >> >  admin: introduce virtio admin virtqueues
> >> 
> >> The ordering seems a bit odd to me. I think that at the beginning, the
> >> admin virtqueue should be introduced with generic concepts (including
> >> the generic command structure), only after that the section introducing
> >> device group stuff and related commands.
> >
> >That's intentional.
> >
> >Jason is working on additional ways to pass commands to
> >devices, using memory mapped registers.
> >
> >Thus admin commands stand by themselves, and the queue is
> >just a way to issue them.
> 
> Okay, in that case, woudnlt it make sense if the generic CMD text
> (including the generic command structure) would go first, then the
> "group" related cmds?
> 

there are not other commands though - they are all group related.

> >
> >
> >
> >> This publicly archived list offers a means to provide input to the
> >> OASIS Virtual I/O Device (VIRTIO) TC.
> >> 
> >> In order to verify user consent to the Feedback License terms and
> >> to minimize spam in the list archive, subscription is required
> >> before posting.
> >> 
> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >> List help: virtio-comment-help@lists.oasis-open.org
> >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >> Committee: https://www.oasis-open.org/committees/virtio/
> >> Join OASIS: https://www.oasis-open.org/join/
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

On Tue, Mar 07, 2023 at 08:21:54AM +0100, Jiri Pirko wrote:
> Mon, Mar 06, 2023 at 11:54:45PM CET, mst@redhat.com wrote:
> >On Mon, Mar 06, 2023 at 01:29:30PM +0100, Jiri Pirko wrote:
> >> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
> >> 
> >> [...]
> >> 
> >> >
> >> >TODO (maybe?) - probably ok to defer until this part is upstream:
> >> >
> >> >	Add "all members" member id.
> >> >
> >> >	Add commands for MSI, feature discovery.
> >> >
> >> >	Add commands for transport vq.
> >> >
> >> >
> >> >My intent is to try and support both SR-IOV and SIOV
> >> >usecases with the same structure and maybe even the same
> >> >VQ.
> >> 
> >> Sorry to be late to the party, I'm trying to catch up. Is it common to
> >> have cover letter for features this brief? I mean, from the cover
> >> letter, I'm totally unable to understand what you are introducing here.
> >> 
> >> Could you elaborate about what you are aiming to achive with this?
> >> Could you shed some usecases perhaps?
> >> 
> >> I have to be missing something obvious, but I don't get why any notion
> >> of SR-IOV could be beneficial for virtio.
> >> 
> >
> >Good point, I'll add a bit of motivation.
> >
> >For SR-IOV, it is not unusual for PFs to excercise control over VFs.
> 
> I understand the concepts of SR-IOV. Yet I fail to see the need of such
> concept in virtio world. SR-IOV is very specific solution for PCI
> functions instantiation, and I believe that it is already considered
> quite limiting in many aspects. Does not make sense to me to introduce
> it for virtio. But again, I may be missing something crucial, I just
> would like to see the motivation, needs, usecases for this crystal
> clear, which is opposite to the current cover letter I'm afraid :/

First people are asking for it because it's out there, however limiting
it is. In fact Nvidia is - why don't you talk to Parav here and tell
him that SR-IOV is legacy and there's no need to support.




> 
> >There is interest in the community to include an interface to allow this
> >in the virtio spec, when the PF is a virtio device.  This is what this
> >patch does.
> 
> Yeah, but why? As I asked before, what are the usecases? The fact there
> is interest in the community does not mean it makes sense to have it :)
> 

If people want to build such hardware it will need some interface.
Far better to have it standard.


But generally e.g. intel already said they will reuse this
same structure with a different group type for SIOV support.
I'll mention this in the cover letter.

> >
> >
> >
> >
> >> >
> >> >For example, it might make sense to split creating/destroying
> >> >SIOV devices from the transport passing data from the guest - the
> >> >driver would then not negotiate VIRTIO_F_SR_IOV (which
> >> >then means auto-provisioning).
> >> >
> >> >This is out of RFC, since we have two commands which are useful
> >> >to discover supported group types (ATM can be none or SR-IOV).
> >> >
> >> >
> >> >Michael S. Tsirkin (10):
> >> >  virtio: document forward compatibility guarantees
> >> >  admin: introduce device group and related concepts
> >> >  admin: introduce group administration commands
> >> >  admin: introduce virtio admin virtqueues
> >> >  pci: add admin vq registers to virtio over pci
> >> >  mmio: document ADMIN_VQ as reserved
> >> >  ccw: document ADMIN_VQ as reserved
> >> >  admin: command list discovery
> >> >  admin: conformance clauses
> >> >  ccw: document more reserved features
> >> >
> >> > admin.tex        | 540 +++++++++++++++++++++++++++++++++++++++++++++++
> >> > content.tex      | 112 +++++++++-
> >> > introduction.tex |   3 +
> >> > 3 files changed, 653 insertions(+), 2 deletions(-)
> >> > create mode 100644 admin.tex
> >> >
> >> >-- 
> >> >MST
> >> >
> >> >
> >> >This publicly archived list offers a means to provide input to the
> >> >OASIS Virtual I/O Device (VIRTIO) TC.
> >> >
> >> >In order to verify user consent to the Feedback License terms and
> >> >to minimize spam in the list archive, subscription is required
> >> >before posting.
> >> >
> >> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >> >List help: virtio-comment-help@lists.oasis-open.org
> >> >List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >> >Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >> >List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >> >Committee: https://www.oasis-open.org/committees/virtio/
> >> >Join OASIS: https://www.oasis-open.org/join/
> >> >
> >> 
> >> This publicly archived list offers a means to provide input to the
> >> OASIS Virtual I/O Device (VIRTIO) TC.
> >> 
> >> In order to verify user consent to the Feedback License terms and
> >> to minimize spam in the list archive, subscription is required
> >> before posting.
> >> 
> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >> List help: virtio-comment-help@lists.oasis-open.org
> >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >> Committee: https://www.oasis-open.org/committees/virtio/
> >> Join OASIS: https://www.oasis-open.org/join/
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 2, 2023 6:47 PM

> > Can you please add other useful error codes in addition to the EINVAL?
> > Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
> 
> We'll add them when there's a use-case.

Errno.h is self-explanatory for the use case.
A device may want the driver to retry the command as it is busy.

> 
> > And with that, VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER moves under
> ENODEV error code.
> 
> 
> No I think that belongs in in EINVAL - command is malformed.
Command is not malformed.
The command prepared is correct, the VF number in the well-formed command which has an invalid VF device number.

> ENODEV is more "The wrong type of device was given to a function that
> expects a particular sort of device."
> 
Yes. Hence the ENODEV when the VF device is accessed is not valid.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Max Gurtovoy <mgurtovoy@nvidia.com>
> Sent: Monday, March 6, 2023 6:05 AM
> 
> On 03/03/2023 15:19, Michael S. Tsirkin wrote:
> > On Fri, Mar 03, 2023 at 08:17:03AM -0500, Stefan Hajnoczi wrote:
> >> On Thu, Mar 02, 2023 at 07:01:56PM -0500, Michael S. Tsirkin wrote:
> >>> On Thu, Mar 02, 2023 at 03:19:12PM -0500, Stefan Hajnoczi wrote:
> >>>> On Thu, Mar 02, 2023 at 06:40:29PM +0000, Parav Pandit wrote:
> >>>>>
> >>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>>> Sent: Thursday, March 2, 2023 8:05 AM
> >>>>>
> >>>>>> +When \field{status} is VIRTIO_ADMIN_STATUS_OK,
> >>>>>> +\field{status_qialifier} is reserved and set to zero by the device.
> >>>>>> +
> >>>>> s/status_qialifier/status_qualifier
> >>>>> Missed from v10 of Feb.
> >>>>>
> >>>>>> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following
> >>>>>> +table describes possible \field{status_qialifier} values:
> >>>>> s/status_qialifier/status_qualifier
> >>>>>
> >>>>> Can you please add other useful error codes in addition to the EINVAL?
> >>>>> Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
> >>>>
> >>>> Please define a unique constant for each error condition that can
> >>>> occur instead of sharing catch-all errno constants between multiple
> >>>> error conditions. If a driver wants to squash them together into an
> >>>> errno, that's fine, but I think doing this at the hardware
> >>>> interface level is just propagating the mistakes of errnos.
> >>>>
> >>>> Only status_qualifier is needed and the vague status field can be
> >>>> dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and
> >>>> ENODEV is useful. They have no meaning to the driver, only the
> >>>> status_qualifier really indicates what is going on.
> >>>
> >>> At a high level at the moment we have only two cases:
> >>> - ok
> >>> - invalid input supplied by driver
> >>>
> >>> maybe we will have more reasons for a failure - remains to be seen.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>
> >>>> I'm sure you guys have discussed this previously, but please provide
> >>>> rationale in the spec because it looks weird to someone with fresh eyes.
> >>>>
> >>>> Stefan
> >>>
> >>> Really most drivers just want to propagate errno to userspace.
> >>> All the detailed reporting is for sure well intentional but
> >>> in the end it is at best printed into log - end to end
> >>> people just end up with a switch statement
> >>> converting these to errno codes.
> >>> So we are passing them from device and this way there will be
> >>> some uniformity.
> >>
> >> Please clarify the rationale in the spec. I don't agree with it, as
> >> explained in my earlier email, but as long as its documented, people can
> >> try to follow it.
> >>
> >> Stefan
> >
> > It's 2:2 for now, you are with Parav, me and Cornelia like it :)
> > Or I will try to document it better.
> I don't understand this status_qualifier as well and it wasn't included
> in my original patch set.
> I vote for "status" that describe generic status codes and
> "command_specific_error" that should be inspected by the driver only if
> "status" is set to "VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR".
> We discussed this so many times before (and already agreed IIRC) and
> adding this new qualifiers mechanism sounds not right to me and not
> intuitive for device and driver developers.
> 
> I suggest:
> 
> 1. VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE
> 2. VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> 3. VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP
> 4. VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER
> 5. VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR (for more info read the
> command_specific_error field).

This is also a good proposal that 3 positives.

1. Tells exactly what was wrong with command or with device
2. generic enough to convert into abstract OS-specific error codes with negligible cost (switch-case)
3. can define command-specific error details

[virtio-dev] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Friday, March 3, 2023 3:34 AM
> 
> On Thu, Mar 02, 2023 at 06:40:55PM +0000, Parav Pandit wrote:
> > Did you miss reviewed-by from [1] or this is an old series reposted?
> > [1]
> > https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html
> 
> As a general rule, we don't strictly need to track reviewed by since there's a
> ballot (and presumably people review before voting).
> 
> People also tack on Signed-off-by: (and I do it too) but as long as we don't
> document what it means it's kind of vague, and the process of subscribing to
> the mailing list is a kind of replacement.
> 
> If you think everyone needs to follow practices like netdev does, we really need
> something written up, and agree on it.
> 
> E.g.  I work on the linux kernel too, so I can copy practices from there, but even
> linux isn't uniform.
> 
> And I wonder whether it's worth it - it definitely makes contributing to Linux
> harder, and even within Linux it pushes contributors away. 
The number of virtio spec contributors are in order of magnitude less than Linux kernel or just the Linux netdev.
Patch split up reduces lot of time author and reviewers.
For example, interrupt moderation at v10 can be very easily split up where prep patch can have RB tag and v11 doesn't need reviewers and author's time.
Your work here with 10 patches is yet another good example of it.
I remember Max and I started with 6 patches... and current 10 are more useful way to split them.

> At least for Linux
> tracking history in a precise way is extremely important since it's helpful with
> stability. Spec is very different.
> 
> Until we have a good contribution documentation I think we should not ask
> people to follow a pseudo Linux work flow with requests like "please split this
> patchset up" or "track changes across patch versions"
> simply because there's no good docs to teach people what exactly is the best
> practice.

Yes, I wanted to update the contributing document. It is in my to-do list.
But given the small crowd of contributors right now, almost everyone is familiar with the RB, ack tag.
At moment it has two main reasons.

1. Acknowledge the work and efforts that go in the review
2. When in question, reach out later to the spec author or reviewers to know about context/design etc.
3. Same reviewer doesn't need to go through the patch which already has RB tag of him/her.





---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5542 bytes --]

On Tue, Mar 07, 2023 at 04:07:54PM +0100, Jiri Pirko wrote:
> Tue, Mar 07, 2023 at 03:39:11PM CET, stefanha@redhat.com wrote:
> >On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
> >> Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
> >> >On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> >> >> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> >> >> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> >> >> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> >> >> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> >> >> > > > > What happens if a command takes 1 second to complete, is the device
> >> >> > > > > allowed to process the next command from the virtqueue during this time,
> >> >> > > > > possibly completing it before the first command?
> >> >> > > > > 
> >> >> > > > > This requires additional clarification in the spec because "they are
> >> >> > > > > processed by the device in the order in which they are queued" does not
> >> >> > > > > explain whether commands block the virtqueue (in order completion) or
> >> >> > > > > not (out of order completion).
> >> >> > > > 
> >> >> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> >> >> > > 
> >> >> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> >> >> > > Several may be processed by the device at the same time.
> >> >> > 
> >> >> > Let's say I submit a write followed by read - is read
> >> >> > guaranteed to return an up to date info?
> >> >> 
> >> >> In general, no. The driver must wait for the write completion before
> >> >> submitting the read if it wants consistency.
> >> >> 
> >> >> Stefan
> >> >
> >> >I see.  I think it's a good design to follow then.
> >> 
> >> Hmm, is it suitable to have this approach for configuration interface?
> >> Storage device is a different beast, having parallel reads and writes
> >> makes complete sense for performance.
> >> 
> >> ->read a req
> >> ->read b req
> >> ->read c req
> >> <-read a rep
> >> <-read b rep
> >> <-read c rep
> >> 
> >> There is no dependency, even between writes.
> >> 
> >> But in case of configuration, does not make any sense to me.
> >> Why is it needed? To pass the burden of consistency of
> >> configuration to driver sounds odd at least.
> >> 
> >> I sense there is no concete idea about what the "admin virtqueue" should
> >> serve for exactly.
> >
> >It's useful for long-running commands because they prevent other
> >commands from executing.
> >
> >An example I've given is that deleting a group member might require
> >waiting for the group member's I/O activity to finish. If that I/O
> >activity cannot be cancelled instantaneously, then it could take an
> >unbounded amount of time to delete the group member. The device would be
> >unable to process futher admin commands.
> 
> I see. Then I believe that the device should handle the dependencies.
> Example 1:
> -> REQ cmd to create group member A
> -> REQ cmd to create group member B
> <- REP cmd to create group member A
> <- REP cmd to create group member B
> 
> The device according to internal implementation can either serialize the
> 2 group member creations or do it in parallel, if it supports it.
> 
> Example 2:
> -> REQ cmd to create group member A
> -> REQ cmd config group member A
> <- REP cmd to create group member A
> <- REP cmd config group member A
> 
> Here the serialization is necessary and the device is the one to take
> care of it.
> 
> Makes sense?

Yes, I understand. The spec would need to define ordering rules for
specific commands and the device must implement them. It allows the
driver to pipeline commands while also allowing out-of-order completion
(parallelism) in some cases. The disadvantage of this approach is
complexity in the spec and implementations.

An alternative is unconditional out-of-order completion, where there are
no per-command ordering rules. The driver must wait for a command to
complete if it relies on the results of that command for its next
command. I like this approach because it's less complex in the spec and
for device implementers, while the burden on the driver implementer is
still reasonable.

> >
> >Group member creation might have similar issues if it involves acquiring
> >remote resources (e.g. connecting to a Ceph cluster or allocating ports
> >on a distributed network switch). It can be impossible to defer resource
> 
> Sidetrack: this is really fuzzy to me, how the new member is going to be
> plugged into backend (network). Over the time, we learned that the
> creation of device from the other side (switch side) makes more sense.
> That is why I asked for motivation to introduce this infra.

Michael, have you already thought about this?

> >acquisition/initialization because because VIRTIO devices must be
> >available as soon as the driver can see them (i.e. how do populate
> >Configuration Space fields if you don't have the details of the resource
> >yet?).
> >
> >So I have raised two questions:
> >
> >1. What are the admin queue command completion semantics: in-order or
> >   out-of-order command completion?
> 
> I would add "dependencies/serialization" here.
> 
> 
> >
> >2. Will there be long-running commands and how will we deal with them
> >   when they hang?
> 
> Yeah, sounds legit to define it in spec.
> 
> >
> >Stefan
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] RE: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: virtio@lists.oasis-open.org <virtio@lists.oasis-open.org> On Behalf Of
> Stefan Hajnoczi

> I'm not saying that it must be possible to abort admin commands. There are
> other options, like requiring the device itself to fail a command after a timeout.
An example where EAGAIN, EBUSY can be useful.

[virtio-dev] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Stefan Hajnoczi <stefanha@redhat.com>
> Sent: Tuesday, March 7, 2023 2:04 PM

> An alternative is unconditional out-of-order completion, where there are no
> per-command ordering rules. The driver must wait for a command to complete
> if it relies on the results of that command for its next command. I like this
> approach because it's less complex in the spec and for device implementers,
> while the burden on the driver implementer is still reasonable.
+1.
This has best of both.
1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
2. Ability to execute multiple unrelated commands using a single AQ.
3. stateless device in AQ command execution


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, March 7, 2023 12:20 PM

[..]
> > >For SR-IOV, it is not unusual for PFs to excercise control over VFs.
> >
> > I understand the concepts of SR-IOV. Yet I fail to see the need of
> > such concept in virtio world. SR-IOV is very specific solution for PCI
> > functions instantiation, and I believe that it is already considered
> > quite limiting in many aspects. Does not make sense to me to introduce
> > it for virtio. But again, I may be missing something crucial, I just
> > would like to see the motivation, needs, usecases for this crystal
> > clear, which is opposite to the current cover letter I'm afraid :/
> 
> First people are asking for it because it's out there, however limiting it is. In
> fact Nvidia is - why don't you talk to Parav here and tell him that SR-IOV is
> legacy and there's no need to support.
> 
Jiri and I discussed offline today.
SR-IOV is widely used today and in the coming future up to thousands of devices.
Many vendors are building it or already built.

SIOV as it stands today is still being crafted at various levels iommu, os, device, pci and more.
Not there yet fully.

Jiri and I built the Linux kernel interfaces for SFs which are equivalent to SIOVs that overcome some of the SR-IOV limitations.
And Jiri's suggestion was to leverage some of the vision and practice from there.

In other words, can AQ command be useful tomorrow for doing SIOV device add/remove, and provisioning from non-owning PF?
The way AQ is crafted today is not there yet, but in the future, it can be extended.
> 
> >
> > >There is interest in the community to include an interface to allow
> > >this in the virtio spec, when the PF is a virtio device.  This is
> > >what this patch does.
> >
> > Yeah, but why? As I asked before, what are the usecases? The fact
> > there is interest in the community does not mean it makes sense to
> > have it :)
> >
> 
> If people want to build such hardware it will need some interface.
> Far better to have it standard.
> 
> 
> But generally e.g. intel already said they will reuse this same structure with a
> different group type for SIOV support.
> I'll mention this in the cover letter.
>

We have the following already identified use cases of a device agnostic VQ.

1. SR-IOV VF device provisioning at virtio layer (features and config)
2. SR IOV (SR-PCIM interface of the PCI spec) for VF provisioning, for example, MSI-X vectors

3. SR-IOV VF Live migration
4. Above #1 to #3 for SIOV devices instead of SRIOV devices

5. cmd q interface for PF/VF/SF/SIOV to transport their own features, config command over queue interface without any parent device

#1 to #3 are well described in past. So, It is worth covering them in the cover letter and things will be clear for new eyes.
#4 is broad use case and like #1 to #3 that will eventually happen.
#5 likely needs more discussion, we can differ to a later point because AQ needs to progress in spec 1.3.



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 01:04:42PM +0200, Max Gurtovoy wrote:
> 
> 
> On 03/03/2023 15:19, Michael S. Tsirkin wrote:
> > On Fri, Mar 03, 2023 at 08:17:03AM -0500, Stefan Hajnoczi wrote:
> > > On Thu, Mar 02, 2023 at 07:01:56PM -0500, Michael S. Tsirkin wrote:
> > > > On Thu, Mar 02, 2023 at 03:19:12PM -0500, Stefan Hajnoczi wrote:
> > > > > On Thu, Mar 02, 2023 at 06:40:29PM +0000, Parav Pandit wrote:
> > > > > > 
> > > > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > > > Sent: Thursday, March 2, 2023 8:05 AM
> > > > > > 
> > > > > > > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> > > > > > > +is reserved and set to zero by the device.
> > > > > > > +
> > > > > > s/status_qialifier/status_qualifier
> > > > > > Missed from v10 of Feb.
> > > > > > 
> > > > > > > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> > > > > > > +describes possible \field{status_qialifier} values:
> > > > > > s/status_qialifier/status_qualifier
> > > > > > 
> > > > > > Can you please add other useful error codes in addition to the EINVAL?
> > > > > > Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
> > > > > 
> > > > > Please define a unique constant for each error condition that can occur
> > > > > instead of sharing catch-all errno constants between multiple error
> > > > > conditions. If a driver wants to squash them together into an errno,
> > > > > that's fine, but I think doing this at the hardware interface level is
> > > > > just propagating the mistakes of errnos.
> > > > > 
> > > > > Only status_qualifier is needed and the vague status field can be
> > > > > dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and
> > > > > ENODEV is useful. They have no meaning to the driver, only the
> > > > > status_qualifier really indicates what is going on.
> > > > 
> > > > At a high level at the moment we have only two cases:
> > > > - ok
> > > > - invalid input supplied by driver
> > > > 
> > > > maybe we will have more reasons for a failure - remains to
> > > > be seen.
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > > 
> > > > > I'm sure you guys have discussed this previously, but please provide
> > > > > rationale in the spec because it looks weird to someone with fresh eyes.
> > > > > 
> > > > > Stefan
> > > > 
> > > > Really most drivers just want to propagate errno to userspace.
> > > > All the detailed reporting is for sure well intentional but
> > > > in the end it is at best printed into log - end to end
> > > > people just end up with a switch statement
> > > > converting these to errno codes.
> > > > So we are passing them from device and this way there will be
> > > > some uniformity.
> > > 
> > > Please clarify the rationale in the spec. I don't agree with it, as
> > > explained in my earlier email, but as long as its documented, people can
> > > try to follow it.
> > > 
> > > Stefan
> > 
> > It's 2:2 for now, you are with Parav, me and Cornelia like it :)
> > Or I will try to document it better.
> I don't understand this status_qualifier as well and it wasn't included in
> my original patch set.

Sounds like you feel I should drop your S.O.B - is this the complaint?
I wanted to give attribution since I started with that but sure, no
problem.

> I vote for "status" that describe generic status codes and
> "command_specific_error" that should be inspected by the driver only if
> "status" is set to "VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR".
> We discussed this so many times before (and already agreed IIRC) and adding
> this new qualifiers mechanism sounds not right to me and not intuitive for
> device and driver developers.
> 
> I suggest:
> 
> 1. VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE
> 2. VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> 3. VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP
> 4. VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER
> 5. VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR (for more info read the
> command_specific_error field).

I don't think it's a good idea, we'll have to agree to disagree.


The point is simple. We can have a detailed virtio specific error.
Nice for debugging most drivers won't know what to do with it.
This is the status_qualifier.
Very detailed but
generally drivers will just have a giant switch statement translating
it to a simple error code for userspace.
So to save everyone work, we also added "status"
a generic kind of error class that is easy to pass to userspace
with a small switch statement.

COMMAND_SPECIFIC_ERR is just way too much detail - commands generally
just should not fail it's a quality of implementation issue.



> 
> 
> > 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, March 7, 2023 7:34 PM
> COMMAND_SPECIFIC_ERR is just way too much detail - commands generally
> just should not fail it's a quality of implementation issue.

I disagree partially.

1. Hw device != sw_hypervisor device.
Device may fail or error out that may need want sw driver to retry.
Like Stefan's example, it may need to return timeout/retry intermittently.
Doesnt means the device is broken at that point.

2. A device implementation may not have imposed a certain locking scheme to synchronize VF enablement with VF provisioning.
ENODEV can reflect two commands not synchronized.

So Boolean 0 = success, 22 = error is not the right way to craft the spec.
Many times, those sub-error codes are good indications of what may have gone wrong in the field.
Useful for the quality issue you pointed out to debug.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 01:01:27AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, March 7, 2023 7:34 PM
> > COMMAND_SPECIFIC_ERR is just way too much detail - commands generally
> > just should not fail it's a quality of implementation issue.
> 
> I disagree partially.
> 
> 1. Hw device != sw_hypervisor device.
> Device may fail or error out that may need want sw driver to retry.
> Like Stefan's example, it may need to return timeout/retry intermittently.
> Doesnt means the device is broken at that point.
> 
> 2. A device implementation may not have imposed a certain locking scheme to synchronize VF enablement with VF provisioning.
> ENODEV can reflect two commands not synchronized.
> 
> So Boolean 0 = success, 22 = error is not the right way to craft the spec.
> Many times, those sub-error codes are good indications of what may have gone wrong in the field.
> Useful for the quality issue you pointed out to debug.

Maybe, but I think we can just leave this stuff for later. Too much
hand-waving, when we add commands that actually need this kind of
ability that is when we will add the relevant error codes.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, March 7, 2023 8:06 PM
> 
> On Wed, Mar 08, 2023 at 01:01:27AM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, March 7, 2023 7:34 PM COMMAND_SPECIFIC_ERR is just
> > > way too much detail - commands generally just should not fail it's a
> > > quality of implementation issue.
> >
> > I disagree partially.
> >
> > 1. Hw device != sw_hypervisor device.
> > Device may fail or error out that may need want sw driver to retry.
> > Like Stefan's example, it may need to return timeout/retry intermittently.
> > Doesnt means the device is broken at that point.
> >
> > 2. A device implementation may not have imposed a certain locking scheme
> to synchronize VF enablement with VF provisioning.
> > ENODEV can reflect two commands not synchronized.
> >
> > So Boolean 0 = success, 22 = error is not the right way to craft the spec.
> > Many times, those sub-error codes are good indications of what may have
> gone wrong in the field.
> > Useful for the quality issue you pointed out to debug.
> 
> Maybe, but I think we can just leave this stuff for later. Too much hand-waving,
> when we add commands that actually need this kind of ability that is when we
> will add the relevant error codes.
Sounds good to me.
I thought the whole reason you wanted to use errno.h was to reuse all applicable error codes from it.
And so doing it in one go appeared natural to me.
As long as adding new error codes doesn't need new feature bits or new/extension of list commands, adding later looks good to me.
As long as we agree that adding more error codes to reflect specific errors is fine, (not Boolean 22), seems like a good direction.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Jason Wang]

On Wed, Mar 8, 2023 at 3:31 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, March 7, 2023 12:20 PM
>
> [..]
> > > >For SR-IOV, it is not unusual for PFs to excercise control over VFs.
> > >
> > > I understand the concepts of SR-IOV. Yet I fail to see the need of
> > > such concept in virtio world. SR-IOV is very specific solution for PCI
> > > functions instantiation, and I believe that it is already considered
> > > quite limiting in many aspects. Does not make sense to me to introduce
> > > it for virtio. But again, I may be missing something crucial, I just
> > > would like to see the motivation, needs, usecases for this crystal
> > > clear, which is opposite to the current cover letter I'm afraid :/
> >
> > First people are asking for it because it's out there, however limiting it is. In
> > fact Nvidia is - why don't you talk to Parav here and tell him that SR-IOV is
> > legacy and there's no need to support.
> >
> Jiri and I discussed offline today.
> SR-IOV is widely used today and in the coming future up to thousands of devices.
> Many vendors are building it or already built.
>
> SIOV as it stands today is still being crafted at various levels iommu, os, device, pci and more.
> Not there yet fully.
>
> Jiri and I built the Linux kernel interfaces for SFs which are equivalent to SIOVs that overcome some of the SR-IOV limitations.
> And Jiri's suggestion was to leverage some of the vision and practice from there.
>
> In other words, can AQ command be useful tomorrow for doing SIOV device add/remove, and provisioning from non-owning PF?
> The way AQ is crafted today is not there yet, but in the future, it can be extended.

There's already a proposal to use the queue as a transport. Any device
that can't use a transport specific configuration or provisioning can
benefit from that. SIOV is only one of the users for that.

> >
> > >
> > > >There is interest in the community to include an interface to allow
> > > >this in the virtio spec, when the PF is a virtio device.  This is
> > > >what this patch does.
> > >
> > > Yeah, but why? As I asked before, what are the usecases? The fact
> > > there is interest in the community does not mean it makes sense to
> > > have it :)
> > >
> >
> > If people want to build such hardware it will need some interface.
> > Far better to have it standard.
> >
> >
> > But generally e.g. intel already said they will reuse this same structure with a
> > different group type for SIOV support.
> > I'll mention this in the cover letter.
> >
>
> We have the following already identified use cases of a device agnostic VQ.
>
> 1. SR-IOV VF device provisioning at virtio layer (features and config)
> 2. SR IOV (SR-PCIM interface of the PCI spec) for VF provisioning, for example, MSI-X vectors
>

The virtio layer device provisioning is transport independent. I don't
see anything that is PCI specific there. Even the MSI-X could be
reused by other transport.

> 3. SR-IOV VF Live migration

I think we don't want a partial solution for live migration that only
works for PCI VF devices and L1. Instead, there needs to be a general
device facility section and the admin virtqueue could be one of the
interfaces for the driver to use the facility.

> 4. Above #1 to #3 for SIOV devices instead of SRIOV devices

If we make the above as a general facility, it would not be hard to
re-use them for transport virtqueue or SIOV.

Thanks

>
> 5. cmd q interface for PF/VF/SF/SIOV to transport their own features, config command over queue interface without any parent device
>
> #1 to #3 are well described in past. So, It is worth covering them in the cover letter and things will be clear for new eyes.
> #4 is broad use case and like #1 to #3 that will eventually happen.
> #5 likely needs more discussion, we can differ to a later point because AQ needs to progress in spec 1.3.
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jason Wang]

On Wed, Mar 8, 2023 at 3:09 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Stefan Hajnoczi <stefanha@redhat.com>
> > Sent: Tuesday, March 7, 2023 2:04 PM
>
> > An alternative is unconditional out-of-order completion, where there are no
> > per-command ordering rules. The driver must wait for a command to complete
> > if it relies on the results of that command for its next command. I like this
> > approach because it's less complex in the spec and for device implementers,
> > while the burden on the driver implementer is still reasonable.
> +1.

Note that this is the way current ctrl virtqueue works.

> This has best of both.
> 1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
> 2. Ability to execute multiple unrelated commands using a single AQ.
> 3. stateless device in AQ command execution

Does this mean if we want to migrate AQ (not use AQ to migrate), we
need to wait for the AQ command to be completed?

Thanks

>
>
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Jiri Pirko]

Tue, Mar 07, 2023 at 06:20:03PM CET, mst@redhat.com wrote:
>On Tue, Mar 07, 2023 at 08:21:54AM +0100, Jiri Pirko wrote:
>> Mon, Mar 06, 2023 at 11:54:45PM CET, mst@redhat.com wrote:
>> >On Mon, Mar 06, 2023 at 01:29:30PM +0100, Jiri Pirko wrote:
>> >> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
>> >> 
>> >> [...]
>> >> 
>> >> >
>> >> >TODO (maybe?) - probably ok to defer until this part is upstream:
>> >> >
>> >> >	Add "all members" member id.
>> >> >
>> >> >	Add commands for MSI, feature discovery.
>> >> >
>> >> >	Add commands for transport vq.
>> >> >
>> >> >
>> >> >My intent is to try and support both SR-IOV and SIOV
>> >> >usecases with the same structure and maybe even the same
>> >> >VQ.
>> >> 
>> >> Sorry to be late to the party, I'm trying to catch up. Is it common to
>> >> have cover letter for features this brief? I mean, from the cover
>> >> letter, I'm totally unable to understand what you are introducing here.
>> >> 
>> >> Could you elaborate about what you are aiming to achive with this?
>> >> Could you shed some usecases perhaps?
>> >> 
>> >> I have to be missing something obvious, but I don't get why any notion
>> >> of SR-IOV could be beneficial for virtio.
>> >> 
>> >
>> >Good point, I'll add a bit of motivation.
>> >
>> >For SR-IOV, it is not unusual for PFs to excercise control over VFs.
>> 
>> I understand the concepts of SR-IOV. Yet I fail to see the need of such
>> concept in virtio world. SR-IOV is very specific solution for PCI
>> functions instantiation, and I believe that it is already considered
>> quite limiting in many aspects. Does not make sense to me to introduce
>> it for virtio. But again, I may be missing something crucial, I just
>> would like to see the motivation, needs, usecases for this crystal
>> clear, which is opposite to the current cover letter I'm afraid :/
>
>First people are asking for it because it's out there, however limiting
>it is. In fact Nvidia is - why don't you talk to Parav here and tell
>him that SR-IOV is legacy and there's no need to support.

Yep, I talked to Parav, makes much more sense now for me as I now
understand the motivation. I was not able to reach there just by reading
the patchset :/


>
>
>
>
>> 
>> >There is interest in the community to include an interface to allow this
>> >in the virtio spec, when the PF is a virtio device.  This is what this
>> >patch does.
>> 
>> Yeah, but why? As I asked before, what are the usecases? The fact there
>> is interest in the community does not mean it makes sense to have it :)
>> 
>
>If people want to build such hardware it will need some interface.
>Far better to have it standard.
>
>
>But generally e.g. intel already said they will reuse this
>same structure with a different group type for SIOV support.
>I'll mention this in the cover letter.

Cool.


>
>> >
>> >
>> >
>> >
>> >> >
>> >> >For example, it might make sense to split creating/destroying
>> >> >SIOV devices from the transport passing data from the guest - the
>> >> >driver would then not negotiate VIRTIO_F_SR_IOV (which
>> >> >then means auto-provisioning).
>> >> >
>> >> >This is out of RFC, since we have two commands which are useful
>> >> >to discover supported group types (ATM can be none or SR-IOV).
>> >> >
>> >> >
>> >> >Michael S. Tsirkin (10):
>> >> >  virtio: document forward compatibility guarantees
>> >> >  admin: introduce device group and related concepts
>> >> >  admin: introduce group administration commands
>> >> >  admin: introduce virtio admin virtqueues
>> >> >  pci: add admin vq registers to virtio over pci
>> >> >  mmio: document ADMIN_VQ as reserved
>> >> >  ccw: document ADMIN_VQ as reserved
>> >> >  admin: command list discovery
>> >> >  admin: conformance clauses
>> >> >  ccw: document more reserved features
>> >> >
>> >> > admin.tex        | 540 +++++++++++++++++++++++++++++++++++++++++++++++
>> >> > content.tex      | 112 +++++++++-
>> >> > introduction.tex |   3 +
>> >> > 3 files changed, 653 insertions(+), 2 deletions(-)
>> >> > create mode 100644 admin.tex
>> >> >
>> >> >-- 
>> >> >MST
>> >> >
>> >> >
>> >> >This publicly archived list offers a means to provide input to the
>> >> >OASIS Virtual I/O Device (VIRTIO) TC.
>> >> >
>> >> >In order to verify user consent to the Feedback License terms and
>> >> >to minimize spam in the list archive, subscription is required
>> >> >before posting.
>> >> >
>> >> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> >> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> >> >List help: virtio-comment-help@lists.oasis-open.org
>> >> >List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> >> >Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> >> >List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> >> >Committee: https://www.oasis-open.org/committees/virtio/
>> >> >Join OASIS: https://www.oasis-open.org/join/
>> >> >
>> >> 
>> >> This publicly archived list offers a means to provide input to the
>> >> OASIS Virtual I/O Device (VIRTIO) TC.
>> >> 
>> >> In order to verify user consent to the Feedback License terms and
>> >> to minimize spam in the list archive, subscription is required
>> >> before posting.
>> >> 
>> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> >> List help: virtio-comment-help@lists.oasis-open.org
>> >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> >> Committee: https://www.oasis-open.org/committees/virtio/
>> >> Join OASIS: https://www.oasis-open.org/join/
>> >
>
>
>---------------------------------------------------------------------
>To unsubscribe from this mail list, you must leave the OASIS TC that 
>generates this mail.  Follow this link to all your TCs in OASIS at:
>https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Tue, Mar 07, 2023 at 05:30:18PM CET, mst@redhat.com wrote:
>On Tue, Mar 07, 2023 at 08:36:41AM +0100, Jiri Pirko wrote:
>> Hmm, if not for now, the future exension would not be so simple, I fear.
>
>Without knowing what it is I can't say.

Yep, so basically you say, for other things if they appear,
let's introduce another queue type? If yes, sounds fair to me.


>
>> 
>> >
>> >Passing commands to devices themselves is already covered in spec
>> >reasonably well though not in a generic way.
>> 
>> You mean using the control queue, correct?
>
>Depends on the device type. network devices have a control queue, yes.
>
>> >From one of the patch description of this patchset I understand that you
>> cannot use control queue for this because control queue is
>> device-specific, yet group control is device-agnostic.
>> 
>> My undestanding therefore was, that the admin queue you are introducing
>> serves as a generic carrier for device-agnostic commands, in parallel
>> for having control queue serving as a generic carrier of device-specific
>> commands. If this is not the case, I think it would be nice to describe
>> the exact monivation and scope of admin queue.
>
>Nope unfortunately.  This queue is just a carrier for admin commands.
>admin commands are commands that talk to one device about other
>devices. There's clearly no mechanism in the spec to do that,
>so we plug this hole.

Okay, in that case "admin" sounds a bit misleading as for me it
implicates that this is for "administration" of the device. Yet is is
for the administration of other devices (slaves).

Perhaps there could be different term used to clarify?
Group leader virtqueue?
Group owner virtqueue?
Group master virtqueue?

>
>
>
>> 
>> >
>> >What we lack is passing commands about one device to another device.
>> >E.g. control VFs through PFs.
>> 
>> Could you provide examples of such commands please?
>
>For example a common feature is to program a vlan and have device
>put a given VF inside this vlan.

I don't follow entirely. The way how the VF is connected to network
should be ouf of the scope of this interface. The eswitch manager should
take care. What you say sounds awfully like the "ip vf" legacy
interface, which should not be considered here I believe.

If PF would be the eswitch manager, there are other means to do network
programming, using eswitch port representors. But I don't think this is
the can of worms we want to open now. I don't think we have a usecase
for it currently. Am I wrong Parav?



>
>In a virtualization scenario host controls this vlan programming giving
>the network a measure of protection from VFs.  If a VF is passed through
>to a VM, IOMMU limits VFs to only access guest memory so host has to do
>this programming through a PF.

Understood. This really looks like "ip vf" legacy. I strongly believe
it should not be supported.

Any other commands you have in mind?


>
>
>
>> 
>> >This is what groups do.
>> >But if we see more uses we can always add them.
>> >
>> >
>> >I'd rather avoid being too generic though.
>> 
>> In that case, why not to avoid using generic terms and stay
>> "group-centric"? What I mean is:
>> "Administration Virtqueues" -> "Group Administration Virtqueues"
>> "struct virtio_admin_cmd" -> "struct virtio_group_admin_cmd"
>> 
>> Etc. Helps to avoid confusion.
>
>Sure, I tried to do that but missed some opportunities.
>Will address.

Cool.


>
>> 
>> >
>> >
>> >
>> >
>> >> 
>> >> >+than one administration virtqueue.
>> 
>> 
>> [...]
>> 
>
>
>This publicly archived list offers a means to provide input to the
>OASIS Virtual I/O Device (VIRTIO) TC.
>
>In order to verify user consent to the Feedback License terms and
>to minimize spam in the list archive, subscription is required
>before posting.
>
>Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>List help: virtio-comment-help@lists.oasis-open.org
>List archive: https://lists.oasis-open.org/archives/virtio-comment/
>Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>Committee: https://www.oasis-open.org/committees/virtio/
>Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Tue, Mar 07, 2023 at 05:13:01PM CET, mst@redhat.com wrote:
>On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
>> I sense there is no concete idea about what the "admin virtqueue" should
>> serve for exactly.
>
>Because the virtqueue is just a virtqueue - a way to pass around
>commands.  Admin commands are used whenever one needs to talk to one
>device about another one.

Yeah, I got that now. Please check my other reply in this thread
suggesting a name change of "admin". Perhaps it would make sense and
makes thing a bit clearer for slower people like myself :)


>-- 
>MST
>
>
>This publicly archived list offers a means to provide input to the
>OASIS Virtual I/O Device (VIRTIO) TC.
>
>In order to verify user consent to the Feedback License terms and
>to minimize spam in the list archive, subscription is required
>before posting.
>
>Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>List help: virtio-comment-help@lists.oasis-open.org
>List archive: https://lists.oasis-open.org/archives/virtio-comment/
>Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>Committee: https://www.oasis-open.org/committees/virtio/
>Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Tue, Mar 07, 2023 at 08:03:47PM CET, stefanha@redhat.com wrote:
>On Tue, Mar 07, 2023 at 04:07:54PM +0100, Jiri Pirko wrote:
>> Tue, Mar 07, 2023 at 03:39:11PM CET, stefanha@redhat.com wrote:
>> >On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
>> >> Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
>> >> >On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
>> >> >> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
>> >> >> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
>> >> >> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
>> >> >> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
>> >> >> > > > > What happens if a command takes 1 second to complete, is the device
>> >> >> > > > > allowed to process the next command from the virtqueue during this time,
>> >> >> > > > > possibly completing it before the first command?
>> >> >> > > > > 
>> >> >> > > > > This requires additional clarification in the spec because "they are
>> >> >> > > > > processed by the device in the order in which they are queued" does not
>> >> >> > > > > explain whether commands block the virtqueue (in order completion) or
>> >> >> > > > > not (out of order completion).
>> >> >> > > > 
>> >> >> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
>> >> >> > > 
>> >> >> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
>> >> >> > > Several may be processed by the device at the same time.
>> >> >> > 
>> >> >> > Let's say I submit a write followed by read - is read
>> >> >> > guaranteed to return an up to date info?
>> >> >> 
>> >> >> In general, no. The driver must wait for the write completion before
>> >> >> submitting the read if it wants consistency.
>> >> >> 
>> >> >> Stefan
>> >> >
>> >> >I see.  I think it's a good design to follow then.
>> >> 
>> >> Hmm, is it suitable to have this approach for configuration interface?
>> >> Storage device is a different beast, having parallel reads and writes
>> >> makes complete sense for performance.
>> >> 
>> >> ->read a req
>> >> ->read b req
>> >> ->read c req
>> >> <-read a rep
>> >> <-read b rep
>> >> <-read c rep
>> >> 
>> >> There is no dependency, even between writes.
>> >> 
>> >> But in case of configuration, does not make any sense to me.
>> >> Why is it needed? To pass the burden of consistency of
>> >> configuration to driver sounds odd at least.
>> >> 
>> >> I sense there is no concete idea about what the "admin virtqueue" should
>> >> serve for exactly.
>> >
>> >It's useful for long-running commands because they prevent other
>> >commands from executing.
>> >
>> >An example I've given is that deleting a group member might require
>> >waiting for the group member's I/O activity to finish. If that I/O
>> >activity cannot be cancelled instantaneously, then it could take an
>> >unbounded amount of time to delete the group member. The device would be
>> >unable to process futher admin commands.
>> 
>> I see. Then I believe that the device should handle the dependencies.
>> Example 1:
>> -> REQ cmd to create group member A
>> -> REQ cmd to create group member B
>> <- REP cmd to create group member A
>> <- REP cmd to create group member B
>> 
>> The device according to internal implementation can either serialize the
>> 2 group member creations or do it in parallel, if it supports it.
>> 
>> Example 2:
>> -> REQ cmd to create group member A
>> -> REQ cmd config group member A
>> <- REP cmd to create group member A
>> <- REP cmd config group member A
>> 
>> Here the serialization is necessary and the device is the one to take
>> care of it.
>> 
>> Makes sense?
>
>Yes, I understand. The spec would need to define ordering rules for
>specific commands and the device must implement them. It allows the
>driver to pipeline commands while also allowing out-of-order completion
>(parallelism) in some cases. The disadvantage of this approach is
>complexity in the spec and implementations.
>
>An alternative is unconditional out-of-order completion, where there are
>no per-command ordering rules. The driver must wait for a command to
>complete if it relies on the results of that command for its next
>command. I like this approach because it's less complex in the spec and
>for device implementers, while the burden on the driver implementer is
>still reasonable.

But isn't this duplicating the burden of maintaining dependencies to
both driver and device? I mean, device should not depend on driver doing
the right thing, that means it has to check the dependencies for every
incoming command anyway. The only difference would be to wait instead of
returning "-EBUSY" in case the dependency is not satisfied yet.

Device knows exactly what are the dependencies. And I believe, those are
device implementation specific. For example, some implementation could
support parallel VF config cmd execution, some implementation might
need to serialize that. Driver has no clue.

Could you please elaborate a bit more what you mean by "complexity in
the spec"?

Thanks!


>
>> >
>> >Group member creation might have similar issues if it involves acquiring
>> >remote resources (e.g. connecting to a Ceph cluster or allocating ports
>> >on a distributed network switch). It can be impossible to defer resource
>> 
>> Sidetrack: this is really fuzzy to me, how the new member is going to be
>> plugged into backend (network). Over the time, we learned that the
>> creation of device from the other side (switch side) makes more sense.
>> That is why I asked for motivation to introduce this infra.
>
>Michael, have you already thought about this?
>
>> >acquisition/initialization because because VIRTIO devices must be
>> >available as soon as the driver can see them (i.e. how do populate
>> >Configuration Space fields if you don't have the details of the resource
>> >yet?).
>> >
>> >So I have raised two questions:
>> >
>> >1. What are the admin queue command completion semantics: in-order or
>> >   out-of-order command completion?
>> 
>> I would add "dependencies/serialization" here.
>> 
>> 
>> >
>> >2. Will there be long-running commands and how will we deal with them
>> >   when they hang?
>> 
>> Yeah, sounds legit to define it in spec.
>> 
>> >
>> >Stefan
>> 
>> 



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 11:08:56AM +0100, Jiri Pirko wrote:
> Tue, Mar 07, 2023 at 05:13:01PM CET, mst@redhat.com wrote:
> >On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
> >> I sense there is no concete idea about what the "admin virtqueue" should
> >> serve for exactly.
> >
> >Because the virtqueue is just a virtqueue - a way to pass around
> >commands.  Admin commands are used whenever one needs to talk to one
> >device about another one.
> 
> Yeah, I got that now. Please check my other reply in this thread
> suggesting a name change of "admin". Perhaps it would make sense and
> makes thing a bit clearer for slower people like myself :)

Yes I think it's a good idea. Replied there too.

> 
> >-- 
> >MST
> >
> >
> >This publicly archived list offers a means to provide input to the
> >OASIS Virtual I/O Device (VIRTIO) TC.
> >
> >In order to verify user consent to the Feedback License terms and
> >to minimize spam in the list archive, subscription is required
> >before posting.
> >
> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >List help: virtio-comment-help@lists.oasis-open.org
> >List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >Committee: https://www.oasis-open.org/committees/virtio/
> >Join OASIS: https://www.oasis-open.org/join/
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 11:05:00AM +0100, Jiri Pirko wrote:
> Tue, Mar 07, 2023 at 05:30:18PM CET, mst@redhat.com wrote:
> >On Tue, Mar 07, 2023 at 08:36:41AM +0100, Jiri Pirko wrote:
> >> Hmm, if not for now, the future exension would not be so simple, I fear.
> >
> >Without knowing what it is I can't say.
> 
> Yep, so basically you say, for other things if they appear,
> let's introduce another queue type? If yes, sounds fair to me.

Yes. For example I find it likely that live migration/failover support
will require a queue where driver pre-adds buffers and then device
supplies information as state changes.

> 
> >
> >> 
> >> >
> >> >Passing commands to devices themselves is already covered in spec
> >> >reasonably well though not in a generic way.
> >> 
> >> You mean using the control queue, correct?
> >
> >Depends on the device type. network devices have a control queue, yes.
> >
> >> >From one of the patch description of this patchset I understand that you
> >> cannot use control queue for this because control queue is
> >> device-specific, yet group control is device-agnostic.
> >> 
> >> My undestanding therefore was, that the admin queue you are introducing
> >> serves as a generic carrier for device-agnostic commands, in parallel
> >> for having control queue serving as a generic carrier of device-specific
> >> commands. If this is not the case, I think it would be nice to describe
> >> the exact monivation and scope of admin queue.
> >
> >Nope unfortunately.  This queue is just a carrier for admin commands.
> >admin commands are commands that talk to one device about other
> >devices. There's clearly no mechanism in the spec to do that,
> >so we plug this hole.
> 
> Okay, in that case "admin" sounds a bit misleading as for me it
> implicates that this is for "administration" of the device. Yet is is
> for the administration of other devices (slaves).
> 
> Perhaps there could be different term used to clarify?
> Group leader virtqueue?
> Group owner virtqueue?
> Group master virtqueue?

I used group administration virtqueue in a couple of places,
just inconsistently. Good enough?


> >
> >
> >
> >> 
> >> >
> >> >What we lack is passing commands about one device to another device.
> >> >E.g. control VFs through PFs.
> >> 
> >> Could you provide examples of such commands please?
> >
> >For example a common feature is to program a vlan and have device
> >put a given VF inside this vlan.
> 
> I don't follow entirely. The way how the VF is connected to network
> should be ouf of the scope of this interface. The eswitch manager should
> take care. What you say sounds awfully like the "ip vf" legacy
> interface, which should not be considered here I believe.
> 
> If PF would be the eswitch manager, there are other means to do network
> programming, using eswitch port representors. But I don't think this is
> the can of worms we want to open now. I don't think we have a usecase
> for it currently. Am I wrong Parav?
> 
> 
> 
> >
> >In a virtualization scenario host controls this vlan programming giving
> >the network a measure of protection from VFs.  If a VF is passed through
> >to a VM, IOMMU limits VFs to only access guest memory so host has to do
> >this programming through a PF.
> 
> Understood. This really looks like "ip vf" legacy. I strongly believe
> it should not be supported.
> 
> Any other commands you have in mind?
> 
> 
> >
> >
> >
> >> 
> >> >This is what groups do.
> >> >But if we see more uses we can always add them.
> >> >
> >> >
> >> >I'd rather avoid being too generic though.
> >> 
> >> In that case, why not to avoid using generic terms and stay
> >> "group-centric"? What I mean is:
> >> "Administration Virtqueues" -> "Group Administration Virtqueues"
> >> "struct virtio_admin_cmd" -> "struct virtio_group_admin_cmd"
> >> 
> >> Etc. Helps to avoid confusion.
> >
> >Sure, I tried to do that but missed some opportunities.
> >Will address.
> 
> Cool.
> 
> 
> >
> >> 
> >> >
> >> >
> >> >
> >> >
> >> >> 
> >> >> >+than one administration virtqueue.
> >> 
> >> 
> >> [...]
> >> 
> >
> >
> >This publicly archived list offers a means to provide input to the
> >OASIS Virtual I/O Device (VIRTIO) TC.
> >
> >In order to verify user consent to the Feedback License terms and
> >to minimize spam in the list archive, subscription is required
> >before posting.
> >
> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >List help: virtio-comment-help@lists.oasis-open.org
> >List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >Committee: https://www.oasis-open.org/committees/virtio/
> >Join OASIS: https://www.oasis-open.org/join/
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 01:22:30PM +0100, Jiri Pirko wrote:
> Thu, Mar 02, 2023 at 02:05:22PM CET, mst@redhat.com wrote:
> >Add commands to find out which commands does each group support,
> >as well as enable their use by driver.
> >This will be especially useful once we have multiple group types.
> >
> >An alternative is per-type VQs. This is possible but will
> >require more per-transport work. Discovery through the vq
> >helps keep things contained.
> >
> >Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> >Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> [...]
> 
> 
> >+
> >+The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> >+query the list of commands valid for this group and before sending
> >+any commands for any member of a group.
> >+
> >+The driver then enables use of some of the opcodes by sending to
> >+the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
> >+of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
> >+both understood and used by the driver.
> 
> To my untrained ear, this sounds somewhat similar to the feature
> negotiantion mechanism. Why the fact that device/driver supports some
> command can't be covered by just another feature? Looks like unnecassary
> complexicity to negotiate supported commands like this.

Absolutely, it is similar. The issue is that a single device can
be an owner for multiple group types.
For example, I think an SRIOV PF itself can have subfunctions with SIOV
as well as virtual functions with SRIOV.
And the set of supported commands might differ.

We thus need a command that is kind of like features but per group type.

As a small bonus as we are building this from scratch, I added
something which was requested for a long time from features
which is blocking access to commands not allowed by features.
With standard virtio features we trust the driver not to send
commands before enabling a feature and this was considered
a maintainance/security problem.


-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

On Tue, Mar 07, 2023 at 11:04:33AM +0000, David Edmondson wrote:
> "Michael S. Tsirkin" <mst@redhat.com> writes:
> 
> > Add conformance clauses for admin commands and admin virtqueues.
> >
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 215 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 1172054..6c4f79c 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  supporting multiple group types, the list of supported commands
> >  might differ between different group types.
> >  
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > +
> > +The device MUST validate \field{opcode}, \field{group_type} and
> > +\field{group_member_id}, and if any of these has an invalid or
> > +unsupported value, set \field{status} to
> > +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
> > +accordingly:
> > +\begin{itemize}
> > +\item if \field{group_type} is invalid, \field{status_qualifier}
> > +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> > +\item otherwise, if \field{opcode} is invalid,
> > +	\field{status_qualifier} MUST be set to
> > +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
> > +\item otherwise, if \field{group_member_id} is used by the
> > +	specific command and is invalid, \field{status_qualifier} MUST be
> > +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER.
> > +\end{itemize}
> > +
> > +If a command completes successfully, the device MUST set
> > +\field{status} to VIRTIO_ADMIN_STATUS_OK.
> > +
> > +If a command fails, the device MUST set
> > +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
> > +
> > +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
> > +device state MUST NOT change, that is the command MUST NOT have
> > +any side effects on the device, in particular the device MUST not
> 
> MUST NOT

/me nods

> > +enter an error state as a result of this command.
> > +
> > +If a command fails, the device state generally SHOULD NOT change,
> > +as far as possible.
> > +
> > +The device MAY enforce additional restrictions and dependencies on
> > +opcodes used by the driver and MAY fail the command
> > +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> > +if the list of commands used violate internal device dependencies.
> > +
> > +If the device supports multiple group types, commands for each group
> > +type MUST operate independently of each other, in particular,
> > +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
> > +for different group types.
> > +
> > +After reset, if the device supports a given group type
> > +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
> > +the device MUST assume
> > +that the list of legal commands used by the driver consists of
> > +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> > +
> > +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
> > +the device MUST set the list of legal commands used by the driver
> > +to the one supplied in \field{command_specific_data}.
> > +
> > +The device MUST set the list of legal commands used by the driver
> > +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
> 
> Are these last two paragraphs not saying the same thing?


They are!

> > +The device MUST validate commands against the list used by
> > +the driver and MUST fail any commands not in the list with
> > +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +and \field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
> > +
> > +The list of supported commands MUST NOT shrink (but MAY expand):
> > +after reporting a given command as supported through
> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> > +as unsupported.  Further, after a given set of commands has been
> > +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
> > +device or system reset the device SHOULD complete successfully
> > +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
> > +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
> > +after a device or system reset, the device MUST not fail it
> > +solely because of the command list used.  Failure to do so would
> > +interfere with resuming from suspend and error recovery.
> > +
> > +When processing a command with the SR-IOV group type,
> > +if the device does not have an SR-IOV Extended Capability or
> > +if \field{VF Enable} is clear
> > +then the device MUST fail all commands with
> > +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> > +\field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> > +otherwise, if \field{group_member_id} is not
> > +between $1$ and \field{NumVFs} inclusive,
> > +the device MUST fail all commands with
> > +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> > +\field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER;
> > +\field{NumVFs}, \field{VF Migration Capable}  and
> > +\field{VF Enable} refer to registers within the SR-IOV Extended
> > +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > +
> > +The driver MAY discover whether device supports a specific group type
> > +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
> > +\field{group_type}.
> > +
> > +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
> > +and wait for it to be completed with status
> > +VIRTIO_ADMIN_STATUS_OK before issuing any commands
> > +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
> > +and VIRTIO_ADMIN_CMD_LIST_USE).
> > +
> > +The driver SHOULD NOT set bits in device_admin_cmds
> > +if it is not familiar with how the command opcode
> > +is used, as dependencies between command opcodes might exist.
> > +
> > +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
> > +the use of any commands not previously reported as
> > +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
> > +
> > +The driver MUST NOT use any commands for a given group type
> > +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
> > +list of command opcodes and group type.
> > +
> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> > +with respective bits cleared in \field{command_specific_data}.
> 
> This runs contrary to the assertion "The list of supported commands MUST
> NOT shrink", given that a driver is told to assume that
> VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
> commands initially available.

Commands are still available just disabled, the meaning of
"MUST NOT shrink" is clarified by the following:

	> > +after reporting a given command as supported through
	> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
	> > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.



> 
> > +The driver MUST handle a command error with a reserved \field{status}
> > +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +(except possibly for different error reporting/diagnostic messages).
> > +
> > +The driver MUST handle a command error with a reserved
> > +\field{status_qualifier} value in the same way as
> > +\field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
> > +different error reporting/diagnostic messages).
> > +
> > +When sending commands with the SR-IOV group type,
> > +the driver specify a value for \field{group_member_id}
> > +between $1$ and \field{NumVFs} inclusive,
> > +the driver MUST also make sure that as long as any such command
> > +is outstanding, \field{VF Migration Capable} is clear and
> > +\field{VF Enable} is set;
> > +\field{NumVFs}, \field{VF Migration Capable}  and
> > +\field{VF Enable} refer to registers within the SR-IOV Extended
> > +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >  
> >  An administration virtqueue of an owner device is used to submit
> > @@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
> >  of the specification are designed, new fields can be added to the
> >  tail of a structure, with the driver/device using the full
> >  structure without concern for versioning.
> > ->>>>>>> 0edc690... admin: introduce virtio admin virtqueues
> 
> Oh, here is where it got removed :-)

/me blushes

> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> > +
> > +The device MUST support device-readable and device-writeable buffers
> > +shorter than described in this specification, by
> > +\begin{enumerate}
> > +\item acting as if any data that would be read outside the
> > +device-readable buffers is set to zero, and
> > +\item discarding data that would be written outside the
> > +specified device-writeable buffers.
> > +\end{enumerate}
> > +
> > +The device MUST support device-readable and device-writeable buffers
> > +longer than described in this specification, by
> > +\begin{enumerate}
> > +\item ignoring any data in device-readable buffers outside
> > +the expected length, and
> > +\item only writing the expected structure to the device-writeable
> > +buffers, ignoring any extra buffers, and reporting the
> > +actual length of data written, in bytes,
> > +as buffer used length.
> > +\end{enumerate}
> > +
> > +The device SHOULD initialize the device-writeable buffer
> > +up to the length of the structure described by this specification or
> > +the length of the buffer supplied by the driver (even if the buffer is
> > +all set to zero), whichever is shorter.
> > +
> > +The device MUST NOT fail a command solely because the buffers
> > +provided are shorter or longer than described in this
> > +specification.
> > +
> > +The device MUST initialize the device-writeable part of
> > +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
> > +size.
> > +
> > +The device MUST initialize \field{status} in \field{struct
> > +virtio_admin_cmd}.
> > +
> > +The device MUST process commands on a given administration virtqueue
> > +in the order in which they are queued.
> > +
> > +If multiple administration virtqueues have been configured,
> > +device MAY process commands on distinct virtqueues with
> > +no order constraints.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> > +
> > +The driver MAY supply device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are longer than described in
> > +this specification.
> > +
> > +The driver SHOULD supply device-readable part of
> > +\field{struct virtio_admin_cmd} that is at least as
> > +large as the structure described by this specification
> > +(even if the structure is all set to zero).
> > +
> > +The driver MUST supply both device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
> > +in length.
> 
> This addresses my question on a previous patch.
> 
> > +The device MUST supply both device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are larger than zero in
> > +length. However, \field{command_specific_data} and
> > +\field{command_specific_result} MAY be zero in length, unless
> > +specified otherwise for the command.
> > +
> > +The driver MUST NOT assume that the device will initialize the whole
> > +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
> > +the driver MUST act as if the structure
> > +outside the part of the buffer used by the device
> > +is set to zero.
> > +
> > +If multiple administration virtqueues have been configured,
> > +the driver MUST ensure ordering for commands
> > +placed on different administration virtqueues.
> > -- 
> > MST
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> -- 
> Music has magic, it's good clear syncopation.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1893 bytes --]

On Wed, Mar 08, 2023 at 01:17:44PM +0800, Jason Wang wrote:
> On Wed, Mar 8, 2023 at 3:09 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> >
> > > From: Stefan Hajnoczi <stefanha@redhat.com>
> > > Sent: Tuesday, March 7, 2023 2:04 PM
> >
> > > An alternative is unconditional out-of-order completion, where there are no
> > > per-command ordering rules. The driver must wait for a command to complete
> > > if it relies on the results of that command for its next command. I like this
> > > approach because it's less complex in the spec and for device implementers,
> > > while the burden on the driver implementer is still reasonable.
> > +1.
> 
> Note that this is the way current ctrl virtqueue works.
> 
> > This has best of both.
> > 1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
> > 2. Ability to execute multiple unrelated commands using a single AQ.
> > 3. stateless device in AQ command execution
> 
> Does this mean if we want to migrate AQ (not use AQ to migrate), we
> need to wait for the AQ command to be completed?

It depends. Today, if the AQ is emulated by the VMM then it might be
possible to migrate while there is a command in-flight. If the AQ is
handled by the device then no because VIRTIO currently does not support
device state migration.

In the future, device state migration will probably be added to
vhost-user and vDPA. In that case a device can migrate in-flight AQ
activity - assuming it's possible to pause it, serialize state,
deserialize it, and resume it. Otherwise it's still necessary to wait.

To be clear, I'm just taking about AQ commands that are currently in the
process of executing. Commands that are in the virtqueue but have not
yet been started by the device can be easily migrated using the existing
virtqueue migration infrastructure.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 01:17:44PM +0800, Jason Wang wrote:
> On Wed, Mar 8, 2023 at 3:09 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> >
> > > From: Stefan Hajnoczi <stefanha@redhat.com>
> > > Sent: Tuesday, March 7, 2023 2:04 PM
> >
> > > An alternative is unconditional out-of-order completion, where there are no
> > > per-command ordering rules. The driver must wait for a command to complete
> > > if it relies on the results of that command for its next command. I like this
> > > approach because it's less complex in the spec and for device implementers,
> > > while the burden on the driver implementer is still reasonable.
> > +1.
> 
> Note that this is the way current ctrl virtqueue works.
> 
> > This has best of both.
> > 1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
> > 2. Ability to execute multiple unrelated commands using a single AQ.
> > 3. stateless device in AQ command execution
> 
> Does this mean if we want to migrate AQ (not use AQ to migrate), we
> need to wait for the AQ command to be completed?
> 
> Thanks

As with any VQ, if you migrate while an operation is in progress
you need to figure out the internal device state.

> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe from this mail list, you must leave the OASIS TC that
> > generates this mail.  Follow this link to all your TCs in OASIS at:
> > https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Wednesday, March 8, 2023 12:12 AM
> > In other words, can AQ command be useful tomorrow for doing SIOV device


> add/remove, and provisioning from non-owning PF?
> > The way AQ is crafted today is not there yet, but in the future, it can be
> extended.
> 
> There's already a proposal to use the queue as a transport. Any device that
> can't use a transport specific configuration or provisioning can benefit from
> that. SIOV is only one of the users for that.
>
o.k. Better to include that link in the cover letter.
 
> > >
> > > >
> > > > >There is interest in the community to include an interface to
> > > > >allow this in the virtio spec, when the PF is a virtio device.
> > > > >This is what this patch does.
> > > >
> > > > Yeah, but why? As I asked before, what are the usecases? The fact
> > > > there is interest in the community does not mean it makes sense to
> > > > have it :)
> > > >
> > >
> > > If people want to build such hardware it will need some interface.
> > > Far better to have it standard.
> > >
> > >
> > > But generally e.g. intel already said they will reuse this same
> > > structure with a different group type for SIOV support.
> > > I'll mention this in the cover letter.
> > >
> >
> > We have the following already identified use cases of a device agnostic VQ.
> >
> > 1. SR-IOV VF device provisioning at virtio layer (features and config)
> > 2. SR IOV (SR-PCIM interface of the PCI spec) for VF provisioning, for
> > example, MSI-X vectors
> >
> 
> The virtio layer device provisioning is transport independent. 
Yes. I listed known use cases above. If there are more that can use, sure.

> I don't see
> anything that is PCI specific there. Even the MSI-X could be reused by other
> transport.
>
Maybe yes.
I just don’t know which other transport of virtio has defined MSI-X.
If there is, sure there are more users of AQ.
 
> > 3. SR-IOV VF Live migration
> 
> I think we don't want a partial solution for live migration that only works for PCI
> VF devices and L1. Instead, there needs to be a general device facility section
> and the admin virtqueue could be one of the interfaces for the driver to use the
> facility.
> 
May be there are more use cases of AQ than what I listed.

> > 4. Above #1 to #3 for SIOV devices instead of SRIOV devices
> 
> If we make the above as a general facility, it would not be hard to re-use them
> for transport virtqueue or SIOV.
>
Ok. thanks.

Re: [virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Tue, Mar 07, 2023 at 12:31:51PM +0100, Jiri Pirko wrote:
> Fri, Mar 03, 2023 at 09:23:14PM CET, stefanha@redhat.com wrote:
> >On Fri, Mar 03, 2023 at 08:18:43AM -0500, Michael S. Tsirkin wrote:
> >> On Fri, Mar 03, 2023 at 08:13:37AM -0500, Stefan Hajnoczi wrote:
> >> > On Thu, Mar 02, 2023 at 06:57:24PM -0500, Michael S. Tsirkin wrote:
> >> > > On Thu, Mar 02, 2023 at 03:10:11PM -0500, Stefan Hajnoczi wrote:
> >> > > > On Thu, Mar 02, 2023 at 08:05:02AM -0500, Michael S. Tsirkin wrote:
> >> > > > > This introduces a general structure for group administration commands,
> >> > > > > used to control device groups through their owner.
> >> > > > > 
> >> > > > > Following patches will introduce specific commands and an interface for
> >> > > > > submitting these commands to the owner.
> >> > > > > 
> >> > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> >> > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >> > > > > ---
> >> > > > >  admin.tex        | 108 +++++++++++++++++++++++++++++++++++++++++++++++
> >> > > > >  introduction.tex |   3 ++
> >> > > > >  2 files changed, 111 insertions(+)
> >> > > > > 
> >> > > > > diff --git a/admin.tex b/admin.tex
> >> > > > > index 3dc47be..7e28b77 100644
> >> > > > > --- a/admin.tex
> >> > > > > +++ b/admin.tex
> >> > > > > @@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
> >> > > > >  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> >> > > > >  \end{description}
> >> > > > >  
> >> > > > > +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> >> > > > >  
> >> > > > > +The driver sends group administration commands to the owner device of
> >> > > > 
> >> > > > I notice that the terminology is simply "the driver". "Owner driver"
> >> > > > and "group member driver" might be clearer because there will be two
> >> > > > (possibly different) drivers involved.
> >> > > 
> >> > > Hmm I don't really want to repeat owner everywhere.
> >> > > I will clarify that in this section simple "driver" and "device" are
> >> > > owner, "member device" and "member driver" is always called explicitly.
> >> > 
> >> > Sounds good.
> >> > 
> >> > > > > +a group to control member devices of the group.
> >> > > > > +This mechanism can
> >> > > > > +be used, for example, to configure a member device before it is
> >> > > > > +initialized by its driver.
> >> > > > > +\footnote{The term "administration" is intended to be interpreted
> >> > > > > +widely to include any kind of control. See specific commands
> >> > > > > +for detail.}
> >> > > > > +
> >> > > > > +All the group administration commands are of the following form:
> >> > > > > +
> >> > > > > +\begin{lstlisting}
> >> > > > > +struct virtio_admin_cmd {
> >> > > > > +        /* Device-readable part */
> >> > > > > +        le16 opcode;
> >> > > > > +        /*
> >> > > > > +         * 1 - SR-IOV
> >> > > > > +         * 2 - 65535 reserved
> >> > > > > +         */
> >> > > > > +        le16 group_type;
> >> > > > > +        /* unused, reserved for future extensions */
> >> > > > > +        u8 reserved1[12];
> >> > > > > +        le64 group_member_id;
> >> > > > > +        u8 command_specific_data[];
> >> > > > > +
> >> > > > > +        /* Device-writable part */
> >> > > > > +        le16 status;
> >> > > > > +        le16 status_qualifier;
> >> > > > > +        /* unused, reserved for future extensions */
> >> > > > > +        u8 reserved2[4];
> >> > > > > +        u8 command_specific_result[];
> >> > > > > +};
> >> > > > > +\end{lstlisting}
> >> > > > > +
> >> > > > > +For all commands, \field{opcode}, \field{group_type} and if
> >> > > > > +necessary \field{group_member_id} and \field{command_specific_data} are
> >> > > > > +set by the driver, and the owner device sets \field{status} and if
> >> > > > > +needed \field{status_qualifier} and
> >> > > > > +\field{command_specific_result}.
> >> > > > > +
> >> > > > > +Generally, any unused device-readable fields are set to zero by the driver
> >> > > > > +and ignored by the device.  Any unused device-writeable fields are set to zero
> >> > > > > +by the device and ignored by the driver.
> >> > > > > +
> >> > > > > +\field{opcode} specifies the command. The valid
> >> > > > > +values for \field{opcode} can be found in the following table:
> >> > > > > +
> >> > > > > +\begin{tabular}{|l|l|}
> >> > > > > +\hline
> >> > > > > +opcode & Name & Command Description \\
> >> > > > > +\hline \hline
> >> > > > > +0x0000 - 0x7FFF & - &  Group administration commands    \\
> >> > > > > +\hline
> >> > > > > +0x8000 - 0xFFFF & - & Reserved    \\
> >> > > > > +\hline
> >> > > > > +\end{tabular}
> >> > > > 
> >> > > > I thought all commands are "group administration commands" but this
> >> > > > table makes it look like they are just a subset (0x0000 - 0x7FFF) of
> >> > > > group administration commands, which is a paradox.
> >> > > 
> >> > > Well the rest are reserved, maybe we will have more command types who
> >> > > knows. No?
> >> > 
> >> > I see. Does that mean the reserved commands don't need to be in the same
> >> > format as struct virtio_admin_cmd?
> >> 
> >> I am not good in predicting future ...
> >> 
> >> > The entire section is called "Group administration commands" but I get
> >> > the impression it's talking both about admin virtqueue commands in
> >> > general and specifically about group administration commands.
> >> > 
> >> > Is it possible to structure this as follows:
> >> > 
> >> >   Admin Commands
> >> >     ...common stuff...
> >> >     Group Administration Commands (0x0000-0x7fff)
> >> >       ...specific to group administration commands...
> >> >     Reserved (0x8000-0xffff)
> >> 
> >> What if I just write it like this:
> >> 
> >> 
> >> > > > > +\hline \hline
> >> > > > > +0x0000 - 0x7FFF & - &  commands using struct virtio_admin_cmd    \\
> >> > > > > +\hline
> >> > > > > +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >> 
> >> 
> >> will this address the comment?
> >
> >Yes, thanks.
> 
> What is the reason for having this table? I fail to see it.

I think it's more a place-holder table in which follow-up patches
add entries. I'll check and see if we can get rid of it.


> 
> >
> >Stefan
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 12:55:37PM +0200, Max Gurtovoy wrote:
> > > 5. VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR (for more info read the
> > > command_specific_error field).
> > 
> > I don't think it's a good idea, we'll have to agree to disagree.
> 
> Ok.
> can you explain why isn't this a good idea please ?

Pointless complexity for spec, devices and drivers. In the end drivers
don't really do anything with all this detailed info.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Wed, Mar 08, 2023 at 12:50:48PM CET, mst@redhat.com wrote:
>On Wed, Mar 08, 2023 at 11:05:00AM +0100, Jiri Pirko wrote:
>> Tue, Mar 07, 2023 at 05:30:18PM CET, mst@redhat.com wrote:
>> >On Tue, Mar 07, 2023 at 08:36:41AM +0100, Jiri Pirko wrote:
>> >> Hmm, if not for now, the future exension would not be so simple, I fear.
>> >
>> >Without knowing what it is I can't say.
>> 
>> Yep, so basically you say, for other things if they appear,
>> let's introduce another queue type? If yes, sounds fair to me.
>
>Yes. For example I find it likely that live migration/failover support
>will require a queue where driver pre-adds buffers and then device
>supplies information as state changes.

I see. So there would be a queue called for example "child state virtqueue"
or something like that for the sole purpose of getting the state of VF?
Hmm, wouldn't it make more sense to have this done as a part of "group
administrarion queue"? I mean, there is already notion of addresing
child/VF here. So from my perspective, it is just another "group
administration" command.

We would have to prealloc buffers and use it for this purpose as
well. I mean, the cmd payload for other group commands
could be put in this buffers as well, making all group command
infrastructure consistent.

Something like:

opcode X
group_member_id 1
cmd_payload-> buffer A

opcode Y
group_member_id 1
cmd_payload-> buffer B

Basically this would replace command_specific_data/command_specific_result


>
>> 
>> >
>> >> 
>> >> >
>> >> >Passing commands to devices themselves is already covered in spec
>> >> >reasonably well though not in a generic way.
>> >> 
>> >> You mean using the control queue, correct?
>> >
>> >Depends on the device type. network devices have a control queue, yes.
>> >
>> >> >From one of the patch description of this patchset I understand that you
>> >> cannot use control queue for this because control queue is
>> >> device-specific, yet group control is device-agnostic.
>> >> 
>> >> My undestanding therefore was, that the admin queue you are introducing
>> >> serves as a generic carrier for device-agnostic commands, in parallel
>> >> for having control queue serving as a generic carrier of device-specific
>> >> commands. If this is not the case, I think it would be nice to describe
>> >> the exact monivation and scope of admin queue.
>> >
>> >Nope unfortunately.  This queue is just a carrier for admin commands.
>> >admin commands are commands that talk to one device about other
>> >devices. There's clearly no mechanism in the spec to do that,
>> >so we plug this hole.
>> 
>> Okay, in that case "admin" sounds a bit misleading as for me it
>> implicates that this is for "administration" of the device. Yet is is
>> for the administration of other devices (slaves).
>> 
>> Perhaps there could be different term used to clarify?
>> Group leader virtqueue?
>> Group owner virtqueue?
>> Group master virtqueue?
>
>I used group administration virtqueue in a couple of places,
>just inconsistently. Good enough?

Yep, sounds good. Thanks!


>
>
>> >
>> >
>> >
>> >> 
>> >> >
>> >> >What we lack is passing commands about one device to another device.
>> >> >E.g. control VFs through PFs.
>> >> 
>> >> Could you provide examples of such commands please?
>> >
>> >For example a common feature is to program a vlan and have device
>> >put a given VF inside this vlan.
>> 
>> I don't follow entirely. The way how the VF is connected to network
>> should be ouf of the scope of this interface. The eswitch manager should
>> take care. What you say sounds awfully like the "ip vf" legacy
>> interface, which should not be considered here I believe.
>> 
>> If PF would be the eswitch manager, there are other means to do network
>> programming, using eswitch port representors. But I don't think this is
>> the can of worms we want to open now. I don't think we have a usecase
>> for it currently. Am I wrong Parav?
>> 
>> 
>> 
>> >
>> >In a virtualization scenario host controls this vlan programming giving
>> >the network a measure of protection from VFs.  If a VF is passed through
>> >to a VM, IOMMU limits VFs to only access guest memory so host has to do
>> >this programming through a PF.
>> 
>> Understood. This really looks like "ip vf" legacy. I strongly believe
>> it should not be supported.
>> 
>> Any other commands you have in mind?
>> 
>> 
>> >
>> >
>> >
>> >> 
>> >> >This is what groups do.
>> >> >But if we see more uses we can always add them.
>> >> >
>> >> >
>> >> >I'd rather avoid being too generic though.
>> >> 
>> >> In that case, why not to avoid using generic terms and stay
>> >> "group-centric"? What I mean is:
>> >> "Administration Virtqueues" -> "Group Administration Virtqueues"
>> >> "struct virtio_admin_cmd" -> "struct virtio_group_admin_cmd"
>> >> 
>> >> Etc. Helps to avoid confusion.
>> >
>> >Sure, I tried to do that but missed some opportunities.
>> >Will address.
>> 
>> Cool.
>> 
>> 
>> >
>> >> 
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >> 
>> >> >> >+than one administration virtqueue.
>> >> 
>> >> 
>> >> [...]
>> >> 
>> >
>> >
>> >This publicly archived list offers a means to provide input to the
>> >OASIS Virtual I/O Device (VIRTIO) TC.
>> >
>> >In order to verify user consent to the Feedback License terms and
>> >to minimize spam in the list archive, subscription is required
>> >before posting.
>> >
>> >Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> >Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> >List help: virtio-comment-help@lists.oasis-open.org
>> >List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> >Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> >List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> >Committee: https://www.oasis-open.org/committees/virtio/
>> >Join OASIS: https://www.oasis-open.org/join/
>> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery

[Jiri Pirko]

Thu, Mar 02, 2023 at 02:05:22PM CET, mst@redhat.com wrote:
>Add commands to find out which commands does each group support,
>as well as enable their use by driver.
>This will be especially useful once we have multiple group types.
>
>An alternative is per-type VQs. This is possible but will
>require more per-transport work. Discovery through the vq
>helps keep things contained.
>
>Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>---
> admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 96 insertions(+), 1 deletion(-)
>
>diff --git a/admin.tex b/admin.tex
>index 3ffac2e..1172054 100644
>--- a/admin.tex
>+++ b/admin.tex
>@@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> \hline
> opcode & Name & Command Description \\
> \hline \hline
>-0x0000 - 0x7FFF & - &  Group administration commands    \\
>+0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>+0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>+0x0002 - 0x7FFF & - &  Group administration commands    \\
> \hline
> 0x8000 - 0xFFFF & - & Reserved    \\

This table would not be needed at all, as all the commands here are
"group administration commands". Could you remove it?


> \hline
>@@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> depends on these structures and is described separately or is
> implicit in the structure description.
> 
>+Before sending any administration commands to the device, the driver
>+needs to communicate to the device which commands it is going to
>+use. Initially (after reset), only two commands are assumed to be used:
>+VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
>+
>+Before sending any other commands for any member of a specific group to
>+the device, the driver queries the supported commands via
>+VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
>+VIRTIO_ADMIN_CMD_LIST_USE.
>+
>+Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
>+VIRTIO_ADMIN_CMD_LIST_USE
>+both use the following structure describing the
>+command opcodes:
>+
>+\begin{lstlisting}
>+struct virtio_admin_cmd_list {
>+       /* Indicates which of the below fields were returned
>+       le64 device_admin_cmds[];
>+};
>+\end{lstlisting}
>+
>+This structure is an array of 64 bit values in little-endian byte
>+order, in which a bit is set if the specific command opcode

Ah wait, you are having this as bitfield. How many commands do you
expect to have? I mean, could this be rather just plain array of
opcodes.

If you want to save some space, the opcode certainly does not need to be
64 bit. I think that 16 bits is more than enough for this :)

Then to make this consistent with struct virtio_admin_cmd,
where you use term "opcode" and with the rest of the text where you use
"CMD" and "cmd", perhaps "cmd_opcode" in struct virtio_admin_cmd
and "cmd_opcodes[]" here would work better?




>+is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value

You say "32", it is "64".


>+in this array corresponding to opcodes 0 to 63,
>+\field{device_admin_cmds[1]} is the second 64-bit value
>+corresponding to opcodes 64 to 127, etc.
>+For example, the array of size 2 including
>+the values 0x3 in \field{device_admin_cmds[0]}
>+and 0x1 in \field{device_admin_cmds[1]} indicates that only
>+opcodes 0, 1 and 64 are supported.
>+The length of the array depends on the supported opcodes - it is
>+large enough to include bits set for all supported opcodes,
>+that is the length can be calculated by starting with the largest
>+supported opcode adding one, dividing by 64 and rounding up.
>+In other words, for
>+VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the
>+length of \field{command_specific_result} and
>+\field{command_specific_data} respectively will be
>+$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division
>+with round up and \field{max_cmd} is the largest available command opcode.
>+
>+The array is also allowed to be larger and to additionally
>+include an arbitrary number of all-zero entries.
>+
>+Accordingly, bits 0 and 1 corresponding to opcode 0
>+(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
>+(VIRTIO_ADMIN_CMD_LIST_USE) are
>+always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
>+
>+For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
>+The \field{group_member_id} is unused. It is set to zero by driver.
>+This command has no command specific data.
>+The device, upon success, returns a result in
>+\field{command_specific_result} in the format
>+\field{struct virtio_admin_cmd_list} describing the
>+list of administration commands supported for the given group.

Hmm, IIUC, the list is per group type (not group). If yes, could
you please consistently use "group type"?

Also, I think you are missing requirement for driver to
specify \field{group_type}.



>+
>+
>+For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
>+is set to 0x1.
>+The \field{group_member_id} is unused. It is set to zero by driver.
>+The \field{command_specific_data} is in the format

Missing \field{group_type} here as well.


>+\field{struct virtio_admin_cmd_list} describing the
>+list of administration commands used by the driver.
>+This command has no command specific result.
>+
>+The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
>+query the list of commands valid for this group and before sending
>+any commands for any member of a group.
>+
>+The driver then enables use of some of the opcodes by sending to
>+the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
>+of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
>+both understood and used by the driver.
>+
>+If the device supports the command list used by the driver, the
>+device completes the command with status VIRTIO_ADMIN_STATUS_OK.

How can the device not support the command list passed by driver? In the
para above you say it is a subset of query list. Wouldn't it be better
to say that driver would have to pass cmd op which was not in the query
list? IDK, maybe better to emphasize this.


>+If the device does not support the command list, the device
>+completes the command with status
>+VIRTIO_ADMIN_STATUS_INVALID_FIELD.
>+
>+Note: drivers are assumed not to set bits in device_admin_cmds
>+if they are not familiar with how the command opcode
>+is used, since devices could have dependencies between
>+command opcodes.
>+
>+It is assumed that all members in a group support and are used
>+with the same list of commands. However, for owner devices
>+supporting multiple group types, the list of supported commands
>+might differ between different group types.
>+
> \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> 
> An administration virtqueue of an owner device is used to submit
>-- 
>MST
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery

[Jiri Pirko]

Wed, Mar 08, 2023 at 12:54:52PM CET, mst@redhat.com wrote:
>On Mon, Mar 06, 2023 at 01:22:30PM +0100, Jiri Pirko wrote:
>> Thu, Mar 02, 2023 at 02:05:22PM CET, mst@redhat.com wrote:
>> >Add commands to find out which commands does each group support,
>> >as well as enable their use by driver.
>> >This will be especially useful once we have multiple group types.
>> >
>> >An alternative is per-type VQs. This is possible but will
>> >require more per-transport work. Discovery through the vq
>> >helps keep things contained.
>> >
>> >Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> >Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> 
>> [...]
>> 
>> 
>> >+
>> >+The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
>> >+query the list of commands valid for this group and before sending
>> >+any commands for any member of a group.
>> >+
>> >+The driver then enables use of some of the opcodes by sending to
>> >+the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
>> >+of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
>> >+both understood and used by the driver.
>> 
>> To my untrained ear, this sounds somewhat similar to the feature
>> negotiantion mechanism. Why the fact that device/driver supports some
>> command can't be covered by just another feature? Looks like unnecassary
>> complexicity to negotiate supported commands like this.
>
>Absolutely, it is similar. The issue is that a single device can
>be an owner for multiple group types.
>For example, I think an SRIOV PF itself can have subfunctions with SIOV
>as well as virtual functions with SRIOV.
>And the set of supported commands might differ.
>
>We thus need a command that is kind of like features but per group type.

I see. But that does not mean it can't be done using existing features:

_F_SRIOV_CMD1
_F_SRIOV_CMD2
..
_F_SIOV_CMD1
_F_SIOV_CMD2
...

>
>As a small bonus as we are building this from scratch, I added
>something which was requested for a long time from features
>which is blocking access to commands not allowed by features.
>With standard virtio features we trust the driver not to send
>commands before enabling a feature and this was considered
>a maintainance/security problem.

I understand. Is it unfixable for features not to break buggy drivers?


>
>
>-- 
>MST
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6456 bytes --]

On Wed, Mar 08, 2023 at 11:17:35AM +0100, Jiri Pirko wrote:
> Tue, Mar 07, 2023 at 08:03:47PM CET, stefanha@redhat.com wrote:
> >On Tue, Mar 07, 2023 at 04:07:54PM +0100, Jiri Pirko wrote:
> >> Tue, Mar 07, 2023 at 03:39:11PM CET, stefanha@redhat.com wrote:
> >> >On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
> >> >> Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
> >> >> >On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> >> >> >> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> >> >> >> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> >> >> >> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> >> >> >> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> >> >> >> > > > > What happens if a command takes 1 second to complete, is the device
> >> >> >> > > > > allowed to process the next command from the virtqueue during this time,
> >> >> >> > > > > possibly completing it before the first command?
> >> >> >> > > > > 
> >> >> >> > > > > This requires additional clarification in the spec because "they are
> >> >> >> > > > > processed by the device in the order in which they are queued" does not
> >> >> >> > > > > explain whether commands block the virtqueue (in order completion) or
> >> >> >> > > > > not (out of order completion).
> >> >> >> > > > 
> >> >> >> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> >> >> >> > > 
> >> >> >> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> >> >> >> > > Several may be processed by the device at the same time.
> >> >> >> > 
> >> >> >> > Let's say I submit a write followed by read - is read
> >> >> >> > guaranteed to return an up to date info?
> >> >> >> 
> >> >> >> In general, no. The driver must wait for the write completion before
> >> >> >> submitting the read if it wants consistency.
> >> >> >> 
> >> >> >> Stefan
> >> >> >
> >> >> >I see.  I think it's a good design to follow then.
> >> >> 
> >> >> Hmm, is it suitable to have this approach for configuration interface?
> >> >> Storage device is a different beast, having parallel reads and writes
> >> >> makes complete sense for performance.
> >> >> 
> >> >> ->read a req
> >> >> ->read b req
> >> >> ->read c req
> >> >> <-read a rep
> >> >> <-read b rep
> >> >> <-read c rep
> >> >> 
> >> >> There is no dependency, even between writes.
> >> >> 
> >> >> But in case of configuration, does not make any sense to me.
> >> >> Why is it needed? To pass the burden of consistency of
> >> >> configuration to driver sounds odd at least.
> >> >> 
> >> >> I sense there is no concete idea about what the "admin virtqueue" should
> >> >> serve for exactly.
> >> >
> >> >It's useful for long-running commands because they prevent other
> >> >commands from executing.
> >> >
> >> >An example I've given is that deleting a group member might require
> >> >waiting for the group member's I/O activity to finish. If that I/O
> >> >activity cannot be cancelled instantaneously, then it could take an
> >> >unbounded amount of time to delete the group member. The device would be
> >> >unable to process futher admin commands.
> >> 
> >> I see. Then I believe that the device should handle the dependencies.
> >> Example 1:
> >> -> REQ cmd to create group member A
> >> -> REQ cmd to create group member B
> >> <- REP cmd to create group member A
> >> <- REP cmd to create group member B
> >> 
> >> The device according to internal implementation can either serialize the
> >> 2 group member creations or do it in parallel, if it supports it.
> >> 
> >> Example 2:
> >> -> REQ cmd to create group member A
> >> -> REQ cmd config group member A
> >> <- REP cmd to create group member A
> >> <- REP cmd config group member A
> >> 
> >> Here the serialization is necessary and the device is the one to take
> >> care of it.
> >> 
> >> Makes sense?
> >
> >Yes, I understand. The spec would need to define ordering rules for
> >specific commands and the device must implement them. It allows the
> >driver to pipeline commands while also allowing out-of-order completion
> >(parallelism) in some cases. The disadvantage of this approach is
> >complexity in the spec and implementations.
> >
> >An alternative is unconditional out-of-order completion, where there are
> >no per-command ordering rules. The driver must wait for a command to
> >complete if it relies on the results of that command for its next
> >command. I like this approach because it's less complex in the spec and
> >for device implementers, while the burden on the driver implementer is
> >still reasonable.
> 
> But isn't this duplicating the burden of maintaining dependencies to
> both driver and device? I mean, device should not depend on driver doing
> the right thing, that means it has to check the dependencies for every
> incoming command anyway. The only difference would be to wait instead of
> returning "-EBUSY" in case the dependency is not satisfied yet.

The device does not need to reject commands that have dependencies with
-EBUSY. The result of commands with dependencies is either A -> B or B
-> A.

For example:
1. Create Group Member A
2. Delete Group Member A

Command 2 might succeed or it might fail with -ENOENT because Group
Member A doesn't exist yet.

> Device knows exactly what are the dependencies. And I believe, those are
> device implementation specific. For example, some implementation could
> support parallel VF config cmd execution, some implementation might
> need to serialize that. Driver has no clue.

Yes, that's up to the device. Out-of-order completion is a superset of
in-order completion. So the device is allowed to run commands in series
when it wants. A driver designed for out-of-order completion will work
fine either way.

> Could you please elaborate a bit more what you mean by "complexity in
> the spec"?

When adding commands to the spec, the dependency relationships with
other commands need to be thought about and documented.

Device implementers need to get those relationships right. That means
they need to remember that command B waits for command A.

Driver implementers have to understand that command B waits for command
A but not command C according to the spec.

That seems complex to me.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

[virtio-dev] Re: [virtio] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Wed, Mar 08, 2023 at 01:44:18PM CET, stefanha@redhat.com wrote:
>On Wed, Mar 08, 2023 at 11:17:35AM +0100, Jiri Pirko wrote:
>> Tue, Mar 07, 2023 at 08:03:47PM CET, stefanha@redhat.com wrote:
>> >On Tue, Mar 07, 2023 at 04:07:54PM +0100, Jiri Pirko wrote:
>> >> Tue, Mar 07, 2023 at 03:39:11PM CET, stefanha@redhat.com wrote:
>> >> >On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
>> >> >> Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
>> >> >> >On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
>> >> >> >> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
>> >> >> >> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
>> >> >> >> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
>> >> >> >> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
>> >> >> >> > > > > What happens if a command takes 1 second to complete, is the device
>> >> >> >> > > > > allowed to process the next command from the virtqueue during this time,
>> >> >> >> > > > > possibly completing it before the first command?
>> >> >> >> > > > > 
>> >> >> >> > > > > This requires additional clarification in the spec because "they are
>> >> >> >> > > > > processed by the device in the order in which they are queued" does not
>> >> >> >> > > > > explain whether commands block the virtqueue (in order completion) or
>> >> >> >> > > > > not (out of order completion).
>> >> >> >> > > > 
>> >> >> >> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
>> >> >> >> > > 
>> >> >> >> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
>> >> >> >> > > Several may be processed by the device at the same time.
>> >> >> >> > 
>> >> >> >> > Let's say I submit a write followed by read - is read
>> >> >> >> > guaranteed to return an up to date info?
>> >> >> >> 
>> >> >> >> In general, no. The driver must wait for the write completion before
>> >> >> >> submitting the read if it wants consistency.
>> >> >> >> 
>> >> >> >> Stefan
>> >> >> >
>> >> >> >I see.  I think it's a good design to follow then.
>> >> >> 
>> >> >> Hmm, is it suitable to have this approach for configuration interface?
>> >> >> Storage device is a different beast, having parallel reads and writes
>> >> >> makes complete sense for performance.
>> >> >> 
>> >> >> ->read a req
>> >> >> ->read b req
>> >> >> ->read c req
>> >> >> <-read a rep
>> >> >> <-read b rep
>> >> >> <-read c rep
>> >> >> 
>> >> >> There is no dependency, even between writes.
>> >> >> 
>> >> >> But in case of configuration, does not make any sense to me.
>> >> >> Why is it needed? To pass the burden of consistency of
>> >> >> configuration to driver sounds odd at least.
>> >> >> 
>> >> >> I sense there is no concete idea about what the "admin virtqueue" should
>> >> >> serve for exactly.
>> >> >
>> >> >It's useful for long-running commands because they prevent other
>> >> >commands from executing.
>> >> >
>> >> >An example I've given is that deleting a group member might require
>> >> >waiting for the group member's I/O activity to finish. If that I/O
>> >> >activity cannot be cancelled instantaneously, then it could take an
>> >> >unbounded amount of time to delete the group member. The device would be
>> >> >unable to process futher admin commands.
>> >> 
>> >> I see. Then I believe that the device should handle the dependencies.
>> >> Example 1:
>> >> -> REQ cmd to create group member A
>> >> -> REQ cmd to create group member B
>> >> <- REP cmd to create group member A
>> >> <- REP cmd to create group member B
>> >> 
>> >> The device according to internal implementation can either serialize the
>> >> 2 group member creations or do it in parallel, if it supports it.
>> >> 
>> >> Example 2:
>> >> -> REQ cmd to create group member A
>> >> -> REQ cmd config group member A
>> >> <- REP cmd to create group member A
>> >> <- REP cmd config group member A
>> >> 
>> >> Here the serialization is necessary and the device is the one to take
>> >> care of it.
>> >> 
>> >> Makes sense?
>> >
>> >Yes, I understand. The spec would need to define ordering rules for
>> >specific commands and the device must implement them. It allows the
>> >driver to pipeline commands while also allowing out-of-order completion
>> >(parallelism) in some cases. The disadvantage of this approach is
>> >complexity in the spec and implementations.
>> >
>> >An alternative is unconditional out-of-order completion, where there are
>> >no per-command ordering rules. The driver must wait for a command to
>> >complete if it relies on the results of that command for its next
>> >command. I like this approach because it's less complex in the spec and
>> >for device implementers, while the burden on the driver implementer is
>> >still reasonable.
>> 
>> But isn't this duplicating the burden of maintaining dependencies to
>> both driver and device? I mean, device should not depend on driver doing
>> the right thing, that means it has to check the dependencies for every
>> incoming command anyway. The only difference would be to wait instead of
>> returning "-EBUSY" in case the dependency is not satisfied yet.
>
>The device does not need to reject commands that have dependencies with
>-EBUSY. The result of commands with dependencies is either A -> B or B
>-> A.
>
>For example:
>1. Create Group Member A
>2. Delete Group Member A
>
>Command 2 might succeed or it might fail with -ENOENT because Group
>Member A doesn't exist yet.

Yeah, correct.

>
>> Device knows exactly what are the dependencies. And I believe, those are
>> device implementation specific. For example, some implementation could
>> support parallel VF config cmd execution, some implementation might
>> need to serialize that. Driver has no clue.
>
>Yes, that's up to the device. Out-of-order completion is a superset of
>in-order completion. So the device is allowed to run commands in series
>when it wants. A driver designed for out-of-order completion will work
>fine either way.

Agreed.


>
>> Could you please elaborate a bit more what you mean by "complexity in
>> the spec"?
>
>When adding commands to the spec, the dependency relationships with
>other commands need to be thought about and documented.
>
>Device implementers need to get those relationships right. That means
>they need to remember that command B waits for command A.

That's my point, this is device implementation specific. Spec should not
define anything like this, that might be limiting in some of the cases.

>
>Driver implementers have to understand that command B waits for command
>A but not command C according to the spec.
>
>That seems complex to me.

Sure, I agree this does not belong to the spec, it can't, really.
For the record, I never suggested it.

I think we are in agreement.


>
>Stefan



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[David Edmondson]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> On Tue, Mar 07, 2023 at 11:04:33AM +0000, David Edmondson wrote:
>> "Michael S. Tsirkin" <mst@redhat.com> writes:
>> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
>> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
>> > +with respective bits cleared in \field{command_specific_data}.
>> 
>> This runs contrary to the assertion "The list of supported commands MUST
>> NOT shrink", given that a driver is told to assume that
>> VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
>> commands initially available.
>
> Commands are still available just disabled, the meaning of
> "MUST NOT shrink" is clarified by the following:
>
> 	> > +after reporting a given command as supported through
> 	> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> 	> > +as unsupported.
> I will stick an "i.e." there to make it hopefully clearer.

This puzzles me.

I can see the point of blocking LIST_USE, but not of blocking
LIST_QUERY. What's the purpose of this?
-- 
Time is waiting to explain, why refuse?

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[Jiri Pirko]

Wed, Mar 08, 2023 at 01:59:10PM CET, david.edmondson@oracle.com wrote:
>"Michael S. Tsirkin" <mst@redhat.com> writes:
>
>> On Tue, Mar 07, 2023 at 11:04:33AM +0000, David Edmondson wrote:
>>> "Michael S. Tsirkin" <mst@redhat.com> writes:
>>> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
>>> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
>>> > +with respective bits cleared in \field{command_specific_data}.
>>> 
>>> This runs contrary to the assertion "The list of supported commands MUST
>>> NOT shrink", given that a driver is told to assume that
>>> VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
>>> commands initially available.
>>
>> Commands are still available just disabled, the meaning of
>> "MUST NOT shrink" is clarified by the following:
>>
>> 	> > +after reporting a given command as supported through
>> 	> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
>> 	> > +as unsupported.
>> I will stick an "i.e." there to make it hopefully clearer.
>
>This puzzles me.
>
>I can see the point of blocking LIST_USE, but not of blocking

What is that point, I'm curious.


>LIST_QUERY. What's the purpose of this?

I think that this might be just for the sake of treating all
commands equally.


>-- 
>Time is waiting to explain, why refuse?
>
>---------------------------------------------------------------------
>To unsubscribe from this mail list, you must leave the OASIS TC that 
>generates this mail.  Follow this link to all your TCs in OASIS at:
>https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 02:05:01PM +0100, Jiri Pirko wrote:
> Wed, Mar 08, 2023 at 01:59:10PM CET, david.edmondson@oracle.com wrote:
> >"Michael S. Tsirkin" <mst@redhat.com> writes:
> >
> >> On Tue, Mar 07, 2023 at 11:04:33AM +0000, David Edmondson wrote:
> >>> "Michael S. Tsirkin" <mst@redhat.com> writes:
> >>> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> >>> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> >>> > +with respective bits cleared in \field{command_specific_data}.
> >>> 
> >>> This runs contrary to the assertion "The list of supported commands MUST
> >>> NOT shrink", given that a driver is told to assume that
> >>> VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
> >>> commands initially available.
> >>
> >> Commands are still available just disabled, the meaning of
> >> "MUST NOT shrink" is clarified by the following:
> >>
> >> 	> > +after reporting a given command as supported through
> >> 	> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> >> 	> > +as unsupported.
> >> I will stick an "i.e." there to make it hopefully clearer.
> >
> >This puzzles me.
> >
> >I can see the point of blocking LIST_USE, but not of blocking
> 
> What is that point, I'm curious.
> 
> 
> >LIST_QUERY. What's the purpose of this?
> 
> I think that this might be just for the sake of treating all
> commands equally.
> 

Right. There was this vague idea that blocking all commands
is good to have. E.g. if you are making some drastic
changes to the device?


> >-- 
> >Time is waiting to explain, why refuse?
> >
> >---------------------------------------------------------------------
> >To unsubscribe from this mail list, you must leave the OASIS TC that 
> >generates this mail.  Follow this link to all your TCs in OASIS at:
> >https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[David Edmondson]

Jiri Pirko <jiri@nvidia.com> writes:

> Wed, Mar 08, 2023 at 01:59:10PM CET, david.edmondson@oracle.com wrote:
>>"Michael S. Tsirkin" <mst@redhat.com> writes:
>>
>>> On Tue, Mar 07, 2023 at 11:04:33AM +0000, David Edmondson wrote:
>>>> "Michael S. Tsirkin" <mst@redhat.com> writes:
>>>> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
>>>> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
>>>> > +with respective bits cleared in \field{command_specific_data}.
>>>> 
>>>> This runs contrary to the assertion "The list of supported commands MUST
>>>> NOT shrink", given that a driver is told to assume that
>>>> VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
>>>> commands initially available.
>>>
>>> Commands are still available just disabled, the meaning of
>>> "MUST NOT shrink" is clarified by the following:
>>>
>>> 	> > +after reporting a given command as supported through
>>> 	> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
>>> 	> > +as unsupported.
>>> I will stick an "i.e." there to make it hopefully clearer.
>>
>>This puzzles me.
>>
>>I can see the point of blocking LIST_USE, but not of blocking
>
> What is that point, I'm curious.

I might restrict the commands permitted by a device before handing it
off to some subordinate less trusted code.

>
>>LIST_QUERY. What's the purpose of this?
>
> I think that this might be just for the sake of treating all
> commands equally.
>
>
>>-- 
>>Time is waiting to explain, why refuse?
>>
>>---------------------------------------------------------------------
>>To unsubscribe from this mail list, you must leave the OASIS TC that 
>>generates this mail.  Follow this link to all your TCs in OASIS at:
>>https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
>>
-- 
I used to get mad at my school, the teachers who taught me weren't cool.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[Jiri Pirko]

Wed, Mar 08, 2023 at 02:44:08PM CET, david.edmondson@oracle.com wrote:
>Jiri Pirko <jiri@nvidia.com> writes:
>
>> Wed, Mar 08, 2023 at 01:59:10PM CET, david.edmondson@oracle.com wrote:
>>>"Michael S. Tsirkin" <mst@redhat.com> writes:
>>>
>>>> On Tue, Mar 07, 2023 at 11:04:33AM +0000, David Edmondson wrote:
>>>>> "Michael S. Tsirkin" <mst@redhat.com> writes:
>>>>> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
>>>>> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
>>>>> > +with respective bits cleared in \field{command_specific_data}.
>>>>> 
>>>>> This runs contrary to the assertion "The list of supported commands MUST
>>>>> NOT shrink", given that a driver is told to assume that
>>>>> VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
>>>>> commands initially available.
>>>>
>>>> Commands are still available just disabled, the meaning of
>>>> "MUST NOT shrink" is clarified by the following:
>>>>
>>>> 	> > +after reporting a given command as supported through
>>>> 	> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
>>>> 	> > +as unsupported.
>>>> I will stick an "i.e." there to make it hopefully clearer.
>>>
>>>This puzzles me.
>>>
>>>I can see the point of blocking LIST_USE, but not of blocking
>>
>> What is that point, I'm curious.
>
>I might restrict the commands permitted by a device before handing it
>off to some subordinate less trusted code.

Hmm, that is theoretical, or is there a usecase for that?


>
>>
>>>LIST_QUERY. What's the purpose of this?
>>
>> I think that this might be just for the sake of treating all
>> commands equally.
>>
>>
>>>-- 
>>>Time is waiting to explain, why refuse?
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe from this mail list, you must leave the OASIS TC that 
>>>generates this mail.  Follow this link to all your TCs in OASIS at:
>>>https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
>>>
>-- 
>I used to get mad at my school, the teachers who taught me weren't cool.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 03:02:05PM +0100, Jiri Pirko wrote:
> Wed, Mar 08, 2023 at 02:44:08PM CET, david.edmondson@oracle.com wrote:
> >Jiri Pirko <jiri@nvidia.com> writes:
> >
> >> Wed, Mar 08, 2023 at 01:59:10PM CET, david.edmondson@oracle.com wrote:
> >>>"Michael S. Tsirkin" <mst@redhat.com> writes:
> >>>
> >>>> On Tue, Mar 07, 2023 at 11:04:33AM +0000, David Edmondson wrote:
> >>>>> "Michael S. Tsirkin" <mst@redhat.com> writes:
> >>>>> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> >>>>> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> >>>>> > +with respective bits cleared in \field{command_specific_data}.
> >>>>> 
> >>>>> This runs contrary to the assertion "The list of supported commands MUST
> >>>>> NOT shrink", given that a driver is told to assume that
> >>>>> VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only
> >>>>> commands initially available.
> >>>>
> >>>> Commands are still available just disabled, the meaning of
> >>>> "MUST NOT shrink" is clarified by the following:
> >>>>
> >>>> 	> > +after reporting a given command as supported through
> >>>> 	> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> >>>> 	> > +as unsupported.
> >>>> I will stick an "i.e." there to make it hopefully clearer.
> >>>
> >>>This puzzles me.
> >>>
> >>>I can see the point of blocking LIST_USE, but not of blocking
> >>
> >> What is that point, I'm curious.
> >
> >I might restrict the commands permitted by a device before handing it
> >off to some subordinate less trusted code.
> 
> Hmm, that is theoretical, or is there a usecase for that?
> 

Maybe there will be a future version replacing VIRTIO_ADMIN_CMD_LIST_QUERY
with a different command. Then we start with a device supporting
both VIRTIO_ADMIN_CMD_LIST_QUERY and the new VIRTIO_ADMIN_FOO,
and we block VIRTIO_ADMIN_CMD_LIST_QUERY for testing, to make sure driver
is able to handle future devices without VIRTIO_ADMIN_CMD_LIST_QUERY.


> >
> >>
> >>>LIST_QUERY. What's the purpose of this?
> >>
> >> I think that this might be just for the sake of treating all
> >> commands equally.
> >>
> >>
> >>>-- 
> >>>Time is waiting to explain, why refuse?
> >>>
> >>>---------------------------------------------------------------------
> >>>To unsubscribe from this mail list, you must leave the OASIS TC that 
> >>>generates this mail.  Follow this link to all your TCs in OASIS at:
> >>>https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
> >>>
> >-- 
> >I used to get mad at my school, the teachers who taught me weren't cool.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6489 bytes --]

On Wed, Mar 08, 2023 at 01:17:33PM +0200, Max Gurtovoy wrote:
> 
> 
> On 06/03/2023 18:25, Stefan Hajnoczi wrote:
> > On Mon, Mar 06, 2023 at 05:28:03PM +0200, Max Gurtovoy wrote:
> > > 
> > > 
> > > On 06/03/2023 13:20, Stefan Hajnoczi wrote:
> > > > On Mon, Mar 06, 2023 at 04:00:50PM +0800, Jason Wang wrote:
> > > > > 
> > > > > 在 2023/3/6 08:03, Stefan Hajnoczi 写道:
> > > > > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > > > > What happens if a command takes 1 second to complete, is the device
> > > > > > > > allowed to process the next command from the virtqueue during this time,
> > > > > > > > possibly completing it before the first command?
> > > > > > > > 
> > > > > > > > This requires additional clarification in the spec because "they are
> > > > > > > > processed by the device in the order in which they are queued" does not
> > > > > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > > > > not (out of order completion).
> > > > > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > > > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > > > > Several may be processed by the device at the same time.
> > > > > > 
> > > > > > They rely on multi-queue for abort operations:
> > > > > > 
> > > > > > In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are
> > > > > > sent on the control virtqueue. The the request identifier namespace is
> > > > > > shared across all virtqueues so it's possible to abort a request that
> > > > > > was submitted to any command virtqueue.
> > > > > > 
> > > > > > NVMe also follows the same design where abort commands are sent on the
> > > > > > Admin Submission Queue instead of an I/O Submission Queue. It's possible
> > > > > > to identify NVMe requests by <Submission Queue ID, Command Identifier>.
> > > > > > 
> > > > > > virtio-blk doesn't support aborting requests.
> > > > > > 
> > > > > > I think the logic behind this design is that if a queue gets stuck
> > > > > > processing long-running requests, then the device should not be forced
> > > > > > to perform lookahead in the queue to find abort commands. A separate
> > > > > > control/admin queue is used for the abort requests.
> > > > > 
> > > > > 
> > > > > Or device need mandate some kind of QOS here, e.g a request must be complete
> > > > > in some time. Otherwise we don't have sufficient reliability for using it as
> > > > > management task?
> > > > 
> > > > Yes, if all commands can be executed in bounded time then a guarantee is
> > > > possible.
> > > > 
> > > > Here is an example where that's hard: imagine a virtio-blk device backed
> > > > by network storage. When an admin queue command is used to delete a
> > > > group member, any of the group member's in-flight I/O requests need to
> > > > be aborted. If the network hangs while the group member is being
> > > > deleted, then the device can't complete an orderly shutdown of I/O
> > > > requests in a reasonable time.
> > > > 
> > > > That example shows a basic group admin command that I think Michael is
> > > > about to propose. We can't avoid this problem by not making it a group
> > > > admin command - it needs to be a group admin command. So I think it's
> > > > likely that there will be admin commands that take an unbounded amount
> > > > of time to complete. One way to achieve what you mentioned is timeouts.
> > > 
> > > I think that you're getting into device specific implementation details and
> > > I'm not sure it's necessary.
> > > 
> > > I don't think we need to abort admin commands. Admin commands can be
> > > flushed/aborted during the device reset phase.
> > > Only IO commands should have the possibility to being aborted as you
> > > mentioned in NVMe and SCSI (and potentially in virtio-blk).
> > 
> > It's a general design issue that should be clarified now rather than
> > being left unspecified.
> > 
> > I'm not saying that it must be possible to abort admin commands. There
> > are other options, like requiring the device itself to fail a command
> > after a timeout.
> 
> do you have an example of timeout today for control vq ?

Do you mean the virtio-net control virtqueue? I don't think it has any
commands with an unbounded execution time.

> > 
> > Or we could say that admin commands must complete within bounded time,
> > but I'm not sure that is implementable for some device types like
> > virtio-blk, virtio-scsi, and virtiofs.
> 
> No we can't.
> Some commands, for example FW upgrade can take 10 minutes and it's perfectly
> fine. Other commands like setting feature bit will take 1 millisec.
> Each device implements commands in a different internal logic so we can't
> expect to complete after X time.

When I say bounded time, I mean that it finishes in a finite amount of
time. I'm not saying there is a specific time X that all device
implementations must satisfy. Unbounded means it might never finish.

> Device can go to so FATAL state in case a command is stuck and causing
> internal errors in it.
> 
> > 
> > > For your example, stopping a member is possible even it there are some
> > > errors in the network. You can for example destroy all the connections to
> > > the remote target and complete all the BIOS with some error.
> > 
> > Forgetting about in-flight requests doesn't necessarily make them go
> > away. It creates a race between forgotten requests and reconnection. In
> > the worst case a forgotten write request takes effect after
> > reconnection, causing data corruption.
> 
> For making it work without data corruption we need a cooperation of the
> target side for sure. But this is fine since the target in that case is part
> of the "virtio-blk backend".
> One solution is that the target can decide it will flush all the requests to
> the storage device before accepting new connections.

This solution shifts the unbounded time from disconnection to
connection. The Group Member Delete command will complete quickly but a
subsequent Group Member Create command for the same underlying storage
device would need to wait until the requests are done.

Therefore I think the admin queue must be designed under the assumption
that some commands take a very long time.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Cornelia Huck]

On Tue, Mar 07 2023, David Edmondson <david.edmondson@oracle.com> wrote:

> "Michael S. Tsirkin" <mst@redhat.com> writes:
>
>> On Mon, Mar 06, 2023 at 01:53:50PM +0000, David Edmondson wrote:
>>> "Michael S. Tsirkin" <mst@redhat.com> writes:
>>> 
>>> > Feature negotiation forms the basis of forward compatibility
>>> > guarantees of virtio but has never been properly documented.
>>> > Do it now.
>>> >
>>> > Suggested-by: Halil Pasic <pasic@linux.ibm.com>
>>> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>> > ---
>>> >  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
>>> >  1 file changed, 42 insertions(+)
>>> >
>>> > diff --git a/content.tex b/content.tex
>>> > index 0e474dd..0c2d917 100644
>>> > --- a/content.tex
>>> > +++ b/content.tex
>>> > @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>>> >  In particular, new fields in the device configuration space are
>>> >  indicated by offering a new feature bit.
>>> >  
>>> > +To keep the feature negotiation mechanism extensible, it is important
>>> > +that devices \em{do not} offer any feature bits that they would not be
>>> > +able to handle if the driver accepted them (even though drivers are not
>>> > +supposed to accept them in the first place even if offered, according to
>>> > +this version of the specification.) 
>>> 
>>> I find this (the bit in parenthesis) confusing.
>>> 
>>> Why are drivers not supposed to accept features that they have been
>>> offered, given that they can't know that the device cannot handle the
>>> feature that it just offered?
>>> 
>>> Is this alluding to the later section:
>>> 
>>> > feature bits not described in this specification, reserved feature
>>> > bits and feature bits reserved or not supported for the specific
>>> > transport or the specific device type
>>> 
>>> ?
>>
>> exactly. how would you put this better? given an example?
>
> Perhaps it would be enough to say:
>
>> (even though drivers are not supposed to accept unrecognised features in
>> the first place even if offered, according to the specification)
>
> "Unrecognised" is intended as a shorthand for the whole "not described,
> reserved, ...". Maybe "unrecognised or reserved"?

Hm, what about

"even though drivers are not supposed to accept any unspecified,
reserved, or unsupported features even if offered..."

?

I'm not sure how we can make this both short and descriptive enough...


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Cornelia Huck]

On Wed, Mar 08 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 08/03/2023 14:07, Michael S. Tsirkin wrote:
>> On Wed, Mar 08, 2023 at 12:55:37PM +0200, Max Gurtovoy wrote:
>>>>> 5. VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR (for more info read the
>>>>> command_specific_error field).
>>>>
>>>> I don't think it's a good idea, we'll have to agree to disagree.
>>>
>>> Ok.
>>> can you explain why isn't this a good idea please ?
>> 
>> Pointless complexity for spec, devices and drivers. In the end drivers
>> don't really do anything with all this detailed info.
>> 
>
> Pointless according to your own opinion. As I mentioned, other 
> specifications and devices work this way. I didn't invent it.
>
> All drivers in SW world ? All drivers in Linux ? or Virtio drivers ?

FWIW, I've spent enough time translating
detailed-but-not-really-that-useful error status codes to -EINVAL for
s390-specific things...

The only thing that might make sense to transmit to a consumer would be
the difference between "this thing won't work" and "this thing might
work if you retry later" (if the consumer is even in a place to retry.)


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 03:43:14PM +0100, Cornelia Huck wrote:
> On Wed, Mar 08 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> 
> > On 08/03/2023 14:07, Michael S. Tsirkin wrote:
> >> On Wed, Mar 08, 2023 at 12:55:37PM +0200, Max Gurtovoy wrote:
> >>>>> 5. VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR (for more info read the
> >>>>> command_specific_error field).
> >>>>
> >>>> I don't think it's a good idea, we'll have to agree to disagree.
> >>>
> >>> Ok.
> >>> can you explain why isn't this a good idea please ?
> >> 
> >> Pointless complexity for spec, devices and drivers. In the end drivers
> >> don't really do anything with all this detailed info.
> >> 
> >
> > Pointless according to your own opinion. As I mentioned, other 
> > specifications and devices work this way. I didn't invent it.
> >
> > All drivers in SW world ? All drivers in Linux ? or Virtio drivers ?
> 
> FWIW, I've spent enough time translating
> detailed-but-not-really-that-useful error status codes to -EINVAL for
> s390-specific things...
> 
> The only thing that might make sense to transmit to a consumer would be
> the difference between "this thing won't work" and "this thing might
> work if you retry later" (if the consumer is even in a place to retry.)

One place where that extra detail might be helpful is device debugging.
But there an error code out of list is too restrictive anyway, you want
"Turbo Encabulator 42 logarithmic casing failure".  I am still looking
for something that drivers are not likely to abuse. Maybe an ASCII
string?

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 03:16:41PM +0100, Cornelia Huck wrote:
> On Tue, Mar 07 2023, David Edmondson <david.edmondson@oracle.com> wrote:
> 
> > "Michael S. Tsirkin" <mst@redhat.com> writes:
> >
> >> On Mon, Mar 06, 2023 at 01:53:50PM +0000, David Edmondson wrote:
> >>> "Michael S. Tsirkin" <mst@redhat.com> writes:
> >>> 
> >>> > Feature negotiation forms the basis of forward compatibility
> >>> > guarantees of virtio but has never been properly documented.
> >>> > Do it now.
> >>> >
> >>> > Suggested-by: Halil Pasic <pasic@linux.ibm.com>
> >>> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >>> > ---
> >>> >  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
> >>> >  1 file changed, 42 insertions(+)
> >>> >
> >>> > diff --git a/content.tex b/content.tex
> >>> > index 0e474dd..0c2d917 100644
> >>> > --- a/content.tex
> >>> > +++ b/content.tex
> >>> > @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
> >>> >  In particular, new fields in the device configuration space are
> >>> >  indicated by offering a new feature bit.
> >>> >  
> >>> > +To keep the feature negotiation mechanism extensible, it is important
> >>> > +that devices \em{do not} offer any feature bits that they would not be
> >>> > +able to handle if the driver accepted them (even though drivers are not
> >>> > +supposed to accept them in the first place even if offered, according to
> >>> > +this version of the specification.) 
> >>> 
> >>> I find this (the bit in parenthesis) confusing.
> >>> 
> >>> Why are drivers not supposed to accept features that they have been
> >>> offered, given that they can't know that the device cannot handle the
> >>> feature that it just offered?
> >>> 
> >>> Is this alluding to the later section:
> >>> 
> >>> > feature bits not described in this specification, reserved feature
> >>> > bits and feature bits reserved or not supported for the specific
> >>> > transport or the specific device type
> >>> 
> >>> ?
> >>
> >> exactly. how would you put this better? given an example?
> >
> > Perhaps it would be enough to say:
> >
> >> (even though drivers are not supposed to accept unrecognised features in
> >> the first place even if offered, according to the specification)
> >
> > "Unrecognised" is intended as a shorthand for the whole "not described,
> > reserved, ...". Maybe "unrecognised or reserved"?
> 
> Hm, what about
> 
> "even though drivers are not supposed to accept any unspecified,
> reserved, or unsupported features even if offered..."
> 
> ?

ok

> I'm not sure how we can make this both short and descriptive enough...



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 03:05:50PM +0200, Max Gurtovoy wrote:
> All drivers in SW world ? All drivers in Linux ? or Virtio drivers ?

Most of the above.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Stefan Hajnoczi <stefanha@redhat.com>
> Sent: Wednesday, March 8, 2023 9:13 AM

> Therefore I think the admin queue must be designed under the assumption that
> some commands take a very long time.

+1

[virtio-dev] Re: [virtio-comment] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Cornelia Huck]

On Tue, Mar 07 2023, Parav Pandit <parav@nvidia.com> wrote:

>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Friday, March 3, 2023 3:34 AM
>> 
>> On Thu, Mar 02, 2023 at 06:40:55PM +0000, Parav Pandit wrote:
>> > Did you miss reviewed-by from [1] or this is an old series reposted?
>> > [1]
>> > https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html
>> 
>> As a general rule, we don't strictly need to track reviewed by since there's a
>> ballot (and presumably people review before voting).
>> 
>> People also tack on Signed-off-by: (and I do it too) but as long as we don't
>> document what it means it's kind of vague, and the process of subscribing to
>> the mailing list is a kind of replacement.
>> 
>> If you think everyone needs to follow practices like netdev does, we really need
>> something written up, and agree on it.
>> 
>> E.g.  I work on the linux kernel too, so I can copy practices from there, but even
>> linux isn't uniform.
>> 
>> And I wonder whether it's worth it - it definitely makes contributing to Linux
>> harder, and even within Linux it pushes contributors away. 
> The number of virtio spec contributors are in order of magnitude less than Linux kernel or just the Linux netdev.
> Patch split up reduces lot of time author and reviewers.
> For example, interrupt moderation at v10 can be very easily split up where prep patch can have RB tag and v11 doesn't need reviewers and author's time.
> Your work here with 10 patches is yet another good example of it.
> I remember Max and I started with 6 patches... and current 10 are more useful way to split them.

I'd argue that splitting changes in a way that makes it easy for
reviewers is a good thing for any project (although practices on many
forges seem to go into another direction...)

>
>> At least for Linux
>> tracking history in a precise way is extremely important since it's helpful with
>> stability. Spec is very different.
>> 
>> Until we have a good contribution documentation I think we should not ask
>> people to follow a pseudo Linux work flow with requests like "please split this
>> patchset up" or "track changes across patch versions"
>> simply because there's no good docs to teach people what exactly is the best
>> practice.
>
> Yes, I wanted to update the contributing document. It is in my to-do list.
> But given the small crowd of contributors right now, almost everyone is familiar with the RB, ack tag.
> At moment it has two main reasons.
>
> 1. Acknowledge the work and efforts that go in the review

I agree, and I try to include tags when applying.

> 2. When in question, reach out later to the spec author or reviewers to know about context/design etc.
> 3. Same reviewer doesn't need to go through the patch which already has RB tag of him/her.

I tend to use R-b in that way as well, but it relies on the patch author
not doing substantial changes between versions...

I guess the usage of R-b et al in the virtio spec stems from the fact
that many contributors are also Linux and/or QEMU contributors  -- not
sure if it makes sense to enshrine their usage, but

4. We can get an R-b from a non-TC member who is an expert on the topic
(and follows standard Linux/QEMU/... practices.)

In the end, how the TC members are voting is the only thing that matters
for inclusion.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

On Tue, Mar 07, 2023 at 06:52:03PM +0000, Parav Pandit wrote:
> > And I wonder whether it's worth it - it definitely makes contributing to Linux
> > harder, and even within Linux it pushes contributors away. 
> The number of virtio spec contributors are in order of magnitude less
> than Linux kernel or just the Linux netdev.  Patch split up reduces
> lot of time author and reviewers.  For example, interrupt moderation
> at v10 can be very easily split up where prep patch can have RB tag
> and v11 doesn't need reviewers and author's time.  Your work here with
> 10 patches is yet another good example of it.  I remember Max and I
> started with 6 patches... and current 10 are more useful way to split
> them.

It's a big project though. All I ask is that we try to be gentler
than netdev to new contributors. If there's a 10 line patch it is
not necessary to be pedantic and ask for it to be split up to
a 3 patch series.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Cornelia Huck]

On Mon, Mar 06 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Mar 06, 2023 at 01:29:30PM +0100, Jiri Pirko wrote:
>> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
>> 
>> [...]
>> 
>> >
>> >TODO (maybe?) - probably ok to defer until this part is upstream:
>> >
>> >	Add "all members" member id.
>> >
>> >	Add commands for MSI, feature discovery.
>> >
>> >	Add commands for transport vq.
>> >
>> >
>> >My intent is to try and support both SR-IOV and SIOV
>> >usecases with the same structure and maybe even the same
>> >VQ.
>> 
>> Sorry to be late to the party, I'm trying to catch up. Is it common to
>> have cover letter for features this brief? I mean, from the cover
>> letter, I'm totally unable to understand what you are introducing here.
>> 
>> Could you elaborate about what you are aiming to achive with this?
>> Could you shed some usecases perhaps?
>> 
>> I have to be missing something obvious, but I don't get why any notion
>> of SR-IOV could be beneficial for virtio.
>> 
>
> Good point, I'll add a bit of motivation.
>
> For SR-IOV, it is not unusual for PFs to excercise control over VFs.
> There is interest in the community to include an interface to allow this
> in the virtio spec, when the PF is a virtio device.  This is what this
> patch does.

Unfortunately, information in the cover letter does not make it into
git -- should things like that go into the github issue (and into the
ballot?) It's useful both for reviewing (cover letter) and understanding
the rationale later (github/ballot).


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Parav Pandit]

> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, March 8, 2023 11:24 AM

> >> And I wonder whether it's worth it - it definitely makes contributing
> >> to Linux harder, and even within Linux it pushes contributors away.
> > The number of virtio spec contributors are in order of magnitude less than
> Linux kernel or just the Linux netdev.
> > Patch split up reduces lot of time author and reviewers.
> > For example, interrupt moderation at v10 can be very easily split up where
> prep patch can have RB tag and v11 doesn't need reviewers and author's time.
> > Your work here with 10 patches is yet another good example of it.
> > I remember Max and I started with 6 patches... and current 10 are more
> useful way to split them.
> 
> I'd argue that splitting changes in a way that makes it easy for reviewers is a
> good thing for any project (although practices on many forges seem to go into
> another direction...)
> 
In my experience in dpdk, Linux netdev, rdma, nvme, tells me that patch splitting is common practice.

> >
> >> At least for Linux
> >> tracking history in a precise way is extremely important since it's
> >> helpful with stability. Spec is very different.
> >>
> >> Until we have a good contribution documentation I think we should not
> >> ask people to follow a pseudo Linux work flow with requests like
> >> "please split this patchset up" or "track changes across patch versions"
> >> simply because there's no good docs to teach people what exactly is
> >> the best practice.
> >
> > Yes, I wanted to update the contributing document. It is in my to-do list.
> > But given the small crowd of contributors right now, almost everyone is
> familiar with the RB, ack tag.
> > At moment it has two main reasons.
> >
> > 1. Acknowledge the work and efforts that go in the review
> 
> I agree, and I try to include tags when applying.
> 
Sure. It is not about you.
It is about the patch author when he/she sends v9 to v10, adding R-b tag in v10 (for the already reviewed work of v9) helps to ack and speed up the v10 review.

> > 2. When in question, reach out later to the spec author or reviewers to know
> about context/design etc.
> > 3. Same reviewer doesn't need to go through the patch which already has RB
> tag of him/her.
> 
> I tend to use R-b in that way as well, but it relies on the patch author not doing
> substantial changes between versions...
> 
> I guess the usage of R-b et al in the virtio spec stems from the fact that many
> contributors are also Linux and/or QEMU contributors  -- not sure if it makes
> sense to enshrine their usage, but
> 
> 4. We can get an R-b from a non-TC member who is an expert on the topic (and
> follows standard Linux/QEMU/... practices.)
> 
The intent here was between vN to v(N+1).
Virtio spec maintainers are likely adding R-b when they merge anyway. No issue there.

> In the end, how the TC members are voting is the only thing that matters for
> inclusion.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 9139 bytes --]

On Wed, Mar 08, 2023 at 06:19:41PM +0200, Max Gurtovoy wrote:
> 
> 
> On 08/03/2023 16:13, Stefan Hajnoczi wrote:
> > On Wed, Mar 08, 2023 at 01:17:33PM +0200, Max Gurtovoy wrote:
> > > 
> > > 
> > > On 06/03/2023 18:25, Stefan Hajnoczi wrote:
> > > > On Mon, Mar 06, 2023 at 05:28:03PM +0200, Max Gurtovoy wrote:
> > > > > 
> > > > > 
> > > > > On 06/03/2023 13:20, Stefan Hajnoczi wrote:
> > > > > > On Mon, Mar 06, 2023 at 04:00:50PM +0800, Jason Wang wrote:
> > > > > > > 
> > > > > > > 在 2023/3/6 08:03, Stefan Hajnoczi 写道:
> > > > > > > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > > > > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > > > > > > What happens if a command takes 1 second to complete, is the device
> > > > > > > > > > allowed to process the next command from the virtqueue during this time,
> > > > > > > > > > possibly completing it before the first command?
> > > > > > > > > > 
> > > > > > > > > > This requires additional clarification in the spec because "they are
> > > > > > > > > > processed by the device in the order in which they are queued" does not
> > > > > > > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > > > > > > not (out of order completion).
> > > > > > > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > > > > > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > > > > > > Several may be processed by the device at the same time.
> > > > > > > > 
> > > > > > > > They rely on multi-queue for abort operations:
> > > > > > > > 
> > > > > > > > In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are
> > > > > > > > sent on the control virtqueue. The the request identifier namespace is
> > > > > > > > shared across all virtqueues so it's possible to abort a request that
> > > > > > > > was submitted to any command virtqueue.
> > > > > > > > 
> > > > > > > > NVMe also follows the same design where abort commands are sent on the
> > > > > > > > Admin Submission Queue instead of an I/O Submission Queue. It's possible
> > > > > > > > to identify NVMe requests by <Submission Queue ID, Command Identifier>.
> > > > > > > > 
> > > > > > > > virtio-blk doesn't support aborting requests.
> > > > > > > > 
> > > > > > > > I think the logic behind this design is that if a queue gets stuck
> > > > > > > > processing long-running requests, then the device should not be forced
> > > > > > > > to perform lookahead in the queue to find abort commands. A separate
> > > > > > > > control/admin queue is used for the abort requests.
> > > > > > > 
> > > > > > > 
> > > > > > > Or device need mandate some kind of QOS here, e.g a request must be complete
> > > > > > > in some time. Otherwise we don't have sufficient reliability for using it as
> > > > > > > management task?
> > > > > > 
> > > > > > Yes, if all commands can be executed in bounded time then a guarantee is
> > > > > > possible.
> > > > > > 
> > > > > > Here is an example where that's hard: imagine a virtio-blk device backed
> > > > > > by network storage. When an admin queue command is used to delete a
> > > > > > group member, any of the group member's in-flight I/O requests need to
> > > > > > be aborted. If the network hangs while the group member is being
> > > > > > deleted, then the device can't complete an orderly shutdown of I/O
> > > > > > requests in a reasonable time.
> > > > > > 
> > > > > > That example shows a basic group admin command that I think Michael is
> > > > > > about to propose. We can't avoid this problem by not making it a group
> > > > > > admin command - it needs to be a group admin command. So I think it's
> > > > > > likely that there will be admin commands that take an unbounded amount
> > > > > > of time to complete. One way to achieve what you mentioned is timeouts.
> > > > > 
> > > > > I think that you're getting into device specific implementation details and
> > > > > I'm not sure it's necessary.
> > > > > 
> > > > > I don't think we need to abort admin commands. Admin commands can be
> > > > > flushed/aborted during the device reset phase.
> > > > > Only IO commands should have the possibility to being aborted as you
> > > > > mentioned in NVMe and SCSI (and potentially in virtio-blk).
> > > > 
> > > > It's a general design issue that should be clarified now rather than
> > > > being left unspecified.
> > > > 
> > > > I'm not saying that it must be possible to abort admin commands. There
> > > > are other options, like requiring the device itself to fail a command
> > > > after a timeout.
> > > 
> > > do you have an example of timeout today for control vq ?
> > 
> > Do you mean the virtio-net control virtqueue? I don't think it has any
> > commands with an unbounded execution time.
> 
> Correct. So why introducing it now ?

The examples I've given are the create and delete group member
operations. I think those operations will take unbounded time in some
device implementations.

> 
> > 
> > > > 
> > > > Or we could say that admin commands must complete within bounded time,
> > > > but I'm not sure that is implementable for some device types like
> > > > virtio-blk, virtio-scsi, and virtiofs.
> > > 
> > > No we can't.
> > > Some commands, for example FW upgrade can take 10 minutes and it's perfectly
> > > fine. Other commands like setting feature bit will take 1 millisec.
> > > Each device implements commands in a different internal logic so we can't
> > > expect to complete after X time.
> > 
> > When I say bounded time, I mean that it finishes in a finite amount of
> > time. I'm not saying there is a specific time X that all device
> > implementations must satisfy. Unbounded means it might never finish.
> 
> There might be a chance that any command for any virtio device type will
> never finish. Nothing new here in the adminq.
> 
> what one can do is to set a timeout for himself and if this timeout expire -
> check the device status. If it needs_reset - do a reset. if status is ok,
> then wait some more time.
> After X retries, unmap buffers or reset the adminq.

Michael: What effect does resetting the group owner device have on group
member devices?

I'm concerned that this approach disrupts all group member devices. For
example, you try to add a new device but the command hangs. In order to
recover you now have to reset the group owner device and this breaks all
the group member devices.

> > 
> > > Device can go to so FATAL state in case a command is stuck and causing
> > > internal errors in it.
> > > 
> > > > 
> > > > > For your example, stopping a member is possible even it there are some
> > > > > errors in the network. You can for example destroy all the connections to
> > > > > the remote target and complete all the BIOS with some error.
> > > > 
> > > > Forgetting about in-flight requests doesn't necessarily make them go
> > > > away. It creates a race between forgotten requests and reconnection. In
> > > > the worst case a forgotten write request takes effect after
> > > > reconnection, causing data corruption.
> > > 
> > > For making it work without data corruption we need a cooperation of the
> > > target side for sure. But this is fine since the target in that case is part
> > > of the "virtio-blk backend".
> > > One solution is that the target can decide it will flush all the requests to
> > > the storage device before accepting new connections.
> > 
> > This solution shifts the unbounded time from disconnection to
> > connection. The Group Member Delete command will complete quickly but a
> > subsequent Group Member Create command for the same underlying storage
> > device would need to wait until the requests are done.
> > 
> > Therefore I think the admin queue must be designed under the assumption
> > that some commands take a very long time.
> 
> For sure an admin command may take long time. FW upgrade can take 10 minutes
> for example.
> But each device is free to implement internal logic as he choose.
> 
> Same for live migration, when we stop/quiesce a device we must make sure it
> doesn't master any DMA operations. Thus, in some implementations we need to
> wait for all inflights to end fast. In others, we can invalidate the access
> to host/guest memory and wait for completions until the freeze state.
> 
> Bottom line, this is device implementation specific consideration.

What I'm asking is that the spec clarifies the command completion order
semantics (in-order or out-of-order), whether there is a mechanism to
abort commands, etc.

Device implementers can then take advantage of those aspects to
implement devices that don't hang (e.g. health monitoring becomes
unavailable when there is a long running command).

If the spec doesn't cover this, then device implementers will not be
able to work around it when implementing standard commands like
create/delete group member.

Does that make sense?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 7425 bytes --]

On Wed, Mar 08, 2023 at 01:57:43PM +0100, Jiri Pirko wrote:
> Wed, Mar 08, 2023 at 01:44:18PM CET, stefanha@redhat.com wrote:
> >On Wed, Mar 08, 2023 at 11:17:35AM +0100, Jiri Pirko wrote:
> >> Tue, Mar 07, 2023 at 08:03:47PM CET, stefanha@redhat.com wrote:
> >> >On Tue, Mar 07, 2023 at 04:07:54PM +0100, Jiri Pirko wrote:
> >> >> Tue, Mar 07, 2023 at 03:39:11PM CET, stefanha@redhat.com wrote:
> >> >> >On Tue, Mar 07, 2023 at 09:03:18AM +0100, Jiri Pirko wrote:
> >> >> >> Mon, Mar 06, 2023 at 07:37:31PM CET, mst@redhat.com wrote:
> >> >> >> >On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> >> >> >> >> On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> >> >> >> >> > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> >> >> >> >> > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> >> >> >> >> > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> >> >> >> >> > > > > What happens if a command takes 1 second to complete, is the device
> >> >> >> >> > > > > allowed to process the next command from the virtqueue during this time,
> >> >> >> >> > > > > possibly completing it before the first command?
> >> >> >> >> > > > > 
> >> >> >> >> > > > > This requires additional clarification in the spec because "they are
> >> >> >> >> > > > > processed by the device in the order in which they are queued" does not
> >> >> >> >> > > > > explain whether commands block the virtqueue (in order completion) or
> >> >> >> >> > > > > not (out of order completion).
> >> >> >> >> > > > 
> >> >> >> >> > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> >> >> >> >> > > 
> >> >> >> >> > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> >> >> >> >> > > Several may be processed by the device at the same time.
> >> >> >> >> > 
> >> >> >> >> > Let's say I submit a write followed by read - is read
> >> >> >> >> > guaranteed to return an up to date info?
> >> >> >> >> 
> >> >> >> >> In general, no. The driver must wait for the write completion before
> >> >> >> >> submitting the read if it wants consistency.
> >> >> >> >> 
> >> >> >> >> Stefan
> >> >> >> >
> >> >> >> >I see.  I think it's a good design to follow then.
> >> >> >> 
> >> >> >> Hmm, is it suitable to have this approach for configuration interface?
> >> >> >> Storage device is a different beast, having parallel reads and writes
> >> >> >> makes complete sense for performance.
> >> >> >> 
> >> >> >> ->read a req
> >> >> >> ->read b req
> >> >> >> ->read c req
> >> >> >> <-read a rep
> >> >> >> <-read b rep
> >> >> >> <-read c rep
> >> >> >> 
> >> >> >> There is no dependency, even between writes.
> >> >> >> 
> >> >> >> But in case of configuration, does not make any sense to me.
> >> >> >> Why is it needed? To pass the burden of consistency of
> >> >> >> configuration to driver sounds odd at least.
> >> >> >> 
> >> >> >> I sense there is no concete idea about what the "admin virtqueue" should
> >> >> >> serve for exactly.
> >> >> >
> >> >> >It's useful for long-running commands because they prevent other
> >> >> >commands from executing.
> >> >> >
> >> >> >An example I've given is that deleting a group member might require
> >> >> >waiting for the group member's I/O activity to finish. If that I/O
> >> >> >activity cannot be cancelled instantaneously, then it could take an
> >> >> >unbounded amount of time to delete the group member. The device would be
> >> >> >unable to process futher admin commands.
> >> >> 
> >> >> I see. Then I believe that the device should handle the dependencies.
> >> >> Example 1:
> >> >> -> REQ cmd to create group member A
> >> >> -> REQ cmd to create group member B
> >> >> <- REP cmd to create group member A
> >> >> <- REP cmd to create group member B
> >> >> 
> >> >> The device according to internal implementation can either serialize the
> >> >> 2 group member creations or do it in parallel, if it supports it.
> >> >> 
> >> >> Example 2:
> >> >> -> REQ cmd to create group member A
> >> >> -> REQ cmd config group member A
> >> >> <- REP cmd to create group member A
> >> >> <- REP cmd config group member A
> >> >> 
> >> >> Here the serialization is necessary and the device is the one to take
> >> >> care of it.
> >> >> 
> >> >> Makes sense?
> >> >
> >> >Yes, I understand. The spec would need to define ordering rules for
> >> >specific commands and the device must implement them. It allows the
> >> >driver to pipeline commands while also allowing out-of-order completion
> >> >(parallelism) in some cases. The disadvantage of this approach is
> >> >complexity in the spec and implementations.
> >> >
> >> >An alternative is unconditional out-of-order completion, where there are
> >> >no per-command ordering rules. The driver must wait for a command to
> >> >complete if it relies on the results of that command for its next
> >> >command. I like this approach because it's less complex in the spec and
> >> >for device implementers, while the burden on the driver implementer is
> >> >still reasonable.
> >> 
> >> But isn't this duplicating the burden of maintaining dependencies to
> >> both driver and device? I mean, device should not depend on driver doing
> >> the right thing, that means it has to check the dependencies for every
> >> incoming command anyway. The only difference would be to wait instead of
> >> returning "-EBUSY" in case the dependency is not satisfied yet.
> >
> >The device does not need to reject commands that have dependencies with
> >-EBUSY. The result of commands with dependencies is either A -> B or B
> >-> A.
> >
> >For example:
> >1. Create Group Member A
> >2. Delete Group Member A
> >
> >Command 2 might succeed or it might fail with -ENOENT because Group
> >Member A doesn't exist yet.
> 
> Yeah, correct.
> 
> >
> >> Device knows exactly what are the dependencies. And I believe, those are
> >> device implementation specific. For example, some implementation could
> >> support parallel VF config cmd execution, some implementation might
> >> need to serialize that. Driver has no clue.
> >
> >Yes, that's up to the device. Out-of-order completion is a superset of
> >in-order completion. So the device is allowed to run commands in series
> >when it wants. A driver designed for out-of-order completion will work
> >fine either way.
> 
> Agreed.
> 
> 
> >
> >> Could you please elaborate a bit more what you mean by "complexity in
> >> the spec"?
> >
> >When adding commands to the spec, the dependency relationships with
> >other commands need to be thought about and documented.
> >
> >Device implementers need to get those relationships right. That means
> >they need to remember that command B waits for command A.
> 
> That's my point, this is device implementation specific. Spec should not
> define anything like this, that might be limiting in some of the cases.
> 
> >
> >Driver implementers have to understand that command B waits for command
> >A but not command C according to the spec.
> >
> >That seems complex to me.
> 
> Sure, I agree this does not belong to the spec, it can't, really.
> For the record, I never suggested it.
> 
> I think we are in agreement.

Okay. In that case I think I misunderstood what you meant. Sorry!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 12:15:23PM -0500, Stefan Hajnoczi wrote:
> > > > > Or we could say that admin commands must complete within bounded time,
> > > > > but I'm not sure that is implementable for some device types like
> > > > > virtio-blk, virtio-scsi, and virtiofs.
> > > > 
> > > > No we can't.
> > > > Some commands, for example FW upgrade can take 10 minutes and it's perfectly
> > > > fine. Other commands like setting feature bit will take 1 millisec.
> > > > Each device implements commands in a different internal logic so we can't
> > > > expect to complete after X time.
> > > 
> > > When I say bounded time, I mean that it finishes in a finite amount of
> > > time. I'm not saying there is a specific time X that all device
> > > implementations must satisfy. Unbounded means it might never finish.
> > 
> > There might be a chance that any command for any virtio device type will
> > never finish. Nothing new here in the adminq.
> > 
> > what one can do is to set a timeout for himself and if this timeout expire -
> > check the device status. If it needs_reset - do a reset. if status is ok,
> > then wait some more time.
> > After X retries, unmap buffers or reset the adminq.
> 
> Michael: What effect does resetting the group owner device have on group
> member devices?

virtio level reset? It's a good question. I'd expect them all to be
reset no?

> I'm concerned that this approach disrupts all group member devices. For
> example, you try to add a new device but the command hangs. In order to
> recover you now have to reset the group owner device and this breaks all
> the group member devices.


I agree. How about a VQ level reset though? Seems like exactly
what's needed here?


> > > 
> > > > Device can go to so FATAL state in case a command is stuck and causing
> > > > internal errors in it.
> > > > 
> > > > > 
> > > > > > For your example, stopping a member is possible even it there are some
> > > > > > errors in the network. You can for example destroy all the connections to
> > > > > > the remote target and complete all the BIOS with some error.
> > > > > 
> > > > > Forgetting about in-flight requests doesn't necessarily make them go
> > > > > away. It creates a race between forgotten requests and reconnection. In
> > > > > the worst case a forgotten write request takes effect after
> > > > > reconnection, causing data corruption.
> > > > 
> > > > For making it work without data corruption we need a cooperation of the
> > > > target side for sure. But this is fine since the target in that case is part
> > > > of the "virtio-blk backend".
> > > > One solution is that the target can decide it will flush all the requests to
> > > > the storage device before accepting new connections.
> > > 
> > > This solution shifts the unbounded time from disconnection to
> > > connection. The Group Member Delete command will complete quickly but a
> > > subsequent Group Member Create command for the same underlying storage
> > > device would need to wait until the requests are done.
> > > 
> > > Therefore I think the admin queue must be designed under the assumption
> > > that some commands take a very long time.
> > 
> > For sure an admin command may take long time. FW upgrade can take 10 minutes
> > for example.
> > But each device is free to implement internal logic as he choose.
> > 
> > Same for live migration, when we stop/quiesce a device we must make sure it
> > doesn't master any DMA operations. Thus, in some implementations we need to
> > wait for all inflights to end fast. In others, we can invalidate the access
> > to host/guest memory and wait for completions until the freeze state.
> > 
> > Bottom line, this is device implementation specific consideration.
> 
> What I'm asking is that the spec clarifies the command completion order
> semantics (in-order or out-of-order), whether there is a mechanism to
> abort commands, etc.
> 
> Device implementers can then take advantage of those aspects to
> implement devices that don't hang (e.g. health monitoring becomes
> unavailable when there is a long running command).
> 
> If the spec doesn't cover this, then device implementers will not be
> able to work around it when implementing standard commands like
> create/delete group member.
> 
> Does that make sense?
> 
> Stefan

It does, to me.



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 05:30:48PM +0100, Cornelia Huck wrote:
> On Mon, Mar 06 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Mar 06, 2023 at 01:29:30PM +0100, Jiri Pirko wrote:
> >> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
> >> 
> >> [...]
> >> 
> >> >
> >> >TODO (maybe?) - probably ok to defer until this part is upstream:
> >> >
> >> >	Add "all members" member id.
> >> >
> >> >	Add commands for MSI, feature discovery.
> >> >
> >> >	Add commands for transport vq.
> >> >
> >> >
> >> >My intent is to try and support both SR-IOV and SIOV
> >> >usecases with the same structure and maybe even the same
> >> >VQ.
> >> 
> >> Sorry to be late to the party, I'm trying to catch up. Is it common to
> >> have cover letter for features this brief? I mean, from the cover
> >> letter, I'm totally unable to understand what you are introducing here.
> >> 
> >> Could you elaborate about what you are aiming to achive with this?
> >> Could you shed some usecases perhaps?
> >> 
> >> I have to be missing something obvious, but I don't get why any notion
> >> of SR-IOV could be beneficial for virtio.
> >> 
> >
> > Good point, I'll add a bit of motivation.
> >
> > For SR-IOV, it is not unusual for PFs to excercise control over VFs.
> > There is interest in the community to include an interface to allow this
> > in the virtio spec, when the PF is a virtio device.  This is what this
> > patch does.
> 
> Unfortunately, information in the cover letter does not make it into
> git -- should things like that go into the github issue (and into the
> ballot?) It's useful both for reviewing (cover letter) and understanding
> the rationale later (github/ballot).

Well the github issue has a link to the archives, but sure it's
possible to copy that.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[David Edmondson]

Max Gurtovoy <mgurtovoy@nvidia.com> writes:

> On 08/03/2023 14:08, Jiri Pirko wrote:
>> Wed, Mar 08, 2023 at 12:50:48PM CET, mst@redhat.com wrote:
>>> On Wed, Mar 08, 2023 at 11:05:00AM +0100, Jiri Pirko wrote:
>>>> Tue, Mar 07, 2023 at 05:30:18PM CET, mst@redhat.com wrote:
>>>>> On Tue, Mar 07, 2023 at 08:36:41AM +0100, Jiri Pirko wrote:
>>>>>> Hmm, if not for now, the future exension would not be so simple, I fear.
>>>>>
>>>>> Without knowing what it is I can't say.
>>>>
>>>> Yep, so basically you say, for other things if they appear,
>>>> let's introduce another queue type? If yes, sounds fair to me.
>>>
>>> Yes. For example I find it likely that live migration/failover support
>>> will require a queue where driver pre-adds buffers and then device
>>> supplies information as state changes.
>> 
>> I see. So there would be a queue called for example "child state virtqueue"
>> or something like that for the sole purpose of getting the state of VF?
>> Hmm, wouldn't it make more sense to have this done as a part of "group
>> administrarion queue"? I mean, there is already notion of addresing
>> child/VF here. So from my perspective, it is just another "group
>> administration" command.
>
> For sure VF Live Migration, MSIX config of VF, VF feature bits config 
> and others should be admin commands on admin vq.
> I don't see any reason introducing another type of admin-like vq.
> Also we don't need to have multiple admin vqs. This AQ is not aimed for 
> performance.

In support of live migration, might we end up moving large amounts of
device state through the admin queue?

If so, that would seem to have some performance requirements, though I
don't know if it would justify multiple admin queues.
-- 
Tonight I think I'll walk alone, I'll find my soul as I go home.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Jiri Pirko]

Wed, Mar 08, 2023 at 05:30:48PM CET, cohuck@redhat.com wrote:
>On Mon, Mar 06 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
>> On Mon, Mar 06, 2023 at 01:29:30PM +0100, Jiri Pirko wrote:
>>> Thu, Mar 02, 2023 at 02:04:48PM CET, mst@redhat.com wrote:
>>> 
>>> [...]
>>> 
>>> >
>>> >TODO (maybe?) - probably ok to defer until this part is upstream:
>>> >
>>> >	Add "all members" member id.
>>> >
>>> >	Add commands for MSI, feature discovery.
>>> >
>>> >	Add commands for transport vq.
>>> >
>>> >
>>> >My intent is to try and support both SR-IOV and SIOV
>>> >usecases with the same structure and maybe even the same
>>> >VQ.
>>> 
>>> Sorry to be late to the party, I'm trying to catch up. Is it common to
>>> have cover letter for features this brief? I mean, from the cover
>>> letter, I'm totally unable to understand what you are introducing here.
>>> 
>>> Could you elaborate about what you are aiming to achive with this?
>>> Could you shed some usecases perhaps?
>>> 
>>> I have to be missing something obvious, but I don't get why any notion
>>> of SR-IOV could be beneficial for virtio.
>>> 
>>
>> Good point, I'll add a bit of motivation.
>>
>> For SR-IOV, it is not unusual for PFs to excercise control over VFs.
>> There is interest in the community to include an interface to allow this
>> in the virtio spec, when the PF is a virtio device.  This is what this
>> patch does.
>
>Unfortunately, information in the cover letter does not make it into
>git -- should things like that go into the github issue (and into the

Why's that? Can't you have a merge commit to include the cover letter,
as we have it in netdev for example?


>ballot?) It's useful both for reviewing (cover letter) and understanding
>the rationale later (github/ballot).
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Wed, Mar 08, 2023 at 07:01:02PM CET, david.edmondson@oracle.com wrote:
>Max Gurtovoy <mgurtovoy@nvidia.com> writes:
>
>> On 08/03/2023 14:08, Jiri Pirko wrote:
>>> Wed, Mar 08, 2023 at 12:50:48PM CET, mst@redhat.com wrote:
>>>> On Wed, Mar 08, 2023 at 11:05:00AM +0100, Jiri Pirko wrote:
>>>>> Tue, Mar 07, 2023 at 05:30:18PM CET, mst@redhat.com wrote:
>>>>>> On Tue, Mar 07, 2023 at 08:36:41AM +0100, Jiri Pirko wrote:
>>>>>>> Hmm, if not for now, the future exension would not be so simple, I fear.
>>>>>>
>>>>>> Without knowing what it is I can't say.
>>>>>
>>>>> Yep, so basically you say, for other things if they appear,
>>>>> let's introduce another queue type? If yes, sounds fair to me.
>>>>
>>>> Yes. For example I find it likely that live migration/failover support
>>>> will require a queue where driver pre-adds buffers and then device
>>>> supplies information as state changes.
>>> 
>>> I see. So there would be a queue called for example "child state virtqueue"
>>> or something like that for the sole purpose of getting the state of VF?
>>> Hmm, wouldn't it make more sense to have this done as a part of "group
>>> administrarion queue"? I mean, there is already notion of addresing
>>> child/VF here. So from my perspective, it is just another "group
>>> administration" command.
>>
>> For sure VF Live Migration, MSIX config of VF, VF feature bits config 
>> and others should be admin commands on admin vq.
>> I don't see any reason introducing another type of admin-like vq.
>> Also we don't need to have multiple admin vqs. This AQ is not aimed for 
>> performance.
>
>In support of live migration, might we end up moving large amounts of
>device state through the admin queue?

Sounds to me that would be the case, yes. Depends on device
implementation, but could be potentially big. We should make sure it
scales good.


>
>If so, that would seem to have some performance requirements, though I
>don't know if it would justify multiple admin queues.

Well, I can imagine when migrating a lots of VFs at the same time (host
shutdown), multiqueue might definitelly help.



>-- 
>Tonight I think I'll walk alone, I'll find my soul as I go home.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Jiri Pirko]

Wed, Mar 08, 2023 at 05:30:21PM CET, mst@redhat.com wrote:
>On Tue, Mar 07, 2023 at 06:52:03PM +0000, Parav Pandit wrote:
>> > And I wonder whether it's worth it - it definitely makes contributing to Linux
>> > harder, and even within Linux it pushes contributors away. 
>> The number of virtio spec contributors are in order of magnitude less
>> than Linux kernel or just the Linux netdev.  Patch split up reduces
>> lot of time author and reviewers.  For example, interrupt moderation
>> at v10 can be very easily split up where prep patch can have RB tag
>> and v11 doesn't need reviewers and author's time.  Your work here with
>> 10 patches is yet another good example of it.  I remember Max and I
>> started with 6 patches... and current 10 are more useful way to split
>> them.
>
>It's a big project though. All I ask is that we try to be gentler
>than netdev to new contributors. If there's a 10 line patch it is
>not necessary to be pedantic and ask for it to be split up to
>a 3 patch series.

One patch per change quite common everywhere. IDK, I don't see why that
would be a problem.


>
>-- 
>MST
>
>
>---------------------------------------------------------------------
>To unsubscribe from this mail list, you must leave the OASIS TC that 
>generates this mail.  Follow this link to all your TCs in OASIS at:
>https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 10:58:59PM +0200, Max Gurtovoy wrote:
> We have a non-spec prototypes for LM so please use our experience in this
> area.

Don't see what the fuss is about.  Nothing will prevent your device from
using a single queue if that is what you want.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, March 8, 2023 4:09 PM
> 
> On Wed, Mar 08, 2023 at 10:58:59PM +0200, Max Gurtovoy wrote:
> > We have a non-spec prototypes for LM so please use our experience in
> > this area.
> 
> Don't see what the fuss is about.  Nothing will prevent your device from using a
> single queue if that is what you want.

I think the use case is not clear in the cover letter on how/when multiple aq are useful or will be used.

I agree with you on this single queue should be enough provided a single queue suffices below design aspects:

The design:
a. to have one aq to support multiple outstanding commands
b. ability to execute unrelated aq commands by device in parallel
c. driver to not issue related commands to the AQ (because they are executed out of order by the device)
(Assumption is IN_ORDER is not negotiated).

Above the actual use case requirements/expectations from the AQ.

The current version indicates that the device will execute commands in-order, hence multi-queue is needed to achieve parallelism - is not serving the purpose of "q" notion fully.
Or do I miss reading v10?



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of David Edmondson

> In support of live migration, might we end up moving large amounts of device
> state through the admin queue?
> 
Correct.

> If so, that would seem to have some performance requirements, though I don't
> know if it would justify multiple admin queues.
DMA of the data through the proposed AQ is supported.

If I understood Max correctly when Max said " This AQ is not aimed for performance ",
he means that AQ doesn't have performance requirements as io/network queues to complete millions of ops/sec.

it is several hundred to maybe (on the higher side) thousand ops/sec during LM, provisioning use case.

DMA perspective as you mentioned, AQ still has the same perf requirements as that of regular nw/blk io queues.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Jiri Pirko <jiri@nvidia.com>
> Sent: Wednesday, March 8, 2023 5:05 AM

> >For example a common feature is to program a vlan and have device put a
> >given VF inside this vlan.
> 
> I don't follow entirely. The way how the VF is connected to network should be
> ouf of the scope of this interface. The eswitch manager should take care. What
> you say sounds awfully like the "ip vf" legacy interface, which should not be
> considered here I believe.
> 
> If PF would be the eswitch manager, there are other means to do network
> programming, using eswitch port representors. But I don't think this is the can
> of worms we want to open now. I don't think we have a usecase for it currently.
> Am I wrong Parav?
> 
> 
We want the ability to program/provision the virtio feature bits and virtio config space parameters of the VF through PF.
These are host-facing parameters that usually migrate from one to another host when a VF migrates.

A future device may even do this out of band, but we are far from it.
At that point in the future virtio management device can take birth and do things over it.

ip vf was legacy interface and it is not applicable here.
virtio net doesn't implement it either and I don't see any user asking for it either.
All users have migrated to using tc mechanism for long time now.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Thu, Mar 09, 2023 at 02:29:31AM +0200, Max Gurtovoy wrote:
> please use "Co-developed-by:" for this patch

Sure.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Wed, Mar 08, 2023 at 10:09:22PM CET, mst@redhat.com wrote:
>On Wed, Mar 08, 2023 at 10:58:59PM +0200, Max Gurtovoy wrote:
>> We have a non-spec prototypes for LM so please use our experience in this
>> area.
>
>Don't see what the fuss is about.  Nothing will prevent your device from
>using a single queue if that is what you want.

Yeah. Plus designing the thing multiq in the spec from start is
future proof. I second that.


>
>-- 
>MST
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
>
>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
>> open.org> On Behalf Of David Edmondson
>
>> In support of live migration, might we end up moving large amounts of device
>> state through the admin queue?
>> 
>Correct.
>
>> If so, that would seem to have some performance requirements, though I don't
>> know if it would justify multiple admin queues.
>DMA of the data through the proposed AQ is supported.
>
>If I understood Max correctly when Max said " This AQ is not aimed for performance ",
>he means that AQ doesn't have performance requirements as io/network queues to complete millions of ops/sec.
>
>it is several hundred to maybe (on the higher side) thousand ops/sec during LM, provisioning use case.

But isn't it good to design it for performance from start? I mean, state
transfer of thousands of VFs at a time is definitelly performance
related, isn't it?


>
>DMA perspective as you mentioned, AQ still has the same perf requirements as that of regular nw/blk io queues.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Wed, Mar 08, 2023 at 10:45:18PM CET, parav@nvidia.com wrote:
>
>> From: Jiri Pirko <jiri@nvidia.com>
>> Sent: Wednesday, March 8, 2023 5:05 AM
>
>> >For example a common feature is to program a vlan and have device put a
>> >given VF inside this vlan.
>> 
>> I don't follow entirely. The way how the VF is connected to network should be
>> ouf of the scope of this interface. The eswitch manager should take care. What
>> you say sounds awfully like the "ip vf" legacy interface, which should not be
>> considered here I believe.
>> 
>> If PF would be the eswitch manager, there are other means to do network
>> programming, using eswitch port representors. But I don't think this is the can
>> of worms we want to open now. I don't think we have a usecase for it currently.
>> Am I wrong Parav?
>> 
>> 
>We want the ability to program/provision the virtio feature bits and virtio config space parameters of the VF through PF.
>These are host-facing parameters that usually migrate from one to another host when a VF migrates.

Here is where we lack to have separate "devlink function" entity,
independend of "devlink port" :/ I mean, for MAC config and irq vectors.
What other host facing parmeters you have in mind. VLAN of "ip vf" is
definitelly one of them, I have to emphasize.


>
>A future device may even do this out of band, but we are far from it.
>At that point in the future virtio management device can take birth and do things over it.
>
>ip vf was legacy interface and it is not applicable here.

Agreed, unacceptable to implement "ip vf" legacy interfaces in
virtio_net from my perspective.


>virtio net doesn't implement it either and I don't see any user asking for it either.
>All users have migrated to using tc mechanism for long time now.

Okay, so the bottom line is, we should take care of host facing config
of VF, but we should not care about network plug of VF - that stays out
of the scope.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1972 bytes --]

On Wed, Mar 08, 2023 at 12:21:46PM -0500, Michael S. Tsirkin wrote:
> On Wed, Mar 08, 2023 at 12:15:23PM -0500, Stefan Hajnoczi wrote:
> > > > > > Or we could say that admin commands must complete within bounded time,
> > > > > > but I'm not sure that is implementable for some device types like
> > > > > > virtio-blk, virtio-scsi, and virtiofs.
> > > > > 
> > > > > No we can't.
> > > > > Some commands, for example FW upgrade can take 10 minutes and it's perfectly
> > > > > fine. Other commands like setting feature bit will take 1 millisec.
> > > > > Each device implements commands in a different internal logic so we can't
> > > > > expect to complete after X time.
> > > > 
> > > > When I say bounded time, I mean that it finishes in a finite amount of
> > > > time. I'm not saying there is a specific time X that all device
> > > > implementations must satisfy. Unbounded means it might never finish.
> > > 
> > > There might be a chance that any command for any virtio device type will
> > > never finish. Nothing new here in the adminq.
> > > 
> > > what one can do is to set a timeout for himself and if this timeout expire -
> > > check the device status. If it needs_reset - do a reset. if status is ok,
> > > then wait some more time.
> > > After X retries, unmap buffers or reset the adminq.
> > 
> > Michael: What effect does resetting the group owner device have on group
> > member devices?
> 
> virtio level reset? It's a good question. I'd expect them all to be
> reset no?
> 
> > I'm concerned that this approach disrupts all group member devices. For
> > example, you try to add a new device but the command hangs. In order to
> > recover you now have to reset the group owner device and this breaks all
> > the group member devices.
> 
> 
> I agree. How about a VQ level reset though? Seems like exactly
> what's needed here?

Yes, a new virtqueue-level reset feature would take care of this case.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Jiri Pirko <jiri@nvidia.com>
> Sent: Thursday, March 9, 2023 2:37 AM


> >We want the ability to program/provision the virtio feature bits and virtio
> config space parameters of the VF through PF.
> >These are host-facing parameters that usually migrate from one to another
> host when a VF migrates.
> 
> Here is where we lack to have separate "devlink function" entity, independend
> of "devlink port" :/ I mean, for MAC config and irq vectors.
> What other host facing parmeters you have in mind. VLAN of "ip vf" is
> definitelly one of them, I have to emphasize.
> 
It is not.

Virtio feature bits defined by the virtio spec.
Virtio config space fields present in the each virtio device specific spec.

> 
> >

> >virtio net doesn't implement it either and I don't see any user asking for it
> either.
> >All users have migrated to using tc mechanism for long time now.
> 
> Okay, so the bottom line is, we should take care of host facing config of VF, but
> we should not care about network plug of VF - that stays out of the scope.
Right.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Jiri Pirko <jiri@nvidia.com>
> Sent: Thursday, March 9, 2023 2:31 AM
> 
> Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
> >
> >> From: virtio-comment@lists.oasis-open.org
> >> <virtio-comment@lists.oasis- open.org> On Behalf Of David Edmondson
> >
> >> In support of live migration, might we end up moving large amounts of
> >> device state through the admin queue?
> >>
> >Correct.
> >
> >> If so, that would seem to have some performance requirements, though
> >> I don't know if it would justify multiple admin queues.
> >DMA of the data through the proposed AQ is supported.
> >
> >If I understood Max correctly when Max said " This AQ is not aimed for
> >performance ", he means that AQ doesn't have performance requirements as
> io/network queues to complete millions of ops/sec.
> >
> >it is several hundred to maybe (on the higher side) thousand ops/sec during
> LM, provisioning use case.
> 
> But isn't it good to design it for performance from start? I mean, state transfer
> of thousands of VFs at a time is definitelly performance related, isn't it?
> 
It is. Which part of the proposed AQ doesn't cover this aspect?
The only issue that I see today is, that a given GET family of commands q contains the read-only and write-only descriptors which require multiple dma allocations on driver side.

> 
> >
> >DMA perspective as you mentioned, AQ still has the same perf requirements
> as that of regular nw/blk io queues.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Thu, Mar 09, 2023 at 07:35:42AM -0500, Stefan Hajnoczi wrote:
> On Wed, Mar 08, 2023 at 12:21:46PM -0500, Michael S. Tsirkin wrote:
> > On Wed, Mar 08, 2023 at 12:15:23PM -0500, Stefan Hajnoczi wrote:
> > > > > > > Or we could say that admin commands must complete within bounded time,
> > > > > > > but I'm not sure that is implementable for some device types like
> > > > > > > virtio-blk, virtio-scsi, and virtiofs.
> > > > > > 
> > > > > > No we can't.
> > > > > > Some commands, for example FW upgrade can take 10 minutes and it's perfectly
> > > > > > fine. Other commands like setting feature bit will take 1 millisec.
> > > > > > Each device implements commands in a different internal logic so we can't
> > > > > > expect to complete after X time.
> > > > > 
> > > > > When I say bounded time, I mean that it finishes in a finite amount of
> > > > > time. I'm not saying there is a specific time X that all device
> > > > > implementations must satisfy. Unbounded means it might never finish.
> > > > 
> > > > There might be a chance that any command for any virtio device type will
> > > > never finish. Nothing new here in the adminq.
> > > > 
> > > > what one can do is to set a timeout for himself and if this timeout expire -
> > > > check the device status. If it needs_reset - do a reset. if status is ok,
> > > > then wait some more time.
> > > > After X retries, unmap buffers or reset the adminq.
> > > 
> > > Michael: What effect does resetting the group owner device have on group
> > > member devices?
> > 
> > virtio level reset? It's a good question. I'd expect them all to be
> > reset no?
> > 
> > > I'm concerned that this approach disrupts all group member devices. For
> > > example, you try to add a new device but the command hangs. In order to
> > > recover you now have to reset the group owner device and this breaks all
> > > the group member devices.
> > 
> > 
> > I agree. How about a VQ level reset though? Seems like exactly
> > what's needed here?
> 
> Yes, a new virtqueue-level reset feature would take care of this case.
> 
> Stefan

Why would we need a new one? We already have VIRTIO_F_RING_RESET.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Thu, Mar 09, 2023 at 01:04:30PM +0000, Parav Pandit wrote:
> 
> > From: Jiri Pirko <jiri@nvidia.com>
> > Sent: Thursday, March 9, 2023 2:31 AM
> > 
> > Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
> > >
> > >> From: virtio-comment@lists.oasis-open.org
> > >> <virtio-comment@lists.oasis- open.org> On Behalf Of David Edmondson
> > >
> > >> In support of live migration, might we end up moving large amounts of
> > >> device state through the admin queue?
> > >>
> > >Correct.
> > >
> > >> If so, that would seem to have some performance requirements, though
> > >> I don't know if it would justify multiple admin queues.
> > >DMA of the data through the proposed AQ is supported.
> > >
> > >If I understood Max correctly when Max said " This AQ is not aimed for
> > >performance ", he means that AQ doesn't have performance requirements as
> > io/network queues to complete millions of ops/sec.
> > >
> > >it is several hundred to maybe (on the higher side) thousand ops/sec during
> > LM, provisioning use case.
> > 
> > But isn't it good to design it for performance from start? I mean, state transfer
> > of thousands of VFs at a time is definitelly performance related, isn't it?
> > 
> It is. Which part of the proposed AQ doesn't cover this aspect?
> The only issue that I see today is, that a given GET family of commands q contains the read-only and write-only descriptors which require multiple dma allocations on driver side.

BTW for a while now I wanted to add a descriptor flag that
will stick data inline in the descriptor. Perfect for passing
small bits of data such as vqn, or the virtio header.


> > 
> > >
> > >DMA perspective as you mentioned, AQ still has the same perf requirements
> > as that of regular nw/blk io queues.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[David Edmondson]

Parav Pandit <parav@nvidia.com> writes:

>> From: Jiri Pirko <jiri@nvidia.com>
>> Sent: Thursday, March 9, 2023 2:31 AM
>> 
>> Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
>> >
>> >> From: virtio-comment@lists.oasis-open.org
>> >> <virtio-comment@lists.oasis- open.org> On Behalf Of David Edmondson
>> >
>> >> In support of live migration, might we end up moving large amounts of
>> >> device state through the admin queue?
>> >>
>> >Correct.
>> >
>> >> If so, that would seem to have some performance requirements, though
>> >> I don't know if it would justify multiple admin queues.
>> >DMA of the data through the proposed AQ is supported.
>> >
>> >If I understood Max correctly when Max said " This AQ is not aimed for
>> >performance ", he means that AQ doesn't have performance requirements as
>> io/network queues to complete millions of ops/sec.
>> >
>> >it is several hundred to maybe (on the higher side) thousand ops/sec during
>> LM, provisioning use case.
>> 
>> But isn't it good to design it for performance from start? I mean, state transfer
>> of thousands of VFs at a time is definitelly performance related, isn't it?
>> 
> It is. Which part of the proposed AQ doesn't cover this aspect?
> The only issue that I see today is, that a given GET family of commands q contains the read-only and write-only descriptors which require multiple dma allocations on driver side.

I don't think that there is any assertion that the current proposal
doesn't cover this.

It was intended as an illustration to try and help determine whether
multiple AQs for a single device was a requirement given existing
anticipated usage, or a safeguard against potential future uses.
-- 
Up eighty flights of stairs to a basement flat.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: David Edmondson <david.edmondson@oracle.com>
> Sent: Thursday, March 9, 2023 9:03 AM
> 
> Parav Pandit <parav@nvidia.com> writes:
> 
> >> From: Jiri Pirko <jiri@nvidia.com>
> >> Sent: Thursday, March 9, 2023 2:31 AM
> >>
> >> Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
> >> >
> >> >> From: virtio-comment@lists.oasis-open.org
> >> >> <virtio-comment@lists.oasis- open.org> On Behalf Of David
> >> >> Edmondson
> >> >
> >> >> In support of live migration, might we end up moving large amounts
> >> >> of device state through the admin queue?
> >> >>
> >> >Correct.
> >> >
> >> >> If so, that would seem to have some performance requirements,
> >> >> though I don't know if it would justify multiple admin queues.
> >> >DMA of the data through the proposed AQ is supported.
> >> >
> >> >If I understood Max correctly when Max said " This AQ is not aimed
> >> >for performance ", he means that AQ doesn't have performance
> >> >requirements as
> >> io/network queues to complete millions of ops/sec.
> >> >
> >> >it is several hundred to maybe (on the higher side) thousand ops/sec
> >> >during
> >> LM, provisioning use case.
> >>
> >> But isn't it good to design it for performance from start? I mean,
> >> state transfer of thousands of VFs at a time is definitelly performance
> related, isn't it?
> >>
> > It is. Which part of the proposed AQ doesn't cover this aspect?
> > The only issue that I see today is, that a given GET family of commands q
> contains the read-only and write-only descriptors which require multiple dma
> allocations on driver side.
> 
> I don't think that there is any assertion that the current proposal doesn't cover
> this.
> 
> It was intended as an illustration to try and help determine whether multiple
> AQs for a single device was a requirement given existing anticipated usage, or
> a safeguard against potential future uses.

Got it.

So far my understanding is:
1. Single AQ is enough for currently cited use cases
2. Having the infrastructure to support multiple AQ is also good.
The extra cost of this infra is only one read-only field = aq count as Michael proposed.

3. Ability to send multiple outstanding commands and execute them out of order by device is/should be supported as long as IN_ORDER is not negotiated.
Should it be relaxed for AQ that even if IN_ORDER is negotiated, AQ by default can be out of order queue?

4. Driver is the sole owner to decide if it is modifying an object through AQ, it should synchronize with ongoing AQ command on the same object or not.
5. If the command gets stuck for a very long time, the driver can recover the AQ using the existing Q RESET facility.

Do we agree with the above design thoughts?
Or there is a better design than this discussed, and I missed it?



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Thu, Mar 09, 2023 at 02:11:54PM +0000, Parav Pandit wrote:
> 
> 
> > From: David Edmondson <david.edmondson@oracle.com>
> > Sent: Thursday, March 9, 2023 9:03 AM
> > 
> > Parav Pandit <parav@nvidia.com> writes:
> > 
> > >> From: Jiri Pirko <jiri@nvidia.com>
> > >> Sent: Thursday, March 9, 2023 2:31 AM
> > >>
> > >> Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
> > >> >
> > >> >> From: virtio-comment@lists.oasis-open.org
> > >> >> <virtio-comment@lists.oasis- open.org> On Behalf Of David
> > >> >> Edmondson
> > >> >
> > >> >> In support of live migration, might we end up moving large amounts
> > >> >> of device state through the admin queue?
> > >> >>
> > >> >Correct.
> > >> >
> > >> >> If so, that would seem to have some performance requirements,
> > >> >> though I don't know if it would justify multiple admin queues.
> > >> >DMA of the data through the proposed AQ is supported.
> > >> >
> > >> >If I understood Max correctly when Max said " This AQ is not aimed
> > >> >for performance ", he means that AQ doesn't have performance
> > >> >requirements as
> > >> io/network queues to complete millions of ops/sec.
> > >> >
> > >> >it is several hundred to maybe (on the higher side) thousand ops/sec
> > >> >during
> > >> LM, provisioning use case.
> > >>
> > >> But isn't it good to design it for performance from start? I mean,
> > >> state transfer of thousands of VFs at a time is definitelly performance
> > related, isn't it?
> > >>
> > > It is. Which part of the proposed AQ doesn't cover this aspect?
> > > The only issue that I see today is, that a given GET family of commands q
> > contains the read-only and write-only descriptors which require multiple dma
> > allocations on driver side.
> > 
> > I don't think that there is any assertion that the current proposal doesn't cover
> > this.
> > 
> > It was intended as an illustration to try and help determine whether multiple
> > AQs for a single device was a requirement given existing anticipated usage, or
> > a safeguard against potential future uses.
> 
> Got it.
> 
> So far my understanding is:
> 1. Single AQ is enough for currently cited use cases

FYI transport VQ work started off with multiple vqs.

> 2. Having the infrastructure to support multiple AQ is also good.
> The extra cost of this infra is only one read-only field = aq count as Michael proposed.
> 
> 3. Ability to send multiple outstanding commands and execute them out of order by device is/should be supported as long as IN_ORDER is not negotiated.
> Should it be relaxed for AQ that even if IN_ORDER is negotiated, AQ by default can be out of order queue?

I feel there's no need: driver can just avoid negotiating IN_ORDER.
Generally I'm not happy with IN_ORDER and have some ideas to
replace it with PARTIAL_ORDER, that will also address this.

> 4. Driver is the sole owner to decide if it is modifying an object through AQ, it should synchronize with ongoing AQ command on the same object or not.
> 5. If the command gets stuck for a very long time, the driver can recover the AQ using the existing Q RESET facility.
> 
> Do we agree with the above design thoughts?
> Or there is a better design than this discussed, and I missed it?
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 9, 2023 9:23 AM
> 
> On Thu, Mar 09, 2023 at 02:11:54PM +0000, Parav Pandit wrote:
> >
> >
> > > From: David Edmondson <david.edmondson@oracle.com>
> > > Sent: Thursday, March 9, 2023 9:03 AM
> > >
> > > Parav Pandit <parav@nvidia.com> writes:
> > >
> > > >> From: Jiri Pirko <jiri@nvidia.com>
> > > >> Sent: Thursday, March 9, 2023 2:31 AM
> > > >>
> > > >> Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
> > > >> >
> > > >> >> From: virtio-comment@lists.oasis-open.org
> > > >> >> <virtio-comment@lists.oasis- open.org> On Behalf Of David
> > > >> >> Edmondson
> > > >> >
> > > >> >> In support of live migration, might we end up moving large
> > > >> >> amounts of device state through the admin queue?
> > > >> >>
> > > >> >Correct.
> > > >> >
> > > >> >> If so, that would seem to have some performance requirements,
> > > >> >> though I don't know if it would justify multiple admin queues.
> > > >> >DMA of the data through the proposed AQ is supported.
> > > >> >
> > > >> >If I understood Max correctly when Max said " This AQ is not
> > > >> >aimed for performance ", he means that AQ doesn't have
> > > >> >performance requirements as
> > > >> io/network queues to complete millions of ops/sec.
> > > >> >
> > > >> >it is several hundred to maybe (on the higher side) thousand
> > > >> >ops/sec during
> > > >> LM, provisioning use case.
> > > >>
> > > >> But isn't it good to design it for performance from start? I
> > > >> mean, state transfer of thousands of VFs at a time is definitelly
> > > >> performance
> > > related, isn't it?
> > > >>
> > > > It is. Which part of the proposed AQ doesn't cover this aspect?
> > > > The only issue that I see today is, that a given GET family of
> > > > commands q
> > > contains the read-only and write-only descriptors which require
> > > multiple dma allocations on driver side.
> > >
> > > I don't think that there is any assertion that the current proposal
> > > doesn't cover this.
> > >
> > > It was intended as an illustration to try and help determine whether
> > > multiple AQs for a single device was a requirement given existing
> > > anticipated usage, or a safeguard against potential future uses.
> >
> > Got it.
> >
> > So far my understanding is:
> > 1. Single AQ is enough for currently cited use cases
> 
> FYI transport VQ work started off with multiple vqs.
> 
Ok. so there may be some use case for transport VQ.
Worth to mention in the cover letter and things will be fine.
Actual review of the transport VQ and its use will be in the respective patchset.

> > 2. Having the infrastructure to support multiple AQ is also good.
> > The extra cost of this infra is only one read-only field = aq count as Michael
> proposed.
> >
> > 3. Ability to send multiple outstanding commands and execute them out of
> order by device is/should be supported as long as IN_ORDER is not negotiated.
> > Should it be relaxed for AQ that even if IN_ORDER is negotiated, AQ by
> default can be out of order queue?
> 
> I feel there's no need: driver can just avoid negotiating IN_ORDER.
I didn't understand the "no need" part.
Do you mean to make use of out-of-order AQ commands, driver should skip IN_ORDER negotiation for the data path queues?
If yes, it may not be a big loss for the PF.

I am considering that since AQ is so fresh, it can keep itself detached from the IN_ORDER from the beginning.
This way AQ and data path queues stay on their own course.

> Generally I'm not happy with IN_ORDER and have some ideas to replace it with
> PARTIAL_ORDER, that will also address this.
> 
> > 4. Driver is the sole owner to decide if it is modifying an object through AQ, it
> should synchronize with ongoing AQ command on the same object or not.
> > 5. If the command gets stuck for a very long time, the driver can recover the
> AQ using the existing Q RESET facility.
> >
> > Do we agree with the above design thoughts?
> > Or there is a better design than this discussed, and I missed it?
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Thu, Mar 09, 2023 at 02:30:52PM +0000, Parav Pandit wrote:
> 
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, March 9, 2023 9:23 AM
> > 
> > On Thu, Mar 09, 2023 at 02:11:54PM +0000, Parav Pandit wrote:
> > >
> > >
> > > > From: David Edmondson <david.edmondson@oracle.com>
> > > > Sent: Thursday, March 9, 2023 9:03 AM
> > > >
> > > > Parav Pandit <parav@nvidia.com> writes:
> > > >
> > > > >> From: Jiri Pirko <jiri@nvidia.com>
> > > > >> Sent: Thursday, March 9, 2023 2:31 AM
> > > > >>
> > > > >> Wed, Mar 08, 2023 at 10:25:32PM CET, parav@nvidia.com wrote:
> > > > >> >
> > > > >> >> From: virtio-comment@lists.oasis-open.org
> > > > >> >> <virtio-comment@lists.oasis- open.org> On Behalf Of David
> > > > >> >> Edmondson
> > > > >> >
> > > > >> >> In support of live migration, might we end up moving large
> > > > >> >> amounts of device state through the admin queue?
> > > > >> >>
> > > > >> >Correct.
> > > > >> >
> > > > >> >> If so, that would seem to have some performance requirements,
> > > > >> >> though I don't know if it would justify multiple admin queues.
> > > > >> >DMA of the data through the proposed AQ is supported.
> > > > >> >
> > > > >> >If I understood Max correctly when Max said " This AQ is not
> > > > >> >aimed for performance ", he means that AQ doesn't have
> > > > >> >performance requirements as
> > > > >> io/network queues to complete millions of ops/sec.
> > > > >> >
> > > > >> >it is several hundred to maybe (on the higher side) thousand
> > > > >> >ops/sec during
> > > > >> LM, provisioning use case.
> > > > >>
> > > > >> But isn't it good to design it for performance from start? I
> > > > >> mean, state transfer of thousands of VFs at a time is definitelly
> > > > >> performance
> > > > related, isn't it?
> > > > >>
> > > > > It is. Which part of the proposed AQ doesn't cover this aspect?
> > > > > The only issue that I see today is, that a given GET family of
> > > > > commands q
> > > > contains the read-only and write-only descriptors which require
> > > > multiple dma allocations on driver side.
> > > >
> > > > I don't think that there is any assertion that the current proposal
> > > > doesn't cover this.
> > > >
> > > > It was intended as an illustration to try and help determine whether
> > > > multiple AQs for a single device was a requirement given existing
> > > > anticipated usage, or a safeguard against potential future uses.
> > >
> > > Got it.
> > >
> > > So far my understanding is:
> > > 1. Single AQ is enough for currently cited use cases
> > 
> > FYI transport VQ work started off with multiple vqs.
> > 
> Ok. so there may be some use case for transport VQ.
> Worth to mention in the cover letter and things will be fine.
> Actual review of the transport VQ and its use will be in the respective patchset.
> 
> > > 2. Having the infrastructure to support multiple AQ is also good.
> > > The extra cost of this infra is only one read-only field = aq count as Michael
> > proposed.
> > >
> > > 3. Ability to send multiple outstanding commands and execute them out of
> > order by device is/should be supported as long as IN_ORDER is not negotiated.
> > > Should it be relaxed for AQ that even if IN_ORDER is negotiated, AQ by
> > default can be out of order queue?
> > 
> > I feel there's no need: driver can just avoid negotiating IN_ORDER.
> I didn't understand the "no need" part.
> Do you mean to make use of out-of-order AQ commands, driver should skip IN_ORDER negotiation for the data path queues?
> If yes, it may not be a big loss for the PF.
> 
> I am considering that since AQ is so fresh, it can keep itself detached from the IN_ORDER from the beginning.
> This way AQ and data path queues stay on their own course.

You can make this claim for different types of data queues too.
I'd rather try to solve it for all queues than special-case AQ.

> > Generally I'm not happy with IN_ORDER and have some ideas to replace it with
> > PARTIAL_ORDER, that will also address this.
> > 
> > > 4. Driver is the sole owner to decide if it is modifying an object through AQ, it
> > should synchronize with ongoing AQ command on the same object or not.
> > > 5. If the command gets stuck for a very long time, the driver can recover the
> > AQ using the existing Q RESET facility.
> > >
> > > Do we agree with the above design thoughts?
> > > Or there is a better design than this discussed, and I missed it?
> > >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2381 bytes --]

On Thu, Mar 09, 2023 at 08:55:15AM -0500, Michael S. Tsirkin wrote:
> On Thu, Mar 09, 2023 at 07:35:42AM -0500, Stefan Hajnoczi wrote:
> > On Wed, Mar 08, 2023 at 12:21:46PM -0500, Michael S. Tsirkin wrote:
> > > On Wed, Mar 08, 2023 at 12:15:23PM -0500, Stefan Hajnoczi wrote:
> > > > > > > > Or we could say that admin commands must complete within bounded time,
> > > > > > > > but I'm not sure that is implementable for some device types like
> > > > > > > > virtio-blk, virtio-scsi, and virtiofs.
> > > > > > > 
> > > > > > > No we can't.
> > > > > > > Some commands, for example FW upgrade can take 10 minutes and it's perfectly
> > > > > > > fine. Other commands like setting feature bit will take 1 millisec.
> > > > > > > Each device implements commands in a different internal logic so we can't
> > > > > > > expect to complete after X time.
> > > > > > 
> > > > > > When I say bounded time, I mean that it finishes in a finite amount of
> > > > > > time. I'm not saying there is a specific time X that all device
> > > > > > implementations must satisfy. Unbounded means it might never finish.
> > > > > 
> > > > > There might be a chance that any command for any virtio device type will
> > > > > never finish. Nothing new here in the adminq.
> > > > > 
> > > > > what one can do is to set a timeout for himself and if this timeout expire -
> > > > > check the device status. If it needs_reset - do a reset. if status is ok,
> > > > > then wait some more time.
> > > > > After X retries, unmap buffers or reset the adminq.
> > > > 
> > > > Michael: What effect does resetting the group owner device have on group
> > > > member devices?
> > > 
> > > virtio level reset? It's a good question. I'd expect them all to be
> > > reset no?
> > > 
> > > > I'm concerned that this approach disrupts all group member devices. For
> > > > example, you try to add a new device but the command hangs. In order to
> > > > recover you now have to reset the group owner device and this breaks all
> > > > the group member devices.
> > > 
> > > 
> > > I agree. How about a VQ level reset though? Seems like exactly
> > > what's needed here?
> > 
> > Yes, a new virtqueue-level reset feature would take care of this case.
> > 
> > Stefan
> 
> Why would we need a new one? We already have VIRTIO_F_RING_RESET.

I was unaware :).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Jiri Pirko]

Thu, Mar 09, 2023 at 01:36:00PM CET, parav@nvidia.com wrote:
>
>
>> From: Jiri Pirko <jiri@nvidia.com>
>> Sent: Thursday, March 9, 2023 2:37 AM
>
>
>> >We want the ability to program/provision the virtio feature bits and virtio
>> config space parameters of the VF through PF.
>> >These are host-facing parameters that usually migrate from one to another
>> host when a VF migrates.
>> 
>> Here is where we lack to have separate "devlink function" entity, independend
>> of "devlink port" :/ I mean, for MAC config and irq vectors.
>> What other host facing parmeters you have in mind. VLAN of "ip vf" is
>> definitelly one of them, I have to emphasize.
>> 
>It is not.
>
>Virtio feature bits defined by the virtio spec.
>Virtio config space fields present in the each virtio device specific spec.

If those are only virtio-specifig things, I'm good.

Mac is part of Virtio_net config space. Good.



>
>> 
>> >
>
>> >virtio net doesn't implement it either and I don't see any user asking for it
>> either.
>> >All users have migrated to using tc mechanism for long time now.
>> 
>> Okay, so the bottom line is, we should take care of host facing config of VF, but
>> we should not care about network plug of VF - that stays out of the scope.
>Right.

Good.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 9, 2023 9:43 AM

> > >
> > Ok. so there may be some use case for transport VQ.
> > Worth to mention in the cover letter and things will be fine.
> > Actual review of the transport VQ and its use will be in the respective
> patchset.
> >
> > > > 2. Having the infrastructure to support multiple AQ is also good.
> > > > The extra cost of this infra is only one read-only field = aq
> > > > count as Michael
> > > proposed.
> > > >
> > > > 3. Ability to send multiple outstanding commands and execute them
> > > > out of
> > > order by device is/should be supported as long as IN_ORDER is not
> negotiated.
> > > > Should it be relaxed for AQ that even if IN_ORDER is negotiated,
> > > > AQ by
> > > default can be out of order queue?
> > >
> > > I feel there's no need: driver can just avoid negotiating IN_ORDER.
> > I didn't understand the "no need" part.
> > Do you mean to make use of out-of-order AQ commands, driver should skip
> IN_ORDER negotiation for the data path queues?
> > If yes, it may not be a big loss for the PF.
> >
> > I am considering that since AQ is so fresh, it can keep itself detached from
> the IN_ORDER from the beginning.
> > This way AQ and data path queues stay on their own course.
> 
> You can make this claim for different types of data queues too.
Those queues are already defined. Without any new feature bit, not sure how can it change.

> I'd rather try to solve it for all queues than special-case AQ.
> 
AQ is newly introduce which has chance of not follow the same limitations as existing queues.

Keeping my inputs aside for moment, what do you propose for v11?

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Thu, Mar 09, 2023 at 04:53:59PM +0000, Parav Pandit wrote:
> Keeping my inputs aside for moment, what do you propose for v11?

Following SCSI and relying on driver to worry about ordering.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, March 9, 2023 12:14 PM
> 
> On Thu, Mar 09, 2023 at 04:53:59PM +0000, Parav Pandit wrote:
> > Keeping my inputs aside for moment, what do you propose for v11?
> 
> Following SCSI and relying on driver to worry about ordering.
Got it. Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 08/10] admin: command list discovery

[Zhu, Lingshan]

On 3/2/2023 9:05 PM, Michael S. Tsirkin wrote:
> Add commands to find out which commands does each group support,
> as well as enable their use by driver.
> This will be especially useful once we have multiple group types.
>
> An alternative is per-type VQs. This is possible but will
> require more per-transport work. Discovery through the vq
> helps keep things contained.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 96 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 3ffac2e..1172054 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   \hline
>   opcode & Name & Command Description \\
>   \hline \hline
> -0x0000 - 0x7FFF & - &  Group administration commands    \\
> +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> +0x0002 - 0x7FFF & - &  Group administration commands    \\
>   \hline
>   0x8000 - 0xFFFF & - & Reserved    \\
>   \hline
> @@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   depends on these structures and is described separately or is
>   implicit in the structure description.
>   
> +Before sending any administration commands to the device, the driver
> +needs to communicate to the device which commands it is going to
> +use. Initially (after reset), only two commands are assumed to be used:
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Before sending any other commands for any member of a specific group to
> +the device, the driver queries the supported commands via
> +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
> +VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE
> +both use the following structure describing the
> +command opcodes:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_list {
> +       /* Indicates which of the below fields were returned
> +       le64 device_admin_cmds[];
> +};
> +\end{lstlisting}
> +
> +This structure is an array of 64 bit values in little-endian byte
> +order, in which a bit is set if the specific command opcode
> +is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value
I think this should be "the first 64-bit"
> +in this array corresponding to opcodes 0 to 63,
> +\field{device_admin_cmds[1]} is the second 64-bit value
> +corresponding to opcodes 64 to 127, etc.
> +For example, the array of size 2 including
> +the values 0x3 in \field{device_admin_cmds[0]}
> +and 0x1 in \field{device_admin_cmds[1]} indicates that only
> +opcodes 0, 1 and 64 are supported.
> +The length of the array depends on the supported opcodes - it is
> +large enough to include bits set for all supported opcodes,
> +that is the length can be calculated by starting with the largest
> +supported opcode adding one, dividing by 64 and rounding up.
> +In other words, for
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the
> +length of \field{command_specific_result} and
> +\field{command_specific_data} respectively will be
> +$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division
> +with round up and \field{max_cmd} is the largest available command opcode.
> +
> +The array is also allowed to be larger and to additionally
> +include an arbitrary number of all-zero entries.
> +
> +Accordingly, bits 0 and 1 corresponding to opcode 0
> +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
> +(VIRTIO_ADMIN_CMD_LIST_USE) are
> +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +This command has no command specific data.
> +The device, upon success, returns a result in
> +\field{command_specific_result} in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands supported for the given group.
> +
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
> +is set to 0x1.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +The \field{command_specific_data} is in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands used by the driver.
> +This command has no command specific result.
> +
> +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> +query the list of commands valid for this group and before sending
> +any commands for any member of a group.
> +
> +The driver then enables use of some of the opcodes by sending to
> +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
> +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
> +both understood and used by the driver.
> +
> +If the device supports the command list used by the driver, the
> +device completes the command with status VIRTIO_ADMIN_STATUS_OK.
> +If the device does not support the command list, the device
> +completes the command with status
> +VIRTIO_ADMIN_STATUS_INVALID_FIELD.
> +
> +Note: drivers are assumed not to set bits in device_admin_cmds
> +if they are not familiar with how the command opcode
> +is used, since devices could have dependencies between
> +command opcodes.
> +
> +It is assumed that all members in a group support and are used
> +with the same list of commands. However, for owner devices
> +supporting multiple group types, the list of supported commands
> +might differ between different group types.
> +
>   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>   
>   An administration virtqueue of an owner device is used to submit


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v10 00/10] Introduce device group and device management

[Zhu, Lingshan]

On 3/8/2023 3:31 AM, Parav Pandit wrote:
>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Tuesday, March 7, 2023 12:20 PM
> [..]
>>>> For SR-IOV, it is not unusual for PFs to excercise control over VFs.
>>> I understand the concepts of SR-IOV. Yet I fail to see the need of
>>> such concept in virtio world. SR-IOV is very specific solution for PCI
>>> functions instantiation, and I believe that it is already considered
>>> quite limiting in many aspects. Does not make sense to me to introduce
>>> it for virtio. But again, I may be missing something crucial, I just
>>> would like to see the motivation, needs, usecases for this crystal
>>> clear, which is opposite to the current cover letter I'm afraid :/
>> First people are asking for it because it's out there, however limiting it is. In
>> fact Nvidia is - why don't you talk to Parav here and tell him that SR-IOV is
>> legacy and there's no need to support.
>>
> Jiri and I discussed offline today.
> SR-IOV is widely used today and in the coming future up to thousands of devices.
> Many vendors are building it or already built.
>
> SIOV as it stands today is still being crafted at various levels iommu, os, device, pci and more.
> Not there yet fully.
>
> Jiri and I built the Linux kernel interfaces for SFs which are equivalent to SIOVs that overcome some of the SR-IOV limitations.
> And Jiri's suggestion was to leverage some of the vision and practice from there.
>
> In other words, can AQ command be useful tomorrow for doing SIOV device add/remove, and provisioning from non-owning PF?
> The way AQ is crafted today is not there yet, but in the future, it can be extended.
I have submitted a series implementing "transport vq" which work for 
ADIs like SIOV devices, MST required a merge with admin q, so we are here.
>>>> There is interest in the community to include an interface to allow
>>>> this in the virtio spec, when the PF is a virtio device.  This is
>>>> what this patch does.
>>> Yeah, but why? As I asked before, what are the usecases? The fact
>>> there is interest in the community does not mean it makes sense to
>>> have it :)
>>>
>> If people want to build such hardware it will need some interface.
>> Far better to have it standard.
>>
>>
>> But generally e.g. intel already said they will reuse this same structure with a
>> different group type for SIOV support.
>> I'll mention this in the cover letter.
>>
> We have the following already identified use cases of a device agnostic VQ.
>
> 1. SR-IOV VF device provisioning at virtio layer (features and config)
> 2. SR IOV (SR-PCIM interface of the PCI spec) for VF provisioning, for example, MSI-X vectors
>
> 3. SR-IOV VF Live migration
> 4. Above #1 to #3 for SIOV devices instead of SRIOV devices
>
> 5. cmd q interface for PF/VF/SF/SIOV to transport their own features, config command over queue interface without any parent device
>
> #1 to #3 are well described in past. So, It is worth covering them in the cover letter and things will be clear for new eyes.
> #4 is broad use case and like #1 to #3 that will eventually happen.
> #5 likely needs more discussion, we can differ to a later point because AQ needs to progress in spec 1.3.
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 09/10] admin: conformance clauses

[Zhu, Lingshan]

On 3/2/2023 9:05 PM, Michael S. Tsirkin wrote:
> Add conformance clauses for admin commands and admin virtqueues.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 215 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 1172054..6c4f79c 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   supporting multiple group types, the list of supported commands
>   might differ between different group types.
>   
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The device MUST validate \field{opcode}, \field{group_type} and
> +\field{group_member_id}, and if any of these has an invalid or
> +unsupported value, set \field{status} to
> +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
> +accordingly:
> +\begin{itemize}
> +\item if \field{group_type} is invalid, \field{status_qualifier}
> +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> +\item otherwise, if \field{opcode} is invalid,
> +	\field{status_qualifier} MUST be set to
> +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
> +\item otherwise, if \field{group_member_id} is used by the
> +	specific command and is invalid, \field{status_qualifier} MUST be
> +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER.
> +\end{itemize}
> +
> +If a command completes successfully, the device MUST set
> +\field{status} to VIRTIO_ADMIN_STATUS_OK.
> +
> +If a command fails, the device MUST set
> +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
> +
> +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
> +device state MUST NOT change, that is the command MUST NOT have
> +any side effects on the device, in particular the device MUST not
> +enter an error state as a result of this command.
> +
> +If a command fails, the device state generally SHOULD NOT change,
> +as far as possible.
> +
> +The device MAY enforce additional restrictions and dependencies on
> +opcodes used by the driver and MAY fail the command
> +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> +if the list of commands used violate internal device dependencies.
> +
> +If the device supports multiple group types, commands for each group
> +type MUST operate independently of each other, in particular,
> +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
> +for different group types.
> +
> +After reset, if the device supports a given group type
> +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
> +the device MUST assume
> +that the list of legal commands used by the driver consists of
> +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
> +the device MUST set the list of legal commands used by the driver
> +to the one supplied in \field{command_specific_data}.
> +
> +The device MUST set the list of legal commands used by the driver
> +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +The device MUST validate commands against the list used by
> +the driver and MUST fail any commands not in the list with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
> +
> +The list of supported commands MUST NOT shrink (but MAY expand):
> +after reporting a given command as supported through
> +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> +as unsupported.  Further, after a given set of commands has been
Can the driver re-negotiate the command list through QUERY/USE
in flight rather than upon a reset? If not, shall forbid this explicitly?
> +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
> +device or system reset the device SHOULD complete successfully
> +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
> +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
I think this requires the device to remember what it had offered even
after a power-cycled reset, shall we say: The device should always offer 
all supported
commands in the list?

Thanks,
Zhu Lingshan
> +after a device or system reset, the device MUST not fail it
> +solely because of the command list used.  Failure to do so would
> +interfere with resuming from suspend and error recovery.
> +
> +When processing a command with the SR-IOV group type,
> +if the device does not have an SR-IOV Extended Capability or
> +if \field{VF Enable} is clear
> +then the device MUST fail all commands with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> +otherwise, if \field{group_member_id} is not
> +between $1$ and \field{NumVFs} inclusive,
> +the device MUST fail all commands with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER;
> +\field{NumVFs}, \field{VF Migration Capable}  and
> +\field{VF Enable} refer to registers within the SR-IOV Extended
> +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The driver MAY discover whether device supports a specific group type
> +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
> +\field{group_type}.
> +
> +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
> +and wait for it to be completed with status
> +VIRTIO_ADMIN_STATUS_OK before issuing any commands
> +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
> +and VIRTIO_ADMIN_CMD_LIST_USE).
> +
> +The driver SHOULD NOT set bits in device_admin_cmds
> +if it is not familiar with how the command opcode
> +is used, as dependencies between command opcodes might exist.
> +
> +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
> +the use of any commands not previously reported as
> +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +The driver MUST NOT use any commands for a given group type
> +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
> +list of command opcodes and group type.
> +
> +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> +with respective bits cleared in \field{command_specific_data}.
> +
> +The driver MUST handle a command error with a reserved \field{status}
> +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +(except possibly for different error reporting/diagnostic messages).
> +
> +The driver MUST handle a command error with a reserved
> +\field{status_qualifier} value in the same way as
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
> +different error reporting/diagnostic messages).
> +
> +When sending commands with the SR-IOV group type,
> +the driver specify a value for \field{group_member_id}
> +between $1$ and \field{NumVFs} inclusive,
> +the driver MUST also make sure that as long as any such command
> +is outstanding, \field{VF Migration Capable} is clear and
> +\field{VF Enable} is set;
> +\field{NumVFs}, \field{VF Migration Capable}  and
> +\field{VF Enable} refer to registers within the SR-IOV Extended
> +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> +
>   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>   
>   An administration virtqueue of an owner device is used to submit
> @@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
>   of the specification are designed, new fields can be added to the
>   tail of a structure, with the driver/device using the full
>   structure without concern for versioning.
> ->>>>>>> 0edc690... admin: introduce virtio admin virtqueues
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The device MUST support device-readable and device-writeable buffers
> +shorter than described in this specification, by
> +\begin{enumerate}
> +\item acting as if any data that would be read outside the
> +device-readable buffers is set to zero, and
> +\item discarding data that would be written outside the
> +specified device-writeable buffers.
> +\end{enumerate}
> +
> +The device MUST support device-readable and device-writeable buffers
> +longer than described in this specification, by
> +\begin{enumerate}
> +\item ignoring any data in device-readable buffers outside
> +the expected length, and
> +\item only writing the expected structure to the device-writeable
> +buffers, ignoring any extra buffers, and reporting the
> +actual length of data written, in bytes,
> +as buffer used length.
> +\end{enumerate}
> +
> +The device SHOULD initialize the device-writeable buffer
> +up to the length of the structure described by this specification or
> +the length of the buffer supplied by the driver (even if the buffer is
> +all set to zero), whichever is shorter.
> +
> +The device MUST NOT fail a command solely because the buffers
> +provided are shorter or longer than described in this
> +specification.
> +
> +The device MUST initialize the device-writeable part of
> +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
> +size.
> +
> +The device MUST initialize \field{status} in \field{struct
> +virtio_admin_cmd}.
> +
> +The device MUST process commands on a given administration virtqueue
> +in the order in which they are queued.
> +
> +If multiple administration virtqueues have been configured,
> +device MAY process commands on distinct virtqueues with
> +no order constraints.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The driver MAY supply device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are longer than described in
> +this specification.
> +
> +The driver SHOULD supply device-readable part of
> +\field{struct virtio_admin_cmd} that is at least as
> +large as the structure described by this specification
> +(even if the structure is all set to zero).
> +
> +The driver MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
> +in length.
> +
> +The device MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are larger than zero in
> +length. However, \field{command_specific_data} and
> +\field{command_specific_result} MAY be zero in length, unless
> +specified otherwise for the command.
> +
> +The driver MUST NOT assume that the device will initialize the whole
> +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
> +the driver MUST act as if the structure
> +outside the part of the buffer used by the device
> +is set to zero.
> +
> +If multiple administration virtqueues have been configured,
> +the driver MUST ensure ordering for commands
> +placed on different administration virtqueues.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

On Fri, Mar 10, 2023 at 05:10:48PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 3/2/2023 9:05 PM, Michael S. Tsirkin wrote:
> > Add conformance clauses for admin commands and admin virtqueues.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >   admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
> >   1 file changed, 215 insertions(+), 1 deletion(-)
> > 
> > diff --git a/admin.tex b/admin.tex
> > index 1172054..6c4f79c 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >   supporting multiple group types, the list of supported commands
> >   might differ between different group types.
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > +
> > +The device MUST validate \field{opcode}, \field{group_type} and
> > +\field{group_member_id}, and if any of these has an invalid or
> > +unsupported value, set \field{status} to
> > +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
> > +accordingly:
> > +\begin{itemize}
> > +\item if \field{group_type} is invalid, \field{status_qualifier}
> > +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> > +\item otherwise, if \field{opcode} is invalid,
> > +	\field{status_qualifier} MUST be set to
> > +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
> > +\item otherwise, if \field{group_member_id} is used by the
> > +	specific command and is invalid, \field{status_qualifier} MUST be
> > +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER.
> > +\end{itemize}
> > +
> > +If a command completes successfully, the device MUST set
> > +\field{status} to VIRTIO_ADMIN_STATUS_OK.
> > +
> > +If a command fails, the device MUST set
> > +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
> > +
> > +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
> > +device state MUST NOT change, that is the command MUST NOT have
> > +any side effects on the device, in particular the device MUST not
> > +enter an error state as a result of this command.
> > +
> > +If a command fails, the device state generally SHOULD NOT change,
> > +as far as possible.
> > +
> > +The device MAY enforce additional restrictions and dependencies on
> > +opcodes used by the driver and MAY fail the command
> > +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> > +if the list of commands used violate internal device dependencies.
> > +
> > +If the device supports multiple group types, commands for each group
> > +type MUST operate independently of each other, in particular,
> > +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
> > +for different group types.
> > +
> > +After reset, if the device supports a given group type
> > +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
> > +the device MUST assume
> > +that the list of legal commands used by the driver consists of
> > +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> > +
> > +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
> > +the device MUST set the list of legal commands used by the driver
> > +to the one supplied in \field{command_specific_data}.
> > +
> > +The device MUST set the list of legal commands used by the driver
> > +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
> > +
> > +The device MUST validate commands against the list used by
> > +the driver and MUST fail any commands not in the list with
> > +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +and \field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
> > +
> > +The list of supported commands MUST NOT shrink (but MAY expand):
> > +after reporting a given command as supported through
> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> > +as unsupported.  Further, after a given set of commands has been
> Can the driver re-negotiate the command list through QUERY/USE
> in flight rather than upon a reset? If not, shall forbid this explicitly?
> > +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
> > +device or system reset the device SHOULD complete successfully
> > +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
> > +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
> I think this requires the device to remember what it had offered even
> after a power-cycled reset,

It just says always return the same.

> shall we say: The device should always offer all
> supported
> commands in the list?

I don't see what will it mean.

> Thanks,
> Zhu Lingshan
> > +after a device or system reset, the device MUST not fail it
> > +solely because of the command list used.  Failure to do so would
> > +interfere with resuming from suspend and error recovery.
> > +
> > +When processing a command with the SR-IOV group type,
> > +if the device does not have an SR-IOV Extended Capability or
> > +if \field{VF Enable} is clear
> > +then the device MUST fail all commands with
> > +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> > +\field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> > +otherwise, if \field{group_member_id} is not
> > +between $1$ and \field{NumVFs} inclusive,
> > +the device MUST fail all commands with
> > +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
> > +\field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER;
> > +\field{NumVFs}, \field{VF Migration Capable}  and
> > +\field{VF Enable} refer to registers within the SR-IOV Extended
> > +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > +
> > +The driver MAY discover whether device supports a specific group type
> > +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
> > +\field{group_type}.
> > +
> > +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
> > +and wait for it to be completed with status
> > +VIRTIO_ADMIN_STATUS_OK before issuing any commands
> > +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
> > +and VIRTIO_ADMIN_CMD_LIST_USE).
> > +
> > +The driver SHOULD NOT set bits in device_admin_cmds
> > +if it is not familiar with how the command opcode
> > +is used, as dependencies between command opcodes might exist.
> > +
> > +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
> > +the use of any commands not previously reported as
> > +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
> > +
> > +The driver MUST NOT use any commands for a given group type
> > +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
> > +list of command opcodes and group type.
> > +
> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> > +with respective bits cleared in \field{command_specific_data}.
> > +
> > +The driver MUST handle a command error with a reserved \field{status}
> > +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +(except possibly for different error reporting/diagnostic messages).
> > +
> > +The driver MUST handle a command error with a reserved
> > +\field{status_qualifier} value in the same way as
> > +\field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
> > +different error reporting/diagnostic messages).
> > +
> > +When sending commands with the SR-IOV group type,
> > +the driver specify a value for \field{group_member_id}
> > +between $1$ and \field{NumVFs} inclusive,
> > +the driver MUST also make sure that as long as any such command
> > +is outstanding, \field{VF Migration Capable} is clear and
> > +\field{VF Enable} is set;
> > +\field{NumVFs}, \field{VF Migration Capable}  and
> > +\field{VF Enable} refer to registers within the SR-IOV Extended
> > +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > +
> >   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >   An administration virtqueue of an owner device is used to submit
> > @@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
> >   of the specification are designed, new fields can be added to the
> >   tail of a structure, with the driver/device using the full
> >   structure without concern for versioning.
> > ->>>>>>> 0edc690... admin: introduce virtio admin virtqueues
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> > +
> > +The device MUST support device-readable and device-writeable buffers
> > +shorter than described in this specification, by
> > +\begin{enumerate}
> > +\item acting as if any data that would be read outside the
> > +device-readable buffers is set to zero, and
> > +\item discarding data that would be written outside the
> > +specified device-writeable buffers.
> > +\end{enumerate}
> > +
> > +The device MUST support device-readable and device-writeable buffers
> > +longer than described in this specification, by
> > +\begin{enumerate}
> > +\item ignoring any data in device-readable buffers outside
> > +the expected length, and
> > +\item only writing the expected structure to the device-writeable
> > +buffers, ignoring any extra buffers, and reporting the
> > +actual length of data written, in bytes,
> > +as buffer used length.
> > +\end{enumerate}
> > +
> > +The device SHOULD initialize the device-writeable buffer
> > +up to the length of the structure described by this specification or
> > +the length of the buffer supplied by the driver (even if the buffer is
> > +all set to zero), whichever is shorter.
> > +
> > +The device MUST NOT fail a command solely because the buffers
> > +provided are shorter or longer than described in this
> > +specification.
> > +
> > +The device MUST initialize the device-writeable part of
> > +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
> > +size.
> > +
> > +The device MUST initialize \field{status} in \field{struct
> > +virtio_admin_cmd}.
> > +
> > +The device MUST process commands on a given administration virtqueue
> > +in the order in which they are queued.
> > +
> > +If multiple administration virtqueues have been configured,
> > +device MAY process commands on distinct virtqueues with
> > +no order constraints.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> > +
> > +The driver MAY supply device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are longer than described in
> > +this specification.
> > +
> > +The driver SHOULD supply device-readable part of
> > +\field{struct virtio_admin_cmd} that is at least as
> > +large as the structure described by this specification
> > +(even if the structure is all set to zero).
> > +
> > +The driver MUST supply both device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
> > +in length.
> > +
> > +The device MUST supply both device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are larger than zero in
> > +length. However, \field{command_specific_data} and
> > +\field{command_specific_result} MAY be zero in length, unless
> > +specified otherwise for the command.
> > +
> > +The driver MUST NOT assume that the device will initialize the whole
> > +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
> > +the driver MUST act as if the structure
> > +outside the part of the buffer used by the device
> > +is set to zero.
> > +
> > +If multiple administration virtqueues have been configured,
> > +the driver MUST ensure ordering for commands
> > +placed on different administration virtqueues.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

On Fri, Mar 10, 2023 at 05:10:48PM +0800, Zhu, Lingshan wrote:
> > +The list of supported commands MUST NOT shrink (but MAY expand):
> > +after reporting a given command as supported through
> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> > +as unsupported.  Further, after a given set of commands has been
> Can the driver re-negotiate the command list through QUERY/USE
> in flight rather than upon a reset? If not, shall forbid this explicitly?

Probably best to allow this.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 09/10] admin: conformance clauses

[Zhu Lingshan]

On 3/10/23 17:13, Michael S. Tsirkin wrote:
> On Fri, Mar 10, 2023 at 05:10:48PM +0800, Zhu, Lingshan wrote:
>>
>> On 3/2/2023 9:05 PM, Michael S. Tsirkin wrote:
>>> Add conformance clauses for admin commands and admin virtqueues.
>>>
>>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>> ---
>>>    admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>>    1 file changed, 215 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/admin.tex b/admin.tex
>>> index 1172054..6c4f79c 100644
>>> --- a/admin.tex
>>> +++ b/admin.tex
>>> @@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>    supporting multiple group types, the list of supported commands
>>>    might differ between different group types.
>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>>> +
>>> +The device MUST validate \field{opcode}, \field{group_type} and
>>> +\field{group_member_id}, and if any of these has an invalid or
>>> +unsupported value, set \field{status} to
>>> +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
>>> +accordingly:
>>> +\begin{itemize}
>>> +\item if \field{group_type} is invalid, \field{status_qualifier}
>>> +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
>>> +\item otherwise, if \field{opcode} is invalid,
>>> +	\field{status_qualifier} MUST be set to
>>> +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
>>> +\item otherwise, if \field{group_member_id} is used by the
>>> +	specific command and is invalid, \field{status_qualifier} MUST be
>>> +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER.
>>> +\end{itemize}
>>> +
>>> +If a command completes successfully, the device MUST set
>>> +\field{status} to VIRTIO_ADMIN_STATUS_OK.
>>> +
>>> +If a command fails, the device MUST set
>>> +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
>>> +
>>> +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
>>> +device state MUST NOT change, that is the command MUST NOT have
>>> +any side effects on the device, in particular the device MUST not
>>> +enter an error state as a result of this command.
>>> +
>>> +If a command fails, the device state generally SHOULD NOT change,
>>> +as far as possible.
>>> +
>>> +The device MAY enforce additional restrictions and dependencies on
>>> +opcodes used by the driver and MAY fail the command
>>> +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
>>> +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
>>> +if the list of commands used violate internal device dependencies.
>>> +
>>> +If the device supports multiple group types, commands for each group
>>> +type MUST operate independently of each other, in particular,
>>> +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
>>> +for different group types.
>>> +
>>> +After reset, if the device supports a given group type
>>> +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
>>> +the device MUST assume
>>> +that the list of legal commands used by the driver consists of
>>> +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
>>> +
>>> +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
>>> +the device MUST set the list of legal commands used by the driver
>>> +to the one supplied in \field{command_specific_data}.
>>> +
>>> +The device MUST set the list of legal commands used by the driver
>>> +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
>>> +
>>> +The device MUST validate commands against the list used by
>>> +the driver and MUST fail any commands not in the list with
>>> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
>>> +and \field{status_qualifier} set to
>>> +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
>>> +
>>> +The list of supported commands MUST NOT shrink (but MAY expand):
>>> +after reporting a given command as supported through
>>> +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
>>> +as unsupported.  Further, after a given set of commands has been
>> Can the driver re-negotiate the command list through QUERY/USE
>> in flight rather than upon a reset? If not, shall forbid this explicitly?
>>> +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
>>> +device or system reset the device SHOULD complete successfully
>>> +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
>>> +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
>> I think this requires the device to remember what it had offered even
>> after a power-cycled reset,
> It just says always return the same.
>
>> shall we say: The device should always offer all
>> supported
>> commands in the list?
> I don't see what will it mean.
For example, a device can support more commands(even can be vendor 
specific commands) through a firmware upgrade,
then after a reset, it should report more supported commands in the list 
through QEURY,
so maybe not a good idea to report the same command set. The device 
should report all supported commands
after a reset.
>
>> Thanks,
>> Zhu Lingshan
>>> +after a device or system reset, the device MUST not fail it
>>> +solely because of the command list used.  Failure to do so would
>>> +interfere with resuming from suspend and error recovery.
>>> +
>>> +When processing a command with the SR-IOV group type,
>>> +if the device does not have an SR-IOV Extended Capability or
>>> +if \field{VF Enable} is clear
>>> +then the device MUST fail all commands with
>>> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
>>> +\field{status_qualifier} set to
>>> +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
>>> +otherwise, if \field{group_member_id} is not
>>> +between $1$ and \field{NumVFs} inclusive,
>>> +the device MUST fail all commands with
>>> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
>>> +\field{status_qualifier} set to
>>> +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER;
>>> +\field{NumVFs}, \field{VF Migration Capable}  and
>>> +\field{VF Enable} refer to registers within the SR-IOV Extended
>>> +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>> +
>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>>> +
>>> +The driver MAY discover whether device supports a specific group type
>>> +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
>>> +\field{group_type}.
>>> +
>>> +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
>>> +and wait for it to be completed with status
>>> +VIRTIO_ADMIN_STATUS_OK before issuing any commands
>>> +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
>>> +and VIRTIO_ADMIN_CMD_LIST_USE).
>>> +
>>> +The driver SHOULD NOT set bits in device_admin_cmds
>>> +if it is not familiar with how the command opcode
>>> +is used, as dependencies between command opcodes might exist.
>>> +
>>> +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
>>> +the use of any commands not previously reported as
>>> +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
>>> +
>>> +The driver MUST NOT use any commands for a given group type
>>> +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
>>> +list of command opcodes and group type.
>>> +
>>> +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
>>> +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
>>> +with respective bits cleared in \field{command_specific_data}.
>>> +
>>> +The driver MUST handle a command error with a reserved \field{status}
>>> +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
>>> +(except possibly for different error reporting/diagnostic messages).
>>> +
>>> +The driver MUST handle a command error with a reserved
>>> +\field{status_qualifier} value in the same way as
>>> +\field{status_qualifier} set to
>>> +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
>>> +different error reporting/diagnostic messages).
>>> +
>>> +When sending commands with the SR-IOV group type,
>>> +the driver specify a value for \field{group_member_id}
>>> +between $1$ and \field{NumVFs} inclusive,
>>> +the driver MUST also make sure that as long as any such command
>>> +is outstanding, \field{VF Migration Capable} is clear and
>>> +\field{VF Enable} is set;
>>> +\field{NumVFs}, \field{VF Migration Capable}  and
>>> +\field{VF Enable} refer to registers within the SR-IOV Extended
>>> +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>> +
>>>    \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>    An administration virtqueue of an owner device is used to submit
>>> @@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
>>>    of the specification are designed, new fields can be added to the
>>>    tail of a structure, with the driver/device using the full
>>>    structure without concern for versioning.
>>> ->>>>>>> 0edc690... admin: introduce virtio admin virtqueues
>>> +
>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
>>> +
>>> +The device MUST support device-readable and device-writeable buffers
>>> +shorter than described in this specification, by
>>> +\begin{enumerate}
>>> +\item acting as if any data that would be read outside the
>>> +device-readable buffers is set to zero, and
>>> +\item discarding data that would be written outside the
>>> +specified device-writeable buffers.
>>> +\end{enumerate}
>>> +
>>> +The device MUST support device-readable and device-writeable buffers
>>> +longer than described in this specification, by
>>> +\begin{enumerate}
>>> +\item ignoring any data in device-readable buffers outside
>>> +the expected length, and
>>> +\item only writing the expected structure to the device-writeable
>>> +buffers, ignoring any extra buffers, and reporting the
>>> +actual length of data written, in bytes,
>>> +as buffer used length.
>>> +\end{enumerate}
>>> +
>>> +The device SHOULD initialize the device-writeable buffer
>>> +up to the length of the structure described by this specification or
>>> +the length of the buffer supplied by the driver (even if the buffer is
>>> +all set to zero), whichever is shorter.
>>> +
>>> +The device MUST NOT fail a command solely because the buffers
>>> +provided are shorter or longer than described in this
>>> +specification.
>>> +
>>> +The device MUST initialize the device-writeable part of
>>> +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
>>> +size.
>>> +
>>> +The device MUST initialize \field{status} in \field{struct
>>> +virtio_admin_cmd}.
>>> +
>>> +The device MUST process commands on a given administration virtqueue
>>> +in the order in which they are queued.
>>> +
>>> +If multiple administration virtqueues have been configured,
>>> +device MAY process commands on distinct virtqueues with
>>> +no order constraints.
>>> +
>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
>>> +
>>> +The driver MAY supply device-readable or device-writeable parts
>>> +of \field{struct virtio_admin_cmd} that are longer than described in
>>> +this specification.
>>> +
>>> +The driver SHOULD supply device-readable part of
>>> +\field{struct virtio_admin_cmd} that is at least as
>>> +large as the structure described by this specification
>>> +(even if the structure is all set to zero).
>>> +
>>> +The driver MUST supply both device-readable or device-writeable parts
>>> +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
>>> +in length.
>>> +
>>> +The device MUST supply both device-readable or device-writeable parts
>>> +of \field{struct virtio_admin_cmd} that are larger than zero in
>>> +length. However, \field{command_specific_data} and
>>> +\field{command_specific_result} MAY be zero in length, unless
>>> +specified otherwise for the command.
>>> +
>>> +The driver MUST NOT assume that the device will initialize the whole
>>> +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
>>> +the driver MUST act as if the structure
>>> +outside the part of the buffer used by the device
>>> +is set to zero.
>>> +
>>> +If multiple administration virtqueues have been configured,
>>> +the driver MUST ensure ordering for commands
>>> +placed on different administration virtqueues.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

On Mon, Mar 06, 2023 at 03:17:59PM -0500, Stefan Hajnoczi wrote:
> On Mon, Mar 06, 2023 at 01:37:31PM -0500, Michael S. Tsirkin wrote:
> > On Mon, Mar 06, 2023 at 06:03:40AM -0500, Stefan Hajnoczi wrote:
> > > On Sun, Mar 05, 2023 at 07:18:24PM -0500, Michael S. Tsirkin wrote:
> > > > On Sun, Mar 05, 2023 at 07:03:02PM -0500, Stefan Hajnoczi wrote:
> > > > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > > > What happens if a command takes 1 second to complete, is the device
> > > > > > > allowed to process the next command from the virtqueue during this time,
> > > > > > > possibly completing it before the first command?
> > > > > > > 
> > > > > > > This requires additional clarification in the spec because "they are
> > > > > > > processed by the device in the order in which they are queued" does not
> > > > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > > > not (out of order completion).
> > > > > > 
> > > > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > > > 
> > > > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > > > Several may be processed by the device at the same time.
> > > > 
> > > > Let's say I submit a write followed by read - is read
> > > > guaranteed to return an up to date info?
> > > 
> > > In general, no. The driver must wait for the write completion before
> > > submitting the read if it wants consistency.
> > > 
> > > Stefan
> > 
> > I see.  I think it's a good design to follow then.
> > 
> > I'll just copy
> > 	The driver queues requests to an arbitrary request queue, and
> > 	they are used by the device on that same queue. It is the
> > 	responsibility of the driver to ensure strict request ordering
> > 	for commands placed on different queues, because they will be
> > 	consumed with no order constraints.
> > 
> > replacing "request" with "admin".
> 
> That sounds like it's only about multi-queue because it says "... for
> commands placed on different queues". What I mentioned about a write
> followed by a read quest also applies within a single queue.
> 
> Can you clarify the semantics in the single queue case?
> 
> Stefan

This will be addressed in the latest version.
However, should not scsi and blk also adjust the wording
to make it clear device does not guarantee consistency?

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v10 08/10] admin: command list discovery

[Michael S. Tsirkin]

On Thu, Mar 02, 2023 at 04:09:29PM -0500, Stefan Hajnoczi wrote:
> Is the purpose of VIRTIO_ADMIN_CMD_LIST_USE so that new commands can be
> added that are mandatory for device operation. Failure to include them
> in the VIRTIO_ADMIN_CMD_LIST_USE command terminates device
> initialization with an error?

Or, switches the device to some legacy mode.

> Are commands evolved by defining flag/version bits in the command
> specific data struct? For example, if a new field needs to be added to a
> command and a 0 value carries a meaning (so it cannot simply be added to
> the end of the existing struct with backwards compatibility).
> 

commands expect to be evolved by extending the command specific
structure at the tail. this is why I spent so much time
on that in the text. if that does not work a new command will
work.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery

[Michael S. Tsirkin]

On Wed, Mar 08, 2023 at 01:41:37PM +0100, Jiri Pirko wrote:
> Wed, Mar 08, 2023 at 12:54:52PM CET, mst@redhat.com wrote:
> >On Mon, Mar 06, 2023 at 01:22:30PM +0100, Jiri Pirko wrote:
> >> Thu, Mar 02, 2023 at 02:05:22PM CET, mst@redhat.com wrote:
> >> >Add commands to find out which commands does each group support,
> >> >as well as enable their use by driver.
> >> >This will be especially useful once we have multiple group types.
> >> >
> >> >An alternative is per-type VQs. This is possible but will
> >> >require more per-transport work. Discovery through the vq
> >> >helps keep things contained.
> >> >
> >> >Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> >> >Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >> 
> >> [...]
> >> 
> >> 
> >> >+
> >> >+The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> >> >+query the list of commands valid for this group and before sending
> >> >+any commands for any member of a group.
> >> >+
> >> >+The driver then enables use of some of the opcodes by sending to
> >> >+the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
> >> >+of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
> >> >+both understood and used by the driver.
> >> 
> >> To my untrained ear, this sounds somewhat similar to the feature
> >> negotiantion mechanism. Why the fact that device/driver supports some
> >> command can't be covered by just another feature? Looks like unnecassary
> >> complexicity to negotiate supported commands like this.
> >
> >Absolutely, it is similar. The issue is that a single device can
> >be an owner for multiple group types.
> >For example, I think an SRIOV PF itself can have subfunctions with SIOV
> >as well as virtual functions with SRIOV.
> >And the set of supported commands might differ.
> >
> >We thus need a command that is kind of like features but per group type.
> 
> I see. But that does not mean it can't be done using existing features:
> 
> _F_SRIOV_CMD1
> _F_SRIOV_CMD2
> ..
> _F_SIOV_CMD1
> _F_SIOV_CMD2
> ...

True. But annoying if each command has to be duplicated.

> >
> >As a small bonus as we are building this from scratch, I added
> >something which was requested for a long time from features
> >which is blocking access to commands not allowed by features.
> >With standard virtio features we trust the driver not to send
> >commands before enabling a feature and this was considered
> >a maintainance/security problem.
> 
> I understand. Is it unfixable for features not to break buggy drivers?
> 

I think it is, feature negotiation is hard to evolve since our
compat strategy is built around feature negotiation.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org