[virtio-dev] [PATCH v10 00/10] Introduce device group and device management

[virtio-dev] [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

Thanks to everyone who commented!  Here's v10 after a longish
hiccup - I hope this is close to being ready. I'm especially
interested to hear whether this is in a good enough shape to
serve as basis for vq transport.

Thanks!

Change log:

since v9:
	addressed review comments:
		multiple typo fixes
		made command list array 64 bit - easier future proofing
		clarifications around group type and admin command concepts
		lots of clarifications around structure
			trucation for forwards and backwards compatibility -
			hopefully things are clear now

since v8:
	addressed comments by Cornelia - as we agreed on list
	
since v7:
	make high level error codes match linux, with virtio specific codes
		in a separate field
	renamed _ACCEPT to _USE since that's what it does
	clarified forward compatibility and non pci transports
	support multiple admin vqs
	conformance statements
	lots of changes all over the place to I changed author from Max
	to myself. Don't need to take credit but also don't want
	to blame Max for my mistakes.

since v6:

	- removed some extentions intended for future use.
	  We'll do them when we get there.

	- brought back command list query from v5 in a simplified form -
	  it's here to address the case where a single parent
	  can address multiple groups, such as PF addressing
	  transport vq and sriov vfs.

	- attempt to make terminology more formal.
	In particular a term for whoever controls the group.
 	I am still going back
	and forth between "parent" and "owner" - owner might
	be better after all since it will work if we ever
	have a self group. For now it's parent.

TODO (maybe?) - probably ok to defer until this part is upstream:

	Add "all members" member id.

	Add commands for MSI, feature discovery.

	Add commands for transport vq.


My intent is to try and support both SR-IOV and SIOV
usecases with the same structure and maybe even the same
VQ.

For example, it might make sense to split creating/destroying
SIOV devices from the transport passing data from the guest - the
driver would then not negotiate VIRTIO_F_SR_IOV (which
then means auto-provisioning).

This is out of RFC, since we have two commands which are useful
to discover supported group types (ATM can be none or SR-IOV).


Michael S. Tsirkin (10):
  virtio: document forward compatibility guarantees
  admin: introduce device group and related concepts
  admin: introduce group administration commands
  admin: introduce virtio admin virtqueues
  pci: add admin vq registers to virtio over pci
  mmio: document ADMIN_VQ as reserved
  ccw: document ADMIN_VQ as reserved
  admin: command list discovery
  admin: conformance clauses
  ccw: document more reserved features

 admin.tex   | 501 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex | 111 +++++++++++-
 2 files changed, 610 insertions(+), 2 deletions(-)
 create mode 100644 admin.tex

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[PATCH v10 01/10] virtio: document forward compatibility guarantees

[Michael S. Tsirkin]

Feature negotiation forms the basis of forward compatibility
guarantees of virtio but has never been properly documented.
Do it now.

Suggested-by: Halil Pasic <pasic@linux.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/content.tex b/content.tex
index 0e474dd..0c2d917 100644
--- a/content.tex
+++ b/content.tex
@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 In particular, new fields in the device configuration space are
 indicated by offering a new feature bit.
 
+To keep the feature negotiation mechanism extensible, it is important
+that devices \em{do not} offer any feature bits that they would not be
+able to handle if the driver accepted them (even though drivers are not
+supposed to accept them in the first place even if offered, according to
+this version of the specification.) Likewise, it is important that
+drivers \em{do not} accept feature bits they do not know how to handle
+(even though devices are not supposed to offer them in the first place,
+according to this version of the specification.) The preferred way for
+handling reserved and unexpected features is that the driver ignores
+them.
+
+In particular, this is
+especially important for features limited to specific transports,
+as enabling these for more transports in future versions of the
+specification is highly likely to require changing the behaviour
+from drivers and devices.  Drivers and devices supporting
+multiple transports need to carefully maintain per-transport
+lists of allowed features.
+
 \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
 The driver MUST NOT accept a feature which the device did not offer,
 and MUST NOT accept a feature which requires another feature which was
 not accepted.
 
+The driver MUST validate the feature bits offered by the device.
+The driver MUST ignore and MUST NOT accept any feature bit that is
+\begin{itemize}
+\item not described in this specification,
+\item marked as reserved,
+\item not supported for the specific transport,
+\item not defined for the device type.
+\end{itemize}
+
 The driver SHOULD go into backwards compatibility mode
 if the device does not offer a feature it understands, otherwise MUST
 set the FAILED \field{device status} bit and cease initialization.
 
+By contrast, the driver MUST NOT fail solely because a feature
+it does not understand has been offered by the device.
+
 \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
 The device MUST NOT offer a feature which requires another feature
 which was not offered.  The device SHOULD accept any valid subset
 of features the driver accepts, otherwise it MUST fail to set the
 FEATURES_OK \field{device status} bit when the driver writes it.
 
+The device MUST NOT offer feature bits corresponding to features
+it would not support if accepted by the driver (even if the
+driver is prohibited from accepting the feature bits by the
+specification); for the sake of clarity, this refers to feature
+bits not described in this specification, reserved feature bits
+and feature bits reserved or not supported for the specific
+transport or the specific device type, but this does not preclude
+devices written to a future version of this specification from
+offering such feature bits should such a specification have a
+provision for devices to support the corresponding features.
+
 If a device has successfully negotiated a set of features
 at least once (by accepting the FEATURES_OK \field{device
 status} bit during device initialization), then it SHOULD
-- 
MST

[PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

Each device group has a type. For now, define one initial group:

SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
PCI SR-IOV physical function (PF). This group may contain one or more
virtio devices.

Each device within a group has a unique identifier. This identifier
is the group member identifier.

Note: one can argue both ways whether the new device group handling
functionality (this and following patches) is closer
to a new device type or a new transport type.

However, I expect that we will add more features in the near future. To
facilitate this as much as possible of the text is located in the new
admin chapter.

I did my best to minimize transport-specific text.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex |  2 ++
 2 files changed, 51 insertions(+)
 create mode 100644 admin.tex

diff --git a/admin.tex b/admin.tex
new file mode 100644
index 0000000..2bc7322
--- /dev/null
+++ b/admin.tex
@@ -0,0 +1,49 @@
+\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
+
+It is occasionally useful to have a device control a group of
+other devices. Terminology used in such cases:
+
+\begin{description}
+\item[Device group]
+        or just group, includes zero or more devices.
+\item[Owner device]
+        or owner, the device controlling the group.
+\item[Member device]
+        a device within a group. The owner device itself is not
+	a member of the group.
+\item[Member identifier]
+        each member has this identifier, unique within the group
+	and used to address it through the owner device.
+\item[Group type identifier]
+	specifies what kind of member devices there are in a
+	group, how is the member identifier is interpreted
+	and what kind of control the owner has.
+	A given owner can control a single group of a given type,
+	thus the type and the owner together identify the group.
+	\footnote{Even though some group types only support
+			specific transports, group type identifiers
+			are global rather than transport-specific -
+			we don't expect a flood of new group types.}
+\end{description}
+
+The following group types, and their identifiers, are currently specified):
+\begin{description}
+\item[SR-IOV group type (0x1)]
+This device group has a PCI Single Root I/O Virtualization
+(SR-IOV) physical function (PF) device as the owner and includes
+all its SR-IOV virtual functions (VFs) as members (see
+\hyperref[intro:PCIe]{[PCIe]}).
+
+The PF device itself is not a member of the group.
+
+The group type identifier for this group is 0x1.
+
+A member identifier for this group can have a value 0x1 to 0xFFFF
+and equals the SR-IOV VF number of the member device (see
+\hyperref[intro:PCIe]{[PCIe]}).
+
+Both owner and member devices for this group type use the Virtio
+PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
+\end{description}
+
+
diff --git a/content.tex b/content.tex
index 0c2d917..ffe45c4 100644
--- a/content.tex
+++ b/content.tex
@@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\input{admin.tex}
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
-- 
MST

[PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

This introduces a general structure for group administration commands,
used to control device groups through their owner.

Following patches will introduce specific commands and an interface for
submitting these commands to the owner.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 101 insertions(+)

diff --git a/admin.tex b/admin.tex
index 2bc7322..46240f8 100644
--- a/admin.tex
+++ b/admin.tex
@@ -46,4 +46,105 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
 PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
 \end{description}
 
+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
 
+Group administration commands can be issued through an owner
+device to control member devices of a group.  This mechanism can
+be used, for example, to configure a member device before it is
+initialized by its driver.
+\footnote{The term "administration" is intended to be interpreted
+widely to include any kind of control. See specific commands
+for detail.}
+
+All the group administration commands are of the following form:
+
+\begin{lstlisting}
+struct virtio_admin_cmd {
+        /* Device-readable part */
+        le16 opcode;
+        /*
+         * 1 - SR-IOV
+         * 2 - 65535 reserved
+         */
+        le16 group_type;
+        /* unused, reserved for future extensions */
+        u8 reserved1[12];
+        le64 group_member_id;
+        u8 command_specific_data[];
+
+        /* Device-writable part */
+        le16 status;
+        le16 status_qualifier;
+        /* unused, reserved for future extensions */
+        u8 reserved2[4];
+        u8 command_specific_result[];
+};
+\end{lstlisting}
+
+For all commands, \field{opcode}, \field{group_type} and if
+necessary \field{group_member_id} and \field{command_specific_data} are
+set by the driver, and the owner device sets \field{status} and if
+needed \field{status_qualifier} and
+\field{command_specific_result}.
+
+As a rule, any unused device-readable fields are set to zero by the driver
+and ignored by the device.  Any unused device-writeable fields are set to zero
+by the device and ignored by the driver.
+
+\field{opcode} specifies the command. The valid
+values for \field{opcode} can be found in the following table:
+
+\begin{tabular}{|l|l|}
+\hline
+opcode & Name & Command Description \\
+\hline \hline
+0x0000 - 0x7FFF & - &  Group administration commands    \\
+\hline
+0x8000 - 0xFFFF & - & Reserved    \\
+\hline
+\end{tabular}
+
+The \field{group_type} specifies the group type identifier.
+The \field{group_member_id} specifies the member identifier within the group.
+See section \ref{sec:Introduction / Terminology / Device group}
+for the definition of the group type identifier and group member
+identifier.
+
+The following table describes possible \field{status} values;
+to simplify common implementations, they are intentionally
+matching common Linux names and error numbers:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Status (decimal) & Name & Description \\
+\hline \hline
+00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
+\hline
+22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
+\hline
+other   & -    & group administration command error  \\
+\hline
+\end{tabular}
+
+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
+is reserved and set to zero by the device.
+
+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
+the following table describes possible \field{status_qialifier} values:
+\begin{tabular}{|l|l|l|}
+\hline
+Status & Name & Description \\
+\hline \hline
+0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
+\hline
+0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
+\hline
+0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
+\hline
+0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
+\hline
+0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEM    & unsupported or invalid \field{group_member_id} \\
+\hline
+0x05-0xFFFF   & -    & reserved for future use \\
+\hline
+\end{tabular}
-- 
MST

[PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Michael S. Tsirkin]

The admin virtqueues will be the first interface to issue admin commands.

Currently virtio specification defines control virtqueue to manipulate
features and configuration of the device it operates on. However,
control virtqueue commands are device type specific, which makes it very
difficult to extend for device agnostic commands.

To support this requirement in a more generic way, this patch introduces
a new admin virtqueue interface.

We also support more than one admin virtqueue, for QoS and
scalability requirements.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex   | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex |  7 +++--
 2 files changed, 78 insertions(+), 2 deletions(-)

diff --git a/admin.tex b/admin.tex
index 46240f8..3026a79 100644
--- a/admin.tex
+++ b/admin.tex
@@ -148,3 +148,76 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 0x05-0xFFFF   & -    & reserved for future use \\
 \hline
 \end{tabular}
+
+\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
+
+An administration virtqueue of an owner device is used to submit
+group administration commands. An owner device can have more
+than one administration virtqueue.
+
+If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
+or more adminstration virtqueues. The number and locations of the
+administration virtqueues are exposed by the owner device in a transport
+specific manner.
+
+The driver submits commands by queueing them to an arbitrary
+administration virtqueue, and they are processed by the
+device in the order in which they are queued. It is the
+responsibility of the driver to ensure ordering for commands
+placed on different administration virtqueues, because they will
+be executed with no order constraints.
+
+Administration virtqueues are used as follows:
+\begin{itemize}
+\item The driver submits the command using the \field{struct virtio_admin_cmd}
+structure using a buffer consisting of two parts: a device-readable one followed by a
+device-writable one.
+\item the device-readable part includes fields from \field{opcode}
+through \field{command_specific_data}.
+\item the device-writeable buffer includes fields from \field{status}
+through \field{command_specific_result} inclusive.
+\end{itemize}
+
+For each command, this specification describes a distinct
+format structure used for \field{command_specific_data} and
+\field{command_specific_result}, the length of these fields
+depends on the command.
+
+However, to ensure forward compatibility
+\begin{itemize}
+\item drivers are allowed to submit buffers that are longer
+than what the device expects
+(that is, longer than the length of
+\field{opcode} through \field{command_specific_data}).
+This allows the driver to maintain
+a single format structure even if some structure fields are
+unused by the device.
+\item drivers are allowed to submit buffers that are shorter
+than what the device expects
+(that is, shorter than the length of \field{status} through
+\field{command_specific_result}). This allows the device to maintain
+a single format structure even if some structure fields are
+unused by the driver.
+\end{itemize}
+
+The device compares the length of each part (device-readable and
+device-writeable) of the buffer as submitted by driver to what it
+expects and then silently truncates the structures to either the
+length submitted by the driver, or the length described in this
+specification, whichever is shorter.  The device silently ignores
+any data falling outside the shorter of the two lengths. Any
+missing fields are interpreted as set to zero.
+
+Similarly, the driver compares the used buffer length
+of the buffer to what it expects and then silently
+truncates the structure to the used buffer length.
+The driver silently ignores any data falling outside
+the used buffer length reported by the device.  Any missing
+fields are interpreted as set to zero.
+
+This simplifies driver and device implementations since the
+driver/device can simply maintain a single large structure (such
+as a C structure) for a command and its result. As new versions
+of the specification are designed, new fields can be added to the
+tail of a structure, with the driver/device using the full
+structure without concern for versioning.
diff --git a/content.tex b/content.tex
index ffe45c4..c8647c9 100644
--- a/content.tex
+++ b/content.tex
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 40] Feature bits reserved for extensions to the queue and
+\item[24 to 41] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver can reset a queue individually.
   See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
 
+  \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
+  administration virtqueues.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
-- 
MST

[PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[Michael S. Tsirkin]

Add new registers to the PCI common configuration structure.

These registers will be used for querying the indices of the admin
virtqueues of the owner device. To configure, reset or enable the admin
virtqueues, the driver should follow existing queue configuration/setup
sequence.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/content.tex b/content.tex
index c8647c9..2d9b903 100644
--- a/content.tex
+++ b/content.tex
@@ -946,6 +946,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
         le16 queue_reset;               /* read-write */
+
+        /* About the administration virtqueue. */
+        le16 admin_queue_index;         /* read-only for driver */
+        le16 admin_queue_num;         /* read-only for driver */
 };
 \end{lstlisting}
 
@@ -1031,6 +1035,19 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         This field exists only if VIRTIO_F_RING_RESET has been
         negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+\item[\field{admin_queue_index}]
+        The device uses this to report the index of the first administration virtqueue.
+        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
+\item[\field{admin_queue_num}]
+	The device uses this to report the number of the
+	supported administration virtqueues.  Virtqueues with index
+	between \field{admin_queue_index} and (\field{admin_queue_index} +
+	\field{admin_queue_num}) inclusive serve as administration
+	virtqueues.
+	Thus the number of administration virtqueues equals
+	(\field{admin_queue_num} + 1).
+	This field is valid only if VIRTIO_F_ADMIN_VQ has been
+	negotiated.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1117,6 +1134,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 were used before the queue reset.
 (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver
+configures any administration virtqueues, the driver MUST
+configure the administration virtqueues using the index
+in the range \field{admin_queue_index} to
+\field{admin_queue_index} + \field{admin_queue_num} inclusive.
+The driver MAY configure less administration virtqueues than
+supported by the device.
+
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
@@ -7684,6 +7709,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
   \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
   administration virtqueues.
+  At the moment this feature is only supported for devices using
+  \ref{sec:Virtio Transport Options / Virtio Over PCI
+	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
+	  as the transport and is reserved for future use for
+	  devices using other transports (see
+	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
+	and
+	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
+	handling features reserved for future use.
 
 \end{description}
 
-- 
MST

[PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

Adding relevant registers needs more work and it's not
clear what the use-case will be as currently only
the PCI transport is supported. But let's keep the
door open on this.
We already say it's reserved in a central place, but it
does not hurt to remind implementers to mask it.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/content.tex b/content.tex
index 2d9b903..f8bd1ab 100644
--- a/content.tex
+++ b/content.tex
@@ -2362,6 +2362,18 @@ \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over M
 
 Notification mechanisms did not change.
 
+\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio Over MMIO / Features reserved for future use}
+
+At this time, devices and drivers utilizing Virtio Over MMIO
+do not support the following features:
+\begin{itemize}
+
+\item VIRTIO_F_ADMIN_VQ
+
+\end{itemize}
+
+These features are reserved for future use.
+
 \section{Virtio Over Channel I/O}\label{sec:Virtio Transport Options / Virtio Over Channel I/O}
 
 S/390 based virtual machines support neither PCI nor MMIO, so a
-- 
MST

[PATCH v10 07/10] ccw: document ADMIN_VQ as reserved

[Michael S. Tsirkin]

Adding relevant registers needs more work and it's not
clear what the use-case will be as currently only
the PCI transport is supported. But let's keep the
door open on this.
We already say it's reserved in a central place, but it
does not hurt to remind implementers to mask it.

Note: there are more features to add to this list.
Will be done later with a patch on top.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/content.tex b/content.tex
index f8bd1ab..6a50b39 100644
--- a/content.tex
+++ b/content.tex
@@ -2976,6 +2976,18 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
 MAY also choose to verify reset completion by reading \field{device status} via
 CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
 
+\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio over channel I/O / Features reserved for future use}
+
+At this time, devices and drivers utilizing Virtio over channel I/O
+do not support the following features:
+\begin{itemize}
+
+\item VIRTIO_F_ADMIN_VQ
+
+\end{itemize}
+
+These features are reserved for future use.
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
-- 
MST

[PATCH v10 08/10] admin: command list discovery

[Michael S. Tsirkin]

Add commands to find out which commands does each group support,
as well as enable their use by driver.
This will be especially useful once we have multiple group types.

An alternative is per-type VQs. This is possible but will
require more per-transport work. Discovery through the vq
helps keep things contained.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 89 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 3026a79..4de606a 100644
--- a/admin.tex
+++ b/admin.tex
@@ -98,7 +98,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline
 opcode & Name & Command Description \\
 \hline \hline
-0x0000 - 0x7FFF & - &  Group administration commands    \\
+0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
+0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
+0x0002 - 0x7FFF & - &  Group administration commands    \\
 \hline
 0x8000 - 0xFFFF & - & Reserved    \\
 \hline
@@ -149,6 +151,92 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline
 \end{tabular}
 
+Before sending any administration commands to the device, the driver
+needs to communicate to the device which commands it is going to
+use. Initially (after reset), only two commands are assumed to be used:
+VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
+
+Before sending any other commands for any member of a specific group to
+the device, the driver queries the supported commands via
+VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
+VIRTIO_ADMIN_CMD_LIST_USE.
+
+Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
+VIRTIO_ADMIN_CMD_LIST_USE
+both use the following structure describing the
+command opcodes:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_list {
+       /* Indicates which of the below fields were returned
+       le64 device_admin_cmds[];
+};
+\end{lstlisting}
+
+This structure is an array of 64 bit values in little-endian byte
+order, in which a bit is set if the specific command opcode
+is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value
+in this array corresponding to opcodes 0 to 63,
+\field{device_admin_cmds[1]} is the second 64-bit value
+corresponding to opcodes 64 to 127, etc.
+For example, the array of size 2 including
+the values 0x3 in \field{device_admin_cmds[0]}
+and 0x1 in \field{device_admin_cmds[1]} indicates that only
+opcodes 0, 1 and 64 are supported.
+The length of the array depends on the supported opcodes - it is
+large enough to include bits set for all supported opcodes,
+that is the length can be calculated by starting with the largest
+supported opcode adding one, dividing by 64 and rounding up.
+The array is also allowed to be larger and to additionally
+include an arbitrary number of all-zero entries.
+
+Accordingly, bits 0 and 1 corresponding to opcode 0
+(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
+(VIRTIO_ADMIN_CMD_LIST_USE) are
+always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
+
+For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
+The \field{group_member_id} is unused. It is set to zero by driver.
+This command has no command specific data.
+The device, upon success, returns a result in
+\field{command_specific_result} in the format
+\field{struct virtio_admin_cmd_list} describing the
+list of administration commands supported for the given group.
+
+
+For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
+is set to 0x1.
+The \field{group_member_id} is unused. It is set to zero by driver.
+The \field{command_specific_data} is in the format
+\field{struct virtio_admin_cmd_list} describing the
+list of administration commands used by the driver.
+This command has no command specific result.
+
+The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
+query the list of commands valid for this group and before sending
+any commands for any member of a group.
+
+The driver then enables use of some of the opcodes by sending to
+the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
+of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
+both understood and used by the driver.
+
+If the device supports the command list used by the driver, the
+device completes the command with status VIRTIO_ADMIN_STATUS_OK.
+If the device does not support the command list, the device
+completes the command with status
+VIRTIO_ADMIN_STATUS_INVALID_FIELD.
+
+Note: drivers are assumed not to set bits in device_admin_cmds
+if they are not familiar with how the command opcode
+is used, since devices could have dependencies between
+command opcodes.
+
+It is assumed that all members in a group support and are used
+with the same list of commands. However, for owner devices
+supporting multiple group types, the list of supported commands
+might differ between different group types.
+
 \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
 
 An administration virtqueue of an owner device is used to submit
-- 
MST

[PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

Add conformance clauses for admin commands and admin virtqueues.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 admin.tex | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 190 insertions(+)

diff --git a/admin.tex b/admin.tex
index 4de606a..f9b1b68 100644
--- a/admin.tex
+++ b/admin.tex
@@ -237,6 +237,119 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 supporting multiple group types, the list of supported commands
 might differ between different group types.
 
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+
+The device MUST validate \field{opcode}, \field{group_type} and
+\field{group_member_id}, and if any of these has an invalid or
+unsupported value, set \field{status} to
+VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
+accordingly:
+\begin{itemize}
+\item if \field{group_type} is invalid, \field{status_qualifier}
+	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
+\item otherwise, if \field{opcode} is invalid,
+	\field{status_qualifier} MUST be set to
+	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
+\item otherwise, if \field{group_member_id} is used by the
+	specific command and is invalid, \field{status_qualifier} MUST be
+	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
+\end{itemize}
+
+If a command completes successfully, the device MUST set
+\field{status} to VIRTIO_ADMIN_STATUS_OK.
+
+If a command fails, the device MUST set
+\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
+
+If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
+device state MUST NOT change, that is the command MUST NOT have
+any side effects on the device, in particular the device MUST not
+enter an error state as a result of this command.
+
+If a command fails, the device state generally SHOULD NOT change,
+as far as possible.
+
+The device MAY enforce additional restrictions and dependencies on
+opcodes used by the driver and MAY fail the command
+VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
+and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
+if the list of commands used violate internal device dependencies.
+
+If the device supports multiple group types, commands for each group
+type MUST operate independently of each other, in particular,
+the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
+for different group types.
+
+After reset, if the device supports a given group type
+and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
+the device MUST assume
+that the list of legal commands used by the driver consists of
+the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
+
+After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
+the device MUST set the list of legal commands used by the driver
+to the one supplied in \field{command_specific_data}.
+
+The device MUST set the list of legal commands used by the driver
+to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
+
+The device MUST validate commands against the list used by
+the driver and MUST fail any commands not in the list with
+\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
+and \field{status_qualifier} set to
+VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
+
+The list of supported commands MUST NOT shrink (but MAY expand):
+after reporting a given command as supported through
+VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
+as unsupported.  Further, after a given set of commands has been
+used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
+device or system reset the device SHOULD complete successfully
+any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
+list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
+after a device or system reset, the device MUST not fail it
+solely because of the command list used.  Failure to do so would
+interfere with resuming from suspend and error recovery.
+
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+
+The driver MAY discover whether device supports a specific group type
+by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
+\field{group_type}.
+
+The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
+and wait for it to be completed with status
+VIRTIO_ADMIN_STATUS_OK before issuing any commands
+(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
+and VIRTIO_ADMIN_CMD_LIST_USE).
+
+The driver SHOULD NOT set bits in device_admin_cmds
+if it is not familiar with how the command opcode
+is used, as dependencies between command opcodes might exist.
+
+The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
+the use of any commands not previously reported as
+supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
+
+The driver MUST NOT use any commands for a given group type
+before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
+list of command opcodes and group type.
+
+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
+VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
+with respective bits cleared in \field{command_specific_data}.
+
+The driver MUST handle a command error with a reserved \field{status}
+value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
+(except possibly for different error reporting/diagnostic messages).
+
+The driver MUST handle a command error with a reserved
+\field{status_qualifier} value in the same way as
+\field{status_qualifier} set to
+VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
+different error reporting/diagnostic messages).
+
 \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
 
 An administration virtqueue of an owner device is used to submit
@@ -309,3 +422,80 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
 of the specification are designed, new fields can be added to the
 tail of a structure, with the driver/device using the full
 structure without concern for versioning.
+
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
+
+The device MUST support device-readable and device-writeable buffers
+shorter than described in this specification, by
+\begin{enumerate}
+\item acting as if any data that would be read outside the
+device-readable buffers is set to zero, and
+\item discarding data that would be written outside the
+specified device-writeable buffers.
+\end{enumerate}
+
+The device MUST support device-readable and device-writeable buffers
+longer than described in this specification, by
+\begin{enumerate}
+\item ignoring any data in device-readable buffers outside
+the expected length, and
+\item only writing the expected structure to the device-writeable
+buffers, ignoring any extra buffers, and reporting the
+actual length of data written, in bytes,
+as buffer used length.
+\end{enumerate}
+
+The device SHOULD initialize the device-writeable buffer
+up to the length of the structure described by this specification or
+the length of the buffer supplied by the driver (even if the buffer is
+all set to zero), whichever is shorter.
+
+The device MUST NOT fail a command solely because the buffers
+provided are shorter or longer than described in this
+specification.
+
+The device MUST initialize the device-writeable part of
+\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
+size.
+
+The device MUST initialize \field{status} in \field{struct
+virtio_admin_cmd}.
+
+The device MUST process commands on a given administration virtqueue
+in the order in which they are queued.
+
+If multiple administration virtqueues have been configured,
+device MAY process commands on distinct virtqueues with
+no order constraints.
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
+
+The driver MAY supply device-readable or device-writeable parts
+of \field{struct virtio_admin_cmd} that are longer than described in
+this specification.
+
+The driver SHOULD supply device-readable part of
+\field{struct virtio_admin_cmd} that is at least as
+large as the structure described by this specification
+(even if the structure is all set to zero).
+
+The driver MUST supply both device-readable or device-writeable parts
+of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
+in length.
+
+The device MUST supply both device-readable or device-writeable parts
+of \field{struct virtio_admin_cmd} that are larger than zero in
+length. However, \field{command_specific_data} and
+\field{command_specific_result} MAY be zero in length, unless
+specified otherwise for the command.
+
+The driver MUST NOT assume that the device will initialize the whole
+device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
+the driver MUST act as if the structure
+outside the part of the buffer used by the device
+is set to zero.
+
+If multiple administration virtqueues have been configured,
+the driver MUST ensure ordering for commands
+placed on different administration virtqueues.
+>>>>>>> 29f6525... admin: conformance clauses
-- 
MST

[PATCH v10 10/10] ccw: document more reserved features

[Michael S. Tsirkin]

vq reset and shared memory are unsupported, too.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---


This is not strictly part of this patchset, but just to
give you the idea of how we can expand this down the road.


 content.tex | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/content.tex b/content.tex
index 6a50b39..b677006 100644
--- a/content.tex
+++ b/content.tex
@@ -2983,6 +2983,8 @@ \subsection{Features reserved for future use}\label{sec:Virtio Transport Options
 \begin{itemize}
 
 \item VIRTIO_F_ADMIN_VQ
+\item VIRTIO_F_RING_RESET
+\item Shared memory regions including VIRTIO_PMEM_F_SHMEM_REGION
 
 \end{itemize}
 
-- 
MST

[virtio-comment] Re: [virtio-dev] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[David Edmondson]

On Thursday, 2023-02-09 at 07:13:32 -05, Michael S. Tsirkin wrote:
> Feature negotiation forms the basis of forward compatibility
> guarantees of virtio but has never been properly documented.
> Do it now.
>
> Suggested-by: Halil Pasic <pasic@linux.ibm.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index 0e474dd..0c2d917 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>  In particular, new fields in the device configuration space are
>  indicated by offering a new feature bit.
>  
> +To keep the feature negotiation mechanism extensible, it is important
> +that devices \em{do not} offer any feature bits that they would not be
> +able to handle if the driver accepted them (even though drivers are not
> +supposed to accept them in the first place even if offered, according to
> +this version of the specification.) Likewise, it is important that
> +drivers \em{do not} accept feature bits they do not know how to handle
> +(even though devices are not supposed to offer them in the first place,
> +according to this version of the specification.) The preferred way for
> +handling reserved and unexpected features is that the driver ignores
> +them.
> +
> +In particular, this is
> +especially important for features limited to specific transports,
> +as enabling these for more transports in future versions of the
> +specification is highly likely to require changing the behaviour
> +from drivers and devices.  Drivers and devices supporting

"changing the behaviour of" or "changed behaviour from" (prefer the
former).

> +multiple transports need to carefully maintain per-transport
> +lists of allowed features.
> +
>  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
>  The driver MUST NOT accept a feature which the device did not offer,
>  and MUST NOT accept a feature which requires another feature which was
>  not accepted.
>  
> +The driver MUST validate the feature bits offered by the device.
> +The driver MUST ignore and MUST NOT accept any feature bit that is
> +\begin{itemize}
> +\item not described in this specification,
> +\item marked as reserved,
> +\item not supported for the specific transport,

What does "supported" mean here? By the driver? By the specification in
respect of this transport?

> +\item not defined for the device type.
> +\end{itemize}
> +
>  The driver SHOULD go into backwards compatibility mode
>  if the device does not offer a feature it understands, otherwise MUST
>  set the FAILED \field{device status} bit and cease initialization.
>  
> +By contrast, the driver MUST NOT fail solely because a feature
> +it does not understand has been offered by the device.
> +
>  \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
>  The device MUST NOT offer a feature which requires another feature
>  which was not offered.  The device SHOULD accept any valid subset
>  of features the driver accepts, otherwise it MUST fail to set the
>  FEATURES_OK \field{device status} bit when the driver writes it.
>  
> +The device MUST NOT offer feature bits corresponding to features
> +it would not support if accepted by the driver (even if the
> +driver is prohibited from accepting the feature bits by the
> +specification); for the sake of clarity, this refers to feature
> +bits not described in this specification, reserved feature bits
> +and feature bits reserved or not supported for the specific
> +transport or the specific device type, but this does not preclude
> +devices written to a future version of this specification from
> +offering such feature bits should such a specification have a
> +provision for devices to support the corresponding features.
> +
>  If a device has successfully negotiated a set of features
>  at least once (by accepting the FEATURES_OK \field{device
>  status} bit during device initialization), then it SHOULD
-- 
Maybe then I'll fade away and not have to face the facts.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[David Edmondson]

On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
> Each device group has a type. For now, define one initial group:
>
> SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
> PCI SR-IOV physical function (PF). This group may contain one or more
> virtio devices.

The specification says zero or more generally and nothing to refine that
for SR-IOV groups.

>
> Each device within a group has a unique identifier. This identifier
> is the group member identifier.
>
> Note: one can argue both ways whether the new device group handling
> functionality (this and following patches) is closer
> to a new device type or a new transport type.
>
> However, I expect that we will add more features in the near future. To
> facilitate this as much as possible of the text is located in the new
> admin chapter.
>
> I did my best to minimize transport-specific text.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex |  2 ++
>  2 files changed, 51 insertions(+)
>  create mode 100644 admin.tex
>
> diff --git a/admin.tex b/admin.tex
> new file mode 100644
> index 0000000..2bc7322
> --- /dev/null
> +++ b/admin.tex
> @@ -0,0 +1,49 @@
> +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
> +
> +It is occasionally useful to have a device control a group of
> +other devices. Terminology used in such cases:
> +
> +\begin{description}
> +\item[Device group]
> +        or just group, includes zero or more devices.
> +\item[Owner device]
> +        or owner, the device controlling the group.
> +\item[Member device]
> +        a device within a group. The owner device itself is not
> +	a member of the group.
> +\item[Member identifier]
> +        each member has this identifier, unique within the group
> +	and used to address it through the owner device.
> +\item[Group type identifier]
> +	specifies what kind of member devices there are in a
> +	group, how is the member identifier is interpreted
> +	and what kind of control the owner has.
> +	A given owner can control a single group of a given type,
> +	thus the type and the owner together identify the group.
> +	\footnote{Even though some group types only support
> +			specific transports, group type identifiers
> +			are global rather than transport-specific -
> +			we don't expect a flood of new group types.}
> +\end{description}
> +
> +The following group types, and their identifiers, are currently specified):
> +\begin{description}
> +\item[SR-IOV group type (0x1)]
> +This device group has a PCI Single Root I/O Virtualization
> +(SR-IOV) physical function (PF) device as the owner and includes
> +all its SR-IOV virtual functions (VFs) as members (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +The PF device itself is not a member of the group.
> +
> +The group type identifier for this group is 0x1.
> +
> +A member identifier for this group can have a value 0x1 to 0xFFFF
> +and equals the SR-IOV VF number of the member device (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +Both owner and member devices for this group type use the Virtio
> +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> +\end{description}
> +
> +
> diff --git a/content.tex b/content.tex
> index 0c2d917..ffe45c4 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  types. It is RECOMMENDED that devices generate version 4
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>  
> +\input{admin.tex}
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
>  We start with an overview of device initialization, then expand on the
-- 
Tell her I'll be waiting, in the usual place.

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
> 
> On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
> > Each device group has a type. For now, define one initial group:
> >
> > SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
> > PCI SR-IOV physical function (PF). This group may contain one or more
> > virtio devices.
> 
> The specification says zero or more generally and nothing to refine that
> for SR-IOV groups.

Well it says:

	(SR-IOV) physical function (PF) device as the owner and includes
	all its SR-IOV virtual functions (VFs) as members (see
	\hyperref[intro:PCIe]{[PCIe]}).

how can that be zero?


> >
> > Each device within a group has a unique identifier. This identifier
> > is the group member identifier.
> >
> > Note: one can argue both ways whether the new device group handling
> > functionality (this and following patches) is closer
> > to a new device type or a new transport type.
> >
> > However, I expect that we will add more features in the near future. To
> > facilitate this as much as possible of the text is located in the new
> > admin chapter.
> >
> > I did my best to minimize transport-specific text.
> >
> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >  admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  content.tex |  2 ++
> >  2 files changed, 51 insertions(+)
> >  create mode 100644 admin.tex
> >
> > diff --git a/admin.tex b/admin.tex
> > new file mode 100644
> > index 0000000..2bc7322
> > --- /dev/null
> > +++ b/admin.tex
> > @@ -0,0 +1,49 @@
> > +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
> > +
> > +It is occasionally useful to have a device control a group of
> > +other devices. Terminology used in such cases:
> > +
> > +\begin{description}
> > +\item[Device group]
> > +        or just group, includes zero or more devices.
> > +\item[Owner device]
> > +        or owner, the device controlling the group.
> > +\item[Member device]
> > +        a device within a group. The owner device itself is not
> > +	a member of the group.
> > +\item[Member identifier]
> > +        each member has this identifier, unique within the group
> > +	and used to address it through the owner device.
> > +\item[Group type identifier]
> > +	specifies what kind of member devices there are in a
> > +	group, how is the member identifier is interpreted
> > +	and what kind of control the owner has.
> > +	A given owner can control a single group of a given type,
> > +	thus the type and the owner together identify the group.
> > +	\footnote{Even though some group types only support
> > +			specific transports, group type identifiers
> > +			are global rather than transport-specific -
> > +			we don't expect a flood of new group types.}
> > +\end{description}
> > +
> > +The following group types, and their identifiers, are currently specified):
> > +\begin{description}
> > +\item[SR-IOV group type (0x1)]
> > +This device group has a PCI Single Root I/O Virtualization
> > +(SR-IOV) physical function (PF) device as the owner and includes
> > +all its SR-IOV virtual functions (VFs) as members (see
> > +\hyperref[intro:PCIe]{[PCIe]}).
> > +
> > +The PF device itself is not a member of the group.
> > +
> > +The group type identifier for this group is 0x1.
> > +
> > +A member identifier for this group can have a value 0x1 to 0xFFFF
> > +and equals the SR-IOV VF number of the member device (see
> > +\hyperref[intro:PCIe]{[PCIe]}).
> > +
> > +Both owner and member devices for this group type use the Virtio
> > +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > +\end{description}
> > +
> > +
> > diff --git a/content.tex b/content.tex
> > index 0c2d917..ffe45c4 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> >  types. It is RECOMMENDED that devices generate version 4
> >  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> >  
> > +\input{admin.tex}
> > +
> >  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> >  
> >  We start with an overview of device initialization, then expand on the
> -- 
> Tell her I'll be waiting, in the usual place.

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[David Edmondson]

On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
> On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
>> 
>> On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
>> > Each device group has a type. For now, define one initial group:
>> >
>> > SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>> > PCI SR-IOV physical function (PF). This group may contain one or more
>> > virtio devices.
>> 
>> The specification says zero or more generally and nothing to refine that
>> for SR-IOV groups.
>
> Well it says:
>
> 	(SR-IOV) physical function (PF) device as the owner and includes
> 	all its SR-IOV virtual functions (VFs) as members (see
> 	\hyperref[intro:PCIe]{[PCIe]}).
>
> how can that be zero?

If I remove all of the VFs of a PF, does the group implicitly disappear?
How about the converse?

>
>> >
>> > Each device within a group has a unique identifier. This identifier
>> > is the group member identifier.
>> >
>> > Note: one can argue both ways whether the new device group handling
>> > functionality (this and following patches) is closer
>> > to a new device type or a new transport type.
>> >
>> > However, I expect that we will add more features in the near future. To
>> > facilitate this as much as possible of the text is located in the new
>> > admin chapter.
>> >
>> > I did my best to minimize transport-specific text.
>> >
>> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> > ---
>> >  admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>> >  content.tex |  2 ++
>> >  2 files changed, 51 insertions(+)
>> >  create mode 100644 admin.tex
>> >
>> > diff --git a/admin.tex b/admin.tex
>> > new file mode 100644
>> > index 0000000..2bc7322
>> > --- /dev/null
>> > +++ b/admin.tex
>> > @@ -0,0 +1,49 @@
>> > +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>> > +
>> > +It is occasionally useful to have a device control a group of
>> > +other devices. Terminology used in such cases:
>> > +
>> > +\begin{description}
>> > +\item[Device group]
>> > +        or just group, includes zero or more devices.
>> > +\item[Owner device]
>> > +        or owner, the device controlling the group.
>> > +\item[Member device]
>> > +        a device within a group. The owner device itself is not
>> > +	a member of the group.
>> > +\item[Member identifier]
>> > +        each member has this identifier, unique within the group
>> > +	and used to address it through the owner device.
>> > +\item[Group type identifier]
>> > +	specifies what kind of member devices there are in a
>> > +	group, how is the member identifier is interpreted
>> > +	and what kind of control the owner has.
>> > +	A given owner can control a single group of a given type,
>> > +	thus the type and the owner together identify the group.
>> > +	\footnote{Even though some group types only support
>> > +			specific transports, group type identifiers
>> > +			are global rather than transport-specific -
>> > +			we don't expect a flood of new group types.}
>> > +\end{description}
>> > +
>> > +The following group types, and their identifiers, are currently specified):
>> > +\begin{description}
>> > +\item[SR-IOV group type (0x1)]
>> > +This device group has a PCI Single Root I/O Virtualization
>> > +(SR-IOV) physical function (PF) device as the owner and includes
>> > +all its SR-IOV virtual functions (VFs) as members (see
>> > +\hyperref[intro:PCIe]{[PCIe]}).
>> > +
>> > +The PF device itself is not a member of the group.
>> > +
>> > +The group type identifier for this group is 0x1.
>> > +
>> > +A member identifier for this group can have a value 0x1 to 0xFFFF
>> > +and equals the SR-IOV VF number of the member device (see
>> > +\hyperref[intro:PCIe]{[PCIe]}).
>> > +
>> > +Both owner and member devices for this group type use the Virtio
>> > +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>> > +\end{description}
>> > +
>> > +
>> > diff --git a/content.tex b/content.tex
>> > index 0c2d917..ffe45c4 100644
>> > --- a/content.tex
>> > +++ b/content.tex
>> > @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>> >  types. It is RECOMMENDED that devices generate version 4
>> >  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>> >  
>> > +\input{admin.tex}
>> > +
>> >  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>> >  
>> >  We start with an overview of device initialization, then expand on the
>> -- 
>> Tell her I'll be waiting, in the usual place.
-- 
Do not leave the building.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [PATCH v10 08/10] admin: command list discovery

[Uminski, Piotr]

> -----Original Message-----
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 1:14 PM
> To: virtio-comment@lists.oasis-open.org; virtio-dev@lists.oasis-open.org;
> jasowang@redhat.com; mst@redhat.com; cohuck@redhat.com;
> sgarzare@redhat.com; stefanha@redhat.com; Jani, Nrupal
> <nrupal.jani@intel.com>; Uminski, Piotr <Piotr.Uminski@intel.com>; Yuan,
> Hang <hang.yuan@intel.com>
> Cc: virtio@lists.oasis-open.org; Zhu, Lingshan <lingshan.zhu@intel.com>;
> pasic@linux.ibm.com; Shahaf Shuler <shahafs@nvidia.com>; Parav Pandit
> <parav@nvidia.com>; Max Gurtovoy <mgurtovoy@nvidia.com>
> Subject: [PATCH v10 08/10] admin: command list discovery
> 
> Add commands to find out which commands does each group support, as
> well as enable their use by driver.
> This will be especially useful once we have multiple group types.
> 
> An alternative is per-type VQs. This is possible but will require more per-
> transport work. Discovery through the vq helps keep things contained.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex | 90
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 89 insertions(+), 1 deletion(-)
> 
> diff --git a/admin.tex b/admin.tex
> index 3026a79..4de606a 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -98,7 +98,9 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti  \hline  opcode & Name &
> Command Description \\  \hline \hline
> -0x0000 - 0x7FFF & - &  Group administration commands    \\
> +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of
> commands supported for this group type    \\
> +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> commands used for this group type \\
> +0x0002 - 0x7FFF & - &  Group administration commands    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved    \\
>  \hline
> @@ -149,6 +151,92 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti  \hline  \end{tabular}
> 
> +Before sending any administration commands to the device, the driver
> +needs to communicate to the device which commands it is going to use.
> +Initially (after reset), only two commands are assumed to be used:
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Before sending any other commands for any member of a specific group to
> +the device, the driver queries the supported commands via
> +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
> +VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
> VIRTIO_ADMIN_CMD_LIST_USE both
> +use the following structure describing the command opcodes:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_list {
> +       /* Indicates which of the below fields were returned
> +       le64 device_admin_cmds[];
> +};
> +\end{lstlisting}
> +
> +This structure is an array of 64 bit values in little-endian byte
> +order, in which a bit is set if the specific command opcode is
> +supported. Thus, \field{device_admin_cmds[0]} refers to the first
> +32-bit value in this array corresponding to opcodes 0 to 63,
Probably should be "64-bit value"
> +\field{device_admin_cmds[1]} is the second 64-bit value corresponding
> +to opcodes 64 to 127, etc.
> +For example, the array of size 2 including the values 0x3 in
> +\field{device_admin_cmds[0]} and 0x1 in \field{device_admin_cmds[1]}
> +indicates that only opcodes 0, 1 and 64 are supported.
> +The length of the array depends on the supported opcodes - it is large
> +enough to include bits set for all supported opcodes, that is the
> +length can be calculated by starting with the largest supported opcode
> +adding one, dividing by 64 and rounding up.
> +The array is also allowed to be larger and to additionally include an
> +arbitrary number of all-zero entries.
> +
> +Accordingly, bits 0 and 1 corresponding to opcode 0
> +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
> +(VIRTIO_ADMIN_CMD_LIST_USE) are
> +always set in \field{device_admin_cmds[0]} returned by
> VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set
> to 0x0.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +This command has no command specific data.
> +The device, upon success, returns a result in
> +\field{command_specific_result} in the format \field{struct
> +virtio_admin_cmd_list} describing the list of administration commands
> +supported for the given group.
> +
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode} is set to
> +0x1.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +The \field{command_specific_data} is in the format \field{struct
> +virtio_admin_cmd_list} describing the list of administration commands
> +used by the driver.
> +This command has no command specific result.
> +
> +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> query the
> +list of commands valid for this group and before sending any commands
> +for any member of a group.
> +
> +The driver then enables use of some of the opcodes by sending to the
> +device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset of the
> list
> +returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is both understood
> and
> +used by the driver.
> +
> +If the device supports the command list used by the driver, the device
> +completes the command with status VIRTIO_ADMIN_STATUS_OK.
> +If the device does not support the command list, the device completes
> +the command with status VIRTIO_ADMIN_STATUS_INVALID_FIELD.
> +
> +Note: drivers are assumed not to set bits in device_admin_cmds if they
> +are not familiar with how the command opcode is used, since devices
> +could have dependencies between command opcodes.
> +
> +It is assumed that all members in a group support and are used with the
> +same list of commands. However, for owner devices supporting multiple
> +group types, the list of supported commands might differ between
> +different group types.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio
> Device / Administration Virtqueues}
> 
>  An administration virtqueue of an owner device is used to submit
> --
> MST

---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Spolka oswiadcza, ze posiada status duzego przedsiebiorcy w rozumieniu ustawy z dnia 8 marca 2013 r. o przeciwdzialaniu nadmiernym opoznieniom w transakcjach handlowych.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Max Gurtovoy]

[-- Attachment #1: Type: text/plain, Size: 5002 bytes --]


On 09/02/2023 17:22, David Edmondson wrote:
> On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
>> On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
>>> On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
>>>> Each device group has a type. For now, define one initial group:
>>>>
>>>> SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>>>> PCI SR-IOV physical function (PF). This group may contain one or more
>>>> virtio devices.
>>> The specification says zero or more generally and nothing to refine that
>>> for SR-IOV groups.
>> Well it says:
>>
>> 	(SR-IOV) physical function (PF) device as the owner and includes
>> 	all its SR-IOV virtual functions (VFs) as members (see
>> 	\hyperref[intro:PCIe]{[PCIe]}).
>>
>> how can that be zero?
> If I remove all of the VFs of a PF, does the group implicitly disappear?

Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.

We can mention that a group include all virtual functions indicated by 
*TotalVFs*RO register.

This group exist as long as the owner PF supports SR-IOV and has admin 
capabilities.
****

> How about the converse?
>
>>>> Each device within a group has a unique identifier. This identifier
>>>> is the group member identifier.
>>>>
>>>> Note: one can argue both ways whether the new device group handling
>>>> functionality (this and following patches) is closer
>>>> to a new device type or a new transport type.
>>>>
>>>> However, I expect that we will add more features in the near future. To
>>>> facilitate this as much as possible of the text is located in the new
>>>> admin chapter.
>>>>
>>>> I did my best to minimize transport-specific text.
>>>>
>>>> Signed-off-by: Max Gurtovoy<mgurtovoy@nvidia.com>
>>>> Signed-off-by: Michael S. Tsirkin<mst@redhat.com>
>>>> ---
>>>>   admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>   content.tex |  2 ++
>>>>   2 files changed, 51 insertions(+)
>>>>   create mode 100644 admin.tex
>>>>
>>>> diff --git a/admin.tex b/admin.tex
>>>> new file mode 100644
>>>> index 0000000..2bc7322
>>>> --- /dev/null
>>>> +++ b/admin.tex
>>>> @@ -0,0 +1,49 @@
>>>> +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>>>> +
>>>> +It is occasionally useful to have a device control a group of
>>>> +other devices. Terminology used in such cases:
>>>> +
>>>> +\begin{description}
>>>> +\item[Device group]
>>>> +        or just group, includes zero or more devices.
>>>> +\item[Owner device]
>>>> +        or owner, the device controlling the group.
>>>> +\item[Member device]
>>>> +        a device within a group. The owner device itself is not
>>>> +	a member of the group.
>>>> +\item[Member identifier]
>>>> +        each member has this identifier, unique within the group
>>>> +	and used to address it through the owner device.
>>>> +\item[Group type identifier]
>>>> +	specifies what kind of member devices there are in a
>>>> +	group, how is the member identifier is interpreted
>>>> +	and what kind of control the owner has.
>>>> +	A given owner can control a single group of a given type,
>>>> +	thus the type and the owner together identify the group.
>>>> +	\footnote{Even though some group types only support
>>>> +			specific transports, group type identifiers
>>>> +			are global rather than transport-specific -
>>>> +			we don't expect a flood of new group types.}
>>>> +\end{description}
>>>> +
>>>> +The following group types, and their identifiers, are currently specified):
>>>> +\begin{description}
>>>> +\item[SR-IOV group type (0x1)]
>>>> +This device group has a PCI Single Root I/O Virtualization
>>>> +(SR-IOV) physical function (PF) device as the owner and includes
>>>> +all its SR-IOV virtual functions (VFs) as members (see
>>>> +\hyperref[intro:PCIe]{[PCIe]}).
>>>> +
>>>> +The PF device itself is not a member of the group.
>>>> +
>>>> +The group type identifier for this group is 0x1.
>>>> +
>>>> +A member identifier for this group can have a value 0x1 to 0xFFFF
>>>> +and equals the SR-IOV VF number of the member device (see
>>>> +\hyperref[intro:PCIe]{[PCIe]}).
>>>> +
>>>> +Both owner and member devices for this group type use the Virtio
>>>> +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>>>> +\end{description}
>>>> +
>>>> +
>>>> diff --git a/content.tex b/content.tex
>>>> index 0c2d917..ffe45c4 100644
>>>> --- a/content.tex
>>>> +++ b/content.tex
>>>> @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>>>   types. It is RECOMMENDED that devices generate version 4
>>>>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>   
>>>> +\input{admin.tex}
>>>> +
>>>>   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>>>   
>>>>   We start with an overview of device initialization, then expand on the
>>> -- 
>>> Tell her I'll be waiting, in the usual place.

[-- Attachment #2: Type: text/html, Size: 9065 bytes --]

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
> 
> On 09/02/2023 17:22, David Edmondson wrote:
> 
>     On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
> 
>         On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
> 
>             On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
> 
>                 Each device group has a type. For now, define one initial group:
> 
>                 SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>                 PCI SR-IOV physical function (PF). This group may contain one or more
>                 virtio devices.
> 
>             The specification says zero or more generally and nothing to refine that
>             for SR-IOV groups.
> 
>         Well it says:
> 
>                 (SR-IOV) physical function (PF) device as the owner and includes
>                 all its SR-IOV virtual functions (VFs) as members (see
>                 \hyperref[intro:PCIe]{[PCIe]}).
> 
>         how can that be zero?
> 
>     If I remove all of the VFs of a PF, does the group implicitly disappear?
> 
> Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.

Are you sure? Not by VF Enable?

> We can mention that a group include all virtual functions indicated by TotalVFs
> RO register.

TotalVFs indicates the maximum number of VFs that could be associated with the PF
but not all of these exist. If we use this we get into issues of
discovering whether VFs can actually be used. I'd rather avoid that.


> This group exist as long as the owner PF supports SR-IOV and has admin
> capabilities.

SRIOV spec says:
	NumVFs controls the number of VFs that are visible.

And one can not play with NumVFs dynamically:

	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
	Set, the results are undefined.

one has to kill all VFs then enable them all again.
As long as we are doing that, we can just unload the driver.

So I would say we should just ask that VF Enable is set
before poking at admin vq, and prohibit clearing VF Enable.


There's also the concern of VF migration under MRIOV where VFs might
not exit. I'm guessing
trying to include that right now is overthinking. So maybe
this should mention that VF Migration Capable should be clear.


I will add all this to the spec.


>     How about the converse?
> 
> 
>                 Each device within a group has a unique identifier. This identifier
>                 is the group member identifier.
> 
>                 Note: one can argue both ways whether the new device group handling
>                 functionality (this and following patches) is closer
>                 to a new device type or a new transport type.
> 
>                 However, I expect that we will add more features in the near future. To
>                 facilitate this as much as possible of the text is located in the new
>                 admin chapter.
> 
>                 I did my best to minimize transport-specific text.
> 
>                 Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>                 Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>                 ---
>                  admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>                  content.tex |  2 ++
>                  2 files changed, 51 insertions(+)
>                  create mode 100644 admin.tex
> 
>                 diff --git a/admin.tex b/admin.tex
>                 new file mode 100644
>                 index 0000000..2bc7322
>                 --- /dev/null
>                 +++ b/admin.tex
>                 @@ -0,0 +1,49 @@
>                 +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>                 +
>                 +It is occasionally useful to have a device control a group of
>                 +other devices. Terminology used in such cases:
>                 +
>                 +\begin{description}
>                 +\item[Device group]
>                 +        or just group, includes zero or more devices.
>                 +\item[Owner device]
>                 +        or owner, the device controlling the group.
>                 +\item[Member device]
>                 +        a device within a group. The owner device itself is not
>                 +       a member of the group.
>                 +\item[Member identifier]
>                 +        each member has this identifier, unique within the group
>                 +       and used to address it through the owner device.
>                 +\item[Group type identifier]
>                 +       specifies what kind of member devices there are in a
>                 +       group, how is the member identifier is interpreted
>                 +       and what kind of control the owner has.
>                 +       A given owner can control a single group of a given type,
>                 +       thus the type and the owner together identify the group.
>                 +       \footnote{Even though some group types only support
>                 +                       specific transports, group type identifiers
>                 +                       are global rather than transport-specific -
>                 +                       we don't expect a flood of new group types.}
>                 +\end{description}
>                 +
>                 +The following group types, and their identifiers, are currently specified):
>                 +\begin{description}
>                 +\item[SR-IOV group type (0x1)]
>                 +This device group has a PCI Single Root I/O Virtualization
>                 +(SR-IOV) physical function (PF) device as the owner and includes
>                 +all its SR-IOV virtual functions (VFs) as members (see
>                 +\hyperref[intro:PCIe]{[PCIe]}).
>                 +
>                 +The PF device itself is not a member of the group.
>                 +
>                 +The group type identifier for this group is 0x1.
>                 +
>                 +A member identifier for this group can have a value 0x1 to 0xFFFF
>                 +and equals the SR-IOV VF number of the member device (see
>                 +\hyperref[intro:PCIe]{[PCIe]}).
>                 +
>                 +Both owner and member devices for this group type use the Virtio
>                 +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>                 +\end{description}
>                 +
>                 +
>                 diff --git a/content.tex b/content.tex
>                 index 0c2d917..ffe45c4 100644
>                 --- a/content.tex
>                 +++ b/content.tex
>                 @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>                  types. It is RECOMMENDED that devices generate version 4
>                  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> 
>                 +\input{admin.tex}
>                 +
>                  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> 
>                  We start with an overview of device initialization, then expand on the
> 
>             --
>             Tell her I'll be waiting, in the usual place.
> 

Re: [virtio-comment] [PATCH v10 03/10] admin: introduce group administration commands

[Zhu Lingshan]

On 2/9/2023 8:13 PM, Michael S. Tsirkin wrote:
> This introduces a general structure for group administration commands,
> used to control device groups through their owner.
>
> Following patches will introduce specific commands and an interface for
> submitting these commands to the owner.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   admin.tex | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 101 insertions(+)
>
> diff --git a/admin.tex b/admin.tex
> index 2bc7322..46240f8 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -46,4 +46,105 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
>   PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>   \end{description}
>   
> +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>   
> +Group administration commands can be issued through an owner
> +device to control member devices of a group.  This mechanism can
> +be used, for example, to configure a member device before it is
> +initialized by its driver.
> +\footnote{The term "administration" is intended to be interpreted
> +widely to include any kind of control. See specific commands
> +for detail.}
> +
> +All the group administration commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        le16 opcode;
> +        /*
> +         * 1 - SR-IOV
> +         * 2 - 65535 reserved
> +         */
> +        le16 group_type;
> +        /* unused, reserved for future extensions */
> +        u8 reserved1[12];
> +        le64 group_member_id;
> +        u8 command_specific_data[];
> +
> +        /* Device-writable part */
> +        le16 status;
> +        le16 status_qualifier;
> +        /* unused, reserved for future extensions */
> +        u8 reserved2[4];
> +        u8 command_specific_result[];
> +};
> +\end{lstlisting}
> +
> +For all commands, \field{opcode}, \field{group_type} and if
> +necessary \field{group_member_id} and \field{command_specific_data} are
> +set by the driver, and the owner device sets \field{status} and if
> +needed \field{status_qualifier} and
> +\field{command_specific_result}.
> +
> +As a rule, any unused device-readable fields are set to zero by the driver
> +and ignored by the device.  Any unused device-writeable fields are set to zero
> +by the device and ignored by the driver.
> +
> +\field{opcode} specifies the command. The valid
> +values for \field{opcode} can be found in the following table:
> +
> +\begin{tabular}{|l|l|}
> +\hline
> +opcode & Name & Command Description \\
> +\hline \hline
> +0x0000 - 0x7FFF & - &  Group administration commands    \\
> +\hline
> +0x8000 - 0xFFFF & - & Reserved    \\
> +\hline
> +\end{tabular}
> +
> +The \field{group_type} specifies the group type identifier.
Hi,

group_type here is used to address a device group, and we don't have a
group_id, so this implies each type of devices belong to only one group,
like there can be only one SRIOV group. Shall we add such description in
patch 2 explicitly?

And there is another concern, if we use this type of command to create a 
SRIOV
VF, do we need to sync with native PCI SRIOV?

Thanks
> +The \field{group_member_id} specifies the member identifier within the group.
> +See section \ref{sec:Introduction / Terminology / Device group}
> +for the definition of the group type identifier and group member
> +identifier.
> +
> +The following table describes possible \field{status} values;
> +to simplify common implementations, they are intentionally
> +matching common Linux names and error numbers:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status (decimal) & Name & Description \\
> +\hline \hline
> +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> +\hline
> +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> +\hline
> +other   & -    & group administration command error  \\
> +\hline
> +\end{tabular}
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> +is reserved and set to zero by the device.
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
> +the following table describes possible \field{status_qialifier} values:
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status & Name & Description \\
> +\hline \hline
> +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
> +\hline
> +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
> +\hline
> +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
> +\hline
> +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
> +\hline
> +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEM    & unsupported or invalid \field{group_member_id} \\
> +\hline
> +0x05-0xFFFF   & -    & reserved for future use \\
> +\hline
> +\end{tabular}


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [PATCH v10 09/10] admin: conformance clauses

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM

[..]
> +>>>>>>> 29f6525... admin: conformance clauses

Merge conflict left over to remove.

RE: [PATCH v10 02/10] admin: introduce device group and related concepts

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM
> 
> Each device group has a type. For now, define one initial group:

> +The following group types, and their identifiers, are currently specified):
Above "):" is that needed?

> +\begin{description}
> +\item[SR-IOV group type (0x1)]
Above 0x1 is duplicate of below.
Better to mention at single place.
> +This device group has a PCI Single Root I/O Virtualization
> +(SR-IOV) physical function (PF) device as the owner and includes all
> +its SR-IOV virtual functions (VFs) as members (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +The PF device itself is not a member of the group.
> +
> +The group type identifier for this group is 0x1.
> +
Current spce for device ID style is better than writing above lines repeatedly.
Something like,

\subsection{Group Type identifier }\label{sec: ... /Group Type Identifier}
1

RE: [PATCH v10 00/10] Introduce device group and device management

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:13 AM

[..]

> My intent is to try and support both SR-IOV and SIOV usecases with the same
> structure and maybe even the same VQ.
> 
Probably it is implicit, but is better for us to mention few examples use cases here something like,

Admin queue is an infrastructure to support use cases like:
a. SIOV devices life cycles management (create, destroy, query, configure)
b. SR-IOV VF device/resource management (SR-PCIM)
c. May be transport

> For example, it might make sense to split creating/destroying SIOV devices
> from the transport passing data from the guest - the driver would then not
> negotiate VIRTIO_F_SR_IOV (which then means auto-provisioning).

Above intermixing of SIOV and SR_IOV is very confusing.
Can you please drop it and have simpler examples like above?

[virtio-dev] RE: [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM
> 
> Feature negotiation forms the basis of forward compatibility guarantees of
> virtio but has never been properly documented.
> Do it now.
> 
> Suggested-by: Halil Pasic <pasic@linux.ibm.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 0e474dd..0c2d917 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> Virtio Device / Feature B  In particular, new fields in the device configuration
> space are  indicated by offering a new feature bit.
> 
> +To keep the feature negotiation mechanism extensible, it is important
> +that devices \em{do not} offer any feature bits that they would not be
> +able to handle if the driver accepted them (even though drivers are not
> +supposed to accept them in the first place even if offered, according
> +to this version of the specification.) Likewise, it is important that
> +drivers \em{do not} accept feature bits they do not know how to handle
> +(even though devices are not supposed to offer them in the first place,
> +according to this version of the specification.) The preferred way for
> +handling reserved and unexpected features is that the driver ignores
> +them.
> +
> +In particular, this is
> +especially important for features limited to specific transports, as
> +enabling these for more transports in future versions of the
> +specification is highly likely to require changing the behaviour from
> +drivers and devices.  Drivers and devices supporting multiple
> +transports need to carefully maintain per-transport lists of allowed
> +features.
> +
>  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
> Feature Bits}  The driver MUST NOT accept a feature which the device did not
> offer,  and MUST NOT accept a feature which requires another feature which
> was  not accepted.
> 
> +The driver MUST validate the feature bits offered by the device.
> +The driver MUST ignore and MUST NOT accept any feature bit that is
> +\begin{itemize} \item not described in this specification, \item marked
> +as reserved, \item not supported for the specific transport, \item not
> +defined for the device type.
> +\end{itemize}
> +
>  The driver SHOULD go into backwards compatibility mode  if the device does
> not offer a feature it understands, otherwise MUST  set the FAILED \field{device
> status} bit and cease initialization.
> 
> +By contrast, the driver MUST NOT fail solely because a feature it does
> +not understand has been offered by the device.
> +
>  \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
> Feature Bits}  The device MUST NOT offer a feature which requires another
> feature  which was not offered.  The device SHOULD accept any valid subset  of
> features the driver accepts, otherwise it MUST fail to set the  FEATURES_OK
> \field{device status} bit when the driver writes it.
> 
> +The device MUST NOT offer feature bits corresponding to features it
> +would not support if accepted by the driver (even if the driver is
> +prohibited from accepting the feature bits by the specification); for
> +the sake of clarity, this refers to feature bits not described in this
> +specification, reserved feature bits and feature bits reserved or not
> +supported for the specific transport or the specific device type, but
> +this does not preclude devices written to a future version of this
> +specification from offering such feature bits should such a
> +specification have a provision for devices to support the corresponding
> +features.
> +
>  If a device has successfully negotiated a set of features  at least once (by
> accepting the FEATURES_OK \field{device  status} bit during device
> initialization), then it SHOULD
> --
> MST

Reviewed-by: Parav Pandit <parav@nvidia.com>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

RE: [PATCH v10 02/10] admin: introduce device group and related concepts

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM

[..]

> +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device /
> +Device groups}
> +
> +It is occasionally useful to have a device control a group of other
> +devices. Terminology used in such cases:
> +
> +\begin{description}
> +\item[Device group]
> +        or just group, includes zero or more devices.
> +\item[Owner device]
> +        or owner, the device controlling the group.
> +\item[Member device]
> +        a device within a group. The owner device itself is not
> +	a member of the group.
> +\item[Member identifier]
> +        each member has this identifier, unique within the group
> +	and used to address it through the owner device.
> +\item[Group type identifier]
> +	specifies what kind of member devices there are in a
> +	group, how is the member identifier is interpreted
> +	and what kind of control the owner has.
> +	A given owner can control a single group of a given type,
> +	thus the type and the owner together identify the group.
> +	\footnote{Even though some group types only support
> +			specific transports, group type identifiers
> +			are global rather than transport-specific -
> +			we don't expect a flood of new group types.}
> \end{description}
> +
> +The following group types, and their identifiers, are currently specified):
> +\begin{description}
> +\item[SR-IOV group type (0x1)]
> +This device group has a PCI Single Root I/O Virtualization
> +(SR-IOV) physical function (PF) device as the owner and includes all
> +its SR-IOV virtual functions (VFs) as members (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +The PF device itself is not a member of the group.
> +
We already mention PC as owner and in 'Owner' definition it is clarifies above line.
It duplicate info, may be can be removed, but doesn't hurt much.

> +The group type identifier for this group is 0x1.
> +
> +A member identifier for this group can have a value 0x1 to 0xFFFF and
> +equals the SR-IOV VF number of the member device (see
> +\hyperref[intro:PCIe]{[PCIe]}).
> +
> +Both owner and member devices for this group type use the Virtio PCI
> +transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> +\end{description}
> +
> +
> diff --git a/content.tex b/content.tex
> index 0c2d917..ffe45c4 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of
> a Virtio Device / Expo  types. It is RECOMMENDED that devices generate
> version 4  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> 
> +\input{admin.tex}
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General
> Initialization And Device Operation}
> 
>  We start with an overview of device initialization, then expand on the
> --
> MST

RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM

[..]

> +Group administration commands can be issued through an owner device to
> +control member devices of a group.  
can be issued by the owner device to control ...

> This mechanism can be used, for
> +example, to configure a member device before it is initialized by its
> +driver.
> +\footnote{The term "administration" is intended to be interpreted
> +widely to include any kind of control. See specific commands for
> +detail.}
> +
> +All the group administration commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        le16 opcode;
> +        /*
> +         * 1 - SR-IOV
> +         * 2 - 65535 reserved
> +         */
> +        le16 group_type;
> +        /* unused, reserved for future extensions */
> +        u8 reserved1[12];
> +        le64 group_member_id;
> +        u8 command_specific_data[];
> +
> +        /* Device-writable part */
> +        le16 status;
> +        le16 status_qualifier;
> +        /* unused, reserved for future extensions */
> +        u8 reserved2[4];
> +        u8 command_specific_result[];
> +};
> +\end{lstlisting}
> +
> +For all commands, \field{opcode}, \field{group_type} and if necessary
> +\field{group_member_id} and \field{command_specific_data} are set by
> +the driver, and the owner device sets \field{status} and if needed
> +\field{status_qualifier} and \field{command_specific_result}.
> +
> +As a rule, any unused device-readable fields are set to zero by the
> +driver and ignored by the device.  Any unused device-writeable fields
> +are set to zero by the device and ignored by the driver.
> +
General spec structure is describing the "rules" as device and driver side requirements.
Can you please move above rule paragraph belongs to device and driver requirements section?

> +\field{opcode} specifies the command. The valid values for
> +\field{opcode} can be found in the following table:
> +
> +\begin{tabular}{|l|l|}
> +\hline
> +opcode & Name & Command Description \\
> +\hline \hline
> +0x0000 - 0x7FFF & - &  Group administration commands    \\
> +\hline
> +0x8000 - 0xFFFF & - & Reserved    \\
> +\hline
> +\end{tabular}
> +
> +The \field{group_type} specifies the group type identifier.
> +The \field{group_member_id} specifies the member identifier within the
> group.
> +See section \ref{sec:Introduction / Terminology / Device group} for the
> +definition of the group type identifier and group member identifier.
> +
> +The following table describes possible \field{status} values; to
> +simplify common implementations, they are intentionally matching common
> +Linux names and error numbers:
> +
I am not sure it should be intentional and mention of Linux.
Any OS would need multiple error values to know the error cause.
Linux doesn't have name "OK" either for well-known value of 0.

For example
$ errno 22
EINVAL 22 Invalid argument

$ errno 0
This has not output for it and doesn't exist in errno.h.

Description is already good enough to describe what they are.
Can we please drop Linux wording?

> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status (decimal) & Name & Description \\ \hline \hline
> +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> +\hline
> +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> +\hline
> +other   & -    & group administration command error  \\
> +\hline
> +\end{tabular}
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
s/status_qialifier/status_qualifier

> +is reserved and set to zero by the device.
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> +describes possible \field{status_qialifier} values:
s/status_qialifier/status_qualifier

RE: [PATCH v10 04/10] admin: introduce virtio admin virtqueues

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM
> +
> +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
An owner device supports one or more..

> +or more adminstration virtqueues. The number and locations of the
> +administration virtqueues are exposed by the owner device in a
> +transport specific manner.
> +
> +The driver submits commands by queueing them to an arbitrary
Do you prefer 'driver adds commands' instead of two verbs of 'submitting' and 'queueuing', like we discussed in other thread for enqueuing/post vs add?

I prefer the driver enqueues command and device process them in order.

> +administration virtqueue, and they are processed by the device in the
> +order in which they are queued. It is the responsibility of the driver
> +to ensure ordering for commands placed on different administration
ordering of commands reads better than ordering for commands to me.

> +virtqueues, because they will be executed with no order constraints.
> +
> +Administration virtqueues are used as follows:
Does it have to be plural to describe an operation?

> +\begin{itemize}
> +\item The driver submits the command using the \field{struct
> +virtio_admin_cmd} structure using a buffer consisting of two parts: a
> +device-readable one followed by a device-writable one.
> +\item the device-readable part includes fields from \field{opcode}
> +through \field{command_specific_data}.
> +\item the device-writeable buffer includes fields from \field{status}
> +through \field{command_specific_result} inclusive.
> +\end{itemize}
> +
> +For each command, this specification describes a distinct format
> +structure used for \field{command_specific_data} and
> +\field{command_specific_result}, the length of these fields depends on
> +the command.
> +
> +However, to ensure forward compatibility \begin{itemize} \item drivers
> +are allowed to submit buffers that are longer than what the device
> +expects (that is, longer than the length of \field{opcode} through
> +\field{command_specific_data}).
> +This allows the driver to maintain
> +a single format structure even if some structure fields are unused by
> +the device.
> +\item drivers are allowed to submit buffers that are shorter than what
> +the device expects (that is, shorter than the length of \field{status}
> +through \field{command_specific_result}). This allows the device to
> +maintain a single format structure even if some structure fields are
> +unused by the driver.
> +\end{itemize}
> +
> +The device compares the length of each part (device-readable and
> +device-writeable) of the buffer as submitted by driver to what it
> +expects and then silently truncates the structures to either the length
> +submitted by the driver, or the length described in this specification,
> +whichever is shorter.  The device silently ignores any data falling
> +outside the shorter of the two lengths. Any missing fields are
> +interpreted as set to zero.
> +
> +Similarly, the driver compares the used buffer length of the buffer to
> +what it expects and then silently truncates the structure to the used
> +buffer length.
> +The driver silently ignores any data falling outside the used buffer
> +length reported by the device.  Any missing fields are interpreted as
> +set to zero.
Above buffer size description is very well written.
I don't remember this in earlier version. May be was there.
Thanks.

RE: [PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM

> diff --git a/content.tex b/content.tex
> index c8647c9..2d9b903 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -946,6 +946,10 @@ \subsubsection{Common configuration structure
> layout}\label{sec:Virtio Transport
>          le64 queue_device;              /* read-write */
>          le16 queue_notify_data;         /* read-only for driver */
>          le16 queue_reset;               /* read-write */
> +
> +        /* About the administration virtqueue. */
> +        le16 admin_queue_index;         /* read-only for driver */
> +        le16 admin_queue_num;         /* read-only for driver */
>  };
>  \end{lstlisting}
> 
> @@ -1031,6 +1035,19 @@ \subsubsection{Common configuration structure
> layout}\label{sec:Virtio Transport
>          This field exists only if VIRTIO_F_RING_RESET has been
>          negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> Virtqueue Reset}).
> 
> +\item[\field{admin_queue_index}]
> +        The device uses this to report the index of the first administration
> virtqueue.
> +        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
> +\item[\field{admin_queue_num}]
> +	The device uses this to report the number of the
> +	supported administration virtqueues.  
admin_queue_count reads more intuitive to me to indicate how many aqs.
It may be even simpler to have start and end vq numbers rather than complication of index + inclusive sum of number.

> Virtqueues with index
> +	between \field{admin_queue_index} and (\field{admin_queue_index} +
> +	\field{admin_queue_num}) inclusive serve as administration
> +	virtqueues.
> +	Thus the number of administration virtqueues equals
> +	(\field{admin_queue_num} + 1).
> +	This field is valid only if VIRTIO_F_ADMIN_VQ has been
> +	negotiated.
>  \end{description}
> 
>  \devicenormative{\paragraph}{Common configuration structure layout}{Virtio
> Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common
> configuration structure layout} @@ -1117,6 +1134,14 @@
> \subsubsection{Common configuration structure layout}\label{sec:Virtio
> Transport  were used before the queue reset.
>  (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
> 
> +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver configures
> +any administration virtqueues, the driver MUST configure the
> +administration virtqueues using the index in the range
> +\field{admin_queue_index} to \field{admin_queue_index} +
> +\field{admin_queue_num} inclusive.
> +The driver MAY configure less administration virtqueues than supported
> +by the device.
> +
>  \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options
> / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
> 
>  The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG @@
> -7684,6 +7709,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> Feature Bits}
> 
>    \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device
> exposes one or more
>    administration virtqueues.
> +  At the moment this feature is only supported for devices using
> +  \ref{sec:Virtio Transport Options / Virtio Over PCI
> +	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
> +	  as the transport and is reserved for future use for
> +	  devices using other transports (see
> +	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
> +	and
> +	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
> for
> +	handling features reserved for future use.
> 
>  \end{description}
> 
> --
> MST
Reviewed-by: Parav Pandit <parav@nvidia.com>

RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM
> 
> Adding relevant registers needs more work and it's not clear what the use-case
> will be as currently only the PCI transport is supported. But let's keep the door
> open on this.
> We already say it's reserved in a central place, but it does not hurt to remind
> implementers to mask it.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 2d9b903..f8bd1ab 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2362,6 +2362,18 @@ \subsection{Legacy interface}\label{sec:Virtio
> Transport Options / Virtio Over M
> 
>  Notification mechanisms did not change.
> 
> +\subsection{Features reserved for future use}\label{sec:Virtio
> +Transport Options / Virtio Over MMIO / Features reserved for future
> +use}
> +
> +At this time, devices and drivers utilizing Virtio Over MMIO do not
> +support the following features:
> +\begin{itemize}
> +
> +\item VIRTIO_F_ADMIN_VQ
> +
> +\end{itemize}
> +
> +These features are reserved for future use.
> +
>  \section{Virtio Over Channel I/O}\label{sec:Virtio Transport Options / Virtio
> Over Channel I/O}
> 
>  S/390 based virtual machines support neither PCI nor MMIO, so a
> --
> MST

Reviewed-by: Parav Pandit <parav@nvidia.com>

RE: [PATCH v10 08/10] admin: command list discovery

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM

[..]
> +This structure is an array of 64 bit values in little-endian byte
LE is already in the above data type. It can be dropped from the description.

> +order, in which a bit is set if the specific command opcode is
> +supported. Thus, \field{device_admin_cmds[0]} refers to the first
> +32-bit value in this array corresponding to opcodes 0 to 63,
s/32-bit/64-bit

Re: [PATCH v10 00/10] Introduce device group and device management

[Michael S. Tsirkin]

On Sat, Feb 11, 2023 at 06:49:52PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, February 9, 2023 7:13 AM
> 
> [..]
> 
> > My intent is to try and support both SR-IOV and SIOV usecases with the same
> > structure and maybe even the same VQ.
> > 
> Probably it is implicit, but is better for us to mention few examples use cases here something like,
> 
> Admin queue is an infrastructure to support use cases like:
> a. SIOV devices life cycles management (create, destroy, query, configure)
> b. SR-IOV VF device/resource management (SR-PCIM)
> c. May be transport
> 
> > For example, it might make sense to split creating/destroying SIOV devices
> > from the transport passing data from the guest - the driver would then not
> > negotiate VIRTIO_F_SR_IOV (which then means auto-provisioning).
> 
> Above intermixing of SIOV and SR_IOV is very confusing.
> Can you please drop it and have simpler examples like above?

You mean in the cover letter? The more info in the cover
letter the better, this stuff does not end up in git.
I'll add your examples too though, thanks!

-- 
MST

Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Sat, Feb 11, 2023 at 06:50:37PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, February 9, 2023 7:14 AM
> 
> [..]
> 
> > +Group administration commands can be issued through an owner device to
> > +control member devices of a group.  
> can be issued by the owner device to control ...

Well it's clearly the driver that issues the commands. Don't see how
can a device issue them. If instead of "through an owner device"
you prefer a more verbose "the driver of an owner device" then
I can live with that. Let me know.

> > This mechanism can be used, for
> > +example, to configure a member device before it is initialized by its
> > +driver.
> > +\footnote{The term "administration" is intended to be interpreted
> > +widely to include any kind of control. See specific commands for
> > +detail.}
> > +
> > +All the group administration commands are of the following form:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd {
> > +        /* Device-readable part */
> > +        le16 opcode;
> > +        /*
> > +         * 1 - SR-IOV
> > +         * 2 - 65535 reserved
> > +         */
> > +        le16 group_type;
> > +        /* unused, reserved for future extensions */
> > +        u8 reserved1[12];
> > +        le64 group_member_id;
> > +        u8 command_specific_data[];
> > +
> > +        /* Device-writable part */
> > +        le16 status;
> > +        le16 status_qualifier;
> > +        /* unused, reserved for future extensions */
> > +        u8 reserved2[4];
> > +        u8 command_specific_result[];
> > +};
> > +\end{lstlisting}
> > +
> > +For all commands, \field{opcode}, \field{group_type} and if necessary
> > +\field{group_member_id} and \field{command_specific_data} are set by
> > +the driver, and the owner device sets \field{status} and if needed
> > +\field{status_qualifier} and \field{command_specific_result}.
> > +
> > +As a rule, any unused device-readable fields are set to zero by the
> > +driver and ignored by the device.  Any unused device-writeable fields
> > +are set to zero by the device and ignored by the driver.
> > +
> General spec structure is describing the "rules" as device and driver side requirements.
> Can you please move above rule paragraph belongs to device and driver requirements section?

Generally we have a bit of duplication. A high level description
showing how things work together. Then per-device/driver conforumance
statements. So I think this should stay but I will look at adding
conformance clauses too.

> > +\field{opcode} specifies the command. The valid values for
> > +\field{opcode} can be found in the following table:
> > +
> > +\begin{tabular}{|l|l|}
> > +\hline
> > +opcode & Name & Command Description \\
> > +\hline \hline
> > +0x0000 - 0x7FFF & - &  Group administration commands    \\
> > +\hline
> > +0x8000 - 0xFFFF & - & Reserved    \\
> > +\hline
> > +\end{tabular}
> > +
> > +The \field{group_type} specifies the group type identifier.
> > +The \field{group_member_id} specifies the member identifier within the
> > group.
> > +See section \ref{sec:Introduction / Terminology / Device group} for the
> > +definition of the group type identifier and group member identifier.
> > +
> > +The following table describes possible \field{status} values; to
> > +simplify common implementations, they are intentionally matching common
> > +Linux names and error numbers:
> > +
> I am not sure it should be intentional and mention of Linux.
> Any OS would need multiple error values to know the error cause.
> Linux doesn't have name "OK" either for well-known value of 0.
> 
> For example
> $ errno 22
> EINVAL 22 Invalid argument
> 
> $ errno 0
> This has not output for it and doesn't exist in errno.h.

No name OK true, but it does use 0 to signal success sometimes:

https://man7.org/linux/man-pages/man3/errno.3.html

For some system calls and library functions (e.g.,
       getpriority(2)), -1 is a valid return on success.  In such cases,
       a successful return can be distinguished from an error return by
       setting errno to zero before the call, and then, if the call
       returns a status that indicates that an error may have occurred,
       checking to see if errno has a nonzero value.



> Description is already good enough to describe what they are.
> Can we please drop Linux wording?

But why should we? It's where 22 comes from so this way people are not
wondering about the value, and it's somewhat helpful for Linux
developers.


> > +\begin{tabular}{|l|l|l|}
> > +\hline
> > +Status (decimal) & Name & Description \\ \hline \hline
> > +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> > +\hline
> > +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> > +\hline
> > +other   & -    & group administration command error  \\
> > +\hline
> > +\end{tabular}
> > +
> > +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> s/status_qialifier/status_qualifier
> 
> > +is reserved and set to zero by the device.
> > +
> > +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
> > +describes possible \field{status_qialifier} values:
> s/status_qialifier/status_qualifier


Thanks!

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Max Gurtovoy]

On 09/02/2023 21:58, Michael S. Tsirkin wrote:
> On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
>>
>> On 09/02/2023 17:22, David Edmondson wrote:
>>
>>      On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
>>
>>          On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
>>
>>              On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
>>
>>                  Each device group has a type. For now, define one initial group:
>>
>>                  SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>>                  PCI SR-IOV physical function (PF). This group may contain one or more
>>                  virtio devices.
>>
>>              The specification says zero or more generally and nothing to refine that
>>              for SR-IOV groups.
>>
>>          Well it says:
>>
>>                  (SR-IOV) physical function (PF) device as the owner and includes
>>                  all its SR-IOV virtual functions (VFs) as members (see
>>                  \hyperref[intro:PCIe]{[PCIe]}).
>>
>>          how can that be zero?
>>
>>      If I remove all of the VFs of a PF, does the group implicitly disappear?
>>
>> Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
> 
> Are you sure? Not by VF Enable?

Right. It's a combination of the VF_Enable and numVfs.

...
"The NumVFs field defines the number of VFs that are enabled when VF 
Enable is Set in the associated PF."

> 
>> We can mention that a group include all virtual functions indicated by TotalVFs
>> RO register.
> 
> TotalVFs indicates the maximum number of VFs that could be associated with the PF
> but not all of these exist. If we use this we get into issues of
> discovering whether VFs can actually be used. I'd rather avoid that.
> 
> 
>> This group exist as long as the owner PF supports SR-IOV and has admin
>> capabilities.
> 
> SRIOV spec says:
> 	NumVFs controls the number of VFs that are visible.
> 
> And one can not play with NumVFs dynamically:
> 
> 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
> 	Set, the results are undefined.
> 
> one has to kill all VFs then enable them all again.
> As long as we are doing that, we can just unload the driver.
what driver to unload ? Not sure I'm following..

> 
> So I would say we should just ask that VF Enable is set
> before poking at admin vq, and prohibit clearing VF Enable.
> 

Sending admin commands shouldn't be restricted and should be allowed 
always. Device should react to incoming command.

We can say that an SR-IOV group is empty while the VF enable is unset 
and is not empty when it's set. The size of the non empty SR-IOV group 
is equal to the numVFs and the VF index is equal to the member 
identifier in this group. I had this definition in my patches.

Sending admin command to a non existing member will result in an ERROR 
VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.

> 
> There's also the concern of VF migration under MRIOV where VFs might
> not exit. I'm guessing
> trying to include that right now is overthinking. So maybe
> this should mention that VF Migration Capable should be clear.
> 
> 
> I will add all this to the spec.
> 
> 
>>      How about the converse?
>>
>>
>>                  Each device within a group has a unique identifier. This identifier
>>                  is the group member identifier.
>>
>>                  Note: one can argue both ways whether the new device group handling
>>                  functionality (this and following patches) is closer
>>                  to a new device type or a new transport type.
>>
>>                  However, I expect that we will add more features in the near future. To
>>                  facilitate this as much as possible of the text is located in the new
>>                  admin chapter.
>>
>>                  I did my best to minimize transport-specific text.
>>
>>                  Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>                  Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>                  ---
>>                   admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>>                   content.tex |  2 ++
>>                   2 files changed, 51 insertions(+)
>>                   create mode 100644 admin.tex
>>
>>                  diff --git a/admin.tex b/admin.tex
>>                  new file mode 100644
>>                  index 0000000..2bc7322
>>                  --- /dev/null
>>                  +++ b/admin.tex
>>                  @@ -0,0 +1,49 @@
>>                  +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>>                  +
>>                  +It is occasionally useful to have a device control a group of
>>                  +other devices. Terminology used in such cases:
>>                  +
>>                  +\begin{description}
>>                  +\item[Device group]
>>                  +        or just group, includes zero or more devices.
>>                  +\item[Owner device]
>>                  +        or owner, the device controlling the group.
>>                  +\item[Member device]
>>                  +        a device within a group. The owner device itself is not
>>                  +       a member of the group.
>>                  +\item[Member identifier]
>>                  +        each member has this identifier, unique within the group
>>                  +       and used to address it through the owner device.
>>                  +\item[Group type identifier]
>>                  +       specifies what kind of member devices there are in a
>>                  +       group, how is the member identifier is interpreted
>>                  +       and what kind of control the owner has.
>>                  +       A given owner can control a single group of a given type,
>>                  +       thus the type and the owner together identify the group.
>>                  +       \footnote{Even though some group types only support
>>                  +                       specific transports, group type identifiers
>>                  +                       are global rather than transport-specific -
>>                  +                       we don't expect a flood of new group types.}
>>                  +\end{description}
>>                  +
>>                  +The following group types, and their identifiers, are currently specified):
>>                  +\begin{description}
>>                  +\item[SR-IOV group type (0x1)]
>>                  +This device group has a PCI Single Root I/O Virtualization
>>                  +(SR-IOV) physical function (PF) device as the owner and includes
>>                  +all its SR-IOV virtual functions (VFs) as members (see
>>                  +\hyperref[intro:PCIe]{[PCIe]}).
>>                  +
>>                  +The PF device itself is not a member of the group.
>>                  +
>>                  +The group type identifier for this group is 0x1.
>>                  +
>>                  +A member identifier for this group can have a value 0x1 to 0xFFFF
>>                  +and equals the SR-IOV VF number of the member device (see
>>                  +\hyperref[intro:PCIe]{[PCIe]}).
>>                  +
>>                  +Both owner and member devices for this group type use the Virtio
>>                  +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>>                  +\end{description}
>>                  +
>>                  +
>>                  diff --git a/content.tex b/content.tex
>>                  index 0c2d917..ffe45c4 100644
>>                  --- a/content.tex
>>                  +++ b/content.tex
>>                  @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>                   types. It is RECOMMENDED that devices generate version 4
>>                   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>
>>                  +\input{admin.tex}
>>                  +
>>                   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>
>>                   We start with an overview of device initialization, then expand on the
>>
>>              --
>>              Tell her I'll be waiting, in the usual place.
>>
> 

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
> 
> 
> On 09/02/2023 21:58, Michael S. Tsirkin wrote:
> > On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
> > > 
> > > On 09/02/2023 17:22, David Edmondson wrote:
> > > 
> > >      On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
> > > 
> > >          On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
> > > 
> > >              On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
> > > 
> > >                  Each device group has a type. For now, define one initial group:
> > > 
> > >                  SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
> > >                  PCI SR-IOV physical function (PF). This group may contain one or more
> > >                  virtio devices.
> > > 
> > >              The specification says zero or more generally and nothing to refine that
> > >              for SR-IOV groups.
> > > 
> > >          Well it says:
> > > 
> > >                  (SR-IOV) physical function (PF) device as the owner and includes
> > >                  all its SR-IOV virtual functions (VFs) as members (see
> > >                  \hyperref[intro:PCIe]{[PCIe]}).
> > > 
> > >          how can that be zero?
> > > 
> > >      If I remove all of the VFs of a PF, does the group implicitly disappear?
> > > 
> > > Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
> > 
> > Are you sure? Not by VF Enable?
> 
> Right. It's a combination of the VF_Enable and numVfs.
> 
> ...
> "The NumVFs field defines the number of VFs that are enabled when VF Enable
> is Set in the associated PF."
> 
> > 
> > > We can mention that a group include all virtual functions indicated by TotalVFs
> > > RO register.
> > 
> > TotalVFs indicates the maximum number of VFs that could be associated with the PF
> > but not all of these exist. If we use this we get into issues of
> > discovering whether VFs can actually be used. I'd rather avoid that.
> > 
> > 
> > > This group exist as long as the owner PF supports SR-IOV and has admin
> > > capabilities.
> > 
> > SRIOV spec says:
> > 	NumVFs controls the number of VFs that are visible.
> > 
> > And one can not play with NumVFs dynamically:
> > 
> > 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
> > 	Set, the results are undefined.
> > 
> > one has to kill all VFs then enable them all again.
> > As long as we are doing that, we can just unload the driver.
> what driver to unload ? Not sure I'm following..

Basically I am saying that if VFs are disabled this destroys all VFs
so it is not
clear that it makes sense to keep some of the configuration
for previous VFs around.

My idea is to say that reset is required to disable VFs.

> > 
> > So I would say we should just ask that VF Enable is set
> > before poking at admin vq, and prohibit clearing VF Enable.
> > 
> 
> Sending admin commands shouldn't be restricted and should be allowed always.
> Device should react to incoming command.
> 
> We can say that an SR-IOV group is empty while the VF enable is unset and is
> not empty when it's set. The size of the non empty SR-IOV group is equal to
> the numVFs and the VF index is equal to the member identifier in this group.
> I had this definition in my patches.
> 
> Sending admin command to a non existing member will result in an ERROR
> VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.


just an example:
- assign resource X to VF 16
- set NumVFs to 8

Does VF 16 keep resource X? I note that
resource X can not be taken back because VF 16
is not accessible.


For sure, all this is solvable but I'd rather just forbid
changes to VF enable for now.


> 
> > 
> > There's also the concern of VF migration under MRIOV where VFs might
> > not exit. I'm guessing
> > trying to include that right now is overthinking. So maybe
> > this should mention that VF Migration Capable should be clear.
> > 
> > 
> > I will add all this to the spec.
> > 
> > 
> > >      How about the converse?
> > > 
> > > 
> > >                  Each device within a group has a unique identifier. This identifier
> > >                  is the group member identifier.
> > > 
> > >                  Note: one can argue both ways whether the new device group handling
> > >                  functionality (this and following patches) is closer
> > >                  to a new device type or a new transport type.
> > > 
> > >                  However, I expect that we will add more features in the near future. To
> > >                  facilitate this as much as possible of the text is located in the new
> > >                  admin chapter.
> > > 
> > >                  I did my best to minimize transport-specific text.
> > > 
> > >                  Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > >                  Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > >                  ---
> > >                   admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
> > >                   content.tex |  2 ++
> > >                   2 files changed, 51 insertions(+)
> > >                   create mode 100644 admin.tex
> > > 
> > >                  diff --git a/admin.tex b/admin.tex
> > >                  new file mode 100644
> > >                  index 0000000..2bc7322
> > >                  --- /dev/null
> > >                  +++ b/admin.tex
> > >                  @@ -0,0 +1,49 @@
> > >                  +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
> > >                  +
> > >                  +It is occasionally useful to have a device control a group of
> > >                  +other devices. Terminology used in such cases:
> > >                  +
> > >                  +\begin{description}
> > >                  +\item[Device group]
> > >                  +        or just group, includes zero or more devices.
> > >                  +\item[Owner device]
> > >                  +        or owner, the device controlling the group.
> > >                  +\item[Member device]
> > >                  +        a device within a group. The owner device itself is not
> > >                  +       a member of the group.
> > >                  +\item[Member identifier]
> > >                  +        each member has this identifier, unique within the group
> > >                  +       and used to address it through the owner device.
> > >                  +\item[Group type identifier]
> > >                  +       specifies what kind of member devices there are in a
> > >                  +       group, how is the member identifier is interpreted
> > >                  +       and what kind of control the owner has.
> > >                  +       A given owner can control a single group of a given type,
> > >                  +       thus the type and the owner together identify the group.
> > >                  +       \footnote{Even though some group types only support
> > >                  +                       specific transports, group type identifiers
> > >                  +                       are global rather than transport-specific -
> > >                  +                       we don't expect a flood of new group types.}
> > >                  +\end{description}
> > >                  +
> > >                  +The following group types, and their identifiers, are currently specified):
> > >                  +\begin{description}
> > >                  +\item[SR-IOV group type (0x1)]
> > >                  +This device group has a PCI Single Root I/O Virtualization
> > >                  +(SR-IOV) physical function (PF) device as the owner and includes
> > >                  +all its SR-IOV virtual functions (VFs) as members (see
> > >                  +\hyperref[intro:PCIe]{[PCIe]}).
> > >                  +
> > >                  +The PF device itself is not a member of the group.
> > >                  +
> > >                  +The group type identifier for this group is 0x1.
> > >                  +
> > >                  +A member identifier for this group can have a value 0x1 to 0xFFFF
> > >                  +and equals the SR-IOV VF number of the member device (see
> > >                  +\hyperref[intro:PCIe]{[PCIe]}).
> > >                  +
> > >                  +Both owner and member devices for this group type use the Virtio
> > >                  +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > >                  +\end{description}
> > >                  +
> > >                  +
> > >                  diff --git a/content.tex b/content.tex
> > >                  index 0c2d917..ffe45c4 100644
> > >                  --- a/content.tex
> > >                  +++ b/content.tex
> > >                  @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> > >                   types. It is RECOMMENDED that devices generate version 4
> > >                   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > > 
> > >                  +\input{admin.tex}
> > >                  +
> > >                   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> > > 
> > >                   We start with an overview of device initialization, then expand on the
> > > 
> > >              --
> > >              Tell her I'll be waiting, in the usual place.
> > > 
> > 

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Max Gurtovoy]

On 12/02/2023 15:15, Michael S. Tsirkin wrote:
> On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
>>
>>
>> On 09/02/2023 21:58, Michael S. Tsirkin wrote:
>>> On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
>>>>
>>>> On 09/02/2023 17:22, David Edmondson wrote:
>>>>
>>>>       On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
>>>>
>>>>           On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
>>>>
>>>>               On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
>>>>
>>>>                   Each device group has a type. For now, define one initial group:
>>>>
>>>>                   SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>>>>                   PCI SR-IOV physical function (PF). This group may contain one or more
>>>>                   virtio devices.
>>>>
>>>>               The specification says zero or more generally and nothing to refine that
>>>>               for SR-IOV groups.
>>>>
>>>>           Well it says:
>>>>
>>>>                   (SR-IOV) physical function (PF) device as the owner and includes
>>>>                   all its SR-IOV virtual functions (VFs) as members (see
>>>>                   \hyperref[intro:PCIe]{[PCIe]}).
>>>>
>>>>           how can that be zero?
>>>>
>>>>       If I remove all of the VFs of a PF, does the group implicitly disappear?
>>>>
>>>> Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
>>>
>>> Are you sure? Not by VF Enable?
>>
>> Right. It's a combination of the VF_Enable and numVfs.
>>
>> ...
>> "The NumVFs field defines the number of VFs that are enabled when VF Enable
>> is Set in the associated PF."
>>
>>>
>>>> We can mention that a group include all virtual functions indicated by TotalVFs
>>>> RO register.
>>>
>>> TotalVFs indicates the maximum number of VFs that could be associated with the PF
>>> but not all of these exist. If we use this we get into issues of
>>> discovering whether VFs can actually be used. I'd rather avoid that.
>>>
>>>
>>>> This group exist as long as the owner PF supports SR-IOV and has admin
>>>> capabilities.
>>>
>>> SRIOV spec says:
>>> 	NumVFs controls the number of VFs that are visible.
>>>
>>> And one can not play with NumVFs dynamically:
>>>
>>> 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
>>> 	Set, the results are undefined.
>>>
>>> one has to kill all VFs then enable them all again.
>>> As long as we are doing that, we can just unload the driver.
>> what driver to unload ? Not sure I'm following..
> 
> Basically I am saying that if VFs are disabled this destroys all VFs
> so it is not
> clear that it makes sense to keep some of the configuration
> for previous VFs around.
> 
> My idea is to say that reset is required to disable VFs.

which reset ?

VFs disabled/enabled using VF_enable PCI register.
Lets follow that please.

Lets define what is happening during VF_enable = 1 and VF_enable = 0.

On VF_enable = 0 --> 1 transition the corresponding device_group, that 
is owned by the PF, grows and contains NumVFs members with initial 
features and resources.

On VF_enable = 1 --> 0 transition the corresponding device_group, that 
is owned by the PF, shrinks and contains 0 members and all the dynamic 
resources are back to the owner.

> 
>>>
>>> So I would say we should just ask that VF Enable is set
>>> before poking at admin vq, and prohibit clearing VF Enable.
>>>
>>
>> Sending admin commands shouldn't be restricted and should be allowed always.
>> Device should react to incoming command.
>>
>> We can say that an SR-IOV group is empty while the VF enable is unset and is
>> not empty when it's set. The size of the non empty SR-IOV group is equal to
>> the numVFs and the VF index is equal to the member identifier in this group.
>> I had this definition in my patches.
>>
>> Sending admin command to a non existing member will result in an ERROR
>> VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
> 
> 
> just an example:
> - assign resource X to VF 16
> - set NumVFs to 8
> 
> Does VF 16 keep resource X? I note that
> resource X can not be taken back because VF 16
> is not accessible.

all dynamic resources should return to the owner when VF_enable moves 1 
--> 0

> 
> 
> For sure, all this is solvable but I'd rather just forbid
> changes to VF enable for now.

how we can forbid it ? setting this register is not related to virtio.

> 
> 
>>
>>>
>>> There's also the concern of VF migration under MRIOV where VFs might
>>> not exit. I'm guessing
>>> trying to include that right now is overthinking. So maybe
>>> this should mention that VF Migration Capable should be clear.
>>>
>>>
>>> I will add all this to the spec.
>>>
>>>
>>>>       How about the converse?
>>>>
>>>>
>>>>                   Each device within a group has a unique identifier. This identifier
>>>>                   is the group member identifier.
>>>>
>>>>                   Note: one can argue both ways whether the new device group handling
>>>>                   functionality (this and following patches) is closer
>>>>                   to a new device type or a new transport type.
>>>>
>>>>                   However, I expect that we will add more features in the near future. To
>>>>                   facilitate this as much as possible of the text is located in the new
>>>>                   admin chapter.
>>>>
>>>>                   I did my best to minimize transport-specific text.
>>>>
>>>>                   Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>                   Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>>                   ---
>>>>                    admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>                    content.tex |  2 ++
>>>>                    2 files changed, 51 insertions(+)
>>>>                    create mode 100644 admin.tex
>>>>
>>>>                   diff --git a/admin.tex b/admin.tex
>>>>                   new file mode 100644
>>>>                   index 0000000..2bc7322
>>>>                   --- /dev/null
>>>>                   +++ b/admin.tex
>>>>                   @@ -0,0 +1,49 @@
>>>>                   +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>>>>                   +
>>>>                   +It is occasionally useful to have a device control a group of
>>>>                   +other devices. Terminology used in such cases:
>>>>                   +
>>>>                   +\begin{description}
>>>>                   +\item[Device group]
>>>>                   +        or just group, includes zero or more devices.
>>>>                   +\item[Owner device]
>>>>                   +        or owner, the device controlling the group.
>>>>                   +\item[Member device]
>>>>                   +        a device within a group. The owner device itself is not
>>>>                   +       a member of the group.
>>>>                   +\item[Member identifier]
>>>>                   +        each member has this identifier, unique within the group
>>>>                   +       and used to address it through the owner device.
>>>>                   +\item[Group type identifier]
>>>>                   +       specifies what kind of member devices there are in a
>>>>                   +       group, how is the member identifier is interpreted
>>>>                   +       and what kind of control the owner has.
>>>>                   +       A given owner can control a single group of a given type,
>>>>                   +       thus the type and the owner together identify the group.
>>>>                   +       \footnote{Even though some group types only support
>>>>                   +                       specific transports, group type identifiers
>>>>                   +                       are global rather than transport-specific -
>>>>                   +                       we don't expect a flood of new group types.}
>>>>                   +\end{description}
>>>>                   +
>>>>                   +The following group types, and their identifiers, are currently specified):
>>>>                   +\begin{description}
>>>>                   +\item[SR-IOV group type (0x1)]
>>>>                   +This device group has a PCI Single Root I/O Virtualization
>>>>                   +(SR-IOV) physical function (PF) device as the owner and includes
>>>>                   +all its SR-IOV virtual functions (VFs) as members (see
>>>>                   +\hyperref[intro:PCIe]{[PCIe]}).
>>>>                   +
>>>>                   +The PF device itself is not a member of the group.
>>>>                   +
>>>>                   +The group type identifier for this group is 0x1.
>>>>                   +
>>>>                   +A member identifier for this group can have a value 0x1 to 0xFFFF
>>>>                   +and equals the SR-IOV VF number of the member device (see
>>>>                   +\hyperref[intro:PCIe]{[PCIe]}).
>>>>                   +
>>>>                   +Both owner and member devices for this group type use the Virtio
>>>>                   +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>>>>                   +\end{description}
>>>>                   +
>>>>                   +
>>>>                   diff --git a/content.tex b/content.tex
>>>>                   index 0c2d917..ffe45c4 100644
>>>>                   --- a/content.tex
>>>>                   +++ b/content.tex
>>>>                   @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>>>                    types. It is RECOMMENDED that devices generate version 4
>>>>                    UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>
>>>>                   +\input{admin.tex}
>>>>                   +
>>>>                    \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>>>
>>>>                    We start with an overview of device initialization, then expand on the
>>>>
>>>>               --
>>>>               Tell her I'll be waiting, in the usual place.
>>>>
>>>
> 

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Sun, Feb 12, 2023 at 04:34:09PM +0200, Max Gurtovoy wrote:
> 
> 
> On 12/02/2023 15:15, Michael S. Tsirkin wrote:
> > On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
> > > 
> > > 
> > > On 09/02/2023 21:58, Michael S. Tsirkin wrote:
> > > > On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
> > > > > 
> > > > > On 09/02/2023 17:22, David Edmondson wrote:
> > > > > 
> > > > >       On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
> > > > > 
> > > > >           On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
> > > > > 
> > > > >               On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
> > > > > 
> > > > >                   Each device group has a type. For now, define one initial group:
> > > > > 
> > > > >                   SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
> > > > >                   PCI SR-IOV physical function (PF). This group may contain one or more
> > > > >                   virtio devices.
> > > > > 
> > > > >               The specification says zero or more generally and nothing to refine that
> > > > >               for SR-IOV groups.
> > > > > 
> > > > >           Well it says:
> > > > > 
> > > > >                   (SR-IOV) physical function (PF) device as the owner and includes
> > > > >                   all its SR-IOV virtual functions (VFs) as members (see
> > > > >                   \hyperref[intro:PCIe]{[PCIe]}).
> > > > > 
> > > > >           how can that be zero?
> > > > > 
> > > > >       If I remove all of the VFs of a PF, does the group implicitly disappear?
> > > > > 
> > > > > Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
> > > > 
> > > > Are you sure? Not by VF Enable?
> > > 
> > > Right. It's a combination of the VF_Enable and numVfs.
> > > 
> > > ...
> > > "The NumVFs field defines the number of VFs that are enabled when VF Enable
> > > is Set in the associated PF."
> > > 
> > > > 
> > > > > We can mention that a group include all virtual functions indicated by TotalVFs
> > > > > RO register.
> > > > 
> > > > TotalVFs indicates the maximum number of VFs that could be associated with the PF
> > > > but not all of these exist. If we use this we get into issues of
> > > > discovering whether VFs can actually be used. I'd rather avoid that.
> > > > 
> > > > 
> > > > > This group exist as long as the owner PF supports SR-IOV and has admin
> > > > > capabilities.
> > > > 
> > > > SRIOV spec says:
> > > > 	NumVFs controls the number of VFs that are visible.
> > > > 
> > > > And one can not play with NumVFs dynamically:
> > > > 
> > > > 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
> > > > 	Set, the results are undefined.
> > > > 
> > > > one has to kill all VFs then enable them all again.
> > > > As long as we are doing that, we can just unload the driver.
> > > what driver to unload ? Not sure I'm following..
> > 
> > Basically I am saying that if VFs are disabled this destroys all VFs
> > so it is not
> > clear that it makes sense to keep some of the configuration
> > for previous VFs around.
> > 
> > My idea is to say that reset is required to disable VFs.
> 
> which reset ?
> 
> VFs disabled/enabled using VF_enable PCI register.
> Lets follow that please.
> 
> Lets define what is happening during VF_enable = 1 and VF_enable = 0.
> 
> On VF_enable = 0 --> 1 transition the corresponding device_group, that is
> owned by the PF, grows and contains NumVFs members with initial features and
> resources.
> 
> On VF_enable = 1 --> 0 transition the corresponding device_group, that is
> owned by the PF, shrinks and contains 0 members and all the dynamic
> resources are back to the owner.

Maybe, it's a possible approach. As current patch set has no
resources I feel we can leave this part to a patch on top
just so that this one can move forward while we have
a more specific example to work from.



> > 
> > > > 
> > > > So I would say we should just ask that VF Enable is set
> > > > before poking at admin vq, and prohibit clearing VF Enable.
> > > > 
> > > 
> > > Sending admin commands shouldn't be restricted and should be allowed always.
> > > Device should react to incoming command.
> > > 
> > > We can say that an SR-IOV group is empty while the VF enable is unset and is
> > > not empty when it's set. The size of the non empty SR-IOV group is equal to
> > > the numVFs and the VF index is equal to the member identifier in this group.
> > > I had this definition in my patches.
> > > 
> > > Sending admin command to a non existing member will result in an ERROR
> > > VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
> > 
> > 
> > just an example:
> > - assign resource X to VF 16
> > - set NumVFs to 8
> > 
> > Does VF 16 keep resource X? I note that
> > resource X can not be taken back because VF 16
> > is not accessible.
> 
> all dynamic resources should return to the owner when VF_enable moves 1 -->
> 0


It's a possible approach but my question would be whether we
need a transport independent way to do this through VQ.
And if we do it's annoying to have multiple ways.

> > 
> > 
> > For sure, all this is solvable but I'd rather just forbid
> > changes to VF enable for now.
> 
> how we can forbid it ? setting this register is not related to virtio.

We can ask driver to write 0 to device reset before changing VF
enable to 0. The advantage is that this kind of reset by definition
frees up all resources.


> > 
> > 
> > > 
> > > > 
> > > > There's also the concern of VF migration under MRIOV where VFs might
> > > > not exit. I'm guessing
> > > > trying to include that right now is overthinking. So maybe
> > > > this should mention that VF Migration Capable should be clear.
> > > > 
> > > > 
> > > > I will add all this to the spec.
> > > > 
> > > > 
> > > > >       How about the converse?
> > > > > 
> > > > > 
> > > > >                   Each device within a group has a unique identifier. This identifier
> > > > >                   is the group member identifier.
> > > > > 
> > > > >                   Note: one can argue both ways whether the new device group handling
> > > > >                   functionality (this and following patches) is closer
> > > > >                   to a new device type or a new transport type.
> > > > > 
> > > > >                   However, I expect that we will add more features in the near future. To
> > > > >                   facilitate this as much as possible of the text is located in the new
> > > > >                   admin chapter.
> > > > > 
> > > > >                   I did my best to minimize transport-specific text.
> > > > > 
> > > > >                   Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > >                   Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > >                   ---
> > > > >                    admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >                    content.tex |  2 ++
> > > > >                    2 files changed, 51 insertions(+)
> > > > >                    create mode 100644 admin.tex
> > > > > 
> > > > >                   diff --git a/admin.tex b/admin.tex
> > > > >                   new file mode 100644
> > > > >                   index 0000000..2bc7322
> > > > >                   --- /dev/null
> > > > >                   +++ b/admin.tex
> > > > >                   @@ -0,0 +1,49 @@
> > > > >                   +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
> > > > >                   +
> > > > >                   +It is occasionally useful to have a device control a group of
> > > > >                   +other devices. Terminology used in such cases:
> > > > >                   +
> > > > >                   +\begin{description}
> > > > >                   +\item[Device group]
> > > > >                   +        or just group, includes zero or more devices.
> > > > >                   +\item[Owner device]
> > > > >                   +        or owner, the device controlling the group.
> > > > >                   +\item[Member device]
> > > > >                   +        a device within a group. The owner device itself is not
> > > > >                   +       a member of the group.
> > > > >                   +\item[Member identifier]
> > > > >                   +        each member has this identifier, unique within the group
> > > > >                   +       and used to address it through the owner device.
> > > > >                   +\item[Group type identifier]
> > > > >                   +       specifies what kind of member devices there are in a
> > > > >                   +       group, how is the member identifier is interpreted
> > > > >                   +       and what kind of control the owner has.
> > > > >                   +       A given owner can control a single group of a given type,
> > > > >                   +       thus the type and the owner together identify the group.
> > > > >                   +       \footnote{Even though some group types only support
> > > > >                   +                       specific transports, group type identifiers
> > > > >                   +                       are global rather than transport-specific -
> > > > >                   +                       we don't expect a flood of new group types.}
> > > > >                   +\end{description}
> > > > >                   +
> > > > >                   +The following group types, and their identifiers, are currently specified):
> > > > >                   +\begin{description}
> > > > >                   +\item[SR-IOV group type (0x1)]
> > > > >                   +This device group has a PCI Single Root I/O Virtualization
> > > > >                   +(SR-IOV) physical function (PF) device as the owner and includes
> > > > >                   +all its SR-IOV virtual functions (VFs) as members (see
> > > > >                   +\hyperref[intro:PCIe]{[PCIe]}).
> > > > >                   +
> > > > >                   +The PF device itself is not a member of the group.
> > > > >                   +
> > > > >                   +The group type identifier for this group is 0x1.
> > > > >                   +
> > > > >                   +A member identifier for this group can have a value 0x1 to 0xFFFF
> > > > >                   +and equals the SR-IOV VF number of the member device (see
> > > > >                   +\hyperref[intro:PCIe]{[PCIe]}).
> > > > >                   +
> > > > >                   +Both owner and member devices for this group type use the Virtio
> > > > >                   +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > > > >                   +\end{description}
> > > > >                   +
> > > > >                   +
> > > > >                   diff --git a/content.tex b/content.tex
> > > > >                   index 0c2d917..ffe45c4 100644
> > > > >                   --- a/content.tex
> > > > >                   +++ b/content.tex
> > > > >                   @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> > > > >                    types. It is RECOMMENDED that devices generate version 4
> > > > >                    UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > > > > 
> > > > >                   +\input{admin.tex}
> > > > >                   +
> > > > >                    \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> > > > > 
> > > > >                    We start with an overview of device initialization, then expand on the
> > > > > 
> > > > >               --
> > > > >               Tell her I'll be waiting, in the usual place.
> > > > > 
> > > > 
> > 

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Max Gurtovoy]

On 12/02/2023 22:19, Michael S. Tsirkin wrote:
> On Sun, Feb 12, 2023 at 04:34:09PM +0200, Max Gurtovoy wrote:
>>
>>
>> On 12/02/2023 15:15, Michael S. Tsirkin wrote:
>>> On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
>>>>
>>>>
>>>> On 09/02/2023 21:58, Michael S. Tsirkin wrote:
>>>>> On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
>>>>>>
>>>>>> On 09/02/2023 17:22, David Edmondson wrote:
>>>>>>
>>>>>>        On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
>>>>>>
>>>>>>            On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
>>>>>>
>>>>>>                On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
>>>>>>
>>>>>>                    Each device group has a type. For now, define one initial group:
>>>>>>
>>>>>>                    SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>>>>>>                    PCI SR-IOV physical function (PF). This group may contain one or more
>>>>>>                    virtio devices.
>>>>>>
>>>>>>                The specification says zero or more generally and nothing to refine that
>>>>>>                for SR-IOV groups.
>>>>>>
>>>>>>            Well it says:
>>>>>>
>>>>>>                    (SR-IOV) physical function (PF) device as the owner and includes
>>>>>>                    all its SR-IOV virtual functions (VFs) as members (see
>>>>>>                    \hyperref[intro:PCIe]{[PCIe]}).
>>>>>>
>>>>>>            how can that be zero?
>>>>>>
>>>>>>        If I remove all of the VFs of a PF, does the group implicitly disappear?
>>>>>>
>>>>>> Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
>>>>>
>>>>> Are you sure? Not by VF Enable?
>>>>
>>>> Right. It's a combination of the VF_Enable and numVfs.
>>>>
>>>> ...
>>>> "The NumVFs field defines the number of VFs that are enabled when VF Enable
>>>> is Set in the associated PF."
>>>>
>>>>>
>>>>>> We can mention that a group include all virtual functions indicated by TotalVFs
>>>>>> RO register.
>>>>>
>>>>> TotalVFs indicates the maximum number of VFs that could be associated with the PF
>>>>> but not all of these exist. If we use this we get into issues of
>>>>> discovering whether VFs can actually be used. I'd rather avoid that.
>>>>>
>>>>>
>>>>>> This group exist as long as the owner PF supports SR-IOV and has admin
>>>>>> capabilities.
>>>>>
>>>>> SRIOV spec says:
>>>>> 	NumVFs controls the number of VFs that are visible.
>>>>>
>>>>> And one can not play with NumVFs dynamically:
>>>>>
>>>>> 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
>>>>> 	Set, the results are undefined.
>>>>>
>>>>> one has to kill all VFs then enable them all again.
>>>>> As long as we are doing that, we can just unload the driver.
>>>> what driver to unload ? Not sure I'm following..
>>>
>>> Basically I am saying that if VFs are disabled this destroys all VFs
>>> so it is not
>>> clear that it makes sense to keep some of the configuration
>>> for previous VFs around.
>>>
>>> My idea is to say that reset is required to disable VFs.
>>
>> which reset ?
>>
>> VFs disabled/enabled using VF_enable PCI register.
>> Lets follow that please.
>>
>> Lets define what is happening during VF_enable = 1 and VF_enable = 0.
>>
>> On VF_enable = 0 --> 1 transition the corresponding device_group, that is
>> owned by the PF, grows and contains NumVFs members with initial features and
>> resources.
>>
>> On VF_enable = 1 --> 0 transition the corresponding device_group, that is
>> owned by the PF, shrinks and contains 0 members and all the dynamic
>> resources are back to the owner.
> 
> Maybe, it's a possible approach. As current patch set has no
> resources I feel we can leave this part to a patch on top
> just so that this one can move forward while we have
> a more specific example to work from.

we can differ the resources part but lets add the part that the group 
becomes empty/non-empty according to the disable/enable of VF_enable 
register as mentioned above.

> 
> 
> 
>>>
>>>>>
>>>>> So I would say we should just ask that VF Enable is set
>>>>> before poking at admin vq, and prohibit clearing VF Enable.
>>>>>
>>>>
>>>> Sending admin commands shouldn't be restricted and should be allowed always.
>>>> Device should react to incoming command.
>>>>
>>>> We can say that an SR-IOV group is empty while the VF enable is unset and is
>>>> not empty when it's set. The size of the non empty SR-IOV group is equal to
>>>> the numVFs and the VF index is equal to the member identifier in this group.
>>>> I had this definition in my patches.
>>>>
>>>> Sending admin command to a non existing member will result in an ERROR
>>>> VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
>>>
>>>
>>> just an example:
>>> - assign resource X to VF 16
>>> - set NumVFs to 8
>>>
>>> Does VF 16 keep resource X? I note that
>>> resource X can not be taken back because VF 16
>>> is not accessible.
>>
>> all dynamic resources should return to the owner when VF_enable moves 1 -->
>> 0
> 
> 
> It's a possible approach but my question would be whether we
> need a transport independent way to do this through VQ.
> And if we do it's annoying to have multiple ways.

We are defining the existence of the SR-IOV group.

Also, the approach of sending commands to a non-existing member is valid 
for any group type and should return error 
VIRTIO_ADMIN_STATUS_Q_INVALID_MEM in that case.

why do we need more ways ?

> 
>>>
>>>
>>> For sure, all this is solvable but I'd rather just forbid
>>> changes to VF enable for now.
>>
>> how we can forbid it ? setting this register is not related to virtio.
> 
> We can ask driver to write 0 to device reset before changing VF
> enable to 0. The advantage is that this kind of reset by definition
> frees up all resources.

why do we need to reset the device ? It is fully functional and should 
be able to run perfectly fine without any need to reset.

The PF device might have a very big role in the hypervisor, so 
restricting it in this way doesn't sounds mandatory.

Controlling the SR-IOV group using SR-IOV register enable/disable and 
NumVFs used as the number of the group size sounds simple from Spec and 
implementation POV.

> 
> 
>>>
>>>
>>>>
>>>>>
>>>>> There's also the concern of VF migration under MRIOV where VFs might
>>>>> not exit. I'm guessing
>>>>> trying to include that right now is overthinking. So maybe
>>>>> this should mention that VF Migration Capable should be clear.
>>>>>
>>>>>
>>>>> I will add all this to the spec.
>>>>>
>>>>>
>>>>>>        How about the converse?
>>>>>>
>>>>>>
>>>>>>                    Each device within a group has a unique identifier. This identifier
>>>>>>                    is the group member identifier.
>>>>>>
>>>>>>                    Note: one can argue both ways whether the new device group handling
>>>>>>                    functionality (this and following patches) is closer
>>>>>>                    to a new device type or a new transport type.
>>>>>>
>>>>>>                    However, I expect that we will add more features in the near future. To
>>>>>>                    facilitate this as much as possible of the text is located in the new
>>>>>>                    admin chapter.
>>>>>>
>>>>>>                    I did my best to minimize transport-specific text.
>>>>>>
>>>>>>                    Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>>                    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>>>>                    ---
>>>>>>                     admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>                     content.tex |  2 ++
>>>>>>                     2 files changed, 51 insertions(+)
>>>>>>                     create mode 100644 admin.tex
>>>>>>
>>>>>>                    diff --git a/admin.tex b/admin.tex
>>>>>>                    new file mode 100644
>>>>>>                    index 0000000..2bc7322
>>>>>>                    --- /dev/null
>>>>>>                    +++ b/admin.tex
>>>>>>                    @@ -0,0 +1,49 @@
>>>>>>                    +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>>>>>>                    +
>>>>>>                    +It is occasionally useful to have a device control a group of
>>>>>>                    +other devices. Terminology used in such cases:
>>>>>>                    +
>>>>>>                    +\begin{description}
>>>>>>                    +\item[Device group]
>>>>>>                    +        or just group, includes zero or more devices.
>>>>>>                    +\item[Owner device]
>>>>>>                    +        or owner, the device controlling the group.
>>>>>>                    +\item[Member device]
>>>>>>                    +        a device within a group. The owner device itself is not
>>>>>>                    +       a member of the group.
>>>>>>                    +\item[Member identifier]
>>>>>>                    +        each member has this identifier, unique within the group
>>>>>>                    +       and used to address it through the owner device.
>>>>>>                    +\item[Group type identifier]
>>>>>>                    +       specifies what kind of member devices there are in a
>>>>>>                    +       group, how is the member identifier is interpreted
>>>>>>                    +       and what kind of control the owner has.
>>>>>>                    +       A given owner can control a single group of a given type,
>>>>>>                    +       thus the type and the owner together identify the group.
>>>>>>                    +       \footnote{Even though some group types only support
>>>>>>                    +                       specific transports, group type identifiers
>>>>>>                    +                       are global rather than transport-specific -
>>>>>>                    +                       we don't expect a flood of new group types.}
>>>>>>                    +\end{description}
>>>>>>                    +
>>>>>>                    +The following group types, and their identifiers, are currently specified):
>>>>>>                    +\begin{description}
>>>>>>                    +\item[SR-IOV group type (0x1)]
>>>>>>                    +This device group has a PCI Single Root I/O Virtualization
>>>>>>                    +(SR-IOV) physical function (PF) device as the owner and includes
>>>>>>                    +all its SR-IOV virtual functions (VFs) as members (see
>>>>>>                    +\hyperref[intro:PCIe]{[PCIe]}).
>>>>>>                    +
>>>>>>                    +The PF device itself is not a member of the group.
>>>>>>                    +
>>>>>>                    +The group type identifier for this group is 0x1.
>>>>>>                    +
>>>>>>                    +A member identifier for this group can have a value 0x1 to 0xFFFF
>>>>>>                    +and equals the SR-IOV VF number of the member device (see
>>>>>>                    +\hyperref[intro:PCIe]{[PCIe]}).
>>>>>>                    +
>>>>>>                    +Both owner and member devices for this group type use the Virtio
>>>>>>                    +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>>>>>>                    +\end{description}
>>>>>>                    +
>>>>>>                    +
>>>>>>                    diff --git a/content.tex b/content.tex
>>>>>>                    index 0c2d917..ffe45c4 100644
>>>>>>                    --- a/content.tex
>>>>>>                    +++ b/content.tex
>>>>>>                    @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>>>>>                     types. It is RECOMMENDED that devices generate version 4
>>>>>>                     UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>>>
>>>>>>                    +\input{admin.tex}
>>>>>>                    +
>>>>>>                     \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>>>>>
>>>>>>                     We start with an overview of device initialization, then expand on the
>>>>>>
>>>>>>                --
>>>>>>                Tell her I'll be waiting, in the usual place.
>>>>>>
>>>>>
>>>
> 

Re: [PATCH v10 03/10] admin: introduce group administration commands

[Max Gurtovoy]

On 12/02/2023 11:49, Michael S. Tsirkin wrote:
> On Sat, Feb 11, 2023 at 06:50:37PM +0000, Parav Pandit wrote:
>>
>>> From: Michael S. Tsirkin <mst@redhat.com>
>>> Sent: Thursday, February 9, 2023 7:14 AM
>>
>> [..]
>>
>>> +Group administration commands can be issued through an owner device to
>>> +control member devices of a group.
>> can be issued by the owner device to control ...
> 
> Well it's clearly the driver that issues the commands. Don't see how
> can a device issue them. If instead of "through an owner device"
> you prefer a more verbose "the driver of an owner device" then
> I can live with that. Let me know.
> 
>>> This mechanism can be used, for
>>> +example, to configure a member device before it is initialized by its
>>> +driver.
>>> +\footnote{The term "administration" is intended to be interpreted
>>> +widely to include any kind of control. See specific commands for
>>> +detail.}
>>> +
>>> +All the group administration commands are of the following form:
>>> +
>>> +\begin{lstlisting}
>>> +struct virtio_admin_cmd {
>>> +        /* Device-readable part */
>>> +        le16 opcode;
>>> +        /*
>>> +         * 1 - SR-IOV
>>> +         * 2 - 65535 reserved
>>> +         */
>>> +        le16 group_type;
>>> +        /* unused, reserved for future extensions */
>>> +        u8 reserved1[12];
>>> +        le64 group_member_id;
>>> +        u8 command_specific_data[];
>>> +
>>> +        /* Device-writable part */
>>> +        le16 status;
>>> +        le16 status_qualifier;
>>> +        /* unused, reserved for future extensions */
>>> +        u8 reserved2[4];
>>> +        u8 command_specific_result[];
>>> +};
>>> +\end{lstlisting}
>>> +
>>> +For all commands, \field{opcode}, \field{group_type} and if necessary
>>> +\field{group_member_id} and \field{command_specific_data} are set by
>>> +the driver, and the owner device sets \field{status} and if needed
>>> +\field{status_qualifier} and \field{command_specific_result}.
>>> +
>>> +As a rule, any unused device-readable fields are set to zero by the
>>> +driver and ignored by the device.  Any unused device-writeable fields
>>> +are set to zero by the device and ignored by the driver.
>>> +
>> General spec structure is describing the "rules" as device and driver side requirements.
>> Can you please move above rule paragraph belongs to device and driver requirements section?
> 
> Generally we have a bit of duplication. A high level description
> showing how things work together. Then per-device/driver conforumance
> statements. So I think this should stay but I will look at adding
> conformance clauses too.
> 
>>> +\field{opcode} specifies the command. The valid values for
>>> +\field{opcode} can be found in the following table:
>>> +
>>> +\begin{tabular}{|l|l|}
>>> +\hline
>>> +opcode & Name & Command Description \\
>>> +\hline \hline
>>> +0x0000 - 0x7FFF & - &  Group administration commands    \\
>>> +\hline
>>> +0x8000 - 0xFFFF & - & Reserved    \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +The \field{group_type} specifies the group type identifier.
>>> +The \field{group_member_id} specifies the member identifier within the
>>> group.
>>> +See section \ref{sec:Introduction / Terminology / Device group} for the
>>> +definition of the group type identifier and group member identifier.
>>> +
>>> +The following table describes possible \field{status} values; to
>>> +simplify common implementations, they are intentionally matching common
>>> +Linux names and error numbers:
>>> +
>> I am not sure it should be intentional and mention of Linux.
>> Any OS would need multiple error values to know the error cause.
>> Linux doesn't have name "OK" either for well-known value of 0.
>>
>> For example
>> $ errno 22
>> EINVAL 22 Invalid argument
>>
>> $ errno 0
>> This has not output for it and doesn't exist in errno.h.
> 
> No name OK true, but it does use 0 to signal success sometimes:
> 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fman7.org%2Flinux%2Fman-pages%2Fman3%2Ferrno.3.html&data=05%7C01%7Cmgurtovoy%40nvidia.com%7Cbe231a7ec67b432a259d08db0cde7e10%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638117921964323355%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4pRK48Hlc%2BTaMAOd8dx1B0jjNf7gvkqJ8SSCMhZVv0o%3D&reserved=0
> 
> For some system calls and library functions (e.g.,
>         getpriority(2)), -1 is a valid return on success.  In such cases,
>         a successful return can be distinguished from an error return by
>         setting errno to zero before the call, and then, if the call
>         returns a status that indicates that an error may have occurred,
>         checking to see if errno has a nonzero value.
> 
> 
> 
>> Description is already good enough to describe what they are.
>> Can we please drop Linux wording?
> 
> But why should we? It's where 22 comes from so this way people are not
> wondering about the value, and it's somewhat helpful for Linux
> developers.
> 

I also think we should not mention Linux. I don't think it's mentioned 
currently in the spec and no good reason to do so now.
Also value of 22 is not mandatory for this EINVAL status code. It can be 
just 1 (the first number after the OK status).
Developers should be able to read the specification and find out the 
meaning of error code.

> 
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Status (decimal) & Name & Description \\ \hline \hline
>>> +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
>>> +\hline
>>> +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
>>> +\hline
>>> +other   & -    & group administration command error  \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
>> s/status_qialifier/status_qualifier
>>
>>> +is reserved and set to zero by the device.
>>> +
>>> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table
>>> +describes possible \field{status_qialifier} values:
>> s/status_qialifier/status_qualifier
> 
> 
> Thanks!
> 

Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery

[Zhu Lingshan]

On 2/9/2023 8:14 PM, Michael S. Tsirkin wrote:
> Add commands to find out which commands does each group support,
> as well as enable their use by driver.
> This will be especially useful once we have multiple group types.
>
> An alternative is per-type VQs. This is possible but will
> require more per-transport work. Discovery through the vq
> helps keep things contained.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   admin.tex | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 89 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 3026a79..4de606a 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -98,7 +98,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   \hline
>   opcode & Name & Command Description \\
>   \hline \hline
> -0x0000 - 0x7FFF & - &  Group administration commands    \\
> +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> +0x0002 - 0x7FFF & - &  Group administration commands    \\
>   \hline
>   0x8000 - 0xFFFF & - & Reserved    \\
>   \hline
> @@ -149,6 +151,92 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   \hline
>   \end{tabular}
>   
> +Before sending any administration commands to the device, the driver
> +needs to communicate to the device which commands it is going to
> +use. Initially (after reset), only two commands are assumed to be used:
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Before sending any other commands for any member of a specific group to
> +the device, the driver queries the supported commands via
> +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
> +VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE
> +both use the following structure describing the
> +command opcodes:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_list {
> +       /* Indicates which of the below fields were returned
> +       le64 device_admin_cmds[];
> +};
> +\end{lstlisting}
> +
> +This structure is an array of 64 bit values in little-endian byte
> +order, in which a bit is set if the specific command opcode
> +is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value
64-bit? And is an LE64 enough for now and near future? Or the the length 
needs to be
calculated from the descriptor length.
> +in this array corresponding to opcodes 0 to 63,
> +\field{device_admin_cmds[1]} is the second 64-bit value
> +corresponding to opcodes 64 to 127, etc.
> +For example, the array of size 2 including
> +the values 0x3 in \field{device_admin_cmds[0]}
> +and 0x1 in \field{device_admin_cmds[1]} indicates that only
> +opcodes 0, 1 and 64 are supported.
> +The length of the array depends on the supported opcodes - it is
> +large enough to include bits set for all supported opcodes,
> +that is the length can be calculated by starting with the largest
> +supported opcode adding one, dividing by 64 and rounding up.
> +The array is also allowed to be larger and to additionally
> +include an arbitrary number of all-zero entries.
> +
> +Accordingly, bits 0 and 1 corresponding to opcode 0
> +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
> +(VIRTIO_ADMIN_CMD_LIST_USE) are
> +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +This command has no command specific data.
> +The device, upon success, returns a result in
> +\field{command_specific_result} in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands supported for the given group.
> +
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
> +is set to 0x1.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +The \field{command_specific_data} is in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands used by the driver.
> +This command has no command specific result.
> +
> +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> +query the list of commands valid for this group and before sending
> +any commands for any member of a group.
> +
> +The driver then enables use of some of the opcodes by sending to
> +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
> +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
> +both understood and used by the driver.
> +
> +If the device supports the command list used by the driver, the
> +device completes the command with status VIRTIO_ADMIN_STATUS_OK.
> +If the device does not support the command list, the device
> +completes the command with status
> +VIRTIO_ADMIN_STATUS_INVALID_FIELD.
> +
> +Note: drivers are assumed not to set bits in device_admin_cmds
> +if they are not familiar with how the command opcode
> +is used, since devices could have dependencies between
> +command opcodes.
> +
> +It is assumed that all members in a group support and are used
> +with the same list of commands. However, for owner devices
> +supporting multiple group types, the list of supported commands
> +might differ between different group types.
> +
>   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>   
>   An administration virtqueue of an owner device is used to submit

[virtio-comment] Re: [virtio-dev] [PATCH v10 09/10] admin: conformance clauses

[Zhu Lingshan]

On 2/9/2023 8:14 PM, Michael S. Tsirkin wrote:
> Add conformance clauses for admin commands and admin virtqueues.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   admin.tex | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 190 insertions(+)
>
> diff --git a/admin.tex b/admin.tex
> index 4de606a..f9b1b68 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -237,6 +237,119 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   supporting multiple group types, the list of supported commands
>   might differ between different group types.
>   
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The device MUST validate \field{opcode}, \field{group_type} and
> +\field{group_member_id}, and if any of these has an invalid or
> +unsupported value, set \field{status} to
> +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
> +accordingly:
> +\begin{itemize}
> +\item if \field{group_type} is invalid, \field{status_qualifier}
> +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> +\item otherwise, if \field{opcode} is invalid,
> +	\field{status_qualifier} MUST be set to
> +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
> +\item otherwise, if \field{group_member_id} is used by the
> +	specific command and is invalid, \field{status_qualifier} MUST be
> +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
> +\end{itemize}
> +
> +If a command completes successfully, the device MUST set
> +\field{status} to VIRTIO_ADMIN_STATUS_OK.
> +
> +If a command fails, the device MUST set
> +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
> +
> +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
> +device state MUST NOT change, that is the command MUST NOT have
> +any side effects on the device, in particular the device MUST not
> +enter an error state as a result of this command.
> +
> +If a command fails, the device state generally SHOULD NOT change,
> +as far as possible.
> +
> +The device MAY enforce additional restrictions and dependencies on
> +opcodes used by the driver and MAY fail the command
> +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> +if the list of commands used violate internal device dependencies.
> +
> +If the device supports multiple group types, commands for each group
> +type MUST operate independently of each other, in particular,
> +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
> +for different group types.
> +
> +After reset, if the device supports a given group type
> +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
> +the device MUST assume
> +that the list of legal commands used by the driver consists of
> +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
> +the device MUST set the list of legal commands used by the driver
> +to the one supplied in \field{command_specific_data}.
> +
> +The device MUST set the list of legal commands used by the driver
> +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +The device MUST validate commands against the list used by
> +the driver and MUST fail any commands not in the list with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
> +
> +The list of supported commands MUST NOT shrink (but MAY expand):
> +after reporting a given command as supported through
> +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> +as unsupported.  Further, after a given set of commands has been
> +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
> +device or system reset the device SHOULD complete successfully
> +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
> +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
> +after a device or system reset, the device MUST not fail it
> +solely because of the command list used.  Failure to do so would
> +interfere with resuming from suspend and error recovery.
> +
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The driver MAY discover whether device supports a specific group type
> +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
> +\field{group_type}.
> +
> +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
> +and wait for it to be completed with status
> +VIRTIO_ADMIN_STATUS_OK before issuing any commands
> +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
> +and VIRTIO_ADMIN_CMD_LIST_USE).
> +
> +The driver SHOULD NOT set bits in device_admin_cmds
> +if it is not familiar with how the command opcode
> +is used, as dependencies between command opcodes might exist.
> +
> +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
> +the use of any commands not previously reported as
> +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +The driver MUST NOT use any commands for a given group type
> +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
> +list of command opcodes and group type.
> +
> +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> +with respective bits cleared in \field{command_specific_data}.
> +
> +The driver MUST handle a command error with a reserved \field{status}
> +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +(except possibly for different error reporting/diagnostic messages).
> +
> +The driver MUST handle a command error with a reserved
> +\field{status_qualifier} value in the same way as
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
> +different error reporting/diagnostic messages).
> +
>   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>   
>   An administration virtqueue of an owner device is used to submit
> @@ -309,3 +422,80 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
>   of the specification are designed, new fields can be added to the
>   tail of a structure, with the driver/device using the full
>   structure without concern for versioning.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The device MUST support device-readable and device-writeable buffers
> +shorter than described in this specification, by
> +\begin{enumerate}
> +\item acting as if any data that would be read outside the
> +device-readable buffers is set to zero, and
> +\item discarding data that would be written outside the
> +specified device-writeable buffers.
> +\end{enumerate}
> +
> +The device MUST support device-readable and device-writeable buffers
> +longer than described in this specification, by
maybe I missed it, where defines the length? Does it refer to the 
descriptor length?
> +\begin{enumerate}
> +\item ignoring any data in device-readable buffers outside
> +the expected length, and
> +\item only writing the expected structure to the device-writeable
> +buffers, ignoring any extra buffers, and reporting the
> +actual length of data written, in bytes,
> +as buffer used length.
> +\end{enumerate}
> +
> +The device SHOULD initialize the device-writeable buffer
> +up to the length of the structure described by this specification or
also this length of the structure
> +the length of the buffer supplied by the driver (even if the buffer is
> +all set to zero), whichever is shorter.
> +
> +The device MUST NOT fail a command solely because the buffers
> +provided are shorter or longer than described in this
> +specification.
> +
> +The device MUST initialize the device-writeable part of
> +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
> +size.
> +
> +The device MUST initialize \field{status} in \field{struct
> +virtio_admin_cmd}.
> +
> +The device MUST process commands on a given administration virtqueue
> +in the order in which they are queued.
> +
> +If multiple administration virtqueues have been configured,
> +device MAY process commands on distinct virtqueues with
> +no order constraints.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The driver MAY supply device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are longer than described in
> +this specification.
> +
> +The driver SHOULD supply device-readable part of
> +\field{struct virtio_admin_cmd} that is at least as
> +large as the structure described by this specification
> +(even if the structure is all set to zero).
> +
> +The driver MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
> +in length.
> +
> +The device MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are larger than zero in
> +length. However, \field{command_specific_data} and
> +\field{command_specific_result} MAY be zero in length, unless
> +specified otherwise for the command.
> +
> +The driver MUST NOT assume that the device will initialize the whole
> +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
> +the driver MUST act as if the structure
> +outside the part of the buffer used by the device
> +is set to zero.
> +
> +If multiple administration virtqueues have been configured,
> +the driver MUST ensure ordering for commands
> +placed on different administration virtqueues.
> +>>>>>>> 29f6525... admin: conformance clauses


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Mon, Feb 13, 2023 at 12:49:44AM +0200, Max Gurtovoy wrote:
> 
> 
> On 12/02/2023 22:19, Michael S. Tsirkin wrote:
> > On Sun, Feb 12, 2023 at 04:34:09PM +0200, Max Gurtovoy wrote:
> > > 
> > > 
> > > On 12/02/2023 15:15, Michael S. Tsirkin wrote:
> > > > On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
> > > > > 
> > > > > 
> > > > > On 09/02/2023 21:58, Michael S. Tsirkin wrote:
> > > > > > On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
> > > > > > > 
> > > > > > > On 09/02/2023 17:22, David Edmondson wrote:
> > > > > > > 
> > > > > > >        On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
> > > > > > > 
> > > > > > >            On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
> > > > > > > 
> > > > > > >                On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
> > > > > > > 
> > > > > > >                    Each device group has a type. For now, define one initial group:
> > > > > > > 
> > > > > > >                    SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
> > > > > > >                    PCI SR-IOV physical function (PF). This group may contain one or more
> > > > > > >                    virtio devices.
> > > > > > > 
> > > > > > >                The specification says zero or more generally and nothing to refine that
> > > > > > >                for SR-IOV groups.
> > > > > > > 
> > > > > > >            Well it says:
> > > > > > > 
> > > > > > >                    (SR-IOV) physical function (PF) device as the owner and includes
> > > > > > >                    all its SR-IOV virtual functions (VFs) as members (see
> > > > > > >                    \hyperref[intro:PCIe]{[PCIe]}).
> > > > > > > 
> > > > > > >            how can that be zero?
> > > > > > > 
> > > > > > >        If I remove all of the VFs of a PF, does the group implicitly disappear?
> > > > > > > 
> > > > > > > Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
> > > > > > 
> > > > > > Are you sure? Not by VF Enable?
> > > > > 
> > > > > Right. It's a combination of the VF_Enable and numVfs.
> > > > > 
> > > > > ...
> > > > > "The NumVFs field defines the number of VFs that are enabled when VF Enable
> > > > > is Set in the associated PF."
> > > > > 
> > > > > > 
> > > > > > > We can mention that a group include all virtual functions indicated by TotalVFs
> > > > > > > RO register.
> > > > > > 
> > > > > > TotalVFs indicates the maximum number of VFs that could be associated with the PF
> > > > > > but not all of these exist. If we use this we get into issues of
> > > > > > discovering whether VFs can actually be used. I'd rather avoid that.
> > > > > > 
> > > > > > 
> > > > > > > This group exist as long as the owner PF supports SR-IOV and has admin
> > > > > > > capabilities.
> > > > > > 
> > > > > > SRIOV spec says:
> > > > > > 	NumVFs controls the number of VFs that are visible.
> > > > > > 
> > > > > > And one can not play with NumVFs dynamically:
> > > > > > 
> > > > > > 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
> > > > > > 	Set, the results are undefined.
> > > > > > 
> > > > > > one has to kill all VFs then enable them all again.
> > > > > > As long as we are doing that, we can just unload the driver.
> > > > > what driver to unload ? Not sure I'm following..
> > > > 
> > > > Basically I am saying that if VFs are disabled this destroys all VFs
> > > > so it is not
> > > > clear that it makes sense to keep some of the configuration
> > > > for previous VFs around.
> > > > 
> > > > My idea is to say that reset is required to disable VFs.
> > > 
> > > which reset ?
> > > 
> > > VFs disabled/enabled using VF_enable PCI register.
> > > Lets follow that please.
> > > 
> > > Lets define what is happening during VF_enable = 1 and VF_enable = 0.
> > > 
> > > On VF_enable = 0 --> 1 transition the corresponding device_group, that is
> > > owned by the PF, grows and contains NumVFs members with initial features and
> > > resources.
> > > 
> > > On VF_enable = 1 --> 0 transition the corresponding device_group, that is
> > > owned by the PF, shrinks and contains 0 members and all the dynamic
> > > resources are back to the owner.
> > 
> > Maybe, it's a possible approach. As current patch set has no
> > resources I feel we can leave this part to a patch on top
> > just so that this one can move forward while we have
> > a more specific example to work from.
> 
> we can differ the resources part but lets add the part that the group
> becomes empty/non-empty according to the disable/enable of VF_enable
> register as mentioned above.

Hmm.  Another option is VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP when VF_enable
is clear - since this disables the whole group not a specific member.
This allows distinguishing between id > NumVFs and VF_enable == 0
so it seems like a nicer option to me.


> > 
> > 
> > 
> > > > 
> > > > > > 
> > > > > > So I would say we should just ask that VF Enable is set
> > > > > > before poking at admin vq, and prohibit clearing VF Enable.
> > > > > > 
> > > > > 
> > > > > Sending admin commands shouldn't be restricted and should be allowed always.
> > > > > Device should react to incoming command.
> > > > > 
> > > > > We can say that an SR-IOV group is empty while the VF enable is unset and is
> > > > > not empty when it's set. The size of the non empty SR-IOV group is equal to
> > > > > the numVFs and the VF index is equal to the member identifier in this group.
> > > > > I had this definition in my patches.
> > > > > 
> > > > > Sending admin command to a non existing member will result in an ERROR
> > > > > VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
> > > > 
> > > > 
> > > > just an example:
> > > > - assign resource X to VF 16
> > > > - set NumVFs to 8
> > > > 
> > > > Does VF 16 keep resource X? I note that
> > > > resource X can not be taken back because VF 16
> > > > is not accessible.
> > > 
> > > all dynamic resources should return to the owner when VF_enable moves 1 -->
> > > 0
> > 
> > 
> > It's a possible approach but my question would be whether we
> > need a transport independent way to do this through VQ.
> > And if we do it's annoying to have multiple ways.
> 
> We are defining the existence of the SR-IOV group.
> 
> Also, the approach of sending commands to a non-existing member is valid for
> any group type and should return error VIRTIO_ADMIN_STATUS_Q_INVALID_MEM in
> that case.
> 
> why do we need more ways ?

Sorry I mean a transport independent way to destroy a group as a whole.


> > 
> > > > 
> > > > 
> > > > For sure, all this is solvable but I'd rather just forbid
> > > > changes to VF enable for now.
> > > 
> > > how we can forbid it ? setting this register is not related to virtio.
> > 
> > We can ask driver to write 0 to device reset before changing VF
> > enable to 0. The advantage is that this kind of reset by definition
> > frees up all resources.
> 
> why do we need to reset the device ? It is fully functional and should be
> able to run perfectly fine without any need to reset.
> 
> The PF device might have a very big role in the hypervisor, so restricting
> it in this way doesn't sounds mandatory.
> 
> Controlling the SR-IOV group using SR-IOV register enable/disable and NumVFs
> used as the number of the group size sounds simple from Spec and
> implementation POV.

Simple, however
- only works for SRIOV
- also limited, for example it's impossible to report an error

I would rather not decide now until we have a use-case.


> > 
> > 
> > > > 
> > > > 
> > > > > 
> > > > > > 
> > > > > > There's also the concern of VF migration under MRIOV where VFs might
> > > > > > not exit. I'm guessing
> > > > > > trying to include that right now is overthinking. So maybe
> > > > > > this should mention that VF Migration Capable should be clear.
> > > > > > 
> > > > > > 
> > > > > > I will add all this to the spec.
> > > > > > 
> > > > > > 
> > > > > > >        How about the converse?
> > > > > > > 
> > > > > > > 
> > > > > > >                    Each device within a group has a unique identifier. This identifier
> > > > > > >                    is the group member identifier.
> > > > > > > 
> > > > > > >                    Note: one can argue both ways whether the new device group handling
> > > > > > >                    functionality (this and following patches) is closer
> > > > > > >                    to a new device type or a new transport type.
> > > > > > > 
> > > > > > >                    However, I expect that we will add more features in the near future. To
> > > > > > >                    facilitate this as much as possible of the text is located in the new
> > > > > > >                    admin chapter.
> > > > > > > 
> > > > > > >                    I did my best to minimize transport-specific text.
> > > > > > > 
> > > > > > >                    Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > > > >                    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > > > >                    ---
> > > > > > >                     admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > >                     content.tex |  2 ++
> > > > > > >                     2 files changed, 51 insertions(+)
> > > > > > >                     create mode 100644 admin.tex
> > > > > > > 
> > > > > > >                    diff --git a/admin.tex b/admin.tex
> > > > > > >                    new file mode 100644
> > > > > > >                    index 0000000..2bc7322
> > > > > > >                    --- /dev/null
> > > > > > >                    +++ b/admin.tex
> > > > > > >                    @@ -0,0 +1,49 @@
> > > > > > >                    +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
> > > > > > >                    +
> > > > > > >                    +It is occasionally useful to have a device control a group of
> > > > > > >                    +other devices. Terminology used in such cases:
> > > > > > >                    +
> > > > > > >                    +\begin{description}
> > > > > > >                    +\item[Device group]
> > > > > > >                    +        or just group, includes zero or more devices.
> > > > > > >                    +\item[Owner device]
> > > > > > >                    +        or owner, the device controlling the group.
> > > > > > >                    +\item[Member device]
> > > > > > >                    +        a device within a group. The owner device itself is not
> > > > > > >                    +       a member of the group.
> > > > > > >                    +\item[Member identifier]
> > > > > > >                    +        each member has this identifier, unique within the group
> > > > > > >                    +       and used to address it through the owner device.
> > > > > > >                    +\item[Group type identifier]
> > > > > > >                    +       specifies what kind of member devices there are in a
> > > > > > >                    +       group, how is the member identifier is interpreted
> > > > > > >                    +       and what kind of control the owner has.
> > > > > > >                    +       A given owner can control a single group of a given type,
> > > > > > >                    +       thus the type and the owner together identify the group.
> > > > > > >                    +       \footnote{Even though some group types only support
> > > > > > >                    +                       specific transports, group type identifiers
> > > > > > >                    +                       are global rather than transport-specific -
> > > > > > >                    +                       we don't expect a flood of new group types.}
> > > > > > >                    +\end{description}
> > > > > > >                    +
> > > > > > >                    +The following group types, and their identifiers, are currently specified):
> > > > > > >                    +\begin{description}
> > > > > > >                    +\item[SR-IOV group type (0x1)]
> > > > > > >                    +This device group has a PCI Single Root I/O Virtualization
> > > > > > >                    +(SR-IOV) physical function (PF) device as the owner and includes
> > > > > > >                    +all its SR-IOV virtual functions (VFs) as members (see
> > > > > > >                    +\hyperref[intro:PCIe]{[PCIe]}).
> > > > > > >                    +
> > > > > > >                    +The PF device itself is not a member of the group.
> > > > > > >                    +
> > > > > > >                    +The group type identifier for this group is 0x1.
> > > > > > >                    +
> > > > > > >                    +A member identifier for this group can have a value 0x1 to 0xFFFF
> > > > > > >                    +and equals the SR-IOV VF number of the member device (see
> > > > > > >                    +\hyperref[intro:PCIe]{[PCIe]}).
> > > > > > >                    +
> > > > > > >                    +Both owner and member devices for this group type use the Virtio
> > > > > > >                    +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > > > > > >                    +\end{description}
> > > > > > >                    +
> > > > > > >                    +
> > > > > > >                    diff --git a/content.tex b/content.tex
> > > > > > >                    index 0c2d917..ffe45c4 100644
> > > > > > >                    --- a/content.tex
> > > > > > >                    +++ b/content.tex
> > > > > > >                    @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> > > > > > >                     types. It is RECOMMENDED that devices generate version 4
> > > > > > >                     UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > > > > > > 
> > > > > > >                    +\input{admin.tex}
> > > > > > >                    +
> > > > > > >                     \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> > > > > > > 
> > > > > > >                     We start with an overview of device initialization, then expand on the
> > > > > > > 
> > > > > > >                --
> > > > > > >                Tell her I'll be waiting, in the usual place.
> > > > > > > 
> > > > > > 
> > > > 
> > 

Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
> > For some system calls and library functions (e.g.,
> >         getpriority(2)), -1 is a valid return on success.  In such cases,
> >         a successful return can be distinguished from an error return by
> >         setting errno to zero before the call, and then, if the call
> >         returns a status that indicates that an error may have occurred,
> >         checking to see if errno has a nonzero value.
> > 
> > 
> > 
> > > Description is already good enough to describe what they are.
> > > Can we please drop Linux wording?
> > 
> > But why should we? It's where 22 comes from so this way people are not
> > wondering about the value, and it's somewhat helpful for Linux
> > developers.
> > 
> 
> I also think we should not mention Linux. I don't think it's mentioned
> currently in the spec and no good reason to do so now.

But we do: fuse, input at least both do.

> Also value of 22 is not mandatory for this EINVAL status code. It can be
> just 1 (the first number after the OK status).

22 makes it a tiny bit easier for kvm. So why not.

> Developers should be able to read the specification and find out the meaning
> of error code.

I think the text does that. 

-- 
MST

Re: [virtio-dev] Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Zhu, Lingshan]

On 2/13/2023 4:12 PM, Michael S. Tsirkin wrote:
> On Mon, Feb 13, 2023 at 12:49:44AM +0200, Max Gurtovoy wrote:
>>
>> On 12/02/2023 22:19, Michael S. Tsirkin wrote:
>>> On Sun, Feb 12, 2023 at 04:34:09PM +0200, Max Gurtovoy wrote:
>>>>
>>>> On 12/02/2023 15:15, Michael S. Tsirkin wrote:
>>>>> On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
>>>>>>
>>>>>> On 09/02/2023 21:58, Michael S. Tsirkin wrote:
>>>>>>> On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
>>>>>>>> On 09/02/2023 17:22, David Edmondson wrote:
>>>>>>>>
>>>>>>>>         On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
>>>>>>>>
>>>>>>>>             On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
>>>>>>>>
>>>>>>>>                 On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
>>>>>>>>
>>>>>>>>                     Each device group has a type. For now, define one initial group:
>>>>>>>>
>>>>>>>>                     SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>>>>>>>>                     PCI SR-IOV physical function (PF). This group may contain one or more
>>>>>>>>                     virtio devices.
>>>>>>>>
>>>>>>>>                 The specification says zero or more generally and nothing to refine that
>>>>>>>>                 for SR-IOV groups.
>>>>>>>>
>>>>>>>>             Well it says:
>>>>>>>>
>>>>>>>>                     (SR-IOV) physical function (PF) device as the owner and includes
>>>>>>>>                     all its SR-IOV virtual functions (VFs) as members (see
>>>>>>>>                     \hyperref[intro:PCIe]{[PCIe]}).
>>>>>>>>
>>>>>>>>             how can that be zero?
>>>>>>>>
>>>>>>>>         If I remove all of the VFs of a PF, does the group implicitly disappear?
>>>>>>>>
>>>>>>>> Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
>>>>>>> Are you sure? Not by VF Enable?
>>>>>> Right. It's a combination of the VF_Enable and numVfs.
>>>>>>
>>>>>> ...
>>>>>> "The NumVFs field defines the number of VFs that are enabled when VF Enable
>>>>>> is Set in the associated PF."
>>>>>>
>>>>>>>> We can mention that a group include all virtual functions indicated by TotalVFs
>>>>>>>> RO register.
>>>>>>> TotalVFs indicates the maximum number of VFs that could be associated with the PF
>>>>>>> but not all of these exist. If we use this we get into issues of
>>>>>>> discovering whether VFs can actually be used. I'd rather avoid that.
>>>>>>>
>>>>>>>
>>>>>>>> This group exist as long as the owner PF supports SR-IOV and has admin
>>>>>>>> capabilities.
>>>>>>> SRIOV spec says:
>>>>>>> 	NumVFs controls the number of VFs that are visible.
>>>>>>>
>>>>>>> And one can not play with NumVFs dynamically:
>>>>>>>
>>>>>>> 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
>>>>>>> 	Set, the results are undefined.
>>>>>>>
>>>>>>> one has to kill all VFs then enable them all again.
>>>>>>> As long as we are doing that, we can just unload the driver.
>>>>>> what driver to unload ? Not sure I'm following..
>>>>> Basically I am saying that if VFs are disabled this destroys all VFs
>>>>> so it is not
>>>>> clear that it makes sense to keep some of the configuration
>>>>> for previous VFs around.
>>>>>
>>>>> My idea is to say that reset is required to disable VFs.
>>>> which reset ?
>>>>
>>>> VFs disabled/enabled using VF_enable PCI register.
>>>> Lets follow that please.
>>>>
>>>> Lets define what is happening during VF_enable = 1 and VF_enable = 0.
>>>>
>>>> On VF_enable = 0 --> 1 transition the corresponding device_group, that is
>>>> owned by the PF, grows and contains NumVFs members with initial features and
>>>> resources.
>>>>
>>>> On VF_enable = 1 --> 0 transition the corresponding device_group, that is
>>>> owned by the PF, shrinks and contains 0 members and all the dynamic
>>>> resources are back to the owner.
>>> Maybe, it's a possible approach. As current patch set has no
>>> resources I feel we can leave this part to a patch on top
>>> just so that this one can move forward while we have
>>> a more specific example to work from.
>> we can differ the resources part but lets add the part that the group
>> becomes empty/non-empty according to the disable/enable of VF_enable
>> register as mentioned above.
> Hmm.  Another option is VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP when VF_enable
> is clear - since this disables the whole group not a specific member.
> This allows distinguishing between id > NumVFs and VF_enable == 0
> so it seems like a nicer option to me.
How to detect whether VF_enable set to 0 or 1, pulling is not a good 
idea, and do we want
SRIOV cap to notify the admin vq on changes?

Thanks
>
>
>>>
>>>
>>>>>>> So I would say we should just ask that VF Enable is set
>>>>>>> before poking at admin vq, and prohibit clearing VF Enable.
>>>>>>>
>>>>>> Sending admin commands shouldn't be restricted and should be allowed always.
>>>>>> Device should react to incoming command.
>>>>>>
>>>>>> We can say that an SR-IOV group is empty while the VF enable is unset and is
>>>>>> not empty when it's set. The size of the non empty SR-IOV group is equal to
>>>>>> the numVFs and the VF index is equal to the member identifier in this group.
>>>>>> I had this definition in my patches.
>>>>>>
>>>>>> Sending admin command to a non existing member will result in an ERROR
>>>>>> VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
>>>>>
>>>>> just an example:
>>>>> - assign resource X to VF 16
>>>>> - set NumVFs to 8
>>>>>
>>>>> Does VF 16 keep resource X? I note that
>>>>> resource X can not be taken back because VF 16
>>>>> is not accessible.
>>>> all dynamic resources should return to the owner when VF_enable moves 1 -->
>>>> 0
>>>
>>> It's a possible approach but my question would be whether we
>>> need a transport independent way to do this through VQ.
>>> And if we do it's annoying to have multiple ways.
>> We are defining the existence of the SR-IOV group.
>>
>> Also, the approach of sending commands to a non-existing member is valid for
>> any group type and should return error VIRTIO_ADMIN_STATUS_Q_INVALID_MEM in
>> that case.
>>
>> why do we need more ways ?
> Sorry I mean a transport independent way to destroy a group as a whole.
>
>
>>>>>
>>>>> For sure, all this is solvable but I'd rather just forbid
>>>>> changes to VF enable for now.
>>>> how we can forbid it ? setting this register is not related to virtio.
>>> We can ask driver to write 0 to device reset before changing VF
>>> enable to 0. The advantage is that this kind of reset by definition
>>> frees up all resources.
>> why do we need to reset the device ? It is fully functional and should be
>> able to run perfectly fine without any need to reset.
>>
>> The PF device might have a very big role in the hypervisor, so restricting
>> it in this way doesn't sounds mandatory.
>>
>> Controlling the SR-IOV group using SR-IOV register enable/disable and NumVFs
>> used as the number of the group size sounds simple from Spec and
>> implementation POV.
> Simple, however
> - only works for SRIOV
> - also limited, for example it's impossible to report an error
>
> I would rather not decide now until we have a use-case.
>
>
>>>
>>>>>
>>>>>>> There's also the concern of VF migration under MRIOV where VFs might
>>>>>>> not exit. I'm guessing
>>>>>>> trying to include that right now is overthinking. So maybe
>>>>>>> this should mention that VF Migration Capable should be clear.
>>>>>>>
>>>>>>>
>>>>>>> I will add all this to the spec.
>>>>>>>
>>>>>>>
>>>>>>>>         How about the converse?
>>>>>>>>
>>>>>>>>
>>>>>>>>                     Each device within a group has a unique identifier. This identifier
>>>>>>>>                     is the group member identifier.
>>>>>>>>
>>>>>>>>                     Note: one can argue both ways whether the new device group handling
>>>>>>>>                     functionality (this and following patches) is closer
>>>>>>>>                     to a new device type or a new transport type.
>>>>>>>>
>>>>>>>>                     However, I expect that we will add more features in the near future. To
>>>>>>>>                     facilitate this as much as possible of the text is located in the new
>>>>>>>>                     admin chapter.
>>>>>>>>
>>>>>>>>                     I did my best to minimize transport-specific text.
>>>>>>>>
>>>>>>>>                     Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>>>>                     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>>>>>>                     ---
>>>>>>>>                      admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>                      content.tex |  2 ++
>>>>>>>>                      2 files changed, 51 insertions(+)
>>>>>>>>                      create mode 100644 admin.tex
>>>>>>>>
>>>>>>>>                     diff --git a/admin.tex b/admin.tex
>>>>>>>>                     new file mode 100644
>>>>>>>>                     index 0000000..2bc7322
>>>>>>>>                     --- /dev/null
>>>>>>>>                     +++ b/admin.tex
>>>>>>>>                     @@ -0,0 +1,49 @@
>>>>>>>>                     +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>>>>>>>>                     +
>>>>>>>>                     +It is occasionally useful to have a device control a group of
>>>>>>>>                     +other devices. Terminology used in such cases:
>>>>>>>>                     +
>>>>>>>>                     +\begin{description}
>>>>>>>>                     +\item[Device group]
>>>>>>>>                     +        or just group, includes zero or more devices.
>>>>>>>>                     +\item[Owner device]
>>>>>>>>                     +        or owner, the device controlling the group.
>>>>>>>>                     +\item[Member device]
>>>>>>>>                     +        a device within a group. The owner device itself is not
>>>>>>>>                     +       a member of the group.
>>>>>>>>                     +\item[Member identifier]
>>>>>>>>                     +        each member has this identifier, unique within the group
>>>>>>>>                     +       and used to address it through the owner device.
>>>>>>>>                     +\item[Group type identifier]
>>>>>>>>                     +       specifies what kind of member devices there are in a
>>>>>>>>                     +       group, how is the member identifier is interpreted
>>>>>>>>                     +       and what kind of control the owner has.
>>>>>>>>                     +       A given owner can control a single group of a given type,
>>>>>>>>                     +       thus the type and the owner together identify the group.
>>>>>>>>                     +       \footnote{Even though some group types only support
>>>>>>>>                     +                       specific transports, group type identifiers
>>>>>>>>                     +                       are global rather than transport-specific -
>>>>>>>>                     +                       we don't expect a flood of new group types.}
>>>>>>>>                     +\end{description}
>>>>>>>>                     +
>>>>>>>>                     +The following group types, and their identifiers, are currently specified):
>>>>>>>>                     +\begin{description}
>>>>>>>>                     +\item[SR-IOV group type (0x1)]
>>>>>>>>                     +This device group has a PCI Single Root I/O Virtualization
>>>>>>>>                     +(SR-IOV) physical function (PF) device as the owner and includes
>>>>>>>>                     +all its SR-IOV virtual functions (VFs) as members (see
>>>>>>>>                     +\hyperref[intro:PCIe]{[PCIe]}).
>>>>>>>>                     +
>>>>>>>>                     +The PF device itself is not a member of the group.
>>>>>>>>                     +
>>>>>>>>                     +The group type identifier for this group is 0x1.
>>>>>>>>                     +
>>>>>>>>                     +A member identifier for this group can have a value 0x1 to 0xFFFF
>>>>>>>>                     +and equals the SR-IOV VF number of the member device (see
>>>>>>>>                     +\hyperref[intro:PCIe]{[PCIe]}).
>>>>>>>>                     +
>>>>>>>>                     +Both owner and member devices for this group type use the Virtio
>>>>>>>>                     +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>>>>>>>>                     +\end{description}
>>>>>>>>                     +
>>>>>>>>                     +
>>>>>>>>                     diff --git a/content.tex b/content.tex
>>>>>>>>                     index 0c2d917..ffe45c4 100644
>>>>>>>>                     --- a/content.tex
>>>>>>>>                     +++ b/content.tex
>>>>>>>>                     @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>>>>>>>                      types. It is RECOMMENDED that devices generate version 4
>>>>>>>>                      UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>>>>>
>>>>>>>>                     +\input{admin.tex}
>>>>>>>>                     +
>>>>>>>>                      \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>>>>>>>
>>>>>>>>                      We start with an overview of device initialization, then expand on the
>>>>>>>>
>>>>>>>>                 --
>>>>>>>>                 Tell her I'll be waiting, in the usual place.
>>>>>>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Max Gurtovoy]

On 13/02/2023 10:12, Michael S. Tsirkin wrote:
> On Mon, Feb 13, 2023 at 12:49:44AM +0200, Max Gurtovoy wrote:
>>
>>
>> On 12/02/2023 22:19, Michael S. Tsirkin wrote:
>>> On Sun, Feb 12, 2023 at 04:34:09PM +0200, Max Gurtovoy wrote:
>>>>
>>>>
>>>> On 12/02/2023 15:15, Michael S. Tsirkin wrote:
>>>>> On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
>>>>>>
>>>>>>
>>>>>> On 09/02/2023 21:58, Michael S. Tsirkin wrote:
>>>>>>> On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
>>>>>>>>
>>>>>>>> On 09/02/2023 17:22, David Edmondson wrote:
>>>>>>>>
>>>>>>>>         On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
>>>>>>>>
>>>>>>>>             On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
>>>>>>>>
>>>>>>>>                 On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
>>>>>>>>
>>>>>>>>                     Each device group has a type. For now, define one initial group:
>>>>>>>>
>>>>>>>>                     SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
>>>>>>>>                     PCI SR-IOV physical function (PF). This group may contain one or more
>>>>>>>>                     virtio devices.
>>>>>>>>
>>>>>>>>                 The specification says zero or more generally and nothing to refine that
>>>>>>>>                 for SR-IOV groups.
>>>>>>>>
>>>>>>>>             Well it says:
>>>>>>>>
>>>>>>>>                     (SR-IOV) physical function (PF) device as the owner and includes
>>>>>>>>                     all its SR-IOV virtual functions (VFs) as members (see
>>>>>>>>                     \hyperref[intro:PCIe]{[PCIe]}).
>>>>>>>>
>>>>>>>>             how can that be zero?
>>>>>>>>
>>>>>>>>         If I remove all of the VFs of a PF, does the group implicitly disappear?
>>>>>>>>
>>>>>>>> Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
>>>>>>>
>>>>>>> Are you sure? Not by VF Enable?
>>>>>>
>>>>>> Right. It's a combination of the VF_Enable and numVfs.
>>>>>>
>>>>>> ...
>>>>>> "The NumVFs field defines the number of VFs that are enabled when VF Enable
>>>>>> is Set in the associated PF."
>>>>>>
>>>>>>>
>>>>>>>> We can mention that a group include all virtual functions indicated by TotalVFs
>>>>>>>> RO register.
>>>>>>>
>>>>>>> TotalVFs indicates the maximum number of VFs that could be associated with the PF
>>>>>>> but not all of these exist. If we use this we get into issues of
>>>>>>> discovering whether VFs can actually be used. I'd rather avoid that.
>>>>>>>
>>>>>>>
>>>>>>>> This group exist as long as the owner PF supports SR-IOV and has admin
>>>>>>>> capabilities.
>>>>>>>
>>>>>>> SRIOV spec says:
>>>>>>> 	NumVFs controls the number of VFs that are visible.
>>>>>>>
>>>>>>> And one can not play with NumVFs dynamically:
>>>>>>>
>>>>>>> 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
>>>>>>> 	Set, the results are undefined.
>>>>>>>
>>>>>>> one has to kill all VFs then enable them all again.
>>>>>>> As long as we are doing that, we can just unload the driver.
>>>>>> what driver to unload ? Not sure I'm following..
>>>>>
>>>>> Basically I am saying that if VFs are disabled this destroys all VFs
>>>>> so it is not
>>>>> clear that it makes sense to keep some of the configuration
>>>>> for previous VFs around.
>>>>>
>>>>> My idea is to say that reset is required to disable VFs.
>>>>
>>>> which reset ?
>>>>
>>>> VFs disabled/enabled using VF_enable PCI register.
>>>> Lets follow that please.
>>>>
>>>> Lets define what is happening during VF_enable = 1 and VF_enable = 0.
>>>>
>>>> On VF_enable = 0 --> 1 transition the corresponding device_group, that is
>>>> owned by the PF, grows and contains NumVFs members with initial features and
>>>> resources.
>>>>
>>>> On VF_enable = 1 --> 0 transition the corresponding device_group, that is
>>>> owned by the PF, shrinks and contains 0 members and all the dynamic
>>>> resources are back to the owner.
>>>
>>> Maybe, it's a possible approach. As current patch set has no
>>> resources I feel we can leave this part to a patch on top
>>> just so that this one can move forward while we have
>>> a more specific example to work from.
>>
>> we can differ the resources part but lets add the part that the group
>> becomes empty/non-empty according to the disable/enable of VF_enable
>> register as mentioned above.
> 
> Hmm.  Another option is VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP when VF_enable
> is clear - since this disables the whole group not a specific member.
> This allows distinguishing between id > NumVFs and VF_enable == 0
> so it seems like a nicer option to me.

The group will be created on VF_enable = 1 and will be destroyed on 
VF_enable = 0.
Members with index >= 1 and index <= NumVFs are part of the group.

sounds good to me.

> 
> 
>>>
>>>
>>>
>>>>>
>>>>>>>
>>>>>>> So I would say we should just ask that VF Enable is set
>>>>>>> before poking at admin vq, and prohibit clearing VF Enable.
>>>>>>>
>>>>>>
>>>>>> Sending admin commands shouldn't be restricted and should be allowed always.
>>>>>> Device should react to incoming command.
>>>>>>
>>>>>> We can say that an SR-IOV group is empty while the VF enable is unset and is
>>>>>> not empty when it's set. The size of the non empty SR-IOV group is equal to
>>>>>> the numVFs and the VF index is equal to the member identifier in this group.
>>>>>> I had this definition in my patches.
>>>>>>
>>>>>> Sending admin command to a non existing member will result in an ERROR
>>>>>> VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
>>>>>
>>>>>
>>>>> just an example:
>>>>> - assign resource X to VF 16
>>>>> - set NumVFs to 8
>>>>>
>>>>> Does VF 16 keep resource X? I note that
>>>>> resource X can not be taken back because VF 16
>>>>> is not accessible.
>>>>
>>>> all dynamic resources should return to the owner when VF_enable moves 1 -->
>>>> 0
>>>
>>>
>>> It's a possible approach but my question would be whether we
>>> need a transport independent way to do this through VQ.
>>> And if we do it's annoying to have multiple ways.
>>
>> We are defining the existence of the SR-IOV group.
>>
>> Also, the approach of sending commands to a non-existing member is valid for
>> any group type and should return error VIRTIO_ADMIN_STATUS_Q_INVALID_MEM in
>> that case.
>>
>> why do we need more ways ?
> 
> Sorry I mean a transport independent way to destroy a group as a whole.

Your proposal above to create/destroy a group during vf_enable = 1/0 is 
transport specific and its ok since the SR-IOV group is only for PCI 
transport.

> 
> 
>>>
>>>>>
>>>>>
>>>>> For sure, all this is solvable but I'd rather just forbid
>>>>> changes to VF enable for now.
>>>>
>>>> how we can forbid it ? setting this register is not related to virtio.
>>>
>>> We can ask driver to write 0 to device reset before changing VF
>>> enable to 0. The advantage is that this kind of reset by definition
>>> frees up all resources.
>>
>> why do we need to reset the device ? It is fully functional and should be
>> able to run perfectly fine without any need to reset.
>>
>> The PF device might have a very big role in the hypervisor, so restricting
>> it in this way doesn't sounds mandatory.
>>
>> Controlling the SR-IOV group using SR-IOV register enable/disable and NumVFs
>> used as the number of the group size sounds simple from Spec and
>> implementation POV.
> 
> Simple, however
> - only works for SRIOV
> - also limited, for example it's impossible to report an error

what error is impossible to report and why ?

> 
> I would rather not decide now until we have a use-case.
> 
> 
>>>
>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> There's also the concern of VF migration under MRIOV where VFs might
>>>>>>> not exit. I'm guessing
>>>>>>> trying to include that right now is overthinking. So maybe
>>>>>>> this should mention that VF Migration Capable should be clear.
>>>>>>>
>>>>>>>
>>>>>>> I will add all this to the spec.
>>>>>>>
>>>>>>>
>>>>>>>>         How about the converse?
>>>>>>>>
>>>>>>>>
>>>>>>>>                     Each device within a group has a unique identifier. This identifier
>>>>>>>>                     is the group member identifier.
>>>>>>>>
>>>>>>>>                     Note: one can argue both ways whether the new device group handling
>>>>>>>>                     functionality (this and following patches) is closer
>>>>>>>>                     to a new device type or a new transport type.
>>>>>>>>
>>>>>>>>                     However, I expect that we will add more features in the near future. To
>>>>>>>>                     facilitate this as much as possible of the text is located in the new
>>>>>>>>                     admin chapter.
>>>>>>>>
>>>>>>>>                     I did my best to minimize transport-specific text.
>>>>>>>>
>>>>>>>>                     Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>>>>                     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>>>>>>                     ---
>>>>>>>>                      admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>                      content.tex |  2 ++
>>>>>>>>                      2 files changed, 51 insertions(+)
>>>>>>>>                      create mode 100644 admin.tex
>>>>>>>>
>>>>>>>>                     diff --git a/admin.tex b/admin.tex
>>>>>>>>                     new file mode 100644
>>>>>>>>                     index 0000000..2bc7322
>>>>>>>>                     --- /dev/null
>>>>>>>>                     +++ b/admin.tex
>>>>>>>>                     @@ -0,0 +1,49 @@
>>>>>>>>                     +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
>>>>>>>>                     +
>>>>>>>>                     +It is occasionally useful to have a device control a group of
>>>>>>>>                     +other devices. Terminology used in such cases:
>>>>>>>>                     +
>>>>>>>>                     +\begin{description}
>>>>>>>>                     +\item[Device group]
>>>>>>>>                     +        or just group, includes zero or more devices.
>>>>>>>>                     +\item[Owner device]
>>>>>>>>                     +        or owner, the device controlling the group.
>>>>>>>>                     +\item[Member device]
>>>>>>>>                     +        a device within a group. The owner device itself is not
>>>>>>>>                     +       a member of the group.
>>>>>>>>                     +\item[Member identifier]
>>>>>>>>                     +        each member has this identifier, unique within the group
>>>>>>>>                     +       and used to address it through the owner device.
>>>>>>>>                     +\item[Group type identifier]
>>>>>>>>                     +       specifies what kind of member devices there are in a
>>>>>>>>                     +       group, how is the member identifier is interpreted
>>>>>>>>                     +       and what kind of control the owner has.
>>>>>>>>                     +       A given owner can control a single group of a given type,
>>>>>>>>                     +       thus the type and the owner together identify the group.
>>>>>>>>                     +       \footnote{Even though some group types only support
>>>>>>>>                     +                       specific transports, group type identifiers
>>>>>>>>                     +                       are global rather than transport-specific -
>>>>>>>>                     +                       we don't expect a flood of new group types.}
>>>>>>>>                     +\end{description}
>>>>>>>>                     +
>>>>>>>>                     +The following group types, and their identifiers, are currently specified):
>>>>>>>>                     +\begin{description}
>>>>>>>>                     +\item[SR-IOV group type (0x1)]
>>>>>>>>                     +This device group has a PCI Single Root I/O Virtualization
>>>>>>>>                     +(SR-IOV) physical function (PF) device as the owner and includes
>>>>>>>>                     +all its SR-IOV virtual functions (VFs) as members (see
>>>>>>>>                     +\hyperref[intro:PCIe]{[PCIe]}).
>>>>>>>>                     +
>>>>>>>>                     +The PF device itself is not a member of the group.
>>>>>>>>                     +
>>>>>>>>                     +The group type identifier for this group is 0x1.
>>>>>>>>                     +
>>>>>>>>                     +A member identifier for this group can have a value 0x1 to 0xFFFF
>>>>>>>>                     +and equals the SR-IOV VF number of the member device (see
>>>>>>>>                     +\hyperref[intro:PCIe]{[PCIe]}).
>>>>>>>>                     +
>>>>>>>>                     +Both owner and member devices for this group type use the Virtio
>>>>>>>>                     +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>>>>>>>>                     +\end{description}
>>>>>>>>                     +
>>>>>>>>                     +
>>>>>>>>                     diff --git a/content.tex b/content.tex
>>>>>>>>                     index 0c2d917..ffe45c4 100644
>>>>>>>>                     --- a/content.tex
>>>>>>>>                     +++ b/content.tex
>>>>>>>>                     @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>>>>>>>                      types. It is RECOMMENDED that devices generate version 4
>>>>>>>>                      UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>>>>>
>>>>>>>>                     +\input{admin.tex}
>>>>>>>>                     +
>>>>>>>>                      \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>>>>>>>
>>>>>>>>                      We start with an overview of device initialization, then expand on the
>>>>>>>>
>>>>>>>>                 --
>>>>>>>>                 Tell her I'll be waiting, in the usual place.
>>>>>>>>
>>>>>>>
>>>>>
>>>
> 

[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Max Gurtovoy]

On 13/02/2023 10:16, Michael S. Tsirkin wrote:
> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>>> For some system calls and library functions (e.g.,
>>>          getpriority(2)), -1 is a valid return on success.  In such cases,
>>>          a successful return can be distinguished from an error return by
>>>          setting errno to zero before the call, and then, if the call
>>>          returns a status that indicates that an error may have occurred,
>>>          checking to see if errno has a nonzero value.
>>>
>>>
>>>
>>>> Description is already good enough to describe what they are.
>>>> Can we please drop Linux wording?
>>>
>>> But why should we? It's where 22 comes from so this way people are not
>>> wondering about the value, and it's somewhat helpful for Linux
>>> developers.
>>>
>>
>> I also think we should not mention Linux. I don't think it's mentioned
>> currently in the spec and no good reason to do so now.
> 
> But we do: fuse, input at least both do.
> 
>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>> just 1 (the first number after the OK status).
> 
> 22 makes it a tiny bit easier for kvm. So why not.

Because people comment on that in the review :) and because a 
specification should be OS independent.
22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.

not a big issue for me, but I prefer to have a cleaner spec than maybe 
simplify something in a very specific OS.

> 
>> Developers should be able to read the specification and find out the meaning
>> of error code.
> 
> I think the text does that.
> 

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-dev] [PATCH v10 09/10] admin: conformance clauses

[Michael S. Tsirkin]

On Mon, Feb 13, 2023 at 02:43:30PM +0800, Zhu Lingshan wrote:
> 
> 
> On 2/9/2023 8:14 PM, Michael S. Tsirkin wrote:
> > Add conformance clauses for admin commands and admin virtqueues.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >   admin.tex | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 190 insertions(+)
> > 
> > diff --git a/admin.tex b/admin.tex
> > index 4de606a..f9b1b68 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -237,6 +237,119 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >   supporting multiple group types, the list of supported commands
> >   might differ between different group types.
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > +
> > +The device MUST validate \field{opcode}, \field{group_type} and
> > +\field{group_member_id}, and if any of these has an invalid or
> > +unsupported value, set \field{status} to
> > +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
> > +accordingly:
> > +\begin{itemize}
> > +\item if \field{group_type} is invalid, \field{status_qualifier}
> > +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> > +\item otherwise, if \field{opcode} is invalid,
> > +	\field{status_qualifier} MUST be set to
> > +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
> > +\item otherwise, if \field{group_member_id} is used by the
> > +	specific command and is invalid, \field{status_qualifier} MUST be
> > +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
> > +\end{itemize}
> > +
> > +If a command completes successfully, the device MUST set
> > +\field{status} to VIRTIO_ADMIN_STATUS_OK.
> > +
> > +If a command fails, the device MUST set
> > +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.
> > +
> > +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
> > +device state MUST NOT change, that is the command MUST NOT have
> > +any side effects on the device, in particular the device MUST not
> > +enter an error state as a result of this command.
> > +
> > +If a command fails, the device state generally SHOULD NOT change,
> > +as far as possible.
> > +
> > +The device MAY enforce additional restrictions and dependencies on
> > +opcodes used by the driver and MAY fail the command
> > +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> > +if the list of commands used violate internal device dependencies.
> > +
> > +If the device supports multiple group types, commands for each group
> > +type MUST operate independently of each other, in particular,
> > +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
> > +for different group types.
> > +
> > +After reset, if the device supports a given group type
> > +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
> > +the device MUST assume
> > +that the list of legal commands used by the driver consists of
> > +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> > +
> > +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
> > +the device MUST set the list of legal commands used by the driver
> > +to the one supplied in \field{command_specific_data}.
> > +
> > +The device MUST set the list of legal commands used by the driver
> > +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
> > +
> > +The device MUST validate commands against the list used by
> > +the driver and MUST fail any commands not in the list with
> > +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +and \field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
> > +
> > +The list of supported commands MUST NOT shrink (but MAY expand):
> > +after reporting a given command as supported through
> > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> > +as unsupported.  Further, after a given set of commands has been
> > +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
> > +device or system reset the device SHOULD complete successfully
> > +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
> > +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
> > +after a device or system reset, the device MUST not fail it
> > +solely because of the command list used.  Failure to do so would
> > +interfere with resuming from suspend and error recovery.
> > +
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> > +
> > +The driver MAY discover whether device supports a specific group type
> > +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
> > +\field{group_type}.
> > +
> > +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
> > +and wait for it to be completed with status
> > +VIRTIO_ADMIN_STATUS_OK before issuing any commands
> > +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
> > +and VIRTIO_ADMIN_CMD_LIST_USE).
> > +
> > +The driver SHOULD NOT set bits in device_admin_cmds
> > +if it is not familiar with how the command opcode
> > +is used, as dependencies between command opcodes might exist.
> > +
> > +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
> > +the use of any commands not previously reported as
> > +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
> > +
> > +The driver MUST NOT use any commands for a given group type
> > +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
> > +list of command opcodes and group type.
> > +
> > +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> > +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> > +with respective bits cleared in \field{command_specific_data}.
> > +
> > +The driver MUST handle a command error with a reserved \field{status}
> > +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> > +(except possibly for different error reporting/diagnostic messages).
> > +
> > +The driver MUST handle a command error with a reserved
> > +\field{status_qualifier} value in the same way as
> > +\field{status_qualifier} set to
> > +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
> > +different error reporting/diagnostic messages).
> > +
> >   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >   An administration virtqueue of an owner device is used to submit
> > @@ -309,3 +422,80 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
> >   of the specification are designed, new fields can be added to the
> >   tail of a structure, with the driver/device using the full
> >   structure without concern for versioning.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> > +
> > +The device MUST support device-readable and device-writeable buffers
> > +shorter than described in this specification, by
> > +\begin{enumerate}
> > +\item acting as if any data that would be read outside the
> > +device-readable buffers is set to zero, and
> > +\item discarding data that would be written outside the
> > +specified device-writeable buffers.
> > +\end{enumerate}
> > +
> > +The device MUST support device-readable and device-writeable buffers
> > +longer than described in this specification, by
> maybe I missed it, where defines the length? Does it refer to the descriptor
> length?

Each command uses a different command specific data and command specific
result structures and the length depends on this and is described
separately or is implicit in the structure description. For example for
VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE it will be
DIV_ROUND_UP(max_cmd, 64) * 8 where DIV_ROUND_UP is integer division
with round up and max_cmd is the largest available command opcode.

I thought it's clear but apprently not - so I will add this explicitly.

> > +\begin{enumerate}
> > +\item ignoring any data in device-readable buffers outside
> > +the expected length, and
> > +\item only writing the expected structure to the device-writeable
> > +buffers, ignoring any extra buffers, and reporting the
> > +actual length of data written, in bytes,
> > +as buffer used length.
> > +\end{enumerate}
> > +
> > +The device SHOULD initialize the device-writeable buffer
> > +up to the length of the structure described by this specification or
> also this length of the structure

same.

> > +the length of the buffer supplied by the driver (even if the buffer is
> > +all set to zero), whichever is shorter.
> > +
> > +The device MUST NOT fail a command solely because the buffers
> > +provided are shorter or longer than described in this
> > +specification.
> > +
> > +The device MUST initialize the device-writeable part of
> > +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
> > +size.
> > +
> > +The device MUST initialize \field{status} in \field{struct
> > +virtio_admin_cmd}.
> > +
> > +The device MUST process commands on a given administration virtqueue
> > +in the order in which they are queued.
> > +
> > +If multiple administration virtqueues have been configured,
> > +device MAY process commands on distinct virtqueues with
> > +no order constraints.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> > +
> > +The driver MAY supply device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are longer than described in
> > +this specification.
> > +
> > +The driver SHOULD supply device-readable part of
> > +\field{struct virtio_admin_cmd} that is at least as
> > +large as the structure described by this specification
> > +(even if the structure is all set to zero).
> > +
> > +The driver MUST supply both device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
> > +in length.
> > +
> > +The device MUST supply both device-readable or device-writeable parts
> > +of \field{struct virtio_admin_cmd} that are larger than zero in
> > +length. However, \field{command_specific_data} and
> > +\field{command_specific_result} MAY be zero in length, unless
> > +specified otherwise for the command.
> > +
> > +The driver MUST NOT assume that the device will initialize the whole
> > +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
> > +the driver MUST act as if the structure
> > +outside the part of the buffer used by the device
> > +is set to zero.
> > +
> > +If multiple administration virtqueues have been configured,
> > +the driver MUST ensure ordering for commands
> > +placed on different administration virtqueues.
> > +>>>>>>> 29f6525... admin: conformance clauses

[virtio] Re: [virtio-dev] Re: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Mon, Feb 13, 2023 at 05:20:55PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 2/13/2023 4:12 PM, Michael S. Tsirkin wrote:
> > On Mon, Feb 13, 2023 at 12:49:44AM +0200, Max Gurtovoy wrote:
> > > 
> > > On 12/02/2023 22:19, Michael S. Tsirkin wrote:
> > > > On Sun, Feb 12, 2023 at 04:34:09PM +0200, Max Gurtovoy wrote:
> > > > > 
> > > > > On 12/02/2023 15:15, Michael S. Tsirkin wrote:
> > > > > > On Sun, Feb 12, 2023 at 02:10:29PM +0200, Max Gurtovoy wrote:
> > > > > > > 
> > > > > > > On 09/02/2023 21:58, Michael S. Tsirkin wrote:
> > > > > > > > On Thu, Feb 09, 2023 at 07:47:56PM +0200, Max Gurtovoy wrote:
> > > > > > > > > On 09/02/2023 17:22, David Edmondson wrote:
> > > > > > > > > 
> > > > > > > > >         On Thursday, 2023-02-09 at 10:13:46 -05, Michael S. Tsirkin wrote:
> > > > > > > > > 
> > > > > > > > >             On Thu, Feb 09, 2023 at 03:00:58PM +0000, David Edmondson wrote:
> > > > > > > > > 
> > > > > > > > >                 On Thursday, 2023-02-09 at 07:13:36 -05, Michael S. Tsirkin wrote:
> > > > > > > > > 
> > > > > > > > >                     Each device group has a type. For now, define one initial group:
> > > > > > > > > 
> > > > > > > > >                     SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given
> > > > > > > > >                     PCI SR-IOV physical function (PF). This group may contain one or more
> > > > > > > > >                     virtio devices.
> > > > > > > > > 
> > > > > > > > >                 The specification says zero or more generally and nothing to refine that
> > > > > > > > >                 for SR-IOV groups.
> > > > > > > > > 
> > > > > > > > >             Well it says:
> > > > > > > > > 
> > > > > > > > >                     (SR-IOV) physical function (PF) device as the owner and includes
> > > > > > > > >                     all its SR-IOV virtual functions (VFs) as members (see
> > > > > > > > >                     \hyperref[intro:PCIe]{[PCIe]}).
> > > > > > > > > 
> > > > > > > > >             how can that be zero?
> > > > > > > > > 
> > > > > > > > >         If I remove all of the VFs of a PF, does the group implicitly disappear?
> > > > > > > > > 
> > > > > > > > > Enable/Disable SR-IOV are dynamic operations controlled by NumVFs register.
> > > > > > > > Are you sure? Not by VF Enable?
> > > > > > > Right. It's a combination of the VF_Enable and numVfs.
> > > > > > > 
> > > > > > > ...
> > > > > > > "The NumVFs field defines the number of VFs that are enabled when VF Enable
> > > > > > > is Set in the associated PF."
> > > > > > > 
> > > > > > > > > We can mention that a group include all virtual functions indicated by TotalVFs
> > > > > > > > > RO register.
> > > > > > > > TotalVFs indicates the maximum number of VFs that could be associated with the PF
> > > > > > > > but not all of these exist. If we use this we get into issues of
> > > > > > > > discovering whether VFs can actually be used. I'd rather avoid that.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > > This group exist as long as the owner PF supports SR-IOV and has admin
> > > > > > > > > capabilities.
> > > > > > > > SRIOV spec says:
> > > > > > > > 	NumVFs controls the number of VFs that are visible.
> > > > > > > > 
> > > > > > > > And one can not play with NumVFs dynamically:
> > > > > > > > 
> > > > > > > > 	NumVFs may only be written while VF Enable is Clear. If NumVFs is written when VF Enable is
> > > > > > > > 	Set, the results are undefined.
> > > > > > > > 
> > > > > > > > one has to kill all VFs then enable them all again.
> > > > > > > > As long as we are doing that, we can just unload the driver.
> > > > > > > what driver to unload ? Not sure I'm following..
> > > > > > Basically I am saying that if VFs are disabled this destroys all VFs
> > > > > > so it is not
> > > > > > clear that it makes sense to keep some of the configuration
> > > > > > for previous VFs around.
> > > > > > 
> > > > > > My idea is to say that reset is required to disable VFs.
> > > > > which reset ?
> > > > > 
> > > > > VFs disabled/enabled using VF_enable PCI register.
> > > > > Lets follow that please.
> > > > > 
> > > > > Lets define what is happening during VF_enable = 1 and VF_enable = 0.
> > > > > 
> > > > > On VF_enable = 0 --> 1 transition the corresponding device_group, that is
> > > > > owned by the PF, grows and contains NumVFs members with initial features and
> > > > > resources.
> > > > > 
> > > > > On VF_enable = 1 --> 0 transition the corresponding device_group, that is
> > > > > owned by the PF, shrinks and contains 0 members and all the dynamic
> > > > > resources are back to the owner.
> > > > Maybe, it's a possible approach. As current patch set has no
> > > > resources I feel we can leave this part to a patch on top
> > > > just so that this one can move forward while we have
> > > > a more specific example to work from.
> > > we can differ the resources part but lets add the part that the group
> > > becomes empty/non-empty according to the disable/enable of VF_enable
> > > register as mentioned above.
> > Hmm.  Another option is VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP when VF_enable
> > is clear - since this disables the whole group not a specific member.
> > This allows distinguishing between id > NumVFs and VF_enable == 0
> > so it seems like a nicer option to me.
> How to detect whether VF_enable set to 0 or 1, pulling is not a good idea,
> and do we want
> SRIOV cap to notify the admin vq on changes?
> 
> Thanks

I don't understand the question.
Who wants to detect this? It's the driver itself setting this bit -
just call some function, there's no reason to involve the spec.



> > 
> > 
> > > > 
> > > > 
> > > > > > > > So I would say we should just ask that VF Enable is set
> > > > > > > > before poking at admin vq, and prohibit clearing VF Enable.
> > > > > > > > 
> > > > > > > Sending admin commands shouldn't be restricted and should be allowed always.
> > > > > > > Device should react to incoming command.
> > > > > > > 
> > > > > > > We can say that an SR-IOV group is empty while the VF enable is unset and is
> > > > > > > not empty when it's set. The size of the non empty SR-IOV group is equal to
> > > > > > > the numVFs and the VF index is equal to the member identifier in this group.
> > > > > > > I had this definition in my patches.
> > > > > > > 
> > > > > > > Sending admin command to a non existing member will result in an ERROR
> > > > > > > VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
> > > > > > 
> > > > > > just an example:
> > > > > > - assign resource X to VF 16
> > > > > > - set NumVFs to 8
> > > > > > 
> > > > > > Does VF 16 keep resource X? I note that
> > > > > > resource X can not be taken back because VF 16
> > > > > > is not accessible.
> > > > > all dynamic resources should return to the owner when VF_enable moves 1 -->
> > > > > 0
> > > > 
> > > > It's a possible approach but my question would be whether we
> > > > need a transport independent way to do this through VQ.
> > > > And if we do it's annoying to have multiple ways.
> > > We are defining the existence of the SR-IOV group.
> > > 
> > > Also, the approach of sending commands to a non-existing member is valid for
> > > any group type and should return error VIRTIO_ADMIN_STATUS_Q_INVALID_MEM in
> > > that case.
> > > 
> > > why do we need more ways ?
> > Sorry I mean a transport independent way to destroy a group as a whole.
> > 
> > 
> > > > > > 
> > > > > > For sure, all this is solvable but I'd rather just forbid
> > > > > > changes to VF enable for now.
> > > > > how we can forbid it ? setting this register is not related to virtio.
> > > > We can ask driver to write 0 to device reset before changing VF
> > > > enable to 0. The advantage is that this kind of reset by definition
> > > > frees up all resources.
> > > why do we need to reset the device ? It is fully functional and should be
> > > able to run perfectly fine without any need to reset.
> > > 
> > > The PF device might have a very big role in the hypervisor, so restricting
> > > it in this way doesn't sounds mandatory.
> > > 
> > > Controlling the SR-IOV group using SR-IOV register enable/disable and NumVFs
> > > used as the number of the group size sounds simple from Spec and
> > > implementation POV.
> > Simple, however
> > - only works for SRIOV
> > - also limited, for example it's impossible to report an error
> > 
> > I would rather not decide now until we have a use-case.
> > 
> > 
> > > > 
> > > > > > 
> > > > > > > > There's also the concern of VF migration under MRIOV where VFs might
> > > > > > > > not exit. I'm guessing
> > > > > > > > trying to include that right now is overthinking. So maybe
> > > > > > > > this should mention that VF Migration Capable should be clear.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > I will add all this to the spec.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > >         How about the converse?
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > >                     Each device within a group has a unique identifier. This identifier
> > > > > > > > >                     is the group member identifier.
> > > > > > > > > 
> > > > > > > > >                     Note: one can argue both ways whether the new device group handling
> > > > > > > > >                     functionality (this and following patches) is closer
> > > > > > > > >                     to a new device type or a new transport type.
> > > > > > > > > 
> > > > > > > > >                     However, I expect that we will add more features in the near future. To
> > > > > > > > >                     facilitate this as much as possible of the text is located in the new
> > > > > > > > >                     admin chapter.
> > > > > > > > > 
> > > > > > > > >                     I did my best to minimize transport-specific text.
> > > > > > > > > 
> > > > > > > > >                     Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > > > > > >                     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > > > > > >                     ---
> > > > > > > > >                      admin.tex   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > >                      content.tex |  2 ++
> > > > > > > > >                      2 files changed, 51 insertions(+)
> > > > > > > > >                      create mode 100644 admin.tex
> > > > > > > > > 
> > > > > > > > >                     diff --git a/admin.tex b/admin.tex
> > > > > > > > >                     new file mode 100644
> > > > > > > > >                     index 0000000..2bc7322
> > > > > > > > >                     --- /dev/null
> > > > > > > > >                     +++ b/admin.tex
> > > > > > > > >                     @@ -0,0 +1,49 @@
> > > > > > > > >                     +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}
> > > > > > > > >                     +
> > > > > > > > >                     +It is occasionally useful to have a device control a group of
> > > > > > > > >                     +other devices. Terminology used in such cases:
> > > > > > > > >                     +
> > > > > > > > >                     +\begin{description}
> > > > > > > > >                     +\item[Device group]
> > > > > > > > >                     +        or just group, includes zero or more devices.
> > > > > > > > >                     +\item[Owner device]
> > > > > > > > >                     +        or owner, the device controlling the group.
> > > > > > > > >                     +\item[Member device]
> > > > > > > > >                     +        a device within a group. The owner device itself is not
> > > > > > > > >                     +       a member of the group.
> > > > > > > > >                     +\item[Member identifier]
> > > > > > > > >                     +        each member has this identifier, unique within the group
> > > > > > > > >                     +       and used to address it through the owner device.
> > > > > > > > >                     +\item[Group type identifier]
> > > > > > > > >                     +       specifies what kind of member devices there are in a
> > > > > > > > >                     +       group, how is the member identifier is interpreted
> > > > > > > > >                     +       and what kind of control the owner has.
> > > > > > > > >                     +       A given owner can control a single group of a given type,
> > > > > > > > >                     +       thus the type and the owner together identify the group.
> > > > > > > > >                     +       \footnote{Even though some group types only support
> > > > > > > > >                     +                       specific transports, group type identifiers
> > > > > > > > >                     +                       are global rather than transport-specific -
> > > > > > > > >                     +                       we don't expect a flood of new group types.}
> > > > > > > > >                     +\end{description}
> > > > > > > > >                     +
> > > > > > > > >                     +The following group types, and their identifiers, are currently specified):
> > > > > > > > >                     +\begin{description}
> > > > > > > > >                     +\item[SR-IOV group type (0x1)]
> > > > > > > > >                     +This device group has a PCI Single Root I/O Virtualization
> > > > > > > > >                     +(SR-IOV) physical function (PF) device as the owner and includes
> > > > > > > > >                     +all its SR-IOV virtual functions (VFs) as members (see
> > > > > > > > >                     +\hyperref[intro:PCIe]{[PCIe]}).
> > > > > > > > >                     +
> > > > > > > > >                     +The PF device itself is not a member of the group.
> > > > > > > > >                     +
> > > > > > > > >                     +The group type identifier for this group is 0x1.
> > > > > > > > >                     +
> > > > > > > > >                     +A member identifier for this group can have a value 0x1 to 0xFFFF
> > > > > > > > >                     +and equals the SR-IOV VF number of the member device (see
> > > > > > > > >                     +\hyperref[intro:PCIe]{[PCIe]}).
> > > > > > > > >                     +
> > > > > > > > >                     +Both owner and member devices for this group type use the Virtio
> > > > > > > > >                     +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> > > > > > > > >                     +\end{description}
> > > > > > > > >                     +
> > > > > > > > >                     +
> > > > > > > > >                     diff --git a/content.tex b/content.tex
> > > > > > > > >                     index 0c2d917..ffe45c4 100644
> > > > > > > > >                     --- a/content.tex
> > > > > > > > >                     +++ b/content.tex
> > > > > > > > >                     @@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> > > > > > > > >                      types. It is RECOMMENDED that devices generate version 4
> > > > > > > > >                      UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > > > > > > > > 
> > > > > > > > >                     +\input{admin.tex}
> > > > > > > > >                     +
> > > > > > > > >                      \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> > > > > > > > > 
> > > > > > > > >                      We start with an overview of device initialization, then expand on the
> > > > > > > > > 
> > > > > > > > >                 --
> > > > > > > > >                 Tell her I'll be waiting, in the usual place.
> > > > > > > > > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > 


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Cornelia Huck]

On Thu, Feb 09 2023, David Edmondson <david.edmondson@oracle.com> wrote:

> On Thursday, 2023-02-09 at 07:13:32 -05, Michael S. Tsirkin wrote:
>> +In particular, this is
>> +especially important for features limited to specific transports,
>> +as enabling these for more transports in future versions of the
>> +specification is highly likely to require changing the behaviour
>> +from drivers and devices.  Drivers and devices supporting
>
> "changing the behaviour of" or "changed behaviour from" (prefer the
> former).

Agreed, I'd prefer the former as well.

>
>> +multiple transports need to carefully maintain per-transport
>> +lists of allowed features.
>> +
>>  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
>>  The driver MUST NOT accept a feature which the device did not offer,
>>  and MUST NOT accept a feature which requires another feature which was
>>  not accepted.
>>  
>> +The driver MUST validate the feature bits offered by the device.
>> +The driver MUST ignore and MUST NOT accept any feature bit that is
>> +\begin{itemize}
>> +\item not described in this specification,
>> +\item marked as reserved,
>> +\item not supported for the specific transport,
>
> What does "supported" mean here? By the driver? By the specification in
> respect of this transport?

Maybe "not specified for the specific transport"?

>
>> +\item not defined for the device type.
>> +\end{itemize}


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [PATCH v10 02/10] admin: introduce device group and related concepts

[Cornelia Huck]

On Thu, Feb 09 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> +A member identifier for this group can have a value 0x1 to 0xFFFF
> +and equals the SR-IOV VF number of the member device (see
> +\hyperref[intro:PCIe]{[PCIe]}).

Maybe

"a value from 0x1 to 0xFFFF"

or

"a value between 0x1 and 0xFFFF inclusive"

?


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-comment] [PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[David Edmondson]

On Thursday, 2023-02-09 at 07:13:48 -05, Michael S. Tsirkin wrote:
> @@ -1031,6 +1035,19 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>          This field exists only if VIRTIO_F_RING_RESET has been
>          negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>  
> +\item[\field{admin_queue_index}]
> +        The device uses this to report the index of the first administration virtqueue.
> +        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
> +\item[\field{admin_queue_num}]
> +	The device uses this to report the number of the
> +	supported administration virtqueues.  Virtqueues with index
> +	between \field{admin_queue_index} and (\field{admin_queue_index} +
> +	\field{admin_queue_num}) inclusive serve as administration
> +	virtqueues.
> +	Thus the number of administration virtqueues equals
> +	(\field{admin_queue_num} + 1).

Really the number of queues is "admin_queue_num + 1"?

That seems ... odd.

> +	This field is valid only if VIRTIO_F_ADMIN_VQ has been
> +	negotiated.
>  \end{description}
>  
>  \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
> @@ -1117,6 +1134,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>  were used before the queue reset.
>  (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>  
> +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver
> +configures any administration virtqueues, the driver MUST
> +configure the administration virtqueues using the index
> +in the range \field{admin_queue_index} to
> +\field{admin_queue_index} + \field{admin_queue_num} inclusive.
> +The driver MAY configure less administration virtqueues than
> +supported by the device.
> +
>  \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
>  
>  The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
> @@ -7684,6 +7709,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  
>    \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
>    administration virtqueues.
> +  At the moment this feature is only supported for devices using
> +  \ref{sec:Virtio Transport Options / Virtio Over PCI
> +	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
> +	  as the transport and is reserved for future use for
> +	  devices using other transports (see
> +	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
> +	and
> +	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> +	handling features reserved for future use.
>  
>  \end{description}
-- 
She's got no name, but she is family.

Re: [virtio-comment] [PATCH v10 08/10] admin: command list discovery

[David Edmondson]

On Thursday, 2023-02-09 at 07:14:00 -05, Michael S. Tsirkin wrote:
> Add commands to find out which commands does each group support,
> as well as enable their use by driver.
> This will be especially useful once we have multiple group types.
>
> An alternative is per-type VQs. This is possible but will
> require more per-transport work. Discovery through the vq
> helps keep things contained.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 89 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 3026a79..4de606a 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -98,7 +98,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline
>  opcode & Name & Command Description \\
>  \hline \hline
> -0x0000 - 0x7FFF & - &  Group administration commands    \\
> +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> +0x0002 - 0x7FFF & - &  Group administration commands    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved    \\
>  \hline
> @@ -149,6 +151,92 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline
>  \end{tabular}
>  
> +Before sending any administration commands to the device, the driver
> +needs to communicate to the device which commands it is going to
> +use. Initially (after reset), only two commands are assumed to be used:
> +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Before sending any other commands for any member of a specific group to
> +the device, the driver queries the supported commands via
> +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
> +VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE
> +both use the following structure describing the
> +command opcodes:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_list {
> +       /* Indicates which of the below fields were returned
> +       le64 device_admin_cmds[];
> +};
> +\end{lstlisting}
> +
> +This structure is an array of 64 bit values in little-endian byte
> +order, in which a bit is set if the specific command opcode
> +is supported. Thus, \field{device_admin_cmds[0]} refers to the first
> 32-bit value

64-bit value.

> +in this array corresponding to opcodes 0 to 63,
> +\field{device_admin_cmds[1]} is the second 64-bit value
> +corresponding to opcodes 64 to 127, etc.
> +For example, the array of size 2 including
> +the values 0x3 in \field{device_admin_cmds[0]}
> +and 0x1 in \field{device_admin_cmds[1]} indicates that only
> +opcodes 0, 1 and 64 are supported.
> +The length of the array depends on the supported opcodes - it is
> +large enough to include bits set for all supported opcodes,
> +that is the length can be calculated by starting with the largest
> +supported opcode adding one, dividing by 64 and rounding up.
> +The array is also allowed to be larger and to additionally
> +include an arbitrary number of all-zero entries.
> +
> +Accordingly, bits 0 and 1 corresponding to opcode 0

"opcodes"

> +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
> +(VIRTIO_ADMIN_CMD_LIST_USE) are
> +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +This command has no command specific data.
> +The device, upon success, returns a result in
> +\field{command_specific_result} in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands supported for the given group.
> +
> +
> +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
> +is set to 0x1.
> +The \field{group_member_id} is unused. It is set to zero by driver.
> +The \field{command_specific_data} is in the format
> +\field{struct virtio_admin_cmd_list} describing the
> +list of administration commands used by the driver.
> +This command has no command specific result.
> +
> +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
> +query the list of commands valid for this group and before sending
> +any commands for any member of a group.
> +
> +The driver then enables use of some of the opcodes by sending to
> +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
> +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
> +both understood and used by the driver.
> +
> +If the device supports the command list used by the driver, the
> +device completes the command with status VIRTIO_ADMIN_STATUS_OK.
> +If the device does not support the command list, the device
> +completes the command with status
> +VIRTIO_ADMIN_STATUS_INVALID_FIELD.
> +
> +Note: drivers are assumed not to set bits in device_admin_cmds

\field{device_admin_cmds}

> +if they are not familiar with how the command opcode
> +is used, since devices could have dependencies between
> +command opcodes.
> +
> +It is assumed that all members in a group support and are used
> +with the same list of commands. However, for owner devices
> +supporting multiple group types, the list of supported commands
> +might differ between different group types.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>  
>  An administration virtqueue of an owner device is used to submit
-- 
They like the smell of it in Hollywood.

Re: [virtio-comment] Re: [virtio-dev] [PATCH v10 01/10] virtio: document forward compatibility guarantees

[Michael S. Tsirkin]

On Mon, Feb 13, 2023 at 01:06:32PM +0100, Cornelia Huck wrote:
> On Thu, Feb 09 2023, David Edmondson <david.edmondson@oracle.com> wrote:
> 
> > On Thursday, 2023-02-09 at 07:13:32 -05, Michael S. Tsirkin wrote:
> >> +In particular, this is
> >> +especially important for features limited to specific transports,
> >> +as enabling these for more transports in future versions of the
> >> +specification is highly likely to require changing the behaviour
> >> +from drivers and devices.  Drivers and devices supporting
> >
> > "changing the behaviour of" or "changed behaviour from" (prefer the
> > former).
> 
> Agreed, I'd prefer the former as well.
> 
> >
> >> +multiple transports need to carefully maintain per-transport
> >> +lists of allowed features.
> >> +
> >>  \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits}
> >>  The driver MUST NOT accept a feature which the device did not offer,
> >>  and MUST NOT accept a feature which requires another feature which was
> >>  not accepted.
> >>  
> >> +The driver MUST validate the feature bits offered by the device.
> >> +The driver MUST ignore and MUST NOT accept any feature bit that is
> >> +\begin{itemize}
> >> +\item not described in this specification,
> >> +\item marked as reserved,
> >> +\item not supported for the specific transport,
> >
> > What does "supported" mean here? By the driver? By the specification in
> > respect of this transport?
> 
> Maybe "not specified for the specific transport"?

given in mmio and ccw sections we say things like
	+At this time, devices and drivers utilizing Virtio Over MMIO
	+do not support the following features:
I tend to agree.

> >
> >> +\item not defined for the device type.
> >> +\end{itemize}

Re: [PATCH v10 02/10] admin: introduce device group and related concepts

[Michael S. Tsirkin]

On Mon, Feb 13, 2023 at 01:20:34PM +0100, Cornelia Huck wrote:
> On Thu, Feb 09 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > +A member identifier for this group can have a value 0x1 to 0xFFFF
> > +and equals the SR-IOV VF number of the member device (see
> > +\hyperref[intro:PCIe]{[PCIe]}).
> 
> Maybe
> 
> "a value from 0x1 to 0xFFFF"

I prefer this one, for brevity.

> or
> 
> "a value between 0x1 and 0xFFFF inclusive"
> 
> ?

[virtio-comment] Re: [virtio-dev] [PATCH v10 03/10] admin: introduce group administration commands

[David Edmondson]

On Thursday, 2023-02-09 at 07:13:39 -05, Michael S. Tsirkin wrote:
> This introduces a general structure for group administration commands,
> used to control device groups through their owner.
>
> Following patches will introduce specific commands and an interface for
> submitting these commands to the owner.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 101 insertions(+)
>
> diff --git a/admin.tex b/admin.tex
> index 2bc7322..46240f8 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -46,4 +46,105 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
>  PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
>  \end{description}
>  
> +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>  
> +Group administration commands can be issued through an owner
> +device to control member devices of a group.  This mechanism can
> +be used, for example, to configure a member device before it is
> +initialized by its driver.
> +\footnote{The term "administration" is intended to be interpreted
> +widely to include any kind of control. See specific commands
> +for detail.}
> +
> +All the group administration commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        le16 opcode;
> +        /*
> +         * 1 - SR-IOV
> +         * 2 - 65535 reserved

This formatting is ambiguous, as a hyphen is used for both ranges and
definitions. Perhaps switch to a colon for definitions? Or lose the
spaces for the range?

> +         */
> +        le16 group_type;
> +        /* unused, reserved for future extensions */
> +        u8 reserved1[12];
> +        le64 group_member_id;
> +        u8 command_specific_data[];
> +
> +        /* Device-writable part */
> +        le16 status;
> +        le16 status_qualifier;
> +        /* unused, reserved for future extensions */
> +        u8 reserved2[4];
> +        u8 command_specific_result[];
> +};
> +\end{lstlisting}
> +
> +For all commands, \field{opcode}, \field{group_type} and if
> +necessary \field{group_member_id} and \field{command_specific_data} are
> +set by the driver, and the owner device sets \field{status} and if
> +needed \field{status_qualifier} and
> +\field{command_specific_result}.
> +
> +As a rule, any unused device-readable fields are set to zero by the driver
> +and ignored by the device.  Any unused device-writeable fields are set to zero
> +by the device and ignored by the driver.
> +
> +\field{opcode} specifies the command. The valid
> +values for \field{opcode} can be found in the following table:
> +
> +\begin{tabular}{|l|l|}
> +\hline
> +opcode & Name & Command Description \\
> +\hline \hline
> +0x0000 - 0x7FFF & - &  Group administration commands    \\
> +\hline
> +0x8000 - 0xFFFF & - & Reserved    \\
> +\hline
> +\end{tabular}
> +
> +The \field{group_type} specifies the group type identifier.
> +The \field{group_member_id} specifies the member identifier within the group.
> +See section \ref{sec:Introduction / Terminology / Device group}
> +for the definition of the group type identifier and group member
> +identifier.
> +
> +The following table describes possible \field{status} values;
> +to simplify common implementations, they are intentionally
> +matching common Linux names and error numbers:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status (decimal) & Name & Description \\
> +\hline \hline
> +00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> +\hline
> +22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
> +\hline
> +other   & -    & group administration command error  \\
> +\hline
> +\end{tabular}
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
> +is reserved and set to zero by the device.
> +
> +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL,
> +the following table describes possible \field{status_qialifier} values:
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Status & Name & Description \\
> +\hline \hline
> +0x00   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND   & command error: no additional information  \\
> +\hline
> +0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE    & unsupported or invalid \field{opcode}  \\
> +\hline
> +0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
> +\hline
> +0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
> +\hline
> +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEM    & unsupported or invalid \field{group_member_id} \\
> +\hline
> +0x05-0xFFFF   & -    & reserved for future use \\
> +\hline
> +\end{tabular}
-- 
I used to worry, thought I was goin' mad in a hurry.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Cornelia Huck]

On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>>>> For some system calls and library functions (e.g.,
>>>>          getpriority(2)), -1 is a valid return on success.  In such cases,
>>>>          a successful return can be distinguished from an error return by
>>>>          setting errno to zero before the call, and then, if the call
>>>>          returns a status that indicates that an error may have occurred,
>>>>          checking to see if errno has a nonzero value.
>>>>
>>>>
>>>>
>>>>> Description is already good enough to describe what they are.
>>>>> Can we please drop Linux wording?
>>>>
>>>> But why should we? It's where 22 comes from so this way people are not
>>>> wondering about the value, and it's somewhat helpful for Linux
>>>> developers.
>>>>
>>>
>>> I also think we should not mention Linux. I don't think it's mentioned
>>> currently in the spec and no good reason to do so now.
>> 
>> But we do: fuse, input at least both do.
>> 
>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>>> just 1 (the first number after the OK status).
>> 
>> 22 makes it a tiny bit easier for kvm. So why not.
>
> Because people comment on that in the review :) and because a 
> specification should be OS independent.
> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
>
> not a big issue for me, but I prefer to have a cleaner spec than maybe 
> simplify something in a very specific OS.

Well, my take is:
- we have to pick *something*
- using EINVAL == 22 is very convenient for one of the most common (the
  most common?) implementors
- noting that we're trying to follow what Linux uses makes it clear what
  to pick for any new return codes


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH v10 07/10] ccw: document ADMIN_VQ as reserved

[Cornelia Huck]

On Thu, Feb 09 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> Adding relevant registers needs more work and it's not
> clear what the use-case will be as currently only
> the PCI transport is supported. But let's keep the
> door open on this.
> We already say it's reserved in a central place, but it
> does not hurt to remind implementers to mask it.
>
> Note: there are more features to add to this list.
> Will be done later with a patch on top.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  content.tex | 12 ++++++++++++
>  1 file changed, 12 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio] Re: [PATCH v10 10/10] ccw: document more reserved features

[Cornelia Huck]

On Thu, Feb 09 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> vq reset and shared memory are unsupported, too.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>
>
> This is not strictly part of this patchset, but just to
> give you the idea of how we can expand this down the road.

We could split this out to a separate change on top. I think I should
just open an issue for that.

>
>
>  content.tex | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index 6a50b39..b677006 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2983,6 +2983,8 @@ \subsection{Features reserved for future use}\label{sec:Virtio Transport Options
>  \begin{itemize}
>  
>  \item VIRTIO_F_ADMIN_VQ
> +\item VIRTIO_F_RING_RESET
> +\item Shared memory regions including VIRTIO_PMEM_F_SHMEM_REGION
>  
>  \end{itemize}


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio] Re: [PATCH v10 10/10] ccw: document more reserved features

[Cornelia Huck]

On Mon, Feb 13 2023, Cornelia Huck <cohuck@redhat.com> wrote:

> On Thu, Feb 09 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
>> vq reset and shared memory are unsupported, too.
>>
>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> ---
>>
>>
>> This is not strictly part of this patchset, but just to
>> give you the idea of how we can expand this down the road.
>
> We could split this out to a separate change on top. I think I should
> just open an issue for that.

Opened https://github.com/oasis-tcs/virtio-spec/issues/160.


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Max Gurtovoy]

On 13/02/2023 14:42, Cornelia Huck wrote:
> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> 
>> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
>>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>>>>> For some system calls and library functions (e.g.,
>>>>>           getpriority(2)), -1 is a valid return on success.  In such cases,
>>>>>           a successful return can be distinguished from an error return by
>>>>>           setting errno to zero before the call, and then, if the call
>>>>>           returns a status that indicates that an error may have occurred,
>>>>>           checking to see if errno has a nonzero value.
>>>>>
>>>>>
>>>>>
>>>>>> Description is already good enough to describe what they are.
>>>>>> Can we please drop Linux wording?
>>>>>
>>>>> But why should we? It's where 22 comes from so this way people are not
>>>>> wondering about the value, and it's somewhat helpful for Linux
>>>>> developers.
>>>>>
>>>>
>>>> I also think we should not mention Linux. I don't think it's mentioned
>>>> currently in the spec and no good reason to do so now.
>>>
>>> But we do: fuse, input at least both do.
>>>
>>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>>>> just 1 (the first number after the OK status).
>>>
>>> 22 makes it a tiny bit easier for kvm. So why not.
>>
>> Because people comment on that in the review :) and because a
>> specification should be OS independent.
>> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
>>
>> not a big issue for me, but I prefer to have a cleaner spec than maybe
>> simplify something in a very specific OS.
> 
> Well, my take is:
> - we have to pick *something*
> - using EINVAL == 22 is very convenient for one of the most common (the
>    most common?) implementors
> - noting that we're trying to follow what Linux uses makes it clear what
>    to pick for any new return codes
> 

Can we agree on 22 without mentioning Linux ?

[virtio] Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Cornelia Huck]

On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 13/02/2023 14:42, Cornelia Huck wrote:
>> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>> 
>>> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
>>>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>>>>>> For some system calls and library functions (e.g.,
>>>>>>           getpriority(2)), -1 is a valid return on success.  In such cases,
>>>>>>           a successful return can be distinguished from an error return by
>>>>>>           setting errno to zero before the call, and then, if the call
>>>>>>           returns a status that indicates that an error may have occurred,
>>>>>>           checking to see if errno has a nonzero value.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Description is already good enough to describe what they are.
>>>>>>> Can we please drop Linux wording?
>>>>>>
>>>>>> But why should we? It's where 22 comes from so this way people are not
>>>>>> wondering about the value, and it's somewhat helpful for Linux
>>>>>> developers.
>>>>>>
>>>>>
>>>>> I also think we should not mention Linux. I don't think it's mentioned
>>>>> currently in the spec and no good reason to do so now.
>>>>
>>>> But we do: fuse, input at least both do.
>>>>
>>>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>>>>> just 1 (the first number after the OK status).
>>>>
>>>> 22 makes it a tiny bit easier for kvm. So why not.
>>>
>>> Because people comment on that in the review :) and because a
>>> specification should be OS independent.
>>> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
>>>
>>> not a big issue for me, but I prefer to have a cleaner spec than maybe
>>> simplify something in a very specific OS.
>> 
>> Well, my take is:
>> - we have to pick *something*
>> - using EINVAL == 22 is very convenient for one of the most common (the
>>    most common?) implementors
>> - noting that we're trying to follow what Linux uses makes it clear what
>>    to pick for any new return codes
>> 
>
> Can we agree on 22 without mentioning Linux ?

See my third point above: we want people to be aware why we chose 22, so
any new values will match up as well.


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Max Gurtovoy]

On 13/02/2023 15:13, Cornelia Huck wrote:
> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> 
>> On 13/02/2023 14:42, Cornelia Huck wrote:
>>> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>
>>>> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
>>>>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>>>>>>> For some system calls and library functions (e.g.,
>>>>>>>            getpriority(2)), -1 is a valid return on success.  In such cases,
>>>>>>>            a successful return can be distinguished from an error return by
>>>>>>>            setting errno to zero before the call, and then, if the call
>>>>>>>            returns a status that indicates that an error may have occurred,
>>>>>>>            checking to see if errno has a nonzero value.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Description is already good enough to describe what they are.
>>>>>>>> Can we please drop Linux wording?
>>>>>>>
>>>>>>> But why should we? It's where 22 comes from so this way people are not
>>>>>>> wondering about the value, and it's somewhat helpful for Linux
>>>>>>> developers.
>>>>>>>
>>>>>>
>>>>>> I also think we should not mention Linux. I don't think it's mentioned
>>>>>> currently in the spec and no good reason to do so now.
>>>>>
>>>>> But we do: fuse, input at least both do.
>>>>>
>>>>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>>>>>> just 1 (the first number after the OK status).
>>>>>
>>>>> 22 makes it a tiny bit easier for kvm. So why not.
>>>>
>>>> Because people comment on that in the review :) and because a
>>>> specification should be OS independent.
>>>> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
>>>>
>>>> not a big issue for me, but I prefer to have a cleaner spec than maybe
>>>> simplify something in a very specific OS.
>>>
>>> Well, my take is:
>>> - we have to pick *something*
>>> - using EINVAL == 22 is very convenient for one of the most common (the
>>>     most common?) implementors
>>> - noting that we're trying to follow what Linux uses makes it clear what
>>>     to pick for any new return codes
>>>
>>
>> Can we agree on 22 without mentioning Linux ?
> 
> See my third point above: we want people to be aware why we chose 22, so
> any new values will match up as well.

Why we need people to be aware ?

In NVMe spec for example there are many status codes (~30-50) and people 
are not aware why the committee choose  0x02 for "Invalid Field in Command".
And frankly, I don't think they care.

My point is that we need to make our lives, as spec developers and 
maintainers, easier.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio] Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Cornelia Huck]

On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 13/02/2023 15:13, Cornelia Huck wrote:
>> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>> 
>>> On 13/02/2023 14:42, Cornelia Huck wrote:
>>>> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>
>>>>> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
>>>>>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>>>>>>>> For some system calls and library functions (e.g.,
>>>>>>>>            getpriority(2)), -1 is a valid return on success.  In such cases,
>>>>>>>>            a successful return can be distinguished from an error return by
>>>>>>>>            setting errno to zero before the call, and then, if the call
>>>>>>>>            returns a status that indicates that an error may have occurred,
>>>>>>>>            checking to see if errno has a nonzero value.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Description is already good enough to describe what they are.
>>>>>>>>> Can we please drop Linux wording?
>>>>>>>>
>>>>>>>> But why should we? It's where 22 comes from so this way people are not
>>>>>>>> wondering about the value, and it's somewhat helpful for Linux
>>>>>>>> developers.
>>>>>>>>
>>>>>>>
>>>>>>> I also think we should not mention Linux. I don't think it's mentioned
>>>>>>> currently in the spec and no good reason to do so now.
>>>>>>
>>>>>> But we do: fuse, input at least both do.
>>>>>>
>>>>>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>>>>>>> just 1 (the first number after the OK status).
>>>>>>
>>>>>> 22 makes it a tiny bit easier for kvm. So why not.
>>>>>
>>>>> Because people comment on that in the review :) and because a
>>>>> specification should be OS independent.
>>>>> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
>>>>>
>>>>> not a big issue for me, but I prefer to have a cleaner spec than maybe
>>>>> simplify something in a very specific OS.
>>>>
>>>> Well, my take is:
>>>> - we have to pick *something*
>>>> - using EINVAL == 22 is very convenient for one of the most common (the
>>>>     most common?) implementors
>>>> - noting that we're trying to follow what Linux uses makes it clear what
>>>>     to pick for any new return codes
>>>>
>>>
>>> Can we agree on 22 without mentioning Linux ?
>> 
>> See my third point above: we want people to be aware why we chose 22, so
>> any new values will match up as well.
>
> Why we need people to be aware ?

people == people adding new error codes

>
> In NVMe spec for example there are many status codes (~30-50) and people 
> are not aware why the committee choose  0x02 for "Invalid Field in Command".
> And frankly, I don't think they care.
>
> My point is that we need to make our lives, as spec developers and 
> maintainers, easier.

And that's why I think we should keep the proposed wording :) I think it
will make the life of Future Me easier.


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses

[David Edmondson]

On Thursday, 2023-02-09 at 07:14:04 -05, Michael S. Tsirkin wrote:
> Add conformance clauses for admin commands and admin virtqueues.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  admin.tex | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 190 insertions(+)
>
> diff --git a/admin.tex b/admin.tex
> index 4de606a..f9b1b68 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -237,6 +237,119 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  supporting multiple group types, the list of supported commands
>  might differ between different group types.
>  
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The device MUST validate \field{opcode}, \field{group_type} and
> +\field{group_member_id}, and if any of these has an invalid or
> +unsupported value, set \field{status} to
> +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
> +accordingly:
> +\begin{itemize}
> +\item if \field{group_type} is invalid, \field{status_qualifier}
> +	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
> +\item otherwise, if \field{opcode} is invalid,
> +	\field{status_qualifier} MUST be set to
> +	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
> +\item otherwise, if \field{group_member_id} is used by the
> +	specific command and is invalid, \field{status_qualifier} MUST be
> +	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEM.
> +\end{itemize}
> +
> +If a command completes successfully, the device MUST set
> +\field{status} to VIRTIO_ADMIN_STATUS_OK.
> +
> +If a command fails, the device MUST set
> +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.

s/from/than/

> +
> +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
> +device state MUST NOT change, that is the command MUST NOT have
> +any side effects on the device, in particular the device MUST not

s/not/NOT/

> +enter an error state as a result of this command.
> +
> +If a command fails, the device state generally SHOULD NOT change,
> +as far as possible.
> +
> +The device MAY enforce additional restrictions and dependencies on
> +opcodes used by the driver and MAY fail the command
> +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
> +if the list of commands used violate internal device dependencies.
> +
> +If the device supports multiple group types, commands for each group
> +type MUST operate independently of each other, in particular,
> +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
> +for different group types.

Is it permitted to have multiple groups of the same type?

If so, can distinct groups of the same type return different results for
VIRTIO_ADMIN_CMD_LIST_QUERY?

> +
> +After reset, if the device supports a given group type
> +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
> +the device MUST assume
> +that the list of legal commands used by the driver consists of
> +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
> +the device MUST set the list of legal commands used by the driver
> +to the one supplied in \field{command_specific_data}.
> +
> +The device MUST set the list of legal commands used by the driver
> +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
> +
> +The device MUST validate commands against the list used by
> +the driver and MUST fail any commands not in the list with
> +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +and \field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.
> +
> +The list of supported commands MUST NOT shrink (but MAY expand):
> +after reporting a given command as supported through
> +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it
> +as unsupported.

Even if it was never used?

> Further, after a given set of commands has been
> +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a
> +device or system reset the device SHOULD complete successfully
> +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same
> +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails
> +after a device or system reset, the device MUST not fail it
> +solely because of the command list used.  Failure to do so would
> +interfere with resuming from suspend and error recovery.
> +
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
> +
> +The driver MAY discover whether device supports a specific group type
> +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
> +\field{group_type}.
> +
> +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
> +and wait for it to be completed with status
> +VIRTIO_ADMIN_STATUS_OK before issuing any commands
> +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
> +and VIRTIO_ADMIN_CMD_LIST_USE).
> +
> +The driver SHOULD NOT set bits in device_admin_cmds

\field{device_admin_cmds}

> +if it is not familiar with how the command opcode
> +is used, as dependencies between command opcodes might exist.
> +
> +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
> +the use of any commands not previously reported as
> +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.
> +
> +The driver MUST NOT use any commands for a given group type
> +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
> +list of command opcodes and group type.
> +
> +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
> +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
> +with respective bits cleared in \field{command_specific_data}.

This seems to run contrary to the paragraph relating to resuming from
suspend above.

> +
> +The driver MUST handle a command error with a reserved \field{status}
> +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
> +(except possibly for different error reporting/diagnostic messages).
> +
> +The driver MUST handle a command error with a reserved
> +\field{status_qualifier} value in the same way as
> +\field{status_qualifier} set to
> +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
> +different error reporting/diagnostic messages).
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>  
>  An administration virtqueue of an owner device is used to submit
> @@ -309,3 +422,80 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic
>  of the specification are designed, new fields can be added to the
>  tail of a structure, with the driver/device using the full
>  structure without concern for versioning.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The device MUST support device-readable and device-writeable buffers
> +shorter than described in this specification, by
> +\begin{enumerate}
> +\item acting as if any data that would be read outside the
> +device-readable buffers is set to zero, and
> +\item discarding data that would be written outside the
> +specified device-writeable buffers.
> +\end{enumerate}
> +
> +The device MUST support device-readable and device-writeable buffers
> +longer than described in this specification, by
> +\begin{enumerate}
> +\item ignoring any data in device-readable buffers outside
> +the expected length, and
> +\item only writing the expected structure to the device-writeable
> +buffers, ignoring any extra buffers, and reporting the
> +actual length of data written, in bytes,
> +as buffer used length.
> +\end{enumerate}
> +
> +The device SHOULD initialize the device-writeable buffer
> +up to the length of the structure described by this specification or
> +the length of the buffer supplied by the driver (even if the buffer is
> +all set to zero), whichever is shorter.
> +
> +The device MUST NOT fail a command solely because the buffers
> +provided are shorter or longer than described in this
> +specification.
> +
> +The device MUST initialize the device-writeable part of
> +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in

s/bit/bits/

> +size.
> +
> +The device MUST initialize \field{status} in \field{struct
> +virtio_admin_cmd}.
> +
> +The device MUST process commands on a given administration virtqueue
> +in the order in which they are queued.
> +
> +If multiple administration virtqueues have been configured,
> +device MAY process commands on distinct virtqueues with
> +no order constraints.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}
> +
> +The driver MAY supply device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are longer than described in
> +this specification.
> +
> +The driver SHOULD supply device-readable part of
> +\field{struct virtio_admin_cmd} that is at least as
> +large as the structure described by this specification
> +(even if the structure is all set to zero).
> +
> +The driver MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit

s/bit/bits/

> +in length.
> +
> +The device MUST supply both device-readable or device-writeable parts
> +of \field{struct virtio_admin_cmd} that are larger than zero in
> +length. However, \field{command_specific_data} and
> +\field{command_specific_result} MAY be zero in length, unless
> +specified otherwise for the command.
> +
> +The driver MUST NOT assume that the device will initialize the whole
> +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
> +the driver MUST act as if the structure
> +outside the part of the buffer used by the device
> +is set to zero.
> +
> +If multiple administration virtqueues have been configured,
> +the driver MUST ensure ordering for commands
> +placed on different administration virtqueues.
> +>>>>>>> 29f6525... admin: conformance clauses

Leftover from a merge?
-- 
I didn't get a lot in class, but I know it don't come in a shot glass.

Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Max Gurtovoy]

On 13/02/2023 15:36, Cornelia Huck wrote:
> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> 
>> On 13/02/2023 15:13, Cornelia Huck wrote:
>>> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>
>>>> On 13/02/2023 14:42, Cornelia Huck wrote:
>>>>> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>
>>>>>> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
>>>>>>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>>>>>>>>> For some system calls and library functions (e.g.,
>>>>>>>>>             getpriority(2)), -1 is a valid return on success.  In such cases,
>>>>>>>>>             a successful return can be distinguished from an error return by
>>>>>>>>>             setting errno to zero before the call, and then, if the call
>>>>>>>>>             returns a status that indicates that an error may have occurred,
>>>>>>>>>             checking to see if errno has a nonzero value.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Description is already good enough to describe what they are.
>>>>>>>>>> Can we please drop Linux wording?
>>>>>>>>>
>>>>>>>>> But why should we? It's where 22 comes from so this way people are not
>>>>>>>>> wondering about the value, and it's somewhat helpful for Linux
>>>>>>>>> developers.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I also think we should not mention Linux. I don't think it's mentioned
>>>>>>>> currently in the spec and no good reason to do so now.
>>>>>>>
>>>>>>> But we do: fuse, input at least both do.
>>>>>>>
>>>>>>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>>>>>>>> just 1 (the first number after the OK status).
>>>>>>>
>>>>>>> 22 makes it a tiny bit easier for kvm. So why not.
>>>>>>
>>>>>> Because people comment on that in the review :) and because a
>>>>>> specification should be OS independent.
>>>>>> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
>>>>>>
>>>>>> not a big issue for me, but I prefer to have a cleaner spec than maybe
>>>>>> simplify something in a very specific OS.
>>>>>
>>>>> Well, my take is:
>>>>> - we have to pick *something*
>>>>> - using EINVAL == 22 is very convenient for one of the most common (the
>>>>>      most common?) implementors
>>>>> - noting that we're trying to follow what Linux uses makes it clear what
>>>>>      to pick for any new return codes
>>>>>
>>>>
>>>> Can we agree on 22 without mentioning Linux ?
>>>
>>> See my third point above: we want people to be aware why we chose 22, so
>>> any new values will match up as well.
>>
>> Why we need people to be aware ?
> 
> people == people adding new error codes

These people should submit a patch for review and get comments from 
maintainers.

> 
>>
>> In NVMe spec for example there are many status codes (~30-50) and people
>> are not aware why the committee choose  0x02 for "Invalid Field in Command".
>> And frankly, I don't think they care.
>>
>> My point is that we need to make our lives, as spec developers and
>> maintainers, easier.
> 
> And that's why I think we should keep the proposed wording :) I think it
> will make the life of Future Me easier.
> 

I see.
Again, my stand for spec wording is not mentioning Linux and not 
following it's errnos.

Having said that, I don't really mind ignoring this sentence, although 
it wasn't part of my original patch.

[virtio] Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Mon, Feb 13, 2023 at 02:13:43PM +0100, Cornelia Huck wrote:
> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> 
> > On 13/02/2023 14:42, Cornelia Huck wrote:
> >> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> >> 
> >>> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
> >>>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
> >>>>>> For some system calls and library functions (e.g.,
> >>>>>>           getpriority(2)), -1 is a valid return on success.  In such cases,
> >>>>>>           a successful return can be distinguished from an error return by
> >>>>>>           setting errno to zero before the call, and then, if the call
> >>>>>>           returns a status that indicates that an error may have occurred,
> >>>>>>           checking to see if errno has a nonzero value.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Description is already good enough to describe what they are.
> >>>>>>> Can we please drop Linux wording?
> >>>>>>
> >>>>>> But why should we? It's where 22 comes from so this way people are not
> >>>>>> wondering about the value, and it's somewhat helpful for Linux
> >>>>>> developers.
> >>>>>>
> >>>>>
> >>>>> I also think we should not mention Linux. I don't think it's mentioned
> >>>>> currently in the spec and no good reason to do so now.
> >>>>
> >>>> But we do: fuse, input at least both do.
> >>>>
> >>>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
> >>>>> just 1 (the first number after the OK status).
> >>>>
> >>>> 22 makes it a tiny bit easier for kvm. So why not.
> >>>
> >>> Because people comment on that in the review :) and because a
> >>> specification should be OS independent.
> >>> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
> >>>
> >>> not a big issue for me, but I prefer to have a cleaner spec than maybe
> >>> simplify something in a very specific OS.
> >> 
> >> Well, my take is:
> >> - we have to pick *something*
> >> - using EINVAL == 22 is very convenient for one of the most common (the
> >>    most common?) implementors
> >> - noting that we're trying to follow what Linux uses makes it clear what
> >>    to pick for any new return codes
> >> 
> >
> > Can we agree on 22 without mentioning Linux ?
> 
> See my third point above: we want people to be aware why we chose 22, so
> any new values will match up as well.

In fact I think it is a good idea to
1. actually link to where one can see EINVAL=22 so people know where to find
   more values
2. ask driver to handle any other error same as EINVAL for now - we don't want
   new feature bits if all we do down the road is add more error codes.

I'll add this in the next version.

-- 
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[virtio-comment] RE: [PATCH v10 00/10] Introduce device group and device management

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Sunday, February 12, 2023 4:42 AM
> 
> On Sat, Feb 11, 2023 at 06:49:52PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, February 9, 2023 7:13 AM
> >
> > [..]
> >
> > > My intent is to try and support both SR-IOV and SIOV usecases with
> > > the same structure and maybe even the same VQ.
> > >
> > Probably it is implicit, but is better for us to mention few examples
> > use cases here something like,
> >
> > Admin queue is an infrastructure to support use cases like:
> > a. SIOV devices life cycles management (create, destroy, query,
> > configure) b. SR-IOV VF device/resource management (SR-PCIM) c. May be
> > transport
> >
> > > For example, it might make sense to split creating/destroying SIOV
> > > devices from the transport passing data from the guest - the driver
> > > would then not negotiate VIRTIO_F_SR_IOV (which then means auto-
> provisioning).
> >
> > Above intermixing of SIOV and SR_IOV is very confusing.
> > Can you please drop it and have simpler examples like above?
> 
> You mean in the cover letter? The more info in the cover letter the better, this
> stuff does not end up in git.
Yes, in the cover letter. Its ok its not in the git.
Maybe we can start also in the git. Sometimes it's useful for series like this.
it is easy to review and digest the use of new feature introduction when those use cases are described in cover letter.

> I'll add your examples too though, thanks!
Ok thanks.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Sunday, February 12, 2023 4:50 AM
> 
> On Sat, Feb 11, 2023 at 06:50:37PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, February 9, 2023 7:14 AM
> >
> > [..]
> >
> > > +Group administration commands can be issued through an owner device
> > > +to control member devices of a group.
> > can be issued by the owner device to control ...
> 
> Well it's clearly the driver that issues the commands. Don't see how
> can a device issue them. If instead of "through an owner device"
> you prefer a more verbose "the driver of an owner device" then
> I can live with that. Let me know.

"through" wording is convoluted and it hints that someone else is preparing the command and its tunneled through the admin vq.
Which is clearly not the scope here.

How about below simple statements like below?

Group administrative commands control the member devices.
The driver sends group administrative commands to the owner device.

> > General spec structure is describing the "rules" as device and driver side
> requirements.
> > Can you please move above rule paragraph belongs to device and driver
> requirements section?
> 
> Generally we have a bit of duplication. A high level description
> showing how things work together. Then per-device/driver conforumance
> statements. So I think this should stay but I will look at adding
> conformance clauses too.
Ok. that will be good.
Yes. General description is good without the "rule" wording so that we have clear separation of conformance and description.

> > I am not sure it should be intentional and mention of Linux.
> > Any OS would need multiple error values to know the error cause.
> > Linux doesn't have name "OK" either for well-known value of 0.
> >
> > For example
> > $ errno 22
> > EINVAL 22 Invalid argument
> >
> > $ errno 0
> > This has not output for it and doesn't exist in errno.h.
> 
> No name OK true, but it does use 0 to signal success sometimes:
Yes, but the wording is "success" and not "ok". :)

> 
> https://man7.org/linux/man-pages/man3/errno.3.html
> 
> For some system calls and library functions (e.g.,
>        getpriority(2)), -1 is a valid return on success.  In such cases,
>        a successful return can be distinguished from an error return by
>        setting errno to zero before the call, and then, if the call
>        returns a status that indicates that an error may have occurred,
>        checking to see if errno has a nonzero value.
> 
> 
> 
> > Description is already good enough to describe what they are.
> > Can we please drop Linux wording?
> 
> But why should we? 
Because of following reasons.

1. If we start putting 22 with Linux annotations, in few weeks virtio Net device section will be full of annotation of ethtool, tc, ip config and more for Linux developers.
For example, the current two interrupt moderation patches need to write ethtool options details to match to following this Linux example here.
It is better to avoid such things and keep spec OS agnostics, even though it is addressing and fitting into the Linux use cases.

2. Virtio error code to Linux error code switch-case is simple routine to have.

3. Every time virtio spec to refer to errno.h for finding right value is opposite of what spec may want to achieve.
If an error code doesn't match to errno.h, now spec developers and reviewers to review what is not defined by errno.h to find as unique value.

3.1 And if that is taken in future by errno.h for something else?

All of this is not worth a simple switch case statement to deal with.

RE: [virtio-comment] [PATCH v10 02/10] admin: introduce device group and related concepts

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 2:58 PM

> There's also the concern of VF migration under MRIOV where VFs might not
> exit. I'm guessing trying to include that right now is overthinking. So maybe this
> should mention that VF Migration Capable should be clear.
> 
MRIOV is already deprecated in PCI spec.
We can ignore VF migration capable wording.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Tue, Feb 14, 2023 at 01:18:42AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Sunday, February 12, 2023 4:50 AM
> > 
> > On Sat, Feb 11, 2023 at 06:50:37PM +0000, Parav Pandit wrote:
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Thursday, February 9, 2023 7:14 AM
> > >
> > > [..]
> > >
> > > > +Group administration commands can be issued through an owner device
> > > > +to control member devices of a group.
> > > can be issued by the owner device to control ...
> > 
> > Well it's clearly the driver that issues the commands. Don't see how
> > can a device issue them. If instead of "through an owner device"
> > you prefer a more verbose "the driver of an owner device" then
> > I can live with that. Let me know.
> 
> "through" wording is convoluted and it hints that someone else is preparing the command and its tunneled through the admin vq.
> Which is clearly not the scope here.

It is exactly someone else - the driver not the owner itself.

> 
> How about below simple statements like below?
> 
> Group administrative commands control the member devices.

> The driver sends group administrative commands to the owner device.

The point is that it's the same group though which this misses.
I think something like:

	The driver sends group administration commands to the owner device of
	the group to control member devices of the group.

will do the trick.

> 
> > > General spec structure is describing the "rules" as device and driver side
> > requirements.
> > > Can you please move above rule paragraph belongs to device and driver
> > requirements section?
> > 
> > Generally we have a bit of duplication. A high level description
> > showing how things work together. Then per-device/driver conforumance
> > statements. So I think this should stay but I will look at adding
> > conformance clauses too.
> Ok. that will be good.
> Yes. General description is good without the "rule" wording so that we have clear separation of conformance and description.

OK I will consider replacing "General rule" with "Generally". Not sure why it is better:
conformance uses rfc2119 which does not include any wording about rule
but oh well.


> > > I am not sure it should be intentional and mention of Linux.
> > > Any OS would need multiple error values to know the error cause.
> > > Linux doesn't have name "OK" either for well-known value of 0.
> > >
> > > For example
> > > $ errno 22
> > > EINVAL 22 Invalid argument
> > >
> > > $ errno 0
> > > This has not output for it and doesn't exist in errno.h.
> > 
> > No name OK true, but it does use 0 to signal success sometimes:
> Yes, but the wording is "success" and not "ok". :)

Hmm. A but wordy and we have _OK elsewhere in the spec, while Linux
does not have either OK ot SUCCESS. Just informal text.
Will consider how to make it clearer.

> > 
> > https://man7.org/linux/man-pages/man3/errno.3.html
> > 
> > For some system calls and library functions (e.g.,
> >        getpriority(2)), -1 is a valid return on success.  In such cases,
> >        a successful return can be distinguished from an error return by
> >        setting errno to zero before the call, and then, if the call
> >        returns a status that indicates that an error may have occurred,
> >        checking to see if errno has a nonzero value.
> > 
> > 
> > 
> > > Description is already good enough to describe what they are.
> > > Can we please drop Linux wording?
> > 
> > But why should we? 
> Because of following reasons.
> 
> 1. If we start putting 22 with Linux annotations, in few weeks virtio Net device section will be full of annotation of ethtool, tc, ip config and more for Linux developers.
> For example, the current two interrupt moderation patches need to write ethtool options details to match to following this Linux example here.
> It is better to avoid such things and keep spec OS agnostics, even though it is addressing and fitting into the Linux use cases.
> 
> 2. Virtio error code to Linux error code switch-case is simple routine to have.
> 
> 3. Every time virtio spec to refer to errno.h for finding right value is opposite of what spec may want to achieve.
> If an error code doesn't match to errno.h, now spec developers and reviewers to review what is not defined by errno.h to find as unique value.
> 
> 3.1 And if that is taken in future by errno.h for something else?
> 
> All of this is not worth a simple switch case statement to deal with.

I think we'll have to agree to disagree here. Years working on virt
taught me that matching some existing interface is usually better than
coming up with our own. Even if it is a bit more work upfront it pays
dividends later. Forcing some discipline in not coming up with
outlandish "EFOOBAR" codes no one driver will end up using is a good idea too.
And yes we can come up with crazy conventions like 0xdead for success.
Does not mean we should.

-- 
MST

[virtio-comment] Re: [virtio] Re: [virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands

[Cornelia Huck]

On Mon, Feb 13 2023, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Feb 13, 2023 at 02:13:43PM +0100, Cornelia Huck wrote:
>> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>> 
>> > On 13/02/2023 14:42, Cornelia Huck wrote:
>> >> On Mon, Feb 13 2023, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>> >> 
>> >>> On 13/02/2023 10:16, Michael S. Tsirkin wrote:
>> >>>> On Mon, Feb 13, 2023 at 02:54:47AM +0200, Max Gurtovoy wrote:
>> >>>>>> For some system calls and library functions (e.g.,
>> >>>>>>           getpriority(2)), -1 is a valid return on success.  In such cases,
>> >>>>>>           a successful return can be distinguished from an error return by
>> >>>>>>           setting errno to zero before the call, and then, if the call
>> >>>>>>           returns a status that indicates that an error may have occurred,
>> >>>>>>           checking to see if errno has a nonzero value.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>> Description is already good enough to describe what they are.
>> >>>>>>> Can we please drop Linux wording?
>> >>>>>>
>> >>>>>> But why should we? It's where 22 comes from so this way people are not
>> >>>>>> wondering about the value, and it's somewhat helpful for Linux
>> >>>>>> developers.
>> >>>>>>
>> >>>>>
>> >>>>> I also think we should not mention Linux. I don't think it's mentioned
>> >>>>> currently in the spec and no good reason to do so now.
>> >>>>
>> >>>> But we do: fuse, input at least both do.
>> >>>>
>> >>>>> Also value of 22 is not mandatory for this EINVAL status code. It can be
>> >>>>> just 1 (the first number after the OK status).
>> >>>>
>> >>>> 22 makes it a tiny bit easier for kvm. So why not.
>> >>>
>> >>> Because people comment on that in the review :) and because a
>> >>> specification should be OS independent.
>> >>> 22 might be EINVAL in Linux but a success in MyOS is my lucky number is 22.
>> >>>
>> >>> not a big issue for me, but I prefer to have a cleaner spec than maybe
>> >>> simplify something in a very specific OS.
>> >> 
>> >> Well, my take is:
>> >> - we have to pick *something*
>> >> - using EINVAL == 22 is very convenient for one of the most common (the
>> >>    most common?) implementors
>> >> - noting that we're trying to follow what Linux uses makes it clear what
>> >>    to pick for any new return codes
>> >> 
>> >
>> > Can we agree on 22 without mentioning Linux ?
>> 
>> See my third point above: we want people to be aware why we chose 22, so
>> any new values will match up as well.
>
> In fact I think it is a good idea to
> 1. actually link to where one can see EINVAL=22 so people know where to find
>    more values
> 2. ask driver to handle any other error same as EINVAL for now - we don't want
>    new feature bits if all we do down the road is add more error codes.
>
> I'll add this in the next version.

Wfm.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, February 14, 2023 2:47 AM
> 
> > >
> > > Well it's clearly the driver that issues the commands. Don't see how
> > > can a device issue them. If instead of "through an owner device"
> > > you prefer a more verbose "the driver of an owner device" then I can
> > > live with that. Let me know.
> >
> > "through" wording is convoluted and it hints that someone else is preparing
> the command and its tunneled through the admin vq.
> > Which is clearly not the scope here.
> 
> It is exactly someone else - the driver not the owner itself.
I completely lost you.
My understanding is below.
For a PCI PF device, AQ is on the PF (owner device who creates the AQ).
PF driver is issuing the AQ command and waiting for the completion to arrive.
PF driver decides which AQ to use to issue the command.
This AQ is no different than net cvq, or tx/rx virtqueue.
Only difference is, it carries generic command instead of device type specific command.

In some other use case that we need to write in the cover letter, that this queue may act as a transport for some SIOV or something.
Even in such scenario, command is still issued to the AQ by the owner PF driver.

What did I miss?
If none, your below description looks good to me.

> 	The driver sends group administration commands to the owner device
> of
> 	the group to control member devices of the group.
> 
> will do the trick.
> 
This looks good to me.

> > > > I am not sure it should be intentional and mention of Linux.
> > > > Any OS would need multiple error values to know the error cause.
> > > > Linux doesn't have name "OK" either for well-known value of 0.
> > > >
> > > > For example
> > > > $ errno 22
> > > > EINVAL 22 Invalid argument
> > > >
> > > > $ errno 0
> > > > This has not output for it and doesn't exist in errno.h.
> > >
> > > No name OK true, but it does use 0 to signal success sometimes:
> > Yes, but the wording is "success" and not "ok". :)
> 
> Hmm. A but wordy and we have _OK elsewhere in the spec, while Linux does
> not have either OK ot SUCCESS. Just informal text.
> Will consider how to make it clearer.
> 
> > >
> > > https://man7.org/linux/man-pages/man3/errno.3.html
> > >
> > > For some system calls and library functions (e.g.,
> > >        getpriority(2)), -1 is a valid return on success.  In such cases,
> > >        a successful return can be distinguished from an error return by
> > >        setting errno to zero before the call, and then, if the call
> > >        returns a status that indicates that an error may have occurred,
> > >        checking to see if errno has a nonzero value.
> > >
> > > > Description is already good enough to describe what they are.
> > > > Can we please drop Linux wording?
> > >
> > > But why should we?
> > Because of following reasons.
> >
> > 1. If we start putting 22 with Linux annotations, in few weeks virtio Net device
> section will be full of annotation of ethtool, tc, ip config and more for Linux
> developers.
> > For example, the current two interrupt moderation patches need to write
> ethtool options details to match to following this Linux example here.
This cannot be relaxed if errno.h and Linux annotation must be added.
And I don't think its correct direction for the spec.

> > It is better to avoid such things and keep spec OS agnostics, even though it is
> addressing and fitting into the Linux use cases.
> >
> > 2. Virtio error code to Linux error code switch-case is simple routine to have.
> >
> > 3. Every time virtio spec to refer to errno.h for finding right value is opposite
> of what spec may want to achieve.
> > If an error code doesn't match to errno.h, now spec developers and reviewers
> to review what is not defined by errno.h to find as unique value.
> >
> > 3.1 And if that is taken in future by errno.h for something else?
> >
> > All of this is not worth a simple switch case statement to deal with.
> 
> I think we'll have to agree to disagree here. Years working on virt taught me
> that matching some existing interface is usually better than coming up with our
> own. 
With all due respect to your working experience on virt, learning from existing technology, standards, and open-source software is certainly good.

However, it is a not correct to imply that virtio community will make mistake if they do not copy error codes from the errno.h file for AQ.

> Even if it is a bit more work upfront it pays dividends later. Forcing some
> discipline in not coming up with outlandish "EFOOBAR" codes no one driver will
> end up using is a good idea too.
Copying GPLv2 errno.h is not the way to enforce that.

Community will review EFOOBAR when/if that appears.
You are implying that a new error code of AQ to be first defined in errno.h.
If so, for sure reference needs to refer that like IEEE, PCI spec and more.
Doesn't sound right to me.

If virtio is copying GPLv2 errno.h error codes, it must be spelled out in licensing area and generic line as errno.h and not just "Linux".
I don't think this is right for virtio spec.

> And yes we can come up with crazy conventions like 0xdead for success.
> Does not mean we should.
That is exactly not the suggestion.
Suggestion is to not to copy GPLv2 definitions to virtio spec.

Avoiding crazy definition can be easily achieved by this community is doing it in OS agnostic manner.

The question is: Can virtio community define error codes by learning from errno.h, netlink errors and by learning from other industry leading specs?
I think yes, community members in this email thread can do that without annotating Linux in the spec.

My humble request is, let's spend more time to solidify the AQ on other aspects than copying error codes.

Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Tue, Feb 14, 2023 at 04:44:59PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, February 14, 2023 2:47 AM
> > 
> > > >
> > > > Well it's clearly the driver that issues the commands. Don't see how
> > > > can a device issue them. If instead of "through an owner device"
> > > > you prefer a more verbose "the driver of an owner device" then I can
> > > > live with that. Let me know.
> > >
> > > "through" wording is convoluted and it hints that someone else is preparing
> > the command and its tunneled through the admin vq.
> > > Which is clearly not the scope here.
> > 
> > It is exactly someone else - the driver not the owner itself.
> I completely lost you.
> My understanding is below.
> For a PCI PF device, AQ is on the PF (owner device who creates the AQ).
> PF driver is issuing the AQ command and waiting for the completion to arrive.
> PF driver decides which AQ to use to issue the command.
> This AQ is no different than net cvq, or tx/rx virtqueue.
> Only difference is, it carries generic command instead of device type specific command.
> 
> In some other use case that we need to write in the cover letter, that this queue may act as a transport for some SIOV or something.
> Even in such scenario, command is still issued to the AQ by the owner PF driver.
> 
> What did I miss?
> If none, your below description looks good to me.

none

> > 	The driver sends group administration commands to the owner device
> > of
> > 	the group to control member devices of the group.
> > 
> > will do the trick.
> > 
> This looks good to me.
> 
> > > > > I am not sure it should be intentional and mention of Linux.
> > > > > Any OS would need multiple error values to know the error cause.
> > > > > Linux doesn't have name "OK" either for well-known value of 0.
> > > > >
> > > > > For example
> > > > > $ errno 22
> > > > > EINVAL 22 Invalid argument
> > > > >
> > > > > $ errno 0
> > > > > This has not output for it and doesn't exist in errno.h.
> > > >
> > > > No name OK true, but it does use 0 to signal success sometimes:
> > > Yes, but the wording is "success" and not "ok". :)
> > 
> > Hmm. A but wordy and we have _OK elsewhere in the spec, while Linux does
> > not have either OK ot SUCCESS. Just informal text.
> > Will consider how to make it clearer.
> > 
> > > >
> > > > https://man7.org/linux/man-pages/man3/errno.3.html
> > > >
> > > > For some system calls and library functions (e.g.,
> > > >        getpriority(2)), -1 is a valid return on success.  In such cases,
> > > >        a successful return can be distinguished from an error return by
> > > >        setting errno to zero before the call, and then, if the call
> > > >        returns a status that indicates that an error may have occurred,
> > > >        checking to see if errno has a nonzero value.
> > > >
> > > > > Description is already good enough to describe what they are.
> > > > > Can we please drop Linux wording?
> > > >
> > > > But why should we?
> > > Because of following reasons.
> > >
> > > 1. If we start putting 22 with Linux annotations, in few weeks virtio Net device
> > section will be full of annotation of ethtool, tc, ip config and more for Linux
> > developers.
> > > For example, the current two interrupt moderation patches need to write
> > ethtool options details to match to following this Linux example here.
> This cannot be relaxed if errno.h and Linux annotation must be added.

I dont really know what does this mean.

> And I don't think its correct direction for the spec.
>
> > > It is better to avoid such things and keep spec OS agnostics, even though it is
> > addressing and fitting into the Linux use cases.
> > >
> > > 2. Virtio error code to Linux error code switch-case is simple routine to have.
> > >
> > > 3. Every time virtio spec to refer to errno.h for finding right value is opposite
> > of what spec may want to achieve.
> > > If an error code doesn't match to errno.h, now spec developers and reviewers
> > to review what is not defined by errno.h to find as unique value.
> > >
> > > 3.1 And if that is taken in future by errno.h for something else?
> > >
> > > All of this is not worth a simple switch case statement to deal with.
> > 
> > I think we'll have to agree to disagree here. Years working on virt taught me
> > that matching some existing interface is usually better than coming up with our
> > own. 
> With all due respect to your working experience on virt, learning from existing technology, standards, and open-source software is certainly good.

Glad we agree on this...

> However, it is a not correct to imply that virtio community will make mistake if they do not copy error codes from the errno.h file for AQ.

unfortunately you seem not to follow this to the logical conclusion.

> > Even if it is a bit more work upfront it pays dividends later. Forcing some
> > discipline in not coming up with outlandish "EFOOBAR" codes no one driver will
> > end up using is a good idea too.
> Copying GPLv2 errno.h is not the way to enforce that.
> 
> Community will review EFOOBAR when/if that appears.
> You are implying that a new error code of AQ to be first defined in errno.h.
> If so, for sure reference needs to refer that like IEEE, PCI spec and more.
> Doesn't sound right to me.
> 
> If virtio is copying GPLv2 errno.h error codes, it must be spelled out in licensing area and generic line as errno.h and not just "Linux".
> I don't think this is right for virtio spec.

Neither EINVAL nor the value 22 are under GPL in any way.


> > And yes we can come up with crazy conventions like 0xdead for success.
> > Does not mean we should.
> That is exactly not the suggestion.
> Suggestion is to not to copy GPLv2 definitions to virtio spec.
> 
> Avoiding crazy definition can be easily achieved by this community is doing it in OS agnostic manner.
> 
> The question is: Can virtio community define error codes by learning from errno.h, netlink errors and by learning from other industry leading specs?
> I think yes, community members in this email thread can do that without annotating Linux in the spec.
> 
> My humble request is, let's spend more time to solidify the AQ on other aspects than copying error codes.

Yea I have no idea why you are wasting your and my time on this either.
Repeating the same thing over and over will not make it right. And no we
do not need more arguments either.

-- 
MST

Re: [virtio-comment] [PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[Max Gurtovoy]

On 13/02/2023 14:21, David Edmondson wrote:
> 
> On Thursday, 2023-02-09 at 07:13:48 -05, Michael S. Tsirkin wrote:
>> @@ -1031,6 +1035,19 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>>           This field exists only if VIRTIO_F_RING_RESET has been
>>           negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>>   
>> +\item[\field{admin_queue_index}]
>> +        The device uses this to report the index of the first administration virtqueue.
>> +        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
>> +\item[\field{admin_queue_num}]
>> +	The device uses this to report the number of the
>> +	supported administration virtqueues.  Virtqueues with index
>> +	between \field{admin_queue_index} and (\field{admin_queue_index} +
>> +	\field{admin_queue_num}) inclusive serve as administration
>> +	virtqueues.
>> +	Thus the number of administration virtqueues equals
>> +	(\field{admin_queue_num} + 1).
> 
> Really the number of queues is "admin_queue_num + 1"?
> 
> That seems ... odd.

It's not odd if we'll mention that this is a zero based value.
MST, can you add it please for the next version ?


> 
>> +	This field is valid only if VIRTIO_F_ADMIN_VQ has been
>> +	negotiated.
>>   \end{description}
>>   
>>   \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
>> @@ -1117,6 +1134,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>>   were used before the queue reset.
>>   (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>>   
>> +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver
>> +configures any administration virtqueues, the driver MUST
>> +configure the administration virtqueues using the index
>> +in the range \field{admin_queue_index} to
>> +\field{admin_queue_index} + \field{admin_queue_num} inclusive.
>> +The driver MAY configure less administration virtqueues than
>> +supported by the device.
>> +
>>   \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
>>   
>>   The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
>> @@ -7684,6 +7709,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>   
>>     \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
>>     administration virtqueues.
>> +  At the moment this feature is only supported for devices using
>> +  \ref{sec:Virtio Transport Options / Virtio Over PCI
>> +	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
>> +	  as the transport and is reserved for future use for
>> +	  devices using other transports (see
>> +	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
>> +	and
>> +	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
>> +	handling features reserved for future use.
>>   
>>   \end{description}

Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved

[Max Gurtovoy]

On 09/02/2023 14:13, Michael S. Tsirkin wrote:
> Adding relevant registers needs more work and it's not
> clear what the use-case will be as currently only
> the PCI transport is supported. But let's keep the
> door open on this.
> We already say it's reserved in a central place, but it
> does not hurt to remind implementers to mask it.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   content.tex | 12 ++++++++++++
>   1 file changed, 12 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 2d9b903..f8bd1ab 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2362,6 +2362,18 @@ \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over M
>   
>   Notification mechanisms did not change.
>   
> +\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio Over MMIO / Features reserved for future use}
> +
> +At this time, devices and drivers utilizing Virtio Over MMIO
> +do not support the following features:
> +\begin{itemize}
> +
> +\item VIRTIO_F_ADMIN_VQ

only ADMIN_VQ is not supported ?
What about SR-IOV :) ?
> +
> +\end{itemize}
> +
> +These features are reserved for future use.
> +
>   \section{Virtio Over Channel I/O}\label{sec:Virtio Transport Options / Virtio Over Channel I/O}
>   
>   S/390 based virtual machines support neither PCI nor MMIO, so a

Re: [PATCH v10 07/10] ccw: document ADMIN_VQ as reserved

[Max Gurtovoy]

On 09/02/2023 14:13, Michael S. Tsirkin wrote:
> Adding relevant registers needs more work and it's not
> clear what the use-case will be as currently only
> the PCI transport is supported. But let's keep the
> door open on this.
> We already say it's reserved in a central place, but it
> does not hurt to remind implementers to mask it.
> 
> Note: there are more features to add to this list.
> Will be done later with a patch on top.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   content.tex | 12 ++++++++++++
>   1 file changed, 12 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index f8bd1ab..6a50b39 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2976,6 +2976,18 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
>   MAY also choose to verify reset completion by reading \field{device status} via
>   CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
>   
> +\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio over channel I/O / Features reserved for future use}
> +
> +At this time, devices and drivers utilizing Virtio over channel I/O
> +do not support the following features:
> +\begin{itemize}
> +
> +\item VIRTIO_F_ADMIN_VQ

same as in previous patch: what about missing support for SR-IOV feature ?
> +
> +\end{itemize}
> +
> +These features are reserved for future use.
> +
>   \chapter{Device Types}\label{sec:Device Types}
>   
>   On top of the queues, config space and feature negotiation facilities

[virtio-comment] RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, February 14, 2023 4:58 PM

> > > > 1. If we start putting 22 with Linux annotations, in few weeks
> > > > virtio Net device
> > > section will be full of annotation of ethtool, tc, ip config and
> > > more for Linux developers.
> > > > For example, the current two interrupt moderation patches need to
> > > > write
> > > ethtool options details to match to following this Linux example here.
> > This cannot be relaxed if errno.h and Linux annotation must be added.
> 
> I dont really know what does this mean.
>
It means why should Linux annotation is limited to error codes of AQ.
Why not annotate Linux for other areas of the spec such as ethtool annotation for interrupt moderation.
 
> > And I don't think its correct direction for the spec.
> >
> > > > It is better to avoid such things and keep spec OS agnostics, even
> > > > though it is
> > > addressing and fitting into the Linux use cases.
> > > >
> > > > 2. Virtio error code to Linux error code switch-case is simple routine to
> have.
> > > >
> > > > 3. Every time virtio spec to refer to errno.h for finding right
> > > > value is opposite
> > > of what spec may want to achieve.
> > > > If an error code doesn't match to errno.h, now spec developers and
> > > > reviewers
> > > to review what is not defined by errno.h to find as unique value.
> > > >
> > > > 3.1 And if that is taken in future by errno.h for something else?
> > > >
> > > > All of this is not worth a simple switch case statement to deal with.
> > >
> > > I think we'll have to agree to disagree here. Years working on virt
> > > taught me that matching some existing interface is usually better
> > > than coming up with our own.
> > With all due respect to your working experience on virt, learning from existing
> technology, standards, and open-source software is certainly good.
> 
> Glad we agree on this...
> 
> > However, it is a not correct to imply that virtio community will make mistake if
> they do not copy error codes from the errno.h file for AQ.
> 
> unfortunately you seem not to follow this to the logical conclusion.
> 
> > > Even if it is a bit more work upfront it pays dividends later.

> > If virtio is copying GPLv2 errno.h error codes, it must be spelled out in
> licensing area and generic line as errno.h and not just "Linux".
> > I don't think this is right for virtio spec.
> 
> Neither EINVAL nor the value 22 are under GPL in any way.
>
I referred to [1] https://elixir.bootlin.com/linux/latest/source/include/uapi/asm-generic/errno-base.h#L26
And missed the sycall note at start.
Thanks for the clarification.

> 
> > > And yes we can come up with crazy conventions like 0xdead for success.
> > > Does not mean we should.
> > The question is: Can virtio community define error codes by learning from
> errno.h, netlink errors and by learning from other industry leading specs?
> > I think yes, community members in this email thread can do that without
> annotating Linux in the spec.
> >
> > My humble request is, let's spend more time to solidify the AQ on other
> aspects than copying error codes.
> 
> Yea I have no idea why you are wasting your and my time on this either.
> Repeating the same thing over and over will not make it right. And no we do not
> need more arguments either.

Not sure what was repeated...

Anyways, if you have so strong preference to use the derivative work of "Linux", 
please add the stable link of it the section 1.1 or 1.2 references sections and refer to it in the text.
This will make things crystal clear for the source of existing and future error codes.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[Michael S. Tsirkin]

On Wed, Feb 15, 2023 at 02:53:34AM +0200, Max Gurtovoy wrote:
> 
> 
> On 13/02/2023 14:21, David Edmondson wrote:
> > 
> > On Thursday, 2023-02-09 at 07:13:48 -05, Michael S. Tsirkin wrote:
> > > @@ -1031,6 +1035,19 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> > >           This field exists only if VIRTIO_F_RING_RESET has been
> > >           negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
> > > +\item[\field{admin_queue_index}]
> > > +        The device uses this to report the index of the first administration virtqueue.
> > > +        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
> > > +\item[\field{admin_queue_num}]
> > > +	The device uses this to report the number of the
> > > +	supported administration virtqueues.  Virtqueues with index
> > > +	between \field{admin_queue_index} and (\field{admin_queue_index} +
> > > +	\field{admin_queue_num}) inclusive serve as administration
> > > +	virtqueues.
> > > +	Thus the number of administration virtqueues equals
> > > +	(\field{admin_queue_num} + 1).
> > 
> > Really the number of queues is "admin_queue_num + 1"?
> > 
> > That seems ... odd.
> 
> It's not odd if we'll mention that this is a zero based value.
> MST, can you add it please for the next version ?

OK I can.

> 
> > 
> > > +	This field is valid only if VIRTIO_F_ADMIN_VQ has been
> > > +	negotiated.
> > >   \end{description}
> > >   \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
> > > @@ -1117,6 +1134,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> > >   were used before the queue reset.
> > >   (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
> > > +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver
> > > +configures any administration virtqueues, the driver MUST
> > > +configure the administration virtqueues using the index
> > > +in the range \field{admin_queue_index} to
> > > +\field{admin_queue_index} + \field{admin_queue_num} inclusive.
> > > +The driver MAY configure less administration virtqueues than
> > > +supported by the device.
> > > +
> > >   \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
> > >   The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
> > > @@ -7684,6 +7709,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > >     \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more
> > >     administration virtqueues.
> > > +  At the moment this feature is only supported for devices using
> > > +  \ref{sec:Virtio Transport Options / Virtio Over PCI
> > > +	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
> > > +	  as the transport and is reserved for future use for
> > > +	  devices using other transports (see
> > > +	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
> > > +	and
> > > +	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> > > +	handling features reserved for future use.
> > >   \end{description}

Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Wed, Feb 15, 2023 at 04:46:26AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, February 14, 2023 4:58 PM
> 
> > > > > 1. If we start putting 22 with Linux annotations, in few weeks
> > > > > virtio Net device
> > > > section will be full of annotation of ethtool, tc, ip config and
> > > > more for Linux developers.
> > > > > For example, the current two interrupt moderation patches need to
> > > > > write
> > > > ethtool options details to match to following this Linux example here.
> > > This cannot be relaxed if errno.h and Linux annotation must be added.
> > 
> > I dont really know what does this mean.
> >
> It means why should Linux annotation is limited to error codes of AQ.
> Why not annotate Linux for other areas of the spec such as ethtool annotation for interrupt moderation.

If it is relevant somehow, bring it in.

> > > And I don't think its correct direction for the spec.
> > >
> > > > > It is better to avoid such things and keep spec OS agnostics, even
> > > > > though it is
> > > > addressing and fitting into the Linux use cases.
> > > > >
> > > > > 2. Virtio error code to Linux error code switch-case is simple routine to
> > have.
> > > > >
> > > > > 3. Every time virtio spec to refer to errno.h for finding right
> > > > > value is opposite
> > > > of what spec may want to achieve.
> > > > > If an error code doesn't match to errno.h, now spec developers and
> > > > > reviewers
> > > > to review what is not defined by errno.h to find as unique value.
> > > > >
> > > > > 3.1 And if that is taken in future by errno.h for something else?
> > > > >
> > > > > All of this is not worth a simple switch case statement to deal with.
> > > >
> > > > I think we'll have to agree to disagree here. Years working on virt
> > > > taught me that matching some existing interface is usually better
> > > > than coming up with our own.
> > > With all due respect to your working experience on virt, learning from existing
> > technology, standards, and open-source software is certainly good.
> > 
> > Glad we agree on this...
> > 
> > > However, it is a not correct to imply that virtio community will make mistake if
> > they do not copy error codes from the errno.h file for AQ.
> > 
> > unfortunately you seem not to follow this to the logical conclusion.
> > 
> > > > Even if it is a bit more work upfront it pays dividends later.
> 
> > > If virtio is copying GPLv2 errno.h error codes, it must be spelled out in
> > licensing area and generic line as errno.h and not just "Linux".
> > > I don't think this is right for virtio spec.
> > 
> > Neither EINVAL nor the value 22 are under GPL in any way.
> >
> I referred to [1] https://elixir.bootlin.com/linux/latest/source/include/uapi/asm-generic/errno-base.h#L26
> And missed the sycall note at start.
> Thanks for the clarification.
> 
> > 
> > > > And yes we can come up with crazy conventions like 0xdead for success.
> > > > Does not mean we should.
> > > The question is: Can virtio community define error codes by learning from
> > errno.h, netlink errors and by learning from other industry leading specs?
> > > I think yes, community members in this email thread can do that without
> > annotating Linux in the spec.
> > >
> > > My humble request is, let's spend more time to solidify the AQ on other
> > aspects than copying error codes.
> > 
> > Yea I have no idea why you are wasting your and my time on this either.
> > Repeating the same thing over and over will not make it right. And no we do not
> > need more arguments either.
> 
> Not sure what was repeated...
> 
> Anyways, if you have so strong preference to use the derivative work of "Linux", 
> please add the stable link of it the section 1.1 or 1.2 references sections and refer to it in the text.
> This will make things crystal clear for the source of existing and future error codes.

Yea that makes sense.

-- 
MST

RE: [PATCH v10 03/10] admin: introduce group administration commands

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, February 9, 2023 7:14 AM

> +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEM    & unsupported or invalid \field{group_member_id}
To avoid confusion with memory, better to replace INVALID_MEM to INVALID_MEMBER.

Re: [PATCH v10 03/10] admin: introduce group administration commands

[Michael S. Tsirkin]

On Wed, Feb 15, 2023 at 05:17:21AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, February 9, 2023 7:14 AM
> 
> > +0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEM    & unsupported or invalid \field{group_member_id}
> To avoid confusion with memory, better to replace INVALID_MEM to INVALID_MEMBER.

Good idea.

Re: [virtio-comment] [PATCH v10 05/10] pci: add admin vq registers to virtio over pci

[David Edmondson]

On Wednesday, 2023-02-15 at 02:53:34 +02, Max Gurtovoy wrote:
> On 13/02/2023 14:21, David Edmondson wrote:
>> On Thursday, 2023-02-09 at 07:13:48 -05, Michael S. Tsirkin wrote:
>>> @@ -1031,6 +1035,19 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>>>           This field exists only if VIRTIO_F_RING_RESET has been
>>>           negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>>>   +\item[\field{admin_queue_index}]
>>> +        The device uses this to report the index of the first administration virtqueue.
>>> +        This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
>>> +\item[\field{admin_queue_num}]
>>> +	The device uses this to report the number of the
>>> +	supported administration virtqueues.  Virtqueues with index
>>> +	between \field{admin_queue_index} and (\field{admin_queue_index} +
>>> +	\field{admin_queue_num}) inclusive serve as administration
>>> +	virtqueues.
>>> +	Thus the number of administration virtqueues equals
>>> +	(\field{admin_queue_num} + 1).
>> Really the number of queues is "admin_queue_num + 1"?
>> That seems ... odd.
>
> It's not odd if we'll mention that this is a zero based value.
> MST, can you add it please for the next version ?

The field is called ...num, which makes it seem like a count of things,
when in reality it's the highest index. Perhaps the name could more
accurately reflect the intent?

>
>> 
>>> +	This field is valid only if VIRTIO_F_ADMIN_VQ has been
>>> +	negotiated.
>>>   \end{description}
>>>     \devicenormative{\paragraph}{Common configuration structure
>>> layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device
>>> Layout / Common configuration structure layout}
>>> @@ -1117,6 +1134,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>>>   were used before the queue reset.
>>>   (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
>>>   +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver
>>> +configures any administration virtqueues, the driver MUST
>>> +configure the administration virtqueues using the index
>>> +in the range \field{admin_queue_index} to
>>> +\field{admin_queue_index} + \field{admin_queue_num} inclusive.
>>> +The driver MAY configure less administration virtqueues than
>>> +supported by the device.
>>> +
>>>   \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
>>>     The notification location is found using the
>>> VIRTIO_PCI_CAP_NOTIFY_CFG
>>> @@ -7684,6 +7709,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>>       \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the
>>> device exposes one or more
>>>     administration virtqueues.
>>> +  At the moment this feature is only supported for devices using
>>> +  \ref{sec:Virtio Transport Options / Virtio Over PCI
>>> +	  Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}
>>> +	  as the transport and is reserved for future use for
>>> +	  devices using other transports (see
>>> +	  \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}
>>> +	and
>>> +	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
>>> +	handling features reserved for future use.
>>>     \end{description}
-- 
Stop the music and go home.