[RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

Admin virtqueues will be used to send administrative commands to
manipulate various features of the device which would not easily map
into the configuration space.

The same Admin command format will be used for all virtio devices. The
Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific class

The above mechanism will enable adding various features to the virtio
specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,
   512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.

As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
for the specification to become flexible and allow a wider feature set.
The corrent ctrl virtq that is defined for some of the virtio devices is
device specific and wasn't designed to be a generic virtq for
admininistration.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex     |   4 +
 2 files changed, 245 insertions(+)
 create mode 100644 admin-virtq.tex

diff --git a/admin-virtq.tex b/admin-virtq.tex
new file mode 100644
index 0000000..ccec2ca
--- /dev/null
+++ b/admin-virtq.tex
@@ -0,0 +1,241 @@
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
+
+Admin virtqueues are used to send administrative commands to manipulate
+various features of the device which would not easily map into the
+configuration space.
+
+Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.
+
+All commands are of the following form:
+
+\begin{lstlisting}
+struct virtio_admin_cmd {
+        /* Device-readable part */
+        u8 class;
+        u8 command;
+        u8 command-specific-data[];
+
+        /* Device-writable part */
+        u8 command-specific-result[];
+        u8 status_type : 4;
+        u8 reserved : 4;
+        u8 status;
+};
+
+/* Status type values */
+#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
+#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
+#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
+#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
+#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
+#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
+
+/* Generic status values */
+#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
+#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
+#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
+#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
+#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
+#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
+\end{lstlisting}
+
+The \field{class}, \field{command} and \field{command-specific-data} are
+set by the driver, and the device sets the \field{status_type}, the
+\field{status} and  the \field{command-specific-result}, if needed.
+
+The virtio Admin command class codes are divided in the following form:
+
+\begin{lstlisting}
+/* class values that are transport, device and vendor independent */
+#define VIRTIO_ADMIN_COMMON_CLASS_START    0
+#define VIRTIO_ADMIN_COMMON_CLASS_END      63
+
+/* class values that are transport specific */
+#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
+#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
+
+/* class values that are device specific */
+#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
+#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
+
+/* class values that are vendor specific */
+#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
+#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
+\end{lstlisting}
+
+\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
+
+Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
+support all the mandatory admin commands. A device MAY support also
+one or more optional admin commands.
+
+\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
+
+The Common command set is a group of classes and commands within each
+of these classes which are transport, device and vendor independent.
+A mandatory class is a class that has at least one mandatory command.
+The Common command set is summarized in following table:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Class  & Description    & M/O \\
+\hline \hline
+0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
+\hline
+1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
+\hline
+2-63  & reserved    & - \\
+\hline
+\end{tabular}
+
+\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
+
+This class (opcode: 0) of commands is used to query generic device
+information. The following table describes the commands supported for
+this class:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Command  & Description    & M/O \\
+\hline \hline
+0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
+\hline
+1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
+\hline
+2-255  & reserved    & - \\
+\hline
+\end{tabular}
+
+\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
+
+This mandatory command should return device identity in the following
+structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bytes  & Description    & M/O \\
+\hline \hline
+03:00  & VIRTIO DEVICE ID    & M \\
+\hline
+05:04  & VIRTIO TRANSPORT ID    & M \\
+\hline
+4095:06  & reserved    & - \\
+ \hline
+\end{tabular}
+
+\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
+
+This mandatory command should return device identity in the following
+structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bytes  & Description    & M/O \\
+\hline \hline
+03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
+\hline
+07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
+ \hline
+11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
+\hline
+...  & ...    & M \\
+\hline
+1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
+\hline
+4095:1024  & reserved    & - \\
+\hline
+\end{tabular}
+
+The above structure is divided to 256 sections of 4B each, that will
+describe whether a device supports a certain class code. The
+following 3072B are reserved. each of the non-reserved 4B sections will
+follow the following data structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bits  & Description    & M/O \\
+\hline \hline
+0  & SUPPORTED (1: yes, 0:no)    & M \\
+\hline
+31:01  & reserved    & - \\
+ \hline
+\end{tabular}
+
+\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
+
+This class (opcode: 1) of commands is used to query supported commands
+for each supported device class. The following table describes the commands
+supported for this class:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Command  & Description    & M/O \\
+\hline \hline
+0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
+\hline
+1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
+\hline
+2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
+\hline
+...  & ...    & M \\
+\hline
+255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
+\hline
+\end{tabular}
+
+Each command in this class, will return class identity in the following structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bytes  & Description    & M/O \\
+\hline \hline
+03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
+\hline
+07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
+ \hline
+11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
+\hline
+...  & ...    & M \\
+\hline
+1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
+\hline
+4095:1024  & reserved    & - \\
+\hline
+\end{tabular}
+
+The above structure is divided to 256 sections of 4B each, that will
+describe whether a class supports a certain command code. The
+following 3072B are reserved. each of the non-reserved 4B sections will
+follow the following data structure:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Bits  & Description    & M/O \\
+\hline \hline
+0  & SUPPORTED (1: yes, 0:no)    & M \\
+\hline
+31:01  & reserved    & - \\
+ \hline
+\end{tabular}
+
+\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
+
+The Transport command set is a group of classes and commands within
+each of these classes which are transport specific. Refer to each
+transport section for more information on the supported commands.
+
+\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
+
+The Device command set is a group of classes and commands within
+each of these classes which are device specific. Refer to each
+device section for more information on the supported commands.
+
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
+
+The Vendor specific command set is a group of classes and commands
+within each of these classes which are vendor specific. Refer to
+each vendor specification for more information on the supported
+commands.
diff --git a/content.tex b/content.tex
index c266fd5..d350175 100644
--- a/content.tex
+++ b/content.tex
@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\input{admin-virtq.tex}
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   data to be provided in driver notification and the delivery method is
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
+  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
+  the device supports administration virtqueue negotiation.
 
 \end{description}
 
-- 
2.21.0

[RFC PATCH v2 2/2] virtio-blk: add support for VIRTIO_F_ADMIN_VQ

[Max Gurtovoy]

Set the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation. Also
restrict the negotiation of VIRTIO_F_ADMIN_VQ to be possible only if
VIRTIO_BLK_F_MQ is negotiated.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 content.tex | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/content.tex b/content.tex
index d350175..03e25da 100644
--- a/content.tex
+++ b/content.tex
@@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
  N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
  \field{num_queues}.
 
+ If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the
+ request queues are shifted as the following:
+\begin{description}
+\item[1] requestq1
+\item[\ldots]
+\item[N] requestqN
+\end{description}
+
+ if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be
+ negotiated.
 \subsection{Feature bits}\label{sec:Device Types / Block Device / Feature bits}
 
 \begin{description}
@@ -4574,6 +4584,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Block Device / Devic
 The driver MUST NOT read \field{writeback} before setting
 the FEATURES_OK \field{device status} bit.
 
+The driver MUST NOT negotiate VIRTIO_F_ADMIN_VQ, unless VIRTIO_BLK_F_MQ
+is negotiated.
+
 \devicenormative{\subsubsection}{Device Initialization}{Device Types / Block Device / Device Initialization}
 
 Devices SHOULD always offer VIRTIO_BLK_F_FLUSH, and MUST offer it
@@ -4585,6 +4598,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Block Device / Devic
 The device MUST initialize padding bytes \field{unused0} and
 \field{unused1} to 0.
 
+The device MUST NOT offer VIRTIO_F_ADMIN_VQ, unless it offer
+VIRTIO_BLK_F_MQ.
+
 \subsubsection{Legacy Interface: Device Initialization}\label{sec:Device Types / Block Device / Device Initialization / Legacy Interface: Device Initialization}
 
 Because legacy devices do not have FEATURES_OK, transitional devices
-- 
2.21.0

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 12614 bytes --]

On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> Admin virtqueues will be used to send administrative commands to
> manipulate various features of the device which would not easily map
> into the configuration space.
> 
> The same Admin command format will be used for all virtio devices. The
> Admin command set will include 4 types of command classes:
> 1. The generic common class
> 2. The transport specific class
> 3. The device specific class
> 4. The vendor specific class
> 
> The above mechanism will enable adding various features to the virtio
> specification, e.g.:
> 1. Format virtio-blk devices in various configurations (512B block size,
>    512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> 2. Live migration management.
> 3. Encrypt/Decrypt descriptors.
> 4. Virtualization management.
> 5. Get device error logs.
> 6. Implement advanced vendor/device/transport specific features.
> 7. Run device health test.
> 8. More.
> 
> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
> for the specification to become flexible and allow a wider feature set.
> The corrent ctrl virtq that is defined for some of the virtio devices is
> device specific and wasn't designed to be a generic virtq for
> admininistration.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>  admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex     |   4 +
>  2 files changed, 245 insertions(+)
>  create mode 100644 admin-virtq.tex
> 
> diff --git a/admin-virtq.tex b/admin-virtq.tex
> new file mode 100644
> index 0000000..ccec2ca
> --- /dev/null
> +++ b/admin-virtq.tex
> @@ -0,0 +1,241 @@
> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> +
> +Admin virtqueues are used to send administrative commands to manipulate
> +various features of the device which would not easily map into the
> +configuration space.
> +
> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
> +feature bit.
> +
> +Admin virtqueue index may vary among different device types.
> +
> +All commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        u8 class;
> +        u8 command;
> +        u8 command-specific-data[];
> +
> +        /* Device-writable part */
> +        u8 command-specific-result[];
> +        u8 status_type : 4;
> +        u8 reserved : 4;
> +        u8 status;
> +};
> +
> +/* Status type values */
> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
> +
> +/* Generic status values */
> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
> +\end{lstlisting}
> +
> +The \field{class}, \field{command} and \field{command-specific-data} are
> +set by the driver, and the device sets the \field{status_type}, the
> +\field{status} and  the \field{command-specific-result}, if needed.
> +
> +The virtio Admin command class codes are divided in the following form:
> +
> +\begin{lstlisting}
> +/* class values that are transport, device and vendor independent */
> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
> +
> +/* class values that are transport specific */
> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
> +
> +/* class values that are device specific */
> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
> +
> +/* class values that are vendor specific */
> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
> +\end{lstlisting}
> +
> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
> +
> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST

"advertises the VIRTIO_F_ADMIN_VQ feature"

> +support all the mandatory admin commands. A device MAY support also
> +one or more optional admin commands.
> +
> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
> +
> +The Common command set is a group of classes and commands within each
> +of these classes which are transport, device and vendor independent.
> +A mandatory class is a class that has at least one mandatory command.
> +The Common command set is summarized in following table:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Class  & Description    & M/O \\
> +\hline \hline
> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
> +\hline
> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
> +\hline
> +2-63  & reserved    & - \\
> +\hline
> +\end{tabular}
> +
> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
> +
> +This class (opcode: 0) of commands is used to query generic device
> +information. The following table describes the commands supported for
> +this class:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Command  & Description    & M/O \\
> +\hline \hline
> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
> +\hline
> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
> +\hline
> +2-255  & reserved    & - \\
> +\hline
> +\end{tabular}
> +
> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
> +
> +This mandatory command should return device identity in the following
> +structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bytes  & Description    & M/O \\
> +\hline \hline
> +03:00  & VIRTIO DEVICE ID    & M \\
> +\hline
> +05:04  & VIRTIO TRANSPORT ID    & M \\

These fields are not defined. I wonder why they are necessary - the
driver should already have this information.

In general, I'm a little concerned that this whole infrastructure will
increase the complexity of VIRTIO significantly with little benefit. I
do think an admin virtqueue makes sense, e.g. for migration, but would
prefer it if we focus on actual commands first instead of
infrastructure. That way it will be clear what infrastructure is needed.

> +\hline
> +4095:06  & reserved    & - \\
> + \hline
> +\end{tabular}
> +
> +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
> +
> +This mandatory command should return device identity in the following
> +structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bytes  & Description    & M/O \\
> +\hline \hline
> +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
> +\hline
> +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
> + \hline
> +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
> +\hline
> +...  & ...    & M \\
> +\hline
> +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
> +\hline
> +4095:1024  & reserved    & - \\
> +\hline
> +\end{tabular}
> +
> +The above structure is divided to 256 sections of 4B each, that will
> +describe whether a device supports a certain class code. The
> +following 3072B are reserved. each of the non-reserved 4B sections will
> +follow the following data structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bits  & Description    & M/O \\
> +\hline \hline
> +0  & SUPPORTED (1: yes, 0:no)    & M \\
> +\hline
> +31:01  & reserved    & - \\
> + \hline
> +\end{tabular}
> +
> +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
> +
> +This class (opcode: 1) of commands is used to query supported commands
> +for each supported device class. The following table describes the commands
> +supported for this class:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Command  & Description    & M/O \\
> +\hline \hline
> +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
> +\hline
> +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
> +\hline
> +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
> +\hline
> +...  & ...    & M \\
> +\hline
> +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
> +\hline
> +\end{tabular}
> +
> +Each command in this class, will return class identity in the following structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bytes  & Description    & M/O \\
> +\hline \hline
> +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
> +\hline
> +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
> + \hline
> +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
> +\hline
> +...  & ...    & M \\
> +\hline
> +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
> +\hline
> +4095:1024  & reserved    & - \\
> +\hline
> +\end{tabular}
> +
> +The above structure is divided to 256 sections of 4B each, that will
> +describe whether a class supports a certain command code. The
> +following 3072B are reserved. each of the non-reserved 4B sections will
> +follow the following data structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bits  & Description    & M/O \\
> +\hline \hline
> +0  & SUPPORTED (1: yes, 0:no)    & M \\
> +\hline
> +31:01  & reserved    & - \\
> + \hline
> +\end{tabular}
> +
> +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
> +
> +The Transport command set is a group of classes and commands within
> +each of these classes which are transport specific. Refer to each
> +transport section for more information on the supported commands.
> +
> +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
> +
> +The Device command set is a group of classes and commands within
> +each of these classes which are device specific. Refer to each
> +device section for more information on the supported commands.
> +
> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
> +
> +The Vendor specific command set is a group of classes and commands
> +within each of these classes which are vendor specific. Refer to
> +each vendor specification for more information on the supported
> +commands.
> diff --git a/content.tex b/content.tex
> index c266fd5..d350175 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  types. It is RECOMMENDED that devices generate version 4
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>  
> +\input{admin-virtq.tex}
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
>  We start with an overview of device initialization, then expand on the
> @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>    data to be provided in driver notification and the delivery method is
>    transport specific.
>    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
> +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
> +  the device supports administration virtqueue negotiation.
>  
>  \end{description}
>  
> -- 
> 2.21.0
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 2/2] virtio-blk: add support for VIRTIO_F_ADMIN_VQ

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

On Mon, Jul 26, 2021 at 07:52:54PM +0300, Max Gurtovoy wrote:
> Set the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation. Also
> restrict the negotiation of VIRTIO_F_ADMIN_VQ to be possible only if
> VIRTIO_BLK_F_MQ is negotiated.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>  content.tex | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index d350175..03e25da 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
>   N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
>   \field{num_queues}.
>  
> + If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the
> + request queues are shifted as the following:

"as follows:"

> +\begin{description}
> +\item[1] requestq1
> +\item[\ldots]
> +\item[N] requestqN
> +\end{description}
> +
> + if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be
> + negotiated.

Why?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>> Admin virtqueues will be used to send administrative commands to
>> manipulate various features of the device which would not easily map
>> into the configuration space.
>> 
>> The same Admin command format will be used for all virtio devices. The
>> Admin command set will include 4 types of command classes:
>> 1. The generic common class
>> 2. The transport specific class
>> 3. The device specific class
>> 4. The vendor specific class
>> 
>> The above mechanism will enable adding various features to the virtio
>> specification, e.g.:
>> 1. Format virtio-blk devices in various configurations (512B block size,
>>    512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>> 2. Live migration management.
>> 3. Encrypt/Decrypt descriptors.
>> 4. Virtualization management.
>> 5. Get device error logs.
>> 6. Implement advanced vendor/device/transport specific features.
>> 7. Run device health test.
>> 8. More.
>> 
>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>> for the specification to become flexible and allow a wider feature set.
>> The corrent ctrl virtq that is defined for some of the virtio devices is
>> device specific and wasn't designed to be a generic virtq for
>> admininistration.
>> 
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> ---
>>  admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>  content.tex     |   4 +
>>  2 files changed, 245 insertions(+)
>>  create mode 100644 admin-virtq.tex
>> 
>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>> new file mode 100644
>> index 0000000..ccec2ca
>> --- /dev/null
>> +++ b/admin-virtq.tex
>> @@ -0,0 +1,241 @@
>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>> +
>> +Admin virtqueues are used to send administrative commands to manipulate
>> +various features of the device which would not easily map into the
>> +configuration space.
>> +
>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>> +feature bit.
>> +
>> +Admin virtqueue index may vary among different device types.
>> +
>> +All commands are of the following form:
>> +
>> +\begin{lstlisting}
>> +struct virtio_admin_cmd {
>> +        /* Device-readable part */
>> +        u8 class;
>> +        u8 command;
>> +        u8 command-specific-data[];
>> +
>> +        /* Device-writable part */
>> +        u8 command-specific-result[];
>> +        u8 status_type : 4;
>> +        u8 reserved : 4;
>> +        u8 status;
>> +};
>> +
>> +/* Status type values */
>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>> +
>> +/* Generic status values */
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>> +\end{lstlisting}

This is very complex, and it feels like we're overengineering this.

>> +
>> +The \field{class}, \field{command} and \field{command-specific-data} are
>> +set by the driver, and the device sets the \field{status_type}, the
>> +\field{status} and  the \field{command-specific-result}, if needed.
>> +
>> +The virtio Admin command class codes are divided in the following form:
>> +
>> +\begin{lstlisting}
>> +/* class values that are transport, device and vendor independent */
>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>> +
>> +/* class values that are transport specific */
>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>> +
>> +/* class values that are device specific */
>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>> +
>> +/* class values that are vendor specific */
>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>> +\end{lstlisting}
>> +
>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>> +
>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>
> "advertises the VIRTIO_F_ADMIN_VQ feature"
>
>> +support all the mandatory admin commands. A device MAY support also
>> +one or more optional admin commands.
>> +
>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>> +
>> +The Common command set is a group of classes and commands within each
>> +of these classes which are transport, device and vendor independent.
>> +A mandatory class is a class that has at least one mandatory command.
>> +The Common command set is summarized in following table:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Class  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>> +\hline
>> +2-63  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>> +
>> +This class (opcode: 0) of commands is used to query generic device
>> +information. The following table describes the commands supported for
>> +this class:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Command  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>> +\hline
>> +2-255  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>> +
>> +This mandatory command should return device identity in the following
>> +structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bytes  & Description    & M/O \\
>> +\hline \hline
>> +03:00  & VIRTIO DEVICE ID    & M \\
>> +\hline
>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>
> These fields are not defined. I wonder why they are necessary - the
> driver should already have this information.

Agreed.

> In general, I'm a little concerned that this whole infrastructure will
> increase the complexity of VIRTIO significantly with little benefit. I
> do think an admin virtqueue makes sense, e.g. for migration, but would
> prefer it if we focus on actual commands first instead of
> infrastructure. That way it will be clear what infrastructure is needed.

A concrete example would be good, but I think we can come up with a
bare-bones spec to start with.

- feature bit for the admin vq, as defined here
- location of the admin vq is device specific
- I think we can get away with two classes, as for feature bits (not
  device specificic and device specific); I don't think we need separate
  classes for transport or vendor specific
- make the format for the request simple (command + length + payload?)

How many different (groups of) commands can we reasonably expect? Do we
need a generic discovery command, or can we get away with a feature bit
covering each new group of commands?


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 7/27/2021 5:28 PM, Cornelia Huck wrote:
> On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>> Admin virtqueues will be used to send administrative commands to
>>> manipulate various features of the device which would not easily map
>>> into the configuration space.
>>>
>>> The same Admin command format will be used for all virtio devices. The
>>> Admin command set will include 4 types of command classes:
>>> 1. The generic common class
>>> 2. The transport specific class
>>> 3. The device specific class
>>> 4. The vendor specific class
>>>
>>> The above mechanism will enable adding various features to the virtio
>>> specification, e.g.:
>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>     512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>> 2. Live migration management.
>>> 3. Encrypt/Decrypt descriptors.
>>> 4. Virtualization management.
>>> 5. Get device error logs.
>>> 6. Implement advanced vendor/device/transport specific features.
>>> 7. Run device health test.
>>> 8. More.
>>>
>>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>>> for the specification to become flexible and allow a wider feature set.
>>> The corrent ctrl virtq that is defined for some of the virtio devices is
>>> device specific and wasn't designed to be a generic virtq for
>>> admininistration.
>>>
>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>> ---
>>>   admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>>   content.tex     |   4 +
>>>   2 files changed, 245 insertions(+)
>>>   create mode 100644 admin-virtq.tex
>>>
>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>> new file mode 100644
>>> index 0000000..ccec2ca
>>> --- /dev/null
>>> +++ b/admin-virtq.tex
>>> @@ -0,0 +1,241 @@
>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>> +
>>> +Admin virtqueues are used to send administrative commands to manipulate
>>> +various features of the device which would not easily map into the
>>> +configuration space.
>>> +
>>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>>> +feature bit.
>>> +
>>> +Admin virtqueue index may vary among different device types.
>>> +
>>> +All commands are of the following form:
>>> +
>>> +\begin{lstlisting}
>>> +struct virtio_admin_cmd {
>>> +        /* Device-readable part */
>>> +        u8 class;
>>> +        u8 command;
>>> +        u8 command-specific-data[];
>>> +
>>> +        /* Device-writable part */
>>> +        u8 command-specific-result[];
>>> +        u8 status_type : 4;
>>> +        u8 reserved : 4;
>>> +        u8 status;
>>> +};
>>> +
>>> +/* Status type values */
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>>> +
>>> +/* Generic status values */
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>>> +\end{lstlisting}
> This is very complex, and it feels like we're overengineering this.

Do you mean the status type and the status ?

>
>>> +
>>> +The \field{class}, \field{command} and \field{command-specific-data} are
>>> +set by the driver, and the device sets the \field{status_type}, the
>>> +\field{status} and  the \field{command-specific-result}, if needed.
>>> +
>>> +The virtio Admin command class codes are divided in the following form:
>>> +
>>> +\begin{lstlisting}
>>> +/* class values that are transport, device and vendor independent */
>>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>>> +
>>> +/* class values that are transport specific */
>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>>> +
>>> +/* class values that are device specific */
>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>>> +
>>> +/* class values that are vendor specific */
>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>>> +\end{lstlisting}
>>> +
>>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>>> +
>>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>> "advertises the VIRTIO_F_ADMIN_VQ feature"
>>
>>> +support all the mandatory admin commands. A device MAY support also
>>> +one or more optional admin commands.
>>> +
>>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>>> +
>>> +The Common command set is a group of classes and commands within each
>>> +of these classes which are transport, device and vendor independent.
>>> +A mandatory class is a class that has at least one mandatory command.
>>> +The Common command set is summarized in following table:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Class  & Description    & M/O \\
>>> +\hline \hline
>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>>> +\hline
>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>>> +\hline
>>> +2-63  & reserved    & - \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>>> +
>>> +This class (opcode: 0) of commands is used to query generic device
>>> +information. The following table describes the commands supported for
>>> +this class:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Command  & Description    & M/O \\
>>> +\hline \hline
>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>>> +\hline
>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>>> +\hline
>>> +2-255  & reserved    & - \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>>> +
>>> +This mandatory command should return device identity in the following
>>> +structure:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Bytes  & Description    & M/O \\
>>> +\hline \hline
>>> +03:00  & VIRTIO DEVICE ID    & M \\
>>> +\hline
>>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>> These fields are not defined. I wonder why they are necessary - the
>> driver should already have this information.
> Agreed.

These are initial fields.

We can add also model, serial_number and more in the future.


>
>> In general, I'm a little concerned that this whole infrastructure will
>> increase the complexity of VIRTIO significantly with little benefit. I
>> do think an admin virtqueue makes sense, e.g. for migration, but would
>> prefer it if we focus on actual commands first instead of
>> infrastructure. That way it will be clear what infrastructure is needed.

admin virtq is not only for migration.

You'll be able to configure virtio device properties using user space 
tools like: virtio-cli.

For example: format a block device, manage virtual function resources 
using its PF, query for error logs, device health and more.

In the SW world maybe all the above were redundant, but now that you 
have more and more HW virtio devices the protocol should be more 
flexible and adjust.

Few weeks ago I've sent a concrete commands for live migration but then 
I was told that new infrastructure (admin virtq) should be developed and 
this is what I did in this RFC.

if you combine the 2 RFCs you can imagine what is needed here for adding 
Live migration support.

But I want to add it step by step.

We need to agree on the infrastructure.

> A concrete example would be good, but I think we can come up with a
> bare-bones spec to start with.
>
> - feature bit for the admin vq, as defined here
> - location of the admin vq is device specific
> - I think we can get away with two classes, as for feature bits (not
>    device specificic and device specific); I don't think we need separate
>    classes for transport or vendor specific

We need it for live migration probably. It will be a transport class.

Vendor specific is also important to allow vendors develop their special 
souse.

> - make the format for the request simple (command + length + payload?)

I used almost the same format as virtio net ctrl queue.

>
> How many different (groups of) commands can we reasonably expect? Do we
> need a generic discovery command, or can we get away with a feature bit
> covering each new group of commands?

I can't predict the future but IMO we need a discovery command.

We have many devices and more can be added in the future.

Re: [virtio-comment] Re: [RFC PATCH v2 2/2] virtio-blk: add support for VIRTIO_F_ADMIN_VQ

[Max Gurtovoy]

On 7/27/2021 3:24 PM, Stefan Hajnoczi wrote:
> On Mon, Jul 26, 2021 at 07:52:54PM +0300, Max Gurtovoy wrote:
>> Set the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation. Also
>> restrict the negotiation of VIRTIO_F_ADMIN_VQ to be possible only if
>> VIRTIO_BLK_F_MQ is negotiated.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> ---
>>   content.tex | 16 ++++++++++++++++
>>   1 file changed, 16 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index d350175..03e25da 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
>>    N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
>>    \field{num_queues}.
>>   
>> + If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the
>> + request queues are shifted as the following:
> "as follows:"
>
>> +\begin{description}
>> +\item[1] requestq1
>> +\item[\ldots]
>> +\item[N] requestqN
>> +\end{description}
>> +
>> + if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be
>> + negotiated.
> Why?

To distinguish between adminq and requestq.

Re: [virtio-comment] Re: [RFC PATCH v2 2/2] virtio-blk: add support for VIRTIO_F_ADMIN_VQ

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

On Tue, Jul 27, 2021 at 07:08:52PM +0300, Max Gurtovoy wrote:
> 
> On 7/27/2021 3:24 PM, Stefan Hajnoczi wrote:
> > On Mon, Jul 26, 2021 at 07:52:54PM +0300, Max Gurtovoy wrote:
> > > Set the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation. Also
> > > restrict the negotiation of VIRTIO_F_ADMIN_VQ to be possible only if
> > > VIRTIO_BLK_F_MQ is negotiated.
> > > 
> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > ---
> > >   content.tex | 16 ++++++++++++++++
> > >   1 file changed, 16 insertions(+)
> > > 
> > > diff --git a/content.tex b/content.tex
> > > index d350175..03e25da 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
> > >    N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
> > >    \field{num_queues}.
> > > + If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the
> > > + request queues are shifted as the following:
> > "as follows:"
> > 
> > > +\begin{description}
> > > +\item[1] requestq1
> > > +\item[\ldots]
> > > +\item[N] requestqN
> > > +\end{description}
> > > +
> > > + if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be
> > > + negotiated.
> > Why?
> 
> To distinguish between adminq and requestq.

I don't see a conflict between VIRTIO_BLK_F_MQ and VIRTIO_F_ADMIN_VQ.
VIRTIO_BLK_F_MQ just indicates to the driver that it can fetch the
num_queues device configuration field for the number of request queues.

In the case where VIRTIO_F_ADMIN_VQ is negotiated but VIRTIO_BLK_F_MQ is
not negotiated the device's virtqueue layout is still:
1. Admin queue
2. requestq1
...
N+1. requestqN

The driver will probably only initialize the admin queue and requestq1.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 12023 bytes --]

On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
> 
> On 7/27/2021 5:28 PM, Cornelia Huck wrote:
> > On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > 
> > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > Admin virtqueues will be used to send administrative commands to
> > > > manipulate various features of the device which would not easily map
> > > > into the configuration space.
> > > > 
> > > > The same Admin command format will be used for all virtio devices. The
> > > > Admin command set will include 4 types of command classes:
> > > > 1. The generic common class
> > > > 2. The transport specific class
> > > > 3. The device specific class
> > > > 4. The vendor specific class
> > > > 
> > > > The above mechanism will enable adding various features to the virtio
> > > > specification, e.g.:
> > > > 1. Format virtio-blk devices in various configurations (512B block size,
> > > >     512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> > > > 2. Live migration management.
> > > > 3. Encrypt/Decrypt descriptors.
> > > > 4. Virtualization management.
> > > > 5. Get device error logs.
> > > > 6. Implement advanced vendor/device/transport specific features.
> > > > 7. Run device health test.
> > > > 8. More.
> > > > 
> > > > As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
> > > > for the specification to become flexible and allow a wider feature set.
> > > > The corrent ctrl virtq that is defined for some of the virtio devices is
> > > > device specific and wasn't designed to be a generic virtq for
> > > > admininistration.
> > > > 
> > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > ---
> > > >   admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
> > > >   content.tex     |   4 +
> > > >   2 files changed, 245 insertions(+)
> > > >   create mode 100644 admin-virtq.tex
> > > > 
> > > > diff --git a/admin-virtq.tex b/admin-virtq.tex
> > > > new file mode 100644
> > > > index 0000000..ccec2ca
> > > > --- /dev/null
> > > > +++ b/admin-virtq.tex
> > > > @@ -0,0 +1,241 @@
> > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > +
> > > > +Admin virtqueues are used to send administrative commands to manipulate
> > > > +various features of the device which would not easily map into the
> > > > +configuration space.
> > > > +
> > > > +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > +feature bit.
> > > > +
> > > > +Admin virtqueue index may vary among different device types.
> > > > +
> > > > +All commands are of the following form:
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd {
> > > > +        /* Device-readable part */
> > > > +        u8 class;
> > > > +        u8 command;
> > > > +        u8 command-specific-data[];
> > > > +
> > > > +        /* Device-writable part */
> > > > +        u8 command-specific-result[];
> > > > +        u8 status_type : 4;
> > > > +        u8 reserved : 4;
> > > > +        u8 status;
> > > > +};
> > > > +
> > > > +/* Status type values */
> > > > +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
> > > > +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
> > > > +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
> > > > +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
> > > > +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
> > > > +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
> > > > +
> > > > +/* Generic status values */
> > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
> > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
> > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
> > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
> > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
> > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
> > > > +\end{lstlisting}
> > This is very complex, and it feels like we're overengineering this.
> 
> Do you mean the status type and the status ?
> 
> > 
> > > > +
> > > > +The \field{class}, \field{command} and \field{command-specific-data} are
> > > > +set by the driver, and the device sets the \field{status_type}, the
> > > > +\field{status} and  the \field{command-specific-result}, if needed.
> > > > +
> > > > +The virtio Admin command class codes are divided in the following form:
> > > > +
> > > > +\begin{lstlisting}
> > > > +/* class values that are transport, device and vendor independent */
> > > > +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
> > > > +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
> > > > +
> > > > +/* class values that are transport specific */
> > > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
> > > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
> > > > +
> > > > +/* class values that are device specific */
> > > > +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
> > > > +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
> > > > +
> > > > +/* class values that are vendor specific */
> > > > +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
> > > > +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
> > > > +\end{lstlisting}
> > > > +
> > > > +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
> > > > +
> > > > +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
> > > "advertises the VIRTIO_F_ADMIN_VQ feature"
> > > 
> > > > +support all the mandatory admin commands. A device MAY support also
> > > > +one or more optional admin commands.
> > > > +
> > > > +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
> > > > +
> > > > +The Common command set is a group of classes and commands within each
> > > > +of these classes which are transport, device and vendor independent.
> > > > +A mandatory class is a class that has at least one mandatory command.
> > > > +The Common command set is summarized in following table:
> > > > +
> > > > +\begin{tabular}{|l|l|l|}
> > > > +\hline
> > > > +Class  & Description    & M/O \\
> > > > +\hline \hline
> > > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
> > > > +\hline
> > > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
> > > > +\hline
> > > > +2-63  & reserved    & - \\
> > > > +\hline
> > > > +\end{tabular}
> > > > +
> > > > +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
> > > > +
> > > > +This class (opcode: 0) of commands is used to query generic device
> > > > +information. The following table describes the commands supported for
> > > > +this class:
> > > > +
> > > > +\begin{tabular}{|l|l|l|}
> > > > +\hline
> > > > +Command  & Description    & M/O \\
> > > > +\hline \hline
> > > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
> > > > +\hline
> > > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
> > > > +\hline
> > > > +2-255  & reserved    & - \\
> > > > +\hline
> > > > +\end{tabular}
> > > > +
> > > > +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
> > > > +
> > > > +This mandatory command should return device identity in the following
> > > > +structure:
> > > > +
> > > > +\begin{tabular}{|l|l|l|}
> > > > +\hline
> > > > +Bytes  & Description    & M/O \\
> > > > +\hline \hline
> > > > +03:00  & VIRTIO DEVICE ID    & M \\
> > > > +\hline
> > > > +05:04  & VIRTIO TRANSPORT ID    & M \\
> > > These fields are not defined. I wonder why they are necessary - the
> > > driver should already have this information.
> > Agreed.
> 
> These are initial fields.
> 
> We can add also model, serial_number and more in the future.
> 
> 
> > 
> > > In general, I'm a little concerned that this whole infrastructure will
> > > increase the complexity of VIRTIO significantly with little benefit. I
> > > do think an admin virtqueue makes sense, e.g. for migration, but would
> > > prefer it if we focus on actual commands first instead of
> > > infrastructure. That way it will be clear what infrastructure is needed.
> 
> admin virtq is not only for migration.
> 
> You'll be able to configure virtio device properties using user space tools
> like: virtio-cli.
> 
> For example: format a block device, manage virtual function resources using
> its PF, query for error logs, device health and more.

That sounds good.

> In the SW world maybe all the above were redundant, but now that you have
> more and more HW virtio devices the protocol should be more flexible and
> adjust.

HW is not special in this regard, I think this will be useful for
software too. In-band admin commands are necessary for nested
virtualization, for example. They also provide a standard admin
interface for out-of-process devices (vhost-user, etc).

> Few weeks ago I've sent a concrete commands for live migration but then I
> was told that new infrastructure (admin virtq) should be developed and this
> is what I did in this RFC.
> 
> if you combine the 2 RFCs you can imagine what is needed here for adding
> Live migration support.
> 
> But I want to add it step by step.
> 
> We need to agree on the infrastructure.
> 
> > A concrete example would be good, but I think we can come up with a
> > bare-bones spec to start with.
> > 
> > - feature bit for the admin vq, as defined here
> > - location of the admin vq is device specific
> > - I think we can get away with two classes, as for feature bits (not
> >    device specificic and device specific); I don't think we need separate
> >    classes for transport or vendor specific
> 
> We need it for live migration probably. It will be a transport class.
> 
> Vendor specific is also important to allow vendors develop their special
> souse.
> 
> > - make the format for the request simple (command + length + payload?)
> 
> I used almost the same format as virtio net ctrl queue.

The virtio_net_ctrl packet format looks good to me, it's close to what
Cornelia's command + length + payload suggestion:

  struct virtio_net_ctrl {
          u8 class;
          u8 command;
          u8 command-specific-data[];
          u8 ack;
  };
  
  /* ack values */
  #define VIRTIO_NET_OK     0
  #define VIRTIO_NET_ERR    1

I'm not sure how vendor commands will be allocated though. Will each
vendor get a unique class id to prevent collisions? If we want to
support cross-implementation migration then it may be necessary to allow
vendor command availability to change while the device is running.

I prefer the simpler struct virtio_net_ctrl format to the more
complicated one proposed in this patch series.

> 
> > 
> > How many different (groups of) commands can we reasonably expect? Do we
> > need a generic discovery command, or can we get away with a feature bit
> > covering each new group of commands?
> 
> I can't predict the future but IMO we need a discovery command.
> 
> We have many devices and more can be added in the future.

A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin
commands would not be included in VIRTIO Feature Bits but instead
reported via a separate admin command that returns up to 8KB of data:

  struct virtio_admin_report_cmds {
      /* Bitmap of available admin commands [Device->Driver]
       * bool command_present =
       *        command_bits[class * 32 + command / 8] & (command % 8);
       */
      u8 command_bits[8192];
  };

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 7/28/2021 11:52 AM, Stefan Hajnoczi wrote:
> On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
>> On 7/27/2021 5:28 PM, Cornelia Huck wrote:
>>> On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>
>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>> Admin virtqueues will be used to send administrative commands to
>>>>> manipulate various features of the device which would not easily map
>>>>> into the configuration space.
>>>>>
>>>>> The same Admin command format will be used for all virtio devices. The
>>>>> Admin command set will include 4 types of command classes:
>>>>> 1. The generic common class
>>>>> 2. The transport specific class
>>>>> 3. The device specific class
>>>>> 4. The vendor specific class
>>>>>
>>>>> The above mechanism will enable adding various features to the virtio
>>>>> specification, e.g.:
>>>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>>>      512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>>>> 2. Live migration management.
>>>>> 3. Encrypt/Decrypt descriptors.
>>>>> 4. Virtualization management.
>>>>> 5. Get device error logs.
>>>>> 6. Implement advanced vendor/device/transport specific features.
>>>>> 7. Run device health test.
>>>>> 8. More.
>>>>>
>>>>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>>>>> for the specification to become flexible and allow a wider feature set.
>>>>> The corrent ctrl virtq that is defined for some of the virtio devices is
>>>>> device specific and wasn't designed to be a generic virtq for
>>>>> admininistration.
>>>>>
>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>> ---
>>>>>    admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>    content.tex     |   4 +
>>>>>    2 files changed, 245 insertions(+)
>>>>>    create mode 100644 admin-virtq.tex
>>>>>
>>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>>> new file mode 100644
>>>>> index 0000000..ccec2ca
>>>>> --- /dev/null
>>>>> +++ b/admin-virtq.tex
>>>>> @@ -0,0 +1,241 @@
>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>> +
>>>>> +Admin virtqueues are used to send administrative commands to manipulate
>>>>> +various features of the device which would not easily map into the
>>>>> +configuration space.
>>>>> +
>>>>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>> +feature bit.
>>>>> +
>>>>> +Admin virtqueue index may vary among different device types.
>>>>> +
>>>>> +All commands are of the following form:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +struct virtio_admin_cmd {
>>>>> +        /* Device-readable part */
>>>>> +        u8 class;
>>>>> +        u8 command;
>>>>> +        u8 command-specific-data[];
>>>>> +
>>>>> +        /* Device-writable part */
>>>>> +        u8 command-specific-result[];
>>>>> +        u8 status_type : 4;
>>>>> +        u8 reserved : 4;
>>>>> +        u8 status;
>>>>> +};
>>>>> +
>>>>> +/* Status type values */
>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>>>>> +
>>>>> +/* Generic status values */
>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>>>>> +\end{lstlisting}
>>> This is very complex, and it feels like we're overengineering this.
>> Do you mean the status type and the status ?
>>
>>>>> +
>>>>> +The \field{class}, \field{command} and \field{command-specific-data} are
>>>>> +set by the driver, and the device sets the \field{status_type}, the
>>>>> +\field{status} and  the \field{command-specific-result}, if needed.
>>>>> +
>>>>> +The virtio Admin command class codes are divided in the following form:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +/* class values that are transport, device and vendor independent */
>>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>>>>> +
>>>>> +/* class values that are transport specific */
>>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>>>>> +
>>>>> +/* class values that are device specific */
>>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>>>>> +
>>>>> +/* class values that are vendor specific */
>>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>>>>> +
>>>>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>>>> "advertises the VIRTIO_F_ADMIN_VQ feature"
>>>>
>>>>> +support all the mandatory admin commands. A device MAY support also
>>>>> +one or more optional admin commands.
>>>>> +
>>>>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>>>>> +
>>>>> +The Common command set is a group of classes and commands within each
>>>>> +of these classes which are transport, device and vendor independent.
>>>>> +A mandatory class is a class that has at least one mandatory command.
>>>>> +The Common command set is summarized in following table:
>>>>> +
>>>>> +\begin{tabular}{|l|l|l|}
>>>>> +\hline
>>>>> +Class  & Description    & M/O \\
>>>>> +\hline \hline
>>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>>>>> +\hline
>>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>>>>> +\hline
>>>>> +2-63  & reserved    & - \\
>>>>> +\hline
>>>>> +\end{tabular}
>>>>> +
>>>>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>>>>> +
>>>>> +This class (opcode: 0) of commands is used to query generic device
>>>>> +information. The following table describes the commands supported for
>>>>> +this class:
>>>>> +
>>>>> +\begin{tabular}{|l|l|l|}
>>>>> +\hline
>>>>> +Command  & Description    & M/O \\
>>>>> +\hline \hline
>>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>>>>> +\hline
>>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>>>>> +\hline
>>>>> +2-255  & reserved    & - \\
>>>>> +\hline
>>>>> +\end{tabular}
>>>>> +
>>>>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>>>>> +
>>>>> +This mandatory command should return device identity in the following
>>>>> +structure:
>>>>> +
>>>>> +\begin{tabular}{|l|l|l|}
>>>>> +\hline
>>>>> +Bytes  & Description    & M/O \\
>>>>> +\hline \hline
>>>>> +03:00  & VIRTIO DEVICE ID    & M \\
>>>>> +\hline
>>>>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>>>> These fields are not defined. I wonder why they are necessary - the
>>>> driver should already have this information.
>>> Agreed.
>> These are initial fields.
>>
>> We can add also model, serial_number and more in the future.
>>
>>
>>>> In general, I'm a little concerned that this whole infrastructure will
>>>> increase the complexity of VIRTIO significantly with little benefit. I
>>>> do think an admin virtqueue makes sense, e.g. for migration, but would
>>>> prefer it if we focus on actual commands first instead of
>>>> infrastructure. That way it will be clear what infrastructure is needed.
>> admin virtq is not only for migration.
>>
>> You'll be able to configure virtio device properties using user space tools
>> like: virtio-cli.
>>
>> For example: format a block device, manage virtual function resources using
>> its PF, query for error logs, device health and more.
> That sounds good.
>
>> In the SW world maybe all the above were redundant, but now that you have
>> more and more HW virtio devices the protocol should be more flexible and
>> adjust.
> HW is not special in this regard, I think this will be useful for
> software too. In-band admin commands are necessary for nested
> virtualization, for example. They also provide a standard admin
> interface for out-of-process devices (vhost-user, etc).
>
>> Few weeks ago I've sent a concrete commands for live migration but then I
>> was told that new infrastructure (admin virtq) should be developed and this
>> is what I did in this RFC.
>>
>> if you combine the 2 RFCs you can imagine what is needed here for adding
>> Live migration support.
>>
>> But I want to add it step by step.
>>
>> We need to agree on the infrastructure.
>>
>>> A concrete example would be good, but I think we can come up with a
>>> bare-bones spec to start with.
>>>
>>> - feature bit for the admin vq, as defined here
>>> - location of the admin vq is device specific
>>> - I think we can get away with two classes, as for feature bits (not
>>>     device specificic and device specific); I don't think we need separate
>>>     classes for transport or vendor specific
>> We need it for live migration probably. It will be a transport class.
>>
>> Vendor specific is also important to allow vendors develop their special
>> souse.
>>
>>> - make the format for the request simple (command + length + payload?)
>> I used almost the same format as virtio net ctrl queue.
> The virtio_net_ctrl packet format looks good to me, it's close to what
> Cornelia's command + length + payload suggestion:

I guess I didn't understand Cornelia suggestion.


>
>    struct virtio_net_ctrl {
>            u8 class;
>            u8 command;
>            u8 command-specific-data[];
>            u8 ack;
>    };
>    
>    /* ack values */
>    #define VIRTIO_NET_OK     0
>    #define VIRTIO_NET_ERR    1
>
> I'm not sure how vendor commands will be allocated though. Will each
> vendor get a unique class id to prevent collisions? If we want to
> support cross-implementation migration then it may be necessary to allow
> vendor command availability to change while the device is running.

vendor specific commands can collide.

Vendor A can implement class 192 to do X and Vendor B can implement 
class 192 to do Y.

what do you mean "support cross-implementation migration" ?


> I prefer the simpler struct virtio_net_ctrl format to the more
> complicated one proposed in this patch series.

This is the same besides adding status type

u8 status_type : 4;
u8 reserved : 4;


I split "u8 command-specific-data[];"
to
"u8 command-specific-data[];
  u8 command-specific-result[];"

to emphasize that there is some data that can be written by the device and some data written by the driver in the same command.
And this is also the case in virtio-net-ctrl, right ?
  

>
>>> How many different (groups of) commands can we reasonably expect? Do we
>>> need a generic discovery command, or can we get away with a feature bit
>>> covering each new group of commands?
>> I can't predict the future but IMO we need a discovery command.
>>
>> We have many devices and more can be added in the future.
> A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin
> commands would not be included in VIRTIO Feature Bits but instead
> reported via a separate admin command that returns up to 8KB of data:
>
>    struct virtio_admin_report_cmds {
>        /* Bitmap of available admin commands [Device->Driver]
>         * bool command_present =
>         *        command_bits[class * 32 + command / 8] & (command % 8);
>         */
>        u8 command_bits[8192];
>    };

Yes, I divided it to multiple commands per class to cover the case we 
will need more than 1 bit to describe a command.

But I guess we can add it later on.

I think the above should be:

bool command_present = command_bits[class * 32 + command / 8] & (1 << (command % 8));

isn't it ?

Also what do you think about renaming <class, command> to <opcode, opmod> ?


>
> Stefan

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> Admin virtqueues will be used to send administrative commands to
> manipulate various features of the device which would not easily map
> into the configuration space.
> 
> The same Admin command format will be used for all virtio devices. The
> Admin command set will include 4 types of command classes:
> 1. The generic common class
> 2. The transport specific class
> 3. The device specific class
> 4. The vendor specific class
> 
> The above mechanism will enable adding various features to the virtio
> specification, e.g.:
> 1. Format virtio-blk devices in various configurations (512B block size,
>    512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> 2. Live migration management.
> 3. Encrypt/Decrypt descriptors.
> 4. Virtualization management.
> 5. Get device error logs.
> 6. Implement advanced vendor/device/transport specific features.
> 7. Run device health test.
> 8. More.
> 
> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
> for the specification to become flexible and allow a wider feature set.
> The corrent ctrl virtq that is defined for some of the virtio devices is
> device specific and wasn't designed to be a generic virtq for
> admininistration.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>

Lots of things on this list seem to make sense when
done from PF and affecting a VF.
I think from this POV the generic structure should include
a way to address one device from another.


> ---
>  admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex     |   4 +
>  2 files changed, 245 insertions(+)
>  create mode 100644 admin-virtq.tex
> 
> diff --git a/admin-virtq.tex b/admin-virtq.tex
> new file mode 100644
> index 0000000..ccec2ca
> --- /dev/null
> +++ b/admin-virtq.tex
> @@ -0,0 +1,241 @@
> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> +
> +Admin virtqueues are used to send administrative commands to manipulate
> +various features of the device which would not easily map into the
> +configuration space.
> +
> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
> +feature bit.
> +
> +Admin virtqueue index may vary among different device types.
> +
> +All commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        u8 class;
> +        u8 command;
> +        u8 command-specific-data[];
> +
> +        /* Device-writable part */
> +        u8 command-specific-result[];
> +        u8 status_type : 4;
> +        u8 reserved : 4;
> +        u8 status;
> +};
> +
> +/* Status type values */
> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
> +
> +/* Generic status values */
> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
> +\end{lstlisting}
> +
> +The \field{class}, \field{command} and \field{command-specific-data} are
> +set by the driver, and the device sets the \field{status_type}, the
> +\field{status} and  the \field{command-specific-result}, if needed.
> +
> +The virtio Admin command class codes are divided in the following form:
> +
> +\begin{lstlisting}
> +/* class values that are transport, device and vendor independent */
> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
> +
> +/* class values that are transport specific */
> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
> +
> +/* class values that are device specific */
> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
> +
> +/* class values that are vendor specific */
> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
> +\end{lstlisting}
> +
> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
> +
> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
> +support all the mandatory admin commands. A device MAY support also
> +one or more optional admin commands.
> +
> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
> +
> +The Common command set is a group of classes and commands within each
> +of these classes which are transport, device and vendor independent.
> +A mandatory class is a class that has at least one mandatory command.
> +The Common command set is summarized in following table:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Class  & Description    & M/O \\
> +\hline \hline
> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
> +\hline
> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
> +\hline
> +2-63  & reserved    & - \\
> +\hline
> +\end{tabular}
> +
> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
> +
> +This class (opcode: 0) of commands is used to query generic device
> +information. The following table describes the commands supported for
> +this class:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Command  & Description    & M/O \\
> +\hline \hline
> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
> +\hline
> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
> +\hline
> +2-255  & reserved    & - \\
> +\hline
> +\end{tabular}

So there are several problems with this approach.
One is that there is no two way negotiation.
No way for device to know what will driver use in the end.
This breaks things like e.g. accelerating some configurations
but not others.

Another is that everything is going through the admin vq.
Hard for hypervisor to intervene and change just some aspects
of the device.

This is also a problem for validation, tricky
to find out during probe what does device support and whether
driver can work with it.


Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.



> +
> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
> +
> +This mandatory command should return device identity in the following
> +structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bytes  & Description    & M/O \\
> +\hline \hline
> +03:00  & VIRTIO DEVICE ID    & M \\
> +\hline
> +05:04  & VIRTIO TRANSPORT ID    & M \\
> +\hline
> +4095:06  & reserved    & - \\
> + \hline
> +\end{tabular}
> +
> +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
> +
> +This mandatory command should return device identity in the following
> +structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bytes  & Description    & M/O \\
> +\hline \hline
> +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
> +\hline
> +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
> + \hline
> +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
> +\hline
> +...  & ...    & M \\
> +\hline
> +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
> +\hline
> +4095:1024  & reserved    & - \\
> +\hline
> +\end{tabular}
> +
> +The above structure is divided to 256 sections of 4B each, that will
> +describe whether a device supports a certain class code. The
> +following 3072B are reserved. each of the non-reserved 4B sections will
> +follow the following data structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bits  & Description    & M/O \\
> +\hline \hline
> +0  & SUPPORTED (1: yes, 0:no)    & M \\
> +\hline
> +31:01  & reserved    & - \\
> + \hline
> +\end{tabular}
> +
> +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
> +
> +This class (opcode: 1) of commands is used to query supported commands
> +for each supported device class.
> The following table describes the commands
> +supported for this class:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Command  & Description    & M/O \\
> +\hline \hline
> +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
> +\hline
> +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
> +\hline
> +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
> +\hline
> +...  & ...    & M \\
> +\hline
> +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
> +\hline
> +\end{tabular}
> +
> +Each command in this class, will return class identity in the following structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bytes  & Description    & M/O \\
> +\hline \hline
> +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
> +\hline
> +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
> + \hline
> +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
> +\hline
> +...  & ...    & M \\
> +\hline
> +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
> +\hline
> +4095:1024  & reserved    & - \\
> +\hline
> +\end{tabular}
> +
> +The above structure is divided to 256 sections of 4B each, that will
> +describe whether a class supports a certain command code. The
> +following 3072B are reserved. each of the non-reserved 4B sections will
> +follow the following data structure:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Bits  & Description    & M/O \\
> +\hline \hline
> +0  & SUPPORTED (1: yes, 0:no)    & M \\
> +\hline
> +31:01  & reserved    & - \\
> + \hline
> +\end{tabular}
> +
> +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
> +
> +The Transport command set is a group of classes and commands within
> +each of these classes which are transport specific. Refer to each
> +transport section for more information on the supported commands.
> +
> +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
> +
> +The Device command set is a group of classes and commands within
> +each of these classes which are device specific. Refer to each
> +device section for more information on the supported commands.
> +
> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
> +
> +The Vendor specific command set is a group of classes and commands
> +within each of these classes which are vendor specific. Refer to
> +each vendor specification for more information on the supported
> +commands.

Here's another example.
It's important that there is a way to make a device completely
generic without vendor specific expensions.
Would be hard to do here.

That's a generic comment.

but specifically I am very reluctant to add vendor specific stuff like
this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why
before we add more vendor specific stuff.


> diff --git a/content.tex b/content.tex
> index c266fd5..d350175 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  types. It is RECOMMENDED that devices generate version 4
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>  
> +\input{admin-virtq.tex}
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
>  We start with an overview of device initialization, then expand on the
> @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>    data to be provided in driver notification and the delivery method is
>    transport specific.
>    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
> +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
> +  the device supports administration virtqueue negotiation.
>  
>  \end{description}
>  
> -- 
> 2.21.0

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
> Vendor specific is also important to allow vendors develop their special
> souse.

Most of the value of virtio is that it's generic.
So we added VIRTIO_PCI_CAP_VENDOR_CFG which is very easy for
the hypervisor to intercept. Maybe that is not enough but that
needs justification, I am not sure it's a good idea to add multiple
ways devices can differ without a lot of thought.

-- 
MST

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 14802 bytes --]

On Wed, Jul 28, 2021 at 01:59:26PM +0300, Max Gurtovoy wrote:
> 
> On 7/28/2021 11:52 AM, Stefan Hajnoczi wrote:
> > On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
> > > On 7/27/2021 5:28 PM, Cornelia Huck wrote:
> > > > On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > 
> > > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > > > Admin virtqueues will be used to send administrative commands to
> > > > > > manipulate various features of the device which would not easily map
> > > > > > into the configuration space.
> > > > > > 
> > > > > > The same Admin command format will be used for all virtio devices. The
> > > > > > Admin command set will include 4 types of command classes:
> > > > > > 1. The generic common class
> > > > > > 2. The transport specific class
> > > > > > 3. The device specific class
> > > > > > 4. The vendor specific class
> > > > > > 
> > > > > > The above mechanism will enable adding various features to the virtio
> > > > > > specification, e.g.:
> > > > > > 1. Format virtio-blk devices in various configurations (512B block size,
> > > > > >      512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> > > > > > 2. Live migration management.
> > > > > > 3. Encrypt/Decrypt descriptors.
> > > > > > 4. Virtualization management.
> > > > > > 5. Get device error logs.
> > > > > > 6. Implement advanced vendor/device/transport specific features.
> > > > > > 7. Run device health test.
> > > > > > 8. More.
> > > > > > 
> > > > > > As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
> > > > > > for the specification to become flexible and allow a wider feature set.
> > > > > > The corrent ctrl virtq that is defined for some of the virtio devices is
> > > > > > device specific and wasn't designed to be a generic virtq for
> > > > > > admininistration.
> > > > > > 
> > > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > > > ---
> > > > > >    admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
> > > > > >    content.tex     |   4 +
> > > > > >    2 files changed, 245 insertions(+)
> > > > > >    create mode 100644 admin-virtq.tex
> > > > > > 
> > > > > > diff --git a/admin-virtq.tex b/admin-virtq.tex
> > > > > > new file mode 100644
> > > > > > index 0000000..ccec2ca
> > > > > > --- /dev/null
> > > > > > +++ b/admin-virtq.tex
> > > > > > @@ -0,0 +1,241 @@
> > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > > > +
> > > > > > +Admin virtqueues are used to send administrative commands to manipulate
> > > > > > +various features of the device which would not easily map into the
> > > > > > +configuration space.
> > > > > > +
> > > > > > +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > > > +feature bit.
> > > > > > +
> > > > > > +Admin virtqueue index may vary among different device types.
> > > > > > +
> > > > > > +All commands are of the following form:
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_admin_cmd {
> > > > > > +        /* Device-readable part */
> > > > > > +        u8 class;
> > > > > > +        u8 command;
> > > > > > +        u8 command-specific-data[];
> > > > > > +
> > > > > > +        /* Device-writable part */
> > > > > > +        u8 command-specific-result[];
> > > > > > +        u8 status_type : 4;
> > > > > > +        u8 reserved : 4;
> > > > > > +        u8 status;
> > > > > > +};
> > > > > > +
> > > > > > +/* Status type values */
> > > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
> > > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
> > > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
> > > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
> > > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
> > > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
> > > > > > +
> > > > > > +/* Generic status values */
> > > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
> > > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
> > > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
> > > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
> > > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
> > > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
> > > > > > +\end{lstlisting}
> > > > This is very complex, and it feels like we're overengineering this.
> > > Do you mean the status type and the status ?
> > > 
> > > > > > +
> > > > > > +The \field{class}, \field{command} and \field{command-specific-data} are
> > > > > > +set by the driver, and the device sets the \field{status_type}, the
> > > > > > +\field{status} and  the \field{command-specific-result}, if needed.
> > > > > > +
> > > > > > +The virtio Admin command class codes are divided in the following form:
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +/* class values that are transport, device and vendor independent */
> > > > > > +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
> > > > > > +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
> > > > > > +
> > > > > > +/* class values that are transport specific */
> > > > > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
> > > > > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
> > > > > > +
> > > > > > +/* class values that are device specific */
> > > > > > +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
> > > > > > +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
> > > > > > +
> > > > > > +/* class values that are vendor specific */
> > > > > > +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
> > > > > > +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
> > > > > > +
> > > > > > +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
> > > > > "advertises the VIRTIO_F_ADMIN_VQ feature"
> > > > > 
> > > > > > +support all the mandatory admin commands. A device MAY support also
> > > > > > +one or more optional admin commands.
> > > > > > +
> > > > > > +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
> > > > > > +
> > > > > > +The Common command set is a group of classes and commands within each
> > > > > > +of these classes which are transport, device and vendor independent.
> > > > > > +A mandatory class is a class that has at least one mandatory command.
> > > > > > +The Common command set is summarized in following table:
> > > > > > +
> > > > > > +\begin{tabular}{|l|l|l|}
> > > > > > +\hline
> > > > > > +Class  & Description    & M/O \\
> > > > > > +\hline \hline
> > > > > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
> > > > > > +\hline
> > > > > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
> > > > > > +\hline
> > > > > > +2-63  & reserved    & - \\
> > > > > > +\hline
> > > > > > +\end{tabular}
> > > > > > +
> > > > > > +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
> > > > > > +
> > > > > > +This class (opcode: 0) of commands is used to query generic device
> > > > > > +information. The following table describes the commands supported for
> > > > > > +this class:
> > > > > > +
> > > > > > +\begin{tabular}{|l|l|l|}
> > > > > > +\hline
> > > > > > +Command  & Description    & M/O \\
> > > > > > +\hline \hline
> > > > > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
> > > > > > +\hline
> > > > > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
> > > > > > +\hline
> > > > > > +2-255  & reserved    & - \\
> > > > > > +\hline
> > > > > > +\end{tabular}
> > > > > > +
> > > > > > +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
> > > > > > +
> > > > > > +This mandatory command should return device identity in the following
> > > > > > +structure:
> > > > > > +
> > > > > > +\begin{tabular}{|l|l|l|}
> > > > > > +\hline
> > > > > > +Bytes  & Description    & M/O \\
> > > > > > +\hline \hline
> > > > > > +03:00  & VIRTIO DEVICE ID    & M \\
> > > > > > +\hline
> > > > > > +05:04  & VIRTIO TRANSPORT ID    & M \\
> > > > > These fields are not defined. I wonder why they are necessary - the
> > > > > driver should already have this information.
> > > > Agreed.
> > > These are initial fields.
> > > 
> > > We can add also model, serial_number and more in the future.
> > > 
> > > 
> > > > > In general, I'm a little concerned that this whole infrastructure will
> > > > > increase the complexity of VIRTIO significantly with little benefit. I
> > > > > do think an admin virtqueue makes sense, e.g. for migration, but would
> > > > > prefer it if we focus on actual commands first instead of
> > > > > infrastructure. That way it will be clear what infrastructure is needed.
> > > admin virtq is not only for migration.
> > > 
> > > You'll be able to configure virtio device properties using user space tools
> > > like: virtio-cli.
> > > 
> > > For example: format a block device, manage virtual function resources using
> > > its PF, query for error logs, device health and more.
> > That sounds good.
> > 
> > > In the SW world maybe all the above were redundant, but now that you have
> > > more and more HW virtio devices the protocol should be more flexible and
> > > adjust.
> > HW is not special in this regard, I think this will be useful for
> > software too. In-band admin commands are necessary for nested
> > virtualization, for example. They also provide a standard admin
> > interface for out-of-process devices (vhost-user, etc).
> > 
> > > Few weeks ago I've sent a concrete commands for live migration but then I
> > > was told that new infrastructure (admin virtq) should be developed and this
> > > is what I did in this RFC.
> > > 
> > > if you combine the 2 RFCs you can imagine what is needed here for adding
> > > Live migration support.
> > > 
> > > But I want to add it step by step.
> > > 
> > > We need to agree on the infrastructure.
> > > 
> > > > A concrete example would be good, but I think we can come up with a
> > > > bare-bones spec to start with.
> > > > 
> > > > - feature bit for the admin vq, as defined here
> > > > - location of the admin vq is device specific
> > > > - I think we can get away with two classes, as for feature bits (not
> > > >     device specificic and device specific); I don't think we need separate
> > > >     classes for transport or vendor specific
> > > We need it for live migration probably. It will be a transport class.
> > > 
> > > Vendor specific is also important to allow vendors develop their special
> > > souse.
> > > 
> > > > - make the format for the request simple (command + length + payload?)
> > > I used almost the same format as virtio net ctrl queue.
> > The virtio_net_ctrl packet format looks good to me, it's close to what
> > Cornelia's command + length + payload suggestion:
> 
> I guess I didn't understand Cornelia suggestion.
> 
> 
> > 
> >    struct virtio_net_ctrl {
> >            u8 class;
> >            u8 command;
> >            u8 command-specific-data[];
> >            u8 ack;
> >    };
> >    /* ack values */
> >    #define VIRTIO_NET_OK     0
> >    #define VIRTIO_NET_ERR    1
> > 
> > I'm not sure how vendor commands will be allocated though. Will each
> > vendor get a unique class id to prevent collisions? If we want to
> > support cross-implementation migration then it may be necessary to allow
> > vendor command availability to change while the device is running.
> 
> vendor specific commands can collide.
> 
> Vendor A can implement class 192 to do X and Vendor B can implement class
> 192 to do Y.
> 
> what do you mean "support cross-implementation migration" ?

Migrating from vhost_net to vDPA virtio-net, for example. Or migrating
between two different vDPA virtio-net implementations.

If vendor commands are all in a single namespace then the guest cannot
use them without the risk of the command accidentally executing on the
migration destination (where it has a different effect because the
vendor has changed!).

> > I prefer the simpler struct virtio_net_ctrl format to the more
> > complicated one proposed in this patch series.
> 
> This is the same besides adding status type
> 
> u8 status_type : 4;
> u8 reserved : 4;

I'm not sure why it's needed.

> I split "u8 command-specific-data[];"
> to
> "u8 command-specific-data[];
>  u8 command-specific-result[];"
> 
> to emphasize that there is some data that can be written by the device and some data written by the driver in the same command.
> And this is also the case in virtio-net-ctrl, right ?

The split makes sense to me.

> > > > How many different (groups of) commands can we reasonably expect? Do we
> > > > need a generic discovery command, or can we get away with a feature bit
> > > > covering each new group of commands?
> > > I can't predict the future but IMO we need a discovery command.
> > > 
> > > We have many devices and more can be added in the future.
> > A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin
> > commands would not be included in VIRTIO Feature Bits but instead
> > reported via a separate admin command that returns up to 8KB of data:
> > 
> >    struct virtio_admin_report_cmds {
> >        /* Bitmap of available admin commands [Device->Driver]
> >         * bool command_present =
> >         *        command_bits[class * 32 + command / 8] & (command % 8);
> >         */
> >        u8 command_bits[8192];
> >    };
> 
> Yes, I divided it to multiple commands per class to cover the case we will
> need more than 1 bit to describe a command.
> 
> But I guess we can add it later on.
> 
> I think the above should be:
> 
> bool command_present = command_bits[class * 32 + command / 8] & (1 << (command % 8));
> 
> isn't it ?

You're right. I forgot to shift the bit :D.

> Also what do you think about renaming <class, command> to <opcode, opmod> ?

I need to understand how opcode and opmod values are used. I'm not sure.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 7/28/2021 4:42 PM, Stefan Hajnoczi wrote:
> On Wed, Jul 28, 2021 at 01:59:26PM +0300, Max Gurtovoy wrote:
>> On 7/28/2021 11:52 AM, Stefan Hajnoczi wrote:
>>> On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
>>>> On 7/27/2021 5:28 PM, Cornelia Huck wrote:
>>>>> On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>>>
>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>> Admin virtqueues will be used to send administrative commands to
>>>>>>> manipulate various features of the device which would not easily map
>>>>>>> into the configuration space.
>>>>>>>
>>>>>>> The same Admin command format will be used for all virtio devices. The
>>>>>>> Admin command set will include 4 types of command classes:
>>>>>>> 1. The generic common class
>>>>>>> 2. The transport specific class
>>>>>>> 3. The device specific class
>>>>>>> 4. The vendor specific class
>>>>>>>
>>>>>>> The above mechanism will enable adding various features to the virtio
>>>>>>> specification, e.g.:
>>>>>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>>>>>       512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>>>>>> 2. Live migration management.
>>>>>>> 3. Encrypt/Decrypt descriptors.
>>>>>>> 4. Virtualization management.
>>>>>>> 5. Get device error logs.
>>>>>>> 6. Implement advanced vendor/device/transport specific features.
>>>>>>> 7. Run device health test.
>>>>>>> 8. More.
>>>>>>>
>>>>>>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>>>>>>> for the specification to become flexible and allow a wider feature set.
>>>>>>> The corrent ctrl virtq that is defined for some of the virtio devices is
>>>>>>> device specific and wasn't designed to be a generic virtq for
>>>>>>> admininistration.
>>>>>>>
>>>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>>> ---
>>>>>>>     admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>     content.tex     |   4 +
>>>>>>>     2 files changed, 245 insertions(+)
>>>>>>>     create mode 100644 admin-virtq.tex
>>>>>>>
>>>>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>>>>> new file mode 100644
>>>>>>> index 0000000..ccec2ca
>>>>>>> --- /dev/null
>>>>>>> +++ b/admin-virtq.tex
>>>>>>> @@ -0,0 +1,241 @@
>>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>>>> +
>>>>>>> +Admin virtqueues are used to send administrative commands to manipulate
>>>>>>> +various features of the device which would not easily map into the
>>>>>>> +configuration space.
>>>>>>> +
>>>>>>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>>> +feature bit.
>>>>>>> +
>>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>>> +
>>>>>>> +All commands are of the following form:
>>>>>>> +
>>>>>>> +\begin{lstlisting}
>>>>>>> +struct virtio_admin_cmd {
>>>>>>> +        /* Device-readable part */
>>>>>>> +        u8 class;
>>>>>>> +        u8 command;
>>>>>>> +        u8 command-specific-data[];
>>>>>>> +
>>>>>>> +        /* Device-writable part */
>>>>>>> +        u8 command-specific-result[];
>>>>>>> +        u8 status_type : 4;
>>>>>>> +        u8 reserved : 4;
>>>>>>> +        u8 status;
>>>>>>> +};
>>>>>>> +
>>>>>>> +/* Status type values */
>>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>>>>>>> +
>>>>>>> +/* Generic status values */
>>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>>>>>>> +\end{lstlisting}
>>>>> This is very complex, and it feels like we're overengineering this.
>>>> Do you mean the status type and the status ?
>>>>
>>>>>>> +
>>>>>>> +The \field{class}, \field{command} and \field{command-specific-data} are
>>>>>>> +set by the driver, and the device sets the \field{status_type}, the
>>>>>>> +\field{status} and  the \field{command-specific-result}, if needed.
>>>>>>> +
>>>>>>> +The virtio Admin command class codes are divided in the following form:
>>>>>>> +
>>>>>>> +\begin{lstlisting}
>>>>>>> +/* class values that are transport, device and vendor independent */
>>>>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>>>>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>>>>>>> +
>>>>>>> +/* class values that are transport specific */
>>>>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>>>>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>>>>>>> +
>>>>>>> +/* class values that are device specific */
>>>>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>>>>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>>>>>>> +
>>>>>>> +/* class values that are vendor specific */
>>>>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>>>>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>>>>>>> +\end{lstlisting}
>>>>>>> +
>>>>>>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>>>>>>> +
>>>>>>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>>>>>> "advertises the VIRTIO_F_ADMIN_VQ feature"
>>>>>>
>>>>>>> +support all the mandatory admin commands. A device MAY support also
>>>>>>> +one or more optional admin commands.
>>>>>>> +
>>>>>>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>>>>>>> +
>>>>>>> +The Common command set is a group of classes and commands within each
>>>>>>> +of these classes which are transport, device and vendor independent.
>>>>>>> +A mandatory class is a class that has at least one mandatory command.
>>>>>>> +The Common command set is summarized in following table:
>>>>>>> +
>>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>>> +\hline
>>>>>>> +Class  & Description    & M/O \\
>>>>>>> +\hline \hline
>>>>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>>>>>>> +\hline
>>>>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>>>>>>> +\hline
>>>>>>> +2-63  & reserved    & - \\
>>>>>>> +\hline
>>>>>>> +\end{tabular}
>>>>>>> +
>>>>>>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>>>>>>> +
>>>>>>> +This class (opcode: 0) of commands is used to query generic device
>>>>>>> +information. The following table describes the commands supported for
>>>>>>> +this class:
>>>>>>> +
>>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>>> +\hline
>>>>>>> +Command  & Description    & M/O \\
>>>>>>> +\hline \hline
>>>>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>>>>>>> +\hline
>>>>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>>>>>>> +\hline
>>>>>>> +2-255  & reserved    & - \\
>>>>>>> +\hline
>>>>>>> +\end{tabular}
>>>>>>> +
>>>>>>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>>>>>>> +
>>>>>>> +This mandatory command should return device identity in the following
>>>>>>> +structure:
>>>>>>> +
>>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>>> +\hline
>>>>>>> +Bytes  & Description    & M/O \\
>>>>>>> +\hline \hline
>>>>>>> +03:00  & VIRTIO DEVICE ID    & M \\
>>>>>>> +\hline
>>>>>>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>>>>>> These fields are not defined. I wonder why they are necessary - the
>>>>>> driver should already have this information.
>>>>> Agreed.
>>>> These are initial fields.
>>>>
>>>> We can add also model, serial_number and more in the future.
>>>>
>>>>
>>>>>> In general, I'm a little concerned that this whole infrastructure will
>>>>>> increase the complexity of VIRTIO significantly with little benefit. I
>>>>>> do think an admin virtqueue makes sense, e.g. for migration, but would
>>>>>> prefer it if we focus on actual commands first instead of
>>>>>> infrastructure. That way it will be clear what infrastructure is needed.
>>>> admin virtq is not only for migration.
>>>>
>>>> You'll be able to configure virtio device properties using user space tools
>>>> like: virtio-cli.
>>>>
>>>> For example: format a block device, manage virtual function resources using
>>>> its PF, query for error logs, device health and more.
>>> That sounds good.
>>>
>>>> In the SW world maybe all the above were redundant, but now that you have
>>>> more and more HW virtio devices the protocol should be more flexible and
>>>> adjust.
>>> HW is not special in this regard, I think this will be useful for
>>> software too. In-band admin commands are necessary for nested
>>> virtualization, for example. They also provide a standard admin
>>> interface for out-of-process devices (vhost-user, etc).
>>>
>>>> Few weeks ago I've sent a concrete commands for live migration but then I
>>>> was told that new infrastructure (admin virtq) should be developed and this
>>>> is what I did in this RFC.
>>>>
>>>> if you combine the 2 RFCs you can imagine what is needed here for adding
>>>> Live migration support.
>>>>
>>>> But I want to add it step by step.
>>>>
>>>> We need to agree on the infrastructure.
>>>>
>>>>> A concrete example would be good, but I think we can come up with a
>>>>> bare-bones spec to start with.
>>>>>
>>>>> - feature bit for the admin vq, as defined here
>>>>> - location of the admin vq is device specific
>>>>> - I think we can get away with two classes, as for feature bits (not
>>>>>      device specificic and device specific); I don't think we need separate
>>>>>      classes for transport or vendor specific
>>>> We need it for live migration probably. It will be a transport class.
>>>>
>>>> Vendor specific is also important to allow vendors develop their special
>>>> souse.
>>>>
>>>>> - make the format for the request simple (command + length + payload?)
>>>> I used almost the same format as virtio net ctrl queue.
>>> The virtio_net_ctrl packet format looks good to me, it's close to what
>>> Cornelia's command + length + payload suggestion:
>> I guess I didn't understand Cornelia suggestion.
>>
>>
>>>     struct virtio_net_ctrl {
>>>             u8 class;
>>>             u8 command;
>>>             u8 command-specific-data[];
>>>             u8 ack;
>>>     };
>>>     /* ack values */
>>>     #define VIRTIO_NET_OK     0
>>>     #define VIRTIO_NET_ERR    1
>>>
>>> I'm not sure how vendor commands will be allocated though. Will each
>>> vendor get a unique class id to prevent collisions? If we want to
>>> support cross-implementation migration then it may be necessary to allow
>>> vendor command availability to change while the device is running.
>> vendor specific commands can collide.
>>
>> Vendor A can implement class 192 to do X and Vendor B can implement class
>> 192 to do Y.
>>
>> what do you mean "support cross-implementation migration" ?
> Migrating from vhost_net to vDPA virtio-net, for example. Or migrating
> between two different vDPA virtio-net implementations.
>
> If vendor commands are all in a single namespace then the guest cannot
> use them without the risk of the command accidentally executing on the
> migration destination (where it has a different effect because the
> vendor has changed!).
>
>>> I prefer the simpler struct virtio_net_ctrl format to the more
>>> complicated one proposed in this patch series.
>> This is the same besides adding status type
>>
>> u8 status_type : 4;
>> u8 reserved : 4;
> I'm not sure why it's needed.

If we can live with 256 status code, I guess we can drop it and divide 
it to groups:

/* status values that are transport, device and vendor independent */
#define VIRTIO_ADMIN_STATUS_GENERIC_START    0
#define VIRTIO_ADMIN_STATUS_GENERIC_END      63

/* status values that are transport specific */
#define VIRTIO_ADMIN_STATUS_TRANSPORT_START  64
#define VIRTIO_ADMIN_STATUS_TRANSPORT_END    127

/* status values that are device specific */
#define VIRTIO_ADMIN_STATUS_DEVICE_START     128
#define VIRTIO_ADMIN_STATUS_DEVICE_END       191

/* status values that are vendor specific */
#define VIRTIO_ADMIN_STATUS_VENDOR_START     192
#define VIRTIO_ADMIN_STATUS_VENDOR_END       255


>
>> I split "u8 command-specific-data[];"
>> to
>> "u8 command-specific-data[];
>>   u8 command-specific-result[];"
>>
>> to emphasize that there is some data that can be written by the device and some data written by the driver in the same command.
>> And this is also the case in virtio-net-ctrl, right ?
> The split makes sense to me.
>
>>>>> How many different (groups of) commands can we reasonably expect? Do we
>>>>> need a generic discovery command, or can we get away with a feature bit
>>>>> covering each new group of commands?
>>>> I can't predict the future but IMO we need a discovery command.
>>>>
>>>> We have many devices and more can be added in the future.
>>> A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin
>>> commands would not be included in VIRTIO Feature Bits but instead
>>> reported via a separate admin command that returns up to 8KB of data:
>>>
>>>     struct virtio_admin_report_cmds {
>>>         /* Bitmap of available admin commands [Device->Driver]
>>>          * bool command_present =
>>>          *        command_bits[class * 32 + command / 8] & (command % 8);
>>>          */
>>>         u8 command_bits[8192];
>>>     };
>> Yes, I divided it to multiple commands per class to cover the case we will
>> need more than 1 bit to describe a command.
>>
>> But I guess we can add it later on.
>>
>> I think the above should be:
>>
>> bool command_present = command_bits[class * 32 + command / 8] & (1 << (command % 8));
>>
>> isn't it ?
> You're right. I forgot to shift the bit :D.
>
>> Also what do you think about renaming <class, command> to <opcode, opmod> ?
> I need to understand how opcode and opmod values are used. I'm not sure.

Same as class and command just with different naming.

>
> Stefan

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]

On Wed, Jul 28, 2021 at 05:20:29PM +0300, Max Gurtovoy wrote:
> On 7/28/2021 4:42 PM, Stefan Hajnoczi wrote:
> > On Wed, Jul 28, 2021 at 01:59:26PM +0300, Max Gurtovoy wrote:
> > > On 7/28/2021 11:52 AM, Stefan Hajnoczi wrote:
> > > > On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
> > > > > On 7/27/2021 5:28 PM, Cornelia Huck wrote:
> > > > > > On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > > > How many different (groups of) commands can we reasonably expect? Do we
> > > > > > need a generic discovery command, or can we get away with a feature bit
> > > > > > covering each new group of commands?
> > > > > I can't predict the future but IMO we need a discovery command.
> > > > > 
> > > > > We have many devices and more can be added in the future.
> > > > A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin
> > > > commands would not be included in VIRTIO Feature Bits but instead
> > > > reported via a separate admin command that returns up to 8KB of data:
> > > > 
> > > >     struct virtio_admin_report_cmds {
> > > >         /* Bitmap of available admin commands [Device->Driver]
> > > >          * bool command_present =
> > > >          *        command_bits[class * 32 + command / 8] & (command % 8);
> > > >          */
> > > >         u8 command_bits[8192];
> > > >     };
> > > Yes, I divided it to multiple commands per class to cover the case we will
> > > need more than 1 bit to describe a command.
> > > 
> > > But I guess we can add it later on.
> > > 
> > > I think the above should be:
> > > 
> > > bool command_present = command_bits[class * 32 + command / 8] & (1 << (command % 8));
> > > 
> > > isn't it ?
> > You're right. I forgot to shift the bit :D.
> > 
> > > Also what do you think about renaming <class, command> to <opcode, opmod> ?
> > I need to understand how opcode and opmod values are used. I'm not sure.
> 
> Same as class and command just with different naming.

Is opcode the same as command? What does opmod do?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>> Admin virtqueues will be used to send administrative commands to
>> manipulate various features of the device which would not easily map
>> into the configuration space.
>>
>> The same Admin command format will be used for all virtio devices. The
>> Admin command set will include 4 types of command classes:
>> 1. The generic common class
>> 2. The transport specific class
>> 3. The device specific class
>> 4. The vendor specific class
>>
>> The above mechanism will enable adding various features to the virtio
>> specification, e.g.:
>> 1. Format virtio-blk devices in various configurations (512B block size,
>>     512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>> 2. Live migration management.
>> 3. Encrypt/Decrypt descriptors.
>> 4. Virtualization management.
>> 5. Get device error logs.
>> 6. Implement advanced vendor/device/transport specific features.
>> 7. Run device health test.
>> 8. More.
>>
>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>> for the specification to become flexible and allow a wider feature set.
>> The corrent ctrl virtq that is defined for some of the virtio devices is
>> device specific and wasn't designed to be a generic virtq for
>> admininistration.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Lots of things on this list seem to make sense when
> done from PF and affecting a VF.
> I think from this POV the generic structure should include
> a way to address one device from another.

This will be defined per command.

For example, funcion_id will be given as command data.


>
>> ---
>>   admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>   content.tex     |   4 +
>>   2 files changed, 245 insertions(+)
>>   create mode 100644 admin-virtq.tex
>>
>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>> new file mode 100644
>> index 0000000..ccec2ca
>> --- /dev/null
>> +++ b/admin-virtq.tex
>> @@ -0,0 +1,241 @@
>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>> +
>> +Admin virtqueues are used to send administrative commands to manipulate
>> +various features of the device which would not easily map into the
>> +configuration space.
>> +
>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>> +feature bit.
>> +
>> +Admin virtqueue index may vary among different device types.
>> +
>> +All commands are of the following form:
>> +
>> +\begin{lstlisting}
>> +struct virtio_admin_cmd {
>> +        /* Device-readable part */
>> +        u8 class;
>> +        u8 command;
>> +        u8 command-specific-data[];
>> +
>> +        /* Device-writable part */
>> +        u8 command-specific-result[];
>> +        u8 status_type : 4;
>> +        u8 reserved : 4;
>> +        u8 status;
>> +};
>> +
>> +/* Status type values */
>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>> +
>> +/* Generic status values */
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>> +\end{lstlisting}
>> +
>> +The \field{class}, \field{command} and \field{command-specific-data} are
>> +set by the driver, and the device sets the \field{status_type}, the
>> +\field{status} and  the \field{command-specific-result}, if needed.
>> +
>> +The virtio Admin command class codes are divided in the following form:
>> +
>> +\begin{lstlisting}
>> +/* class values that are transport, device and vendor independent */
>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>> +
>> +/* class values that are transport specific */
>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>> +
>> +/* class values that are device specific */
>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>> +
>> +/* class values that are vendor specific */
>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>> +\end{lstlisting}
>> +
>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>> +
>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>> +support all the mandatory admin commands. A device MAY support also
>> +one or more optional admin commands.
>> +
>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>> +
>> +The Common command set is a group of classes and commands within each
>> +of these classes which are transport, device and vendor independent.
>> +A mandatory class is a class that has at least one mandatory command.
>> +The Common command set is summarized in following table:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Class  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>> +\hline
>> +2-63  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>> +
>> +This class (opcode: 0) of commands is used to query generic device
>> +information. The following table describes the commands supported for
>> +this class:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Command  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>> +\hline
>> +2-255  & reserved    & - \\
>> +\hline
>> +\end{tabular}
> So there are several problems with this approach.
> One is that there is no two way negotiation.

you negotiate the adminq feature.

then you can send admin commands and discover the supported commands.


> No way for device to know what will driver use in the end.

device will be designed to support mandatory admin commands and some 
optional.

It doesn't need to care whether the driver will use it or not.

> This breaks things like e.g. accelerating some configurations
> but not others.

I don't understand what it breaks exactly.

>
> Another is that everything is going through the admin vq.
> Hard for hypervisor to intervene and change just some aspects
> of the device.

we'll develop virtio-cli tool.

I don't know what should be changed in the device by the hypervisor.


>
> This is also a problem for validation, tricky
> to find out during probe what does device support and whether
> driver can work with it.

device will support admin q with mandatory commands.

Old driver shouldn't care and it won't negotiate this feature.

New drivers will adjust and will be extended.


>
>
> Can't we map this stuff into memory instead?
> Maybe have an admin config structure
> that is separate from regular config?
> Only issue we can't then make it too big.
> But so far I don't see a lot of info used, most is reserved.

This is initial commit to add infrastructure. I can't add 255 commands 
at once.

We need to add it incrementally.

>
>
>
>> +
>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>> +
>> +This mandatory command should return device identity in the following
>> +structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bytes  & Description    & M/O \\
>> +\hline \hline
>> +03:00  & VIRTIO DEVICE ID    & M \\
>> +\hline
>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>> +\hline
>> +4095:06  & reserved    & - \\
>> + \hline
>> +\end{tabular}
>> +
>> +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
>> +
>> +This mandatory command should return device identity in the following
>> +structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bytes  & Description    & M/O \\
>> +\hline \hline
>> +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
>> +\hline
>> +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
>> + \hline
>> +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
>> +\hline
>> +...  & ...    & M \\
>> +\hline
>> +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
>> +\hline
>> +4095:1024  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +The above structure is divided to 256 sections of 4B each, that will
>> +describe whether a device supports a certain class code. The
>> +following 3072B are reserved. each of the non-reserved 4B sections will
>> +follow the following data structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bits  & Description    & M/O \\
>> +\hline \hline
>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>> +\hline
>> +31:01  & reserved    & - \\
>> + \hline
>> +\end{tabular}
>> +
>> +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
>> +
>> +This class (opcode: 1) of commands is used to query supported commands
>> +for each supported device class.
>> The following table describes the commands
>> +supported for this class:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Command  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +...  & ...    & M \\
>> +\hline
>> +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +\end{tabular}
>> +
>> +Each command in this class, will return class identity in the following structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bytes  & Description    & M/O \\
>> +\hline \hline
>> +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
>> +\hline
>> +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
>> + \hline
>> +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
>> +\hline
>> +...  & ...    & M \\
>> +\hline
>> +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
>> +\hline
>> +4095:1024  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +The above structure is divided to 256 sections of 4B each, that will
>> +describe whether a class supports a certain command code. The
>> +following 3072B are reserved. each of the non-reserved 4B sections will
>> +follow the following data structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bits  & Description    & M/O \\
>> +\hline \hline
>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>> +\hline
>> +31:01  & reserved    & - \\
>> + \hline
>> +\end{tabular}
>> +
>> +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
>> +
>> +The Transport command set is a group of classes and commands within
>> +each of these classes which are transport specific. Refer to each
>> +transport section for more information on the supported commands.
>> +
>> +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
>> +
>> +The Device command set is a group of classes and commands within
>> +each of these classes which are device specific. Refer to each
>> +device section for more information on the supported commands.
>> +
>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>> +
>> +The Vendor specific command set is a group of classes and commands
>> +within each of these classes which are vendor specific. Refer to
>> +each vendor specification for more information on the supported
>> +commands.
> Here's another example.
> It's important that there is a way to make a device completely
> generic without vendor specific expensions.
> Would be hard to do here.
>
> That's a generic comment.
>
> but specifically I am very reluctant to add vendor specific stuff like
> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
> and if that is not sufficient I would like to know why
> before we add more vendor specific stuff.

We are adding an option to add vendor specific commands. We're not 
adding it to the spec since each vendor will have its own manual for that.

For example, we can use virtio-cli to pass command from command line to 
device in pass-through manner without changing driver.


>
>
>> diff --git a/content.tex b/content.tex
>> index c266fd5..d350175 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>   types. It is RECOMMENDED that devices generate version 4
>>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>   
>> +\input{admin-virtq.tex}
>> +
>>   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>   
>>   We start with an overview of device initialization, then expand on the
>> @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>     data to be provided in driver notification and the delivery method is
>>     transport specific.
>>     For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
>> +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
>> +  the device supports administration virtqueue negotiation.
>>   
>>   \end{description}
>>   
>> -- 
>> 2.21.0

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Wed, Jul 28 2021, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
>> Vendor specific is also important to allow vendors develop their special
>> souse.
>
> Most of the value of virtio is that it's generic.
> So we added VIRTIO_PCI_CAP_VENDOR_CFG which is very easy for
> the hypervisor to intercept. Maybe that is not enough but that
> needs justification, I am not sure it's a good idea to add multiple
> ways devices can differ without a lot of thought.

Agreed, we should be careful about adding devices that differ in a lot
of little things.

My thought is that the default should be to concentrate on common
features that can be specified, just to keep things movable between
different implementations. If a device is really very different from the
generic one, it should probably become a different device type. But my
preference would be that everyone tries to make their device stick to
a common specification.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>>> +
>>> +The Vendor specific command set is a group of classes and commands
>>> +within each of these classes which are vendor specific. Refer to
>>> +each vendor specification for more information on the supported
>>> +commands.
>> Here's another example.
>> It's important that there is a way to make a device completely
>> generic without vendor specific expensions.
>> Would be hard to do here.
>>
>> That's a generic comment.
>>
>> but specifically I am very reluctant to add vendor specific stuff like
>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>> and if that is not sufficient I would like to know why
>> before we add more vendor specific stuff.
>
> We are adding an option to add vendor specific commands. We're not 
> adding it to the spec since each vendor will have its own manual for
> that.

IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a
vendor has some special feature they want to support, put it in the
spec, so that it is possible to actually implement it.

>
> For example, we can use virtio-cli to pass command from command line to 
> device in pass-through manner without changing driver.

Things like that are part of the driver as in the spec sense. The spec
does not care how you actually split the implementation, or what
controls you are giving to whom. We need a defined interface.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
> 
> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
> > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > Admin virtqueues will be used to send administrative commands to
> > > manipulate various features of the device which would not easily map
> > > into the configuration space.
> > > 
> > > The same Admin command format will be used for all virtio devices. The
> > > Admin command set will include 4 types of command classes:
> > > 1. The generic common class
> > > 2. The transport specific class
> > > 3. The device specific class
> > > 4. The vendor specific class
> > > 
> > > The above mechanism will enable adding various features to the virtio
> > > specification, e.g.:
> > > 1. Format virtio-blk devices in various configurations (512B block size,
> > >     512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> > > 2. Live migration management.
> > > 3. Encrypt/Decrypt descriptors.
> > > 4. Virtualization management.
> > > 5. Get device error logs.
> > > 6. Implement advanced vendor/device/transport specific features.
> > > 7. Run device health test.
> > > 8. More.
> > > 
> > > As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
> > > for the specification to become flexible and allow a wider feature set.
> > > The corrent ctrl virtq that is defined for some of the virtio devices is
> > > device specific and wasn't designed to be a generic virtq for
> > > admininistration.
> > > 
> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > Lots of things on this list seem to make sense when
> > done from PF and affecting a VF.
> > I think from this POV the generic structure should include
> > a way to address one device from another.
> 
> This will be defined per command.
> 
> For example, funcion_id will be given as command data.

Why? Sounds like a generic thing to me.


> 
> > 
> > > ---
> > >   admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
> > >   content.tex     |   4 +
> > >   2 files changed, 245 insertions(+)
> > >   create mode 100644 admin-virtq.tex
> > > 
> > > diff --git a/admin-virtq.tex b/admin-virtq.tex
> > > new file mode 100644
> > > index 0000000..ccec2ca
> > > --- /dev/null
> > > +++ b/admin-virtq.tex
> > > @@ -0,0 +1,241 @@
> > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > +
> > > +Admin virtqueues are used to send administrative commands to manipulate
> > > +various features of the device which would not easily map into the
> > > +configuration space.
> > > +
> > > +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
> > > +feature bit.
> > > +
> > > +Admin virtqueue index may vary among different device types.
> > > +
> > > +All commands are of the following form:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd {
> > > +        /* Device-readable part */
> > > +        u8 class;
> > > +        u8 command;
> > > +        u8 command-specific-data[];
> > > +
> > > +        /* Device-writable part */
> > > +        u8 command-specific-result[];
> > > +        u8 status_type : 4;
> > > +        u8 reserved : 4;
> > > +        u8 status;
> > > +};
> > > +
> > > +/* Status type values */
> > > +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
> > > +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
> > > +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
> > > +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
> > > +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
> > > +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
> > > +
> > > +/* Generic status values */
> > > +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
> > > +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
> > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
> > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
> > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
> > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
> > > +\end{lstlisting}
> > > +
> > > +The \field{class}, \field{command} and \field{command-specific-data} are
> > > +set by the driver, and the device sets the \field{status_type}, the
> > > +\field{status} and  the \field{command-specific-result}, if needed.
> > > +
> > > +The virtio Admin command class codes are divided in the following form:
> > > +
> > > +\begin{lstlisting}
> > > +/* class values that are transport, device and vendor independent */
> > > +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
> > > +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
> > > +
> > > +/* class values that are transport specific */
> > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
> > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
> > > +
> > > +/* class values that are device specific */
> > > +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
> > > +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
> > > +
> > > +/* class values that are vendor specific */
> > > +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
> > > +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
> > > +\end{lstlisting}
> > > +
> > > +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
> > > +
> > > +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
> > > +support all the mandatory admin commands. A device MAY support also
> > > +one or more optional admin commands.
> > > +
> > > +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
> > > +
> > > +The Common command set is a group of classes and commands within each
> > > +of these classes which are transport, device and vendor independent.
> > > +A mandatory class is a class that has at least one mandatory command.
> > > +The Common command set is summarized in following table:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Class  & Description    & M/O \\
> > > +\hline \hline
> > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
> > > +\hline
> > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
> > > +\hline
> > > +2-63  & reserved    & - \\
> > > +\hline
> > > +\end{tabular}
> > > +
> > > +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
> > > +
> > > +This class (opcode: 0) of commands is used to query generic device
> > > +information. The following table describes the commands supported for
> > > +this class:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Command  & Description    & M/O \\
> > > +\hline \hline
> > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
> > > +\hline
> > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
> > > +\hline
> > > +2-255  & reserved    & - \\
> > > +\hline
> > > +\end{tabular}
> > So there are several problems with this approach.
> > One is that there is no two way negotiation.
> 
> you negotiate the adminq feature.
> 
> then you can send admin commands and discover the supported commands.
> 
> > No way for device to know what will driver use in the end.
> 
> device will be designed to support mandatory admin commands and some
> optional.
> 
> It doesn't need to care whether the driver will use it or not.

> > This breaks things like e.g. accelerating some configurations
> > but not others.
> 
> I don't understand what it breaks exactly.

Long practice taught us that it is good for device to know
what is driver going to use.
For example, some features can be implemented in hardware
and some in hypervisor software. If driver is going to use software
features then you need to switch to a slower software
implementation. Doing that dynamically at time of use is
often much harder that up-front at negotiation time.

> 
> > 
> > Another is that everything is going through the admin vq.
> > Hard for hypervisor to intervene and change just some aspects
> > of the device.
> 
> we'll develop virtio-cli tool.

will that be part of the spec somehow then?
I'm not sure it can be ...

> I don't know what should be changed in the device by the hypervisor.

Look at vdpa as an example, guest device can be different from
host device.


> 
> > 
> > This is also a problem for validation, tricky
> > to find out during probe what does device support and whether
> > driver can work with it.
> 
> device will support admin q with mandatory commands.
> 
> Old driver shouldn't care and it won't negotiate this feature.
> 
> New drivers will adjust and will be extended.

Does not sound like I got the issue across.

Operating VQs just to know whether to bind to device is
problematic since there's an assumption that once
DRIVER_OK is set driver is happy. Before DRIVER_OK
device will not consume buffers.



> 
> > 
> > 
> > Can't we map this stuff into memory instead?
> > Maybe have an admin config structure
> > that is separate from regular config?
> > Only issue we can't then make it too big.
> > But so far I don't see a lot of info used, most is reserved.
> 
> This is initial commit to add infrastructure. I can't add 255 commands at
> once.
> 
> We need to add it incrementally.

How about a list of future extensions?


> > 
> > 
> > 
> > > +
> > > +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
> > > +
> > > +This mandatory command should return device identity in the following
> > > +structure:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Bytes  & Description    & M/O \\
> > > +\hline \hline
> > > +03:00  & VIRTIO DEVICE ID    & M \\
> > > +\hline
> > > +05:04  & VIRTIO TRANSPORT ID    & M \\
> > > +\hline
> > > +4095:06  & reserved    & - \\
> > > + \hline
> > > +\end{tabular}
> > > +
> > > +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
> > > +
> > > +This mandatory command should return device identity in the following
> > > +structure:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Bytes  & Description    & M/O \\
> > > +\hline \hline
> > > +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
> > > +\hline
> > > +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
> > > + \hline
> > > +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
> > > +\hline
> > > +...  & ...    & M \\
> > > +\hline
> > > +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
> > > +\hline
> > > +4095:1024  & reserved    & - \\
> > > +\hline
> > > +\end{tabular}
> > > +
> > > +The above structure is divided to 256 sections of 4B each, that will
> > > +describe whether a device supports a certain class code. The
> > > +following 3072B are reserved. each of the non-reserved 4B sections will
> > > +follow the following data structure:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Bits  & Description    & M/O \\
> > > +\hline \hline
> > > +0  & SUPPORTED (1: yes, 0:no)    & M \\
> > > +\hline
> > > +31:01  & reserved    & - \\
> > > + \hline
> > > +\end{tabular}
> > > +
> > > +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
> > > +
> > > +This class (opcode: 1) of commands is used to query supported commands
> > > +for each supported device class.
> > > The following table describes the commands
> > > +supported for this class:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Command  & Description    & M/O \\
> > > +\hline \hline
> > > +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
> > > +\hline
> > > +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
> > > +\hline
> > > +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
> > > +\hline
> > > +...  & ...    & M \\
> > > +\hline
> > > +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
> > > +\hline
> > > +\end{tabular}
> > > +
> > > +Each command in this class, will return class identity in the following structure:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Bytes  & Description    & M/O \\
> > > +\hline \hline
> > > +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
> > > +\hline
> > > +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
> > > + \hline
> > > +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
> > > +\hline
> > > +...  & ...    & M \\
> > > +\hline
> > > +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
> > > +\hline
> > > +4095:1024  & reserved    & - \\
> > > +\hline
> > > +\end{tabular}
> > > +
> > > +The above structure is divided to 256 sections of 4B each, that will
> > > +describe whether a class supports a certain command code. The
> > > +following 3072B are reserved. each of the non-reserved 4B sections will
> > > +follow the following data structure:
> > > +
> > > +\begin{tabular}{|l|l|l|}
> > > +\hline
> > > +Bits  & Description    & M/O \\
> > > +\hline \hline
> > > +0  & SUPPORTED (1: yes, 0:no)    & M \\
> > > +\hline
> > > +31:01  & reserved    & - \\
> > > + \hline
> > > +\end{tabular}
> > > +
> > > +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
> > > +
> > > +The Transport command set is a group of classes and commands within
> > > +each of these classes which are transport specific. Refer to each
> > > +transport section for more information on the supported commands.
> > > +
> > > +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
> > > +
> > > +The Device command set is a group of classes and commands within
> > > +each of these classes which are device specific. Refer to each
> > > +device section for more information on the supported commands.
> > > +
> > > +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
> > > +
> > > +The Vendor specific command set is a group of classes and commands
> > > +within each of these classes which are vendor specific. Refer to
> > > +each vendor specification for more information on the supported
> > > +commands.
> > Here's another example.
> > It's important that there is a way to make a device completely
> > generic without vendor specific expensions.
> > Would be hard to do here.
> > 
> > That's a generic comment.
> > 
> > but specifically I am very reluctant to add vendor specific stuff like
> > this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
> > and if that is not sufficient I would like to know why
> > before we add more vendor specific stuff.
> 
> We are adding an option to add vendor specific commands. We're not adding it
> to the spec since each vendor will have its own manual for that.

You didn't actually answer the question though.

> For example, we can use virtio-cli to pass command from command line to
> device in pass-through manner without changing driver.
> 

Opaque strings passed all the way from guest userspace to host device?
Not sure why is that a good idea.

> > 
> > 
> > > diff --git a/content.tex b/content.tex
> > > index c266fd5..d350175 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> > >   types. It is RECOMMENDED that devices generate version 4
> > >   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > > +\input{admin-virtq.tex}
> > > +
> > >   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> > >   We start with an overview of device initialization, then expand on the
> > > @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > >     data to be provided in driver notification and the delivery method is
> > >     transport specific.
> > >     For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
> > > +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
> > > +  the device supports administration virtqueue negotiation.
> > >   \end{description}
> > > -- 
> > > 2.21.0

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 7/30/2021 10:05 AM, Cornelia Huck wrote:
> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>
>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>>>> +
>>>> +The Vendor specific command set is a group of classes and commands
>>>> +within each of these classes which are vendor specific. Refer to
>>>> +each vendor specification for more information on the supported
>>>> +commands.
>>> Here's another example.
>>> It's important that there is a way to make a device completely
>>> generic without vendor specific expensions.
>>> Would be hard to do here.
>>>
>>> That's a generic comment.
>>>
>>> but specifically I am very reluctant to add vendor specific stuff like
>>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>>> and if that is not sufficient I would like to know why
>>> before we add more vendor specific stuff.
>> We are adding an option to add vendor specific commands. We're not
>> adding it to the spec since each vendor will have its own manual for
>> that.
> IMHO, that way madness lies. I want to be able to look at the spec and
> be able to implement a compliant device or a compliant driver. If a
> vendor has some special feature they want to support, put it in the
> spec, so that it is possible to actually implement it.

You can implement a compliant device and a compliant. why do you think 
you can't ?

Some features are vendor/sub-vendor specific.

And as mentioned, you already added it to  the spec so I guess it was 
for a reason.

If someone would like to add its special souse to virtio device and we 
have a flexible admin queue to manage it with a spec compliant manner, 
why not ?

The feature might not get support by the working group so do you really 
want to limit vendor from implementing features that are not agreed here 
on the mailing list ?

Adding a vendor specific command set is trivial.

>
>> For example, we can use virtio-cli to pass command from command line to
>> device in pass-through manner without changing driver.
> Things like that are part of the driver as in the spec sense. The spec
> does not care how you actually split the implementation, or what
> controls you are giving to whom. We need a defined interface.

We have an interface. Its the admin queue.

Virtio-cli will be a user space tool to send commands via this admin 
queue and configure the device.

The driver will just open a char dev to supply a channel to this admin 
queue from user space.

Examples:

1. format a virtio-blk device to add data integrity checks (T-10)

2. set 5 msix to VF_1, 12 msix to VF_2 and 2 msix to VF_3 before 
enabling SRIOV

3. Get telemetry log

4. Get error log

5. Get vendor specific HW health report

6. Get generic virtio device health report

7. set 5 queues to VF_1, 12 queues to VF_2 and 2 queues to VF_3 before 
enabling SRIOV


This knowledge is needed for the sys-admin and not the driver. The 
driver will provide the channel to enable this configuration by an 
orchestrator/admin.



>

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
> On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>> Admin virtqueues will be used to send administrative commands to
>>>> manipulate various features of the device which would not easily map
>>>> into the configuration space.
>>>>
>>>> The same Admin command format will be used for all virtio devices. The
>>>> Admin command set will include 4 types of command classes:
>>>> 1. The generic common class
>>>> 2. The transport specific class
>>>> 3. The device specific class
>>>> 4. The vendor specific class
>>>>
>>>> The above mechanism will enable adding various features to the virtio
>>>> specification, e.g.:
>>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>>      512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>>> 2. Live migration management.
>>>> 3. Encrypt/Decrypt descriptors.
>>>> 4. Virtualization management.
>>>> 5. Get device error logs.
>>>> 6. Implement advanced vendor/device/transport specific features.
>>>> 7. Run device health test.
>>>> 8. More.
>>>>
>>>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>>>> for the specification to become flexible and allow a wider feature set.
>>>> The corrent ctrl virtq that is defined for some of the virtio devices is
>>>> device specific and wasn't designed to be a generic virtq for
>>>> admininistration.
>>>>
>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>> Lots of things on this list seem to make sense when
>>> done from PF and affecting a VF.
>>> I think from this POV the generic structure should include
>>> a way to address one device from another.
>> This will be defined per command.
>>
>> For example, funcion_id will be given as command data.
> Why? Sounds like a generic thing to me.

Generic to a command that handles virtualization management.

It has nothing to do with a discovery command for the device itself.

I guess we can add few bytes to be mandatory after the <u8 class, u8 
command> pair.

Something like <u8 class, u8 command, u16 mandatory_command_data >.

But I think it's not a must and it's already covered by:  "u8 
command-specific-data[];"


>
>
>>>> ---
>>>>    admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>    content.tex     |   4 +
>>>>    2 files changed, 245 insertions(+)
>>>>    create mode 100644 admin-virtq.tex
>>>>
>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>> new file mode 100644
>>>> index 0000000..ccec2ca
>>>> --- /dev/null
>>>> +++ b/admin-virtq.tex
>>>> @@ -0,0 +1,241 @@
>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>> +
>>>> +Admin virtqueues are used to send administrative commands to manipulate
>>>> +various features of the device which would not easily map into the
>>>> +configuration space.
>>>> +
>>>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>>>> +feature bit.
>>>> +
>>>> +Admin virtqueue index may vary among different device types.
>>>> +
>>>> +All commands are of the following form:
>>>> +
>>>> +\begin{lstlisting}
>>>> +struct virtio_admin_cmd {
>>>> +        /* Device-readable part */
>>>> +        u8 class;
>>>> +        u8 command;
>>>> +        u8 command-specific-data[];
>>>> +
>>>> +        /* Device-writable part */
>>>> +        u8 command-specific-result[];
>>>> +        u8 status_type : 4;
>>>> +        u8 reserved : 4;
>>>> +        u8 status;
>>>> +};
>>>> +
>>>> +/* Status type values */
>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>>>> +
>>>> +/* Generic status values */
>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>>>> +\end{lstlisting}
>>>> +
>>>> +The \field{class}, \field{command} and \field{command-specific-data} are
>>>> +set by the driver, and the device sets the \field{status_type}, the
>>>> +\field{status} and  the \field{command-specific-result}, if needed.
>>>> +
>>>> +The virtio Admin command class codes are divided in the following form:
>>>> +
>>>> +\begin{lstlisting}
>>>> +/* class values that are transport, device and vendor independent */
>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>>>> +
>>>> +/* class values that are transport specific */
>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>>>> +
>>>> +/* class values that are device specific */
>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>>>> +
>>>> +/* class values that are vendor specific */
>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>>>> +\end{lstlisting}
>>>> +
>>>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>>>> +
>>>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>>>> +support all the mandatory admin commands. A device MAY support also
>>>> +one or more optional admin commands.
>>>> +
>>>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>>>> +
>>>> +The Common command set is a group of classes and commands within each
>>>> +of these classes which are transport, device and vendor independent.
>>>> +A mandatory class is a class that has at least one mandatory command.
>>>> +The Common command set is summarized in following table:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Class  & Description    & M/O \\
>>>> +\hline \hline
>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>>>> +\hline
>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>>>> +\hline
>>>> +2-63  & reserved    & - \\
>>>> +\hline
>>>> +\end{tabular}
>>>> +
>>>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>>>> +
>>>> +This class (opcode: 0) of commands is used to query generic device
>>>> +information. The following table describes the commands supported for
>>>> +this class:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Command  & Description    & M/O \\
>>>> +\hline \hline
>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>>>> +\hline
>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>>>> +\hline
>>>> +2-255  & reserved    & - \\
>>>> +\hline
>>>> +\end{tabular}
>>> So there are several problems with this approach.
>>> One is that there is no two way negotiation.
>> you negotiate the adminq feature.
>>
>> then you can send admin commands and discover the supported commands.
>>
>>> No way for device to know what will driver use in the end.
>> device will be designed to support mandatory admin commands and some
>> optional.
>>
>> It doesn't need to care whether the driver will use it or not.
>>> This breaks things like e.g. accelerating some configurations
>>> but not others.
>> I don't understand what it breaks exactly.
> Long practice taught us that it is good for device to know
> what is driver going to use.
> For example, some features can be implemented in hardware
> and some in hypervisor software. If driver is going to use software
> features then you need to switch to a slower software
> implementation. Doing that dynamically at time of use is
> often much harder that up-front at negotiation time.

I don't think we should write specifications that should consider 
hypervisor SW.

You might use virtio device without hypervisor at all.


>
>>> Another is that everything is going through the admin vq.
>>> Hard for hypervisor to intervene and change just some aspects
>>> of the device.
>> we'll develop virtio-cli tool.
> will that be part of the spec somehow then?
> I'm not sure it can be ...

of course not.

This is just a channel for the admin to send admin command to the virtio 
device.

>
>> I don't know what should be changed in the device by the hypervisor.
> Look at vdpa as an example, guest device can be different from
> host device.

VDPA is an acceleration. I can write different acceleration on to of 
virtio device.

Is shouldn't be considered in the specification.

Nevertheless, there is nothing in this RFC that VDPA can't handle.


>
>>> This is also a problem for validation, tricky
>>> to find out during probe what does device support and whether
>>> driver can work with it.
>> device will support admin q with mandatory commands.
>>
>> Old driver shouldn't care and it won't negotiate this feature.
>>
>> New drivers will adjust and will be extended.
> Does not sound like I got the issue across.
>
> Operating VQs just to know whether to bind to device is
> problematic since there's an assumption that once
> DRIVER_OK is set driver is happy. Before DRIVER_OK
> device will not consume buffers.

Nothing is changed in DRIVER_OK negotiation rules.

>
>
>>>
>>> Can't we map this stuff into memory instead?
>>> Maybe have an admin config structure
>>> that is separate from regular config?
>>> Only issue we can't then make it too big.
>>> But so far I don't see a lot of info used, most is reserved.
>> This is initial commit to add infrastructure. I can't add 255 commands at
>> once.
>>
>> We need to add it incrementally.
> How about a list of future extensions?

Added a partial list in the commit message and in the answer to Cornelia.


>
>
>>>
>>>
>>>> +
>>>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>>>> +
>>>> +This mandatory command should return device identity in the following
>>>> +structure:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Bytes  & Description    & M/O \\
>>>> +\hline \hline
>>>> +03:00  & VIRTIO DEVICE ID    & M \\
>>>> +\hline
>>>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>>>> +\hline
>>>> +4095:06  & reserved    & - \\
>>>> + \hline
>>>> +\end{tabular}
>>>> +
>>>> +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
>>>> +
>>>> +This mandatory command should return device identity in the following
>>>> +structure:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Bytes  & Description    & M/O \\
>>>> +\hline \hline
>>>> +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
>>>> +\hline
>>>> +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
>>>> + \hline
>>>> +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
>>>> +\hline
>>>> +...  & ...    & M \\
>>>> +\hline
>>>> +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
>>>> +\hline
>>>> +4095:1024  & reserved    & - \\
>>>> +\hline
>>>> +\end{tabular}
>>>> +
>>>> +The above structure is divided to 256 sections of 4B each, that will
>>>> +describe whether a device supports a certain class code. The
>>>> +following 3072B are reserved. each of the non-reserved 4B sections will
>>>> +follow the following data structure:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Bits  & Description    & M/O \\
>>>> +\hline \hline
>>>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>>>> +\hline
>>>> +31:01  & reserved    & - \\
>>>> + \hline
>>>> +\end{tabular}
>>>> +
>>>> +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
>>>> +
>>>> +This class (opcode: 1) of commands is used to query supported commands
>>>> +for each supported device class.
>>>> The following table describes the commands
>>>> +supported for this class:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Command  & Description    & M/O \\
>>>> +\hline \hline
>>>> +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
>>>> +\hline
>>>> +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
>>>> +\hline
>>>> +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
>>>> +\hline
>>>> +...  & ...    & M \\
>>>> +\hline
>>>> +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
>>>> +\hline
>>>> +\end{tabular}
>>>> +
>>>> +Each command in this class, will return class identity in the following structure:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Bytes  & Description    & M/O \\
>>>> +\hline \hline
>>>> +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
>>>> +\hline
>>>> +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
>>>> + \hline
>>>> +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
>>>> +\hline
>>>> +...  & ...    & M \\
>>>> +\hline
>>>> +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
>>>> +\hline
>>>> +4095:1024  & reserved    & - \\
>>>> +\hline
>>>> +\end{tabular}
>>>> +
>>>> +The above structure is divided to 256 sections of 4B each, that will
>>>> +describe whether a class supports a certain command code. The
>>>> +following 3072B are reserved. each of the non-reserved 4B sections will
>>>> +follow the following data structure:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Bits  & Description    & M/O \\
>>>> +\hline \hline
>>>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>>>> +\hline
>>>> +31:01  & reserved    & - \\
>>>> + \hline
>>>> +\end{tabular}
>>>> +
>>>> +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
>>>> +
>>>> +The Transport command set is a group of classes and commands within
>>>> +each of these classes which are transport specific. Refer to each
>>>> +transport section for more information on the supported commands.
>>>> +
>>>> +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
>>>> +
>>>> +The Device command set is a group of classes and commands within
>>>> +each of these classes which are device specific. Refer to each
>>>> +device section for more information on the supported commands.
>>>> +
>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>>>> +
>>>> +The Vendor specific command set is a group of classes and commands
>>>> +within each of these classes which are vendor specific. Refer to
>>>> +each vendor specification for more information on the supported
>>>> +commands.
>>> Here's another example.
>>> It's important that there is a way to make a device completely
>>> generic without vendor specific expensions.
>>> Would be hard to do here.
>>>
>>> That's a generic comment.
>>>
>>> but specifically I am very reluctant to add vendor specific stuff like
>>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>>> and if that is not sufficient I would like to know why
>>> before we add more vendor specific stuff.
>> We are adding an option to add vendor specific commands. We're not adding it
>> to the spec since each vendor will have its own manual for that.
> You didn't actually answer the question though.
>
>> For example, we can use virtio-cli to pass command from command line to
>> device in pass-through manner without changing driver.
>>
> Opaque strings passed all the way from guest userspace to host device?
> Not sure why is that a good idea.

Where did I mentioned a guest ?

Virtio-cli will control a device on the same host that it runs.

If it's a bare metal host so it will manage the virtio attached device.

If it's a guest it will manage the device attached to the guest.

>
>>>
>>>> diff --git a/content.tex b/content.tex
>>>> index c266fd5..d350175 100644
>>>> --- a/content.tex
>>>> +++ b/content.tex
>>>> @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>>>    types. It is RECOMMENDED that devices generate version 4
>>>>    UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>> +\input{admin-virtq.tex}
>>>> +
>>>>    \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>>>    We start with an overview of device initialization, then expand on the
>>>> @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>>>      data to be provided in driver notification and the delivery method is
>>>>      transport specific.
>>>>      For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
>>>> +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
>>>> +  the device supports administration virtqueue negotiation.
>>>>    \end{description}
>>>> -- 
>>>> 2.21.0

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
> 
> On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
> > On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
> > > On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
> > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > > Admin virtqueues will be used to send administrative commands to
> > > > > manipulate various features of the device which would not easily map
> > > > > into the configuration space.
> > > > > 
> > > > > The same Admin command format will be used for all virtio devices. The
> > > > > Admin command set will include 4 types of command classes:
> > > > > 1. The generic common class
> > > > > 2. The transport specific class
> > > > > 3. The device specific class
> > > > > 4. The vendor specific class
> > > > > 
> > > > > The above mechanism will enable adding various features to the virtio
> > > > > specification, e.g.:
> > > > > 1. Format virtio-blk devices in various configurations (512B block size,
> > > > >      512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> > > > > 2. Live migration management.
> > > > > 3. Encrypt/Decrypt descriptors.
> > > > > 4. Virtualization management.
> > > > > 5. Get device error logs.
> > > > > 6. Implement advanced vendor/device/transport specific features.
> > > > > 7. Run device health test.
> > > > > 8. More.

I still don't really see what do all these things have in common?
Why are they grouped behind the same feature bit? Share same VQ?

> > > > > 
> > > > > As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
> > > > > for the specification to become flexible and allow a wider feature set.
> > > > > The corrent ctrl virtq that is defined for some of the virtio devices is
> > > > > device specific and wasn't designed to be a generic virtq for
> > > > > admininistration.
> > > > > 
> > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > Lots of things on this list seem to make sense when
> > > > done from PF and affecting a VF.
> > > > I think from this POV the generic structure should include
> > > > a way to address one device from another.
> > > This will be defined per command.
> > > 
> > > For example, funcion_id will be given as command data.
> > Why? Sounds like a generic thing to me.
> 
> Generic to a command that handles virtualization management.


It could be that mixing up virtualization management and arbitrary
other management commands in the same interface is a mistake.
Or maybe not - do we want host to have ability to run health tests
for a VF without loading VF driver? Get error logs?


In fact besides migration and virtualization management the rest of
examples that you give all seem to be more or less device specific, with
the possible exception of 3. Encrypt/Decrypt descriptors.  what does
this one imply, exactly?



> It has nothing to do with a discovery command for the device itself.
> 
> I guess we can add few bytes to be mandatory after the <u8 class, u8
> command> pair.
> 
> Something like <u8 class, u8 command, u16 mandatory_command_data >.
> 
> But I think it's not a must and it's already covered by:  "u8
> command-specific-data[];"

I would say we should include the device ID as well as a definition
what the ID is at least for SRIOV. This isn't a command
specific data, any virt management command will need this.


> 
> > 
> > 
> > > > > ---
> > > > >    admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >    content.tex     |   4 +
> > > > >    2 files changed, 245 insertions(+)
> > > > >    create mode 100644 admin-virtq.tex
> > > > > 
> > > > > diff --git a/admin-virtq.tex b/admin-virtq.tex
> > > > > new file mode 100644
> > > > > index 0000000..ccec2ca
> > > > > --- /dev/null
> > > > > +++ b/admin-virtq.tex
> > > > > @@ -0,0 +1,241 @@
> > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > > +
> > > > > +Admin virtqueues are used to send administrative commands to manipulate
> > > > > +various features of the device which would not easily map into the
> > > > > +configuration space.
> > > > > +
> > > > > +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > > +feature bit.
> > > > > +
> > > > > +Admin virtqueue index may vary among different device types.
> > > > > +
> > > > > +All commands are of the following form:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_admin_cmd {
> > > > > +        /* Device-readable part */
> > > > > +        u8 class;
> > > > > +        u8 command;
> > > > > +        u8 command-specific-data[];
> > > > > +
> > > > > +        /* Device-writable part */
> > > > > +        u8 command-specific-result[];
> > > > > +        u8 status_type : 4;
> > > > > +        u8 reserved : 4;
> > > > > +        u8 status;
> > > > > +};
> > > > > +
> > > > > +/* Status type values */
> > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
> > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
> > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
> > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
> > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
> > > > > +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
> > > > > +
> > > > > +/* Generic status values */
> > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
> > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
> > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
> > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
> > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
> > > > > +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +The \field{class}, \field{command} and \field{command-specific-data} are
> > > > > +set by the driver, and the device sets the \field{status_type}, the
> > > > > +\field{status} and  the \field{command-specific-result}, if needed.
> > > > > +
> > > > > +The virtio Admin command class codes are divided in the following form:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +/* class values that are transport, device and vendor independent */
> > > > > +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
> > > > > +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
> > > > > +
> > > > > +/* class values that are transport specific */
> > > > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
> > > > > +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
> > > > > +
> > > > > +/* class values that are device specific */
> > > > > +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
> > > > > +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
> > > > > +
> > > > > +/* class values that are vendor specific */
> > > > > +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
> > > > > +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
> > > > > +
> > > > > +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
> > > > > +support all the mandatory admin commands. A device MAY support also
> > > > > +one or more optional admin commands.
> > > > > +
> > > > > +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
> > > > > +
> > > > > +The Common command set is a group of classes and commands within each
> > > > > +of these classes which are transport, device and vendor independent.
> > > > > +A mandatory class is a class that has at least one mandatory command.
> > > > > +The Common command set is summarized in following table:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Class  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
> > > > > +\hline
> > > > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
> > > > > +\hline
> > > > > +2-63  & reserved    & - \\
> > > > > +\hline
> > > > > +\end{tabular}
> > > > > +
> > > > > +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
> > > > > +
> > > > > +This class (opcode: 0) of commands is used to query generic device
> > > > > +information. The following table describes the commands supported for
> > > > > +this class:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Command  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
> > > > > +\hline
> > > > > +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
> > > > > +\hline
> > > > > +2-255  & reserved    & - \\
> > > > > +\hline
> > > > > +\end{tabular}
> > > > So there are several problems with this approach.
> > > > One is that there is no two way negotiation.
> > > you negotiate the adminq feature.
> > > 
> > > then you can send admin commands and discover the supported commands.
> > > 
> > > > No way for device to know what will driver use in the end.
> > > device will be designed to support mandatory admin commands and some
> > > optional.
> > > 
> > > It doesn't need to care whether the driver will use it or not.
> > > > This breaks things like e.g. accelerating some configurations
> > > > but not others.
> > > I don't understand what it breaks exactly.
> > Long practice taught us that it is good for device to know
> > what is driver going to use.
> > For example, some features can be implemented in hardware
> > and some in hypervisor software. If driver is going to use software
> > features then you need to switch to a slower software
> > implementation. Doing that dynamically at time of use is
> > often much harder that up-front at negotiation time.
> 
> I don't think we should write specifications that should consider hypervisor
> SW.

Well considering hypervisors is clearly one of the purposes of virtio TC.
Look it up in the charter, section 2. Statement of Purpose.


> You might use virtio device without hypervisor at all.

Yes and supporting that is also clearly an objective:

	With the 1.1, 1.2 and future revisions of the Specification, we aim to
	evolve the VIRTIO standard further, addressing such new requirements
	while both continuing to honor the goals of the 1.0 Specification and
	including new objectives.




> 
> > 
> > > > Another is that everything is going through the admin vq.
> > > > Hard for hypervisor to intervene and change just some aspects
> > > > of the device.
> > > we'll develop virtio-cli tool.
> > will that be part of the spec somehow then?
> > I'm not sure it can be ...
> 
> of course not.
> 
> This is just a channel for the admin to send admin command to the virtio
> device.
> 
> > 
> > > I don't know what should be changed in the device by the hypervisor.
> > Look at vdpa as an example, guest device can be different from
> > host device.
> 
> VDPA is an acceleration. I can write different acceleration on to of virtio
> device.
> 
> Is shouldn't be considered in the specification.
> 
> Nevertheless, there is nothing in this RFC that VDPA can't handle.
> 
> 
> > 
> > > > This is also a problem for validation, tricky
> > > > to find out during probe what does device support and whether
> > > > driver can work with it.
> > > device will support admin q with mandatory commands.
> > > 
> > > Old driver shouldn't care and it won't negotiate this feature.
> > > 
> > > New drivers will adjust and will be extended.
> > Does not sound like I got the issue across.
> > 
> > Operating VQs just to know whether to bind to device is
> > problematic since there's an assumption that once
> > DRIVER_OK is set driver is happy. Before DRIVER_OK
> > device will not consume buffers.
> 
> Nothing is changed in DRIVER_OK negotiation rules.

Right and that means you can't really use the admin vq to validate
device during driver probe.


> > 
> > 
> > > > 
> > > > Can't we map this stuff into memory instead?
> > > > Maybe have an admin config structure
> > > > that is separate from regular config?
> > > > Only issue we can't then make it too big.
> > > > But so far I don't see a lot of info used, most is reserved.
> > > This is initial commit to add infrastructure. I can't add 255 commands at
> > > once.
> > > 
> > > We need to add it incrementally.
> > How about a list of future extensions?
> 
> Added a partial list in the commit message and in the answer to Cornelia.
> 
> 
> > 
> > 
> > > > 
> > > > 
> > > > > +
> > > > > +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
> > > > > +
> > > > > +This mandatory command should return device identity in the following
> > > > > +structure:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Bytes  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +03:00  & VIRTIO DEVICE ID    & M \\
> > > > > +\hline
> > > > > +05:04  & VIRTIO TRANSPORT ID    & M \\
> > > > > +\hline
> > > > > +4095:06  & reserved    & - \\
> > > > > + \hline
> > > > > +\end{tabular}
> > > > > +
> > > > > +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
> > > > > +
> > > > > +This mandatory command should return device identity in the following
> > > > > +structure:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Bytes  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
> > > > > +\hline
> > > > > +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
> > > > > + \hline
> > > > > +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
> > > > > +\hline
> > > > > +...  & ...    & M \\
> > > > > +\hline
> > > > > +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
> > > > > +\hline
> > > > > +4095:1024  & reserved    & - \\
> > > > > +\hline
> > > > > +\end{tabular}
> > > > > +
> > > > > +The above structure is divided to 256 sections of 4B each, that will
> > > > > +describe whether a device supports a certain class code. The
> > > > > +following 3072B are reserved. each of the non-reserved 4B sections will
> > > > > +follow the following data structure:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Bits  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +0  & SUPPORTED (1: yes, 0:no)    & M \\
> > > > > +\hline
> > > > > +31:01  & reserved    & - \\
> > > > > + \hline
> > > > > +\end{tabular}
> > > > > +
> > > > > +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
> > > > > +
> > > > > +This class (opcode: 1) of commands is used to query supported commands
> > > > > +for each supported device class.
> > > > > The following table describes the commands
> > > > > +supported for this class:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Command  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
> > > > > +\hline
> > > > > +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
> > > > > +\hline
> > > > > +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
> > > > > +\hline
> > > > > +...  & ...    & M \\
> > > > > +\hline
> > > > > +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
> > > > > +\hline
> > > > > +\end{tabular}
> > > > > +
> > > > > +Each command in this class, will return class identity in the following structure:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Bytes  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
> > > > > +\hline
> > > > > +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
> > > > > + \hline
> > > > > +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
> > > > > +\hline
> > > > > +...  & ...    & M \\
> > > > > +\hline
> > > > > +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
> > > > > +\hline
> > > > > +4095:1024  & reserved    & - \\
> > > > > +\hline
> > > > > +\end{tabular}
> > > > > +
> > > > > +The above structure is divided to 256 sections of 4B each, that will
> > > > > +describe whether a class supports a certain command code. The
> > > > > +following 3072B are reserved. each of the non-reserved 4B sections will
> > > > > +follow the following data structure:
> > > > > +
> > > > > +\begin{tabular}{|l|l|l|}
> > > > > +\hline
> > > > > +Bits  & Description    & M/O \\
> > > > > +\hline \hline
> > > > > +0  & SUPPORTED (1: yes, 0:no)    & M \\
> > > > > +\hline
> > > > > +31:01  & reserved    & - \\
> > > > > + \hline
> > > > > +\end{tabular}
> > > > > +
> > > > > +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
> > > > > +
> > > > > +The Transport command set is a group of classes and commands within
> > > > > +each of these classes which are transport specific. Refer to each
> > > > > +transport section for more information on the supported commands.
> > > > > +
> > > > > +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
> > > > > +
> > > > > +The Device command set is a group of classes and commands within
> > > > > +each of these classes which are device specific. Refer to each
> > > > > +device section for more information on the supported commands.
> > > > > +
> > > > > +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
> > > > > +
> > > > > +The Vendor specific command set is a group of classes and commands
> > > > > +within each of these classes which are vendor specific. Refer to
> > > > > +each vendor specification for more information on the supported
> > > > > +commands.
> > > > Here's another example.
> > > > It's important that there is a way to make a device completely
> > > > generic without vendor specific expensions.
> > > > Would be hard to do here.
> > > > 
> > > > That's a generic comment.
> > > > 
> > > > but specifically I am very reluctant to add vendor specific stuff like
> > > > this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
> > > > and if that is not sufficient I would like to know why
> > > > before we add more vendor specific stuff.
> > > We are adding an option to add vendor specific commands. We're not adding it
> > > to the spec since each vendor will have its own manual for that.
> > You didn't actually answer the question though.
> > 
> > > For example, we can use virtio-cli to pass command from command line to
> > > device in pass-through manner without changing driver.
> > > 
> > Opaque strings passed all the way from guest userspace to host device?
> > Not sure why is that a good idea.
> 
> Where did I mentioned a guest ?
> 
> Virtio-cli will control a device on the same host that it runs.
> 
> If it's a bare metal host so it will manage the virtio attached device.
> 
> If it's a guest it will manage the device attached to the guest.

Opaqueness of all this is what worries at least me and Cornelia.


> > 
> > > > 
> > > > > diff --git a/content.tex b/content.tex
> > > > > index c266fd5..d350175 100644
> > > > > --- a/content.tex
> > > > > +++ b/content.tex
> > > > > @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> > > > >    types. It is RECOMMENDED that devices generate version 4
> > > > >    UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> > > > > +\input{admin-virtq.tex}
> > > > > +
> > > > >    \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> > > > >    We start with an overview of device initialization, then expand on the
> > > > > @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > >      data to be provided in driver notification and the delivery method is
> > > > >      transport specific.
> > > > >      For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
> > > > > +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
> > > > > +  the device supports administration virtqueue negotiation.
> > > > >    \end{description}
> > > > > -- 
> > > > > 2.21.0

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
> 
> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
> > On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > 
> > > On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
> > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > > +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
> > > > > +
> > > > > +The Vendor specific command set is a group of classes and commands
> > > > > +within each of these classes which are vendor specific. Refer to
> > > > > +each vendor specification for more information on the supported
> > > > > +commands.
> > > > Here's another example.
> > > > It's important that there is a way to make a device completely
> > > > generic without vendor specific expensions.
> > > > Would be hard to do here.
> > > > 
> > > > That's a generic comment.
> > > > 
> > > > but specifically I am very reluctant to add vendor specific stuff like
> > > > this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
> > > > and if that is not sufficient I would like to know why
> > > > before we add more vendor specific stuff.
> > > We are adding an option to add vendor specific commands. We're not
> > > adding it to the spec since each vendor will have its own manual for
> > > that.
> > IMHO, that way madness lies. I want to be able to look at the spec and
> > be able to implement a compliant device or a compliant driver. If a
> > vendor has some special feature they want to support, put it in the
> > spec, so that it is possible to actually implement it.
> 
> You can implement a compliant device and a compliant. why do you think you
> can't ?
> 
> Some features are vendor/sub-vendor specific.
> 
> And as mentioned, you already added it to  the spec so I guess it was for a
> reason.

Well we basically just reserved a capability ID so if people add their
vendor specific ones they don't conflict with each other or capabilities
we'll add in the future.
And such a capability can prove useful for things like
identifying the device to enable workarounds, etc.

That is a far cry from a fat interface which generic drivers will need to
support and who's sole purpose is vendor specific extensions.

And yes, it also allows you to unload the generic driver and load
a vendor driver. Which can then go wild if it wants to - nothing
we can or want to do about it.


> If someone would like to add its special souse to virtio device and we have
> a flexible admin queue to manage it with a spec compliant manner, why not ?
>
> The feature might not get support by the working group so do you really want
> to limit vendor from implementing features that are not agreed here on the
> mailing list ?
> 
> Adding a vendor specific command set is trivial.

Supporting all this mess leter isn't. So we better make sure just how
is all this vendor stuff going to be limited.

> > 
> > > For example, we can use virtio-cli to pass command from command line to
> > > device in pass-through manner without changing driver.
> > Things like that are part of the driver as in the spec sense. The spec
> > does not care how you actually split the implementation, or what
> > controls you are giving to whom. We need a defined interface.
> 
> We have an interface. Its the admin queue.
> 
> Virtio-cli will be a user space tool to send commands via this admin queue
> and configure the device.
> 
> The driver will just open a char dev to supply a channel to this admin queue
> from user space.
> 
> Examples:
> 
> 1. format a virtio-blk device to add data integrity checks (T-10)
> 
> 2. set 5 msix to VF_1, 12 msix to VF_2 and 2 msix to VF_3 before enabling
> SRIOV
> 
> 3. Get telemetry log
> 
> 4. Get error log
> 
> 5. Get vendor specific HW health report
> 
> 6. Get generic virtio device health report
> 
> 7. set 5 queues to VF_1, 12 queues to VF_2 and 2 queues to VF_3 before
> enabling SRIOV
> 
> 
> This knowledge is needed for the sys-admin and not the driver. The driver
> will provide the channel to enable this configuration by an
> orchestrator/admin.
> 
> 
> 
> > 

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>
>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>>>>>> +
>>>>>> +The Vendor specific command set is a group of classes and commands
>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>> +each vendor specification for more information on the supported
>>>>>> +commands.
>>>>> Here's another example.
>>>>> It's important that there is a way to make a device completely
>>>>> generic without vendor specific expensions.
>>>>> Would be hard to do here.
>>>>>
>>>>> That's a generic comment.
>>>>>
>>>>> but specifically I am very reluctant to add vendor specific stuff like
>>>>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>>>>> and if that is not sufficient I would like to know why
>>>>> before we add more vendor specific stuff.
>>>> We are adding an option to add vendor specific commands. We're not
>>>> adding it to the spec since each vendor will have its own manual for
>>>> that.
>>> IMHO, that way madness lies. I want to be able to look at the spec and
>>> be able to implement a compliant device or a compliant driver. If a
>>> vendor has some special feature they want to support, put it in the
>>> spec, so that it is possible to actually implement it.
>> You can implement a compliant device and a compliant. why do you think you
>> can't ?
>>
>> Some features are vendor/sub-vendor specific.
>>
>> And as mentioned, you already added it to  the spec so I guess it was for a
>> reason.
> Well we basically just reserved a capability ID so if people add their
> vendor specific ones they don't conflict with each other or capabilities
> we'll add in the future.

it shouldn't bother you as a driver maintainer. I don't think vendor 
specific code should go to Linux kernel.

No conflict will happen.

There is no harm in having a virtio-cli tool that will pass-through 
commands from command line to the device using the admin queue.

All the HW devices I know have sw tool that can access it and virtio 
shouldn't be different.

Also, I can develop my own vendor specific user space driver for 
virtio-blk for example (lets say in SPDK framework). And this driver 
will be aware of this vendor specific capabilities.


> And such a capability can prove useful for things like
> identifying the device to enable workarounds, etc.
>
> That is a far cry from a fat interface which generic drivers will need to
> support and who's sole purpose is vendor specific extensions.

No vendor specific commands will be added to generic driver. I never 
said this.

I'll enable the option to create devices that support vendor specific 
commands.


>
> And yes, it also allows you to unload the generic driver and load
> a vendor driver. Which can then go wild if it wants to - nothing
> we can or want to do about it.

I don't get your point.

If someone would like to do crazy stuff, you can't stop him.


>
>
>> If someone would like to add its special souse to virtio device and we have
>> a flexible admin queue to manage it with a spec compliant manner, why not ?
>>
>> The feature might not get support by the working group so do you really want
>> to limit vendor from implementing features that are not agreed here on the
>> mailing list ?
>>
>> Adding a vendor specific command set is trivial.
> Supporting all this mess leter isn't. So we better make sure just how
> is all this vendor stuff going to be limited.

You define a range of commands for vendors and that's it.

I don't see a scenario to add some vendor code to virtio generic drivers.


>
>>>> For example, we can use virtio-cli to pass command from command line to
>>>> device in pass-through manner without changing driver.
>>> Things like that are part of the driver as in the spec sense. The spec
>>> does not care how you actually split the implementation, or what
>>> controls you are giving to whom. We need a defined interface.
>> We have an interface. Its the admin queue.
>>
>> Virtio-cli will be a user space tool to send commands via this admin queue
>> and configure the device.
>>
>> The driver will just open a char dev to supply a channel to this admin queue
>> from user space.
>>
>> Examples:
>>
>> 1. format a virtio-blk device to add data integrity checks (T-10)
>>
>> 2. set 5 msix to VF_1, 12 msix to VF_2 and 2 msix to VF_3 before enabling
>> SRIOV
>>
>> 3. Get telemetry log
>>
>> 4. Get error log
>>
>> 5. Get vendor specific HW health report
>>
>> 6. Get generic virtio device health report
>>
>> 7. set 5 queues to VF_1, 12 queues to VF_2 and 2 queues to VF_3 before
>> enabling SRIOV
>>
>>
>> This knowledge is needed for the sys-admin and not the driver. The driver
>> will provide the channel to enable this configuration by an
>> orchestrator/admin.
>>
>>
>>

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/1/2021 1:17 AM, Michael S. Tsirkin wrote:
> On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
>> On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
>>> On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>> Admin virtqueues will be used to send administrative commands to
>>>>>> manipulate various features of the device which would not easily map
>>>>>> into the configuration space.
>>>>>>
>>>>>> The same Admin command format will be used for all virtio devices. The
>>>>>> Admin command set will include 4 types of command classes:
>>>>>> 1. The generic common class
>>>>>> 2. The transport specific class
>>>>>> 3. The device specific class
>>>>>> 4. The vendor specific class
>>>>>>
>>>>>> The above mechanism will enable adding various features to the virtio
>>>>>> specification, e.g.:
>>>>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>>>>       512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>>>>> 2. Live migration management.
>>>>>> 3. Encrypt/Decrypt descriptors.
>>>>>> 4. Virtualization management.
>>>>>> 5. Get device error logs.
>>>>>> 6. Implement advanced vendor/device/transport specific features.
>>>>>> 7. Run device health test.
>>>>>> 8. More.
> I still don't really see what do all these things have in common?

I don't think you need to look on this in that direction.

This is a queue for administration.

Cornelia and Stefan already agreed on this approach.

Please lets progress and not go back to the beginning.

> Why are they grouped behind the same feature bit? Share same VQ?

They are grouped behind an admin q.


>
>>>>>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>>>>>> for the specification to become flexible and allow a wider feature set.
>>>>>> The corrent ctrl virtq that is defined for some of the virtio devices is
>>>>>> device specific and wasn't designed to be a generic virtq for
>>>>>> admininistration.
>>>>>>
>>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>> Lots of things on this list seem to make sense when
>>>>> done from PF and affecting a VF.
>>>>> I think from this POV the generic structure should include
>>>>> a way to address one device from another.
>>>> This will be defined per command.
>>>>
>>>> For example, funcion_id will be given as command data.
>>> Why? Sounds like a generic thing to me.
>> Generic to a command that handles virtualization management.
>
> It could be that mixing up virtualization management and arbitrary
> other management commands in the same interface is a mistake.

It's not a mistake.

This is the right design.

Creating a new interface for each feature is madness.


> Or maybe not - do we want host to have ability to run health tests
> for a VF without loading VF driver? Get error logs?

This can be an optional feature.

We need to define it in the future. We can't define the entire command 
set now.

We need to define the infrastructure.

>
> In fact besides migration and virtualization management the rest of
> examples that you give all seem to be more or less device specific, with
> the possible exception of 3. Encrypt/Decrypt descriptors.  what does
> this one imply, exactly?

For storage devices there is an option to have a self encrypted drive.

Maybe we can develop encryption/decryption also  for net devices.

This will be developed in the future.

But the infrastructure will allow it. This is the beauty if it, you 
create infrastructure today and add optional commands tomorrow.

Nobody can think of thousands of features and commands today, and we 
also can't push thousands of pages to the spec.

Lets push 2-3 mandatory commands with the infrastructure and build new 
features incrementally.

>
>
>
>> It has nothing to do with a discovery command for the device itself.
>>
>> I guess we can add few bytes to be mandatory after the <u8 class, u8
>> command> pair.
>>
>> Something like <u8 class, u8 command, u16 mandatory_command_data >.
>>
>> But I think it's not a must and it's already covered by:  "u8
>> command-specific-data[];"
> I would say we should include the device ID as well as a definition
> what the ID is at least for SRIOV. This isn't a command
> specific data, any virt management command will need this.

of course it will be defined for virt management. But is only for PCI 
devices.

Not needed for MMIO devices, right ?

>
>
>>>
>>>>>> ---
>>>>>>     admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>     content.tex     |   4 +
>>>>>>     2 files changed, 245 insertions(+)
>>>>>>     create mode 100644 admin-virtq.tex
>>>>>>
>>>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>>>> new file mode 100644
>>>>>> index 0000000..ccec2ca
>>>>>> --- /dev/null
>>>>>> +++ b/admin-virtq.tex
>>>>>> @@ -0,0 +1,241 @@
>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>>> +
>>>>>> +Admin virtqueues are used to send administrative commands to manipulate
>>>>>> +various features of the device which would not easily map into the
>>>>>> +configuration space.
>>>>>> +
>>>>>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>> +feature bit.
>>>>>> +
>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>> +
>>>>>> +All commands are of the following form:
>>>>>> +
>>>>>> +\begin{lstlisting}
>>>>>> +struct virtio_admin_cmd {
>>>>>> +        /* Device-readable part */
>>>>>> +        u8 class;
>>>>>> +        u8 command;
>>>>>> +        u8 command-specific-data[];
>>>>>> +
>>>>>> +        /* Device-writable part */
>>>>>> +        u8 command-specific-result[];
>>>>>> +        u8 status_type : 4;
>>>>>> +        u8 reserved : 4;
>>>>>> +        u8 status;
>>>>>> +};
>>>>>> +
>>>>>> +/* Status type values */
>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>>>>>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>>>>>> +
>>>>>> +/* Generic status values */
>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>>>>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>>>>>> +\end{lstlisting}
>>>>>> +
>>>>>> +The \field{class}, \field{command} and \field{command-specific-data} are
>>>>>> +set by the driver, and the device sets the \field{status_type}, the
>>>>>> +\field{status} and  the \field{command-specific-result}, if needed.
>>>>>> +
>>>>>> +The virtio Admin command class codes are divided in the following form:
>>>>>> +
>>>>>> +\begin{lstlisting}
>>>>>> +/* class values that are transport, device and vendor independent */
>>>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>>>>>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>>>>>> +
>>>>>> +/* class values that are transport specific */
>>>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>>>>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>>>>>> +
>>>>>> +/* class values that are device specific */
>>>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>>>>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>>>>>> +
>>>>>> +/* class values that are vendor specific */
>>>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>>>>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>>>>>> +\end{lstlisting}
>>>>>> +
>>>>>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>>>>>> +
>>>>>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>>>>>> +support all the mandatory admin commands. A device MAY support also
>>>>>> +one or more optional admin commands.
>>>>>> +
>>>>>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>>>>>> +
>>>>>> +The Common command set is a group of classes and commands within each
>>>>>> +of these classes which are transport, device and vendor independent.
>>>>>> +A mandatory class is a class that has at least one mandatory command.
>>>>>> +The Common command set is summarized in following table:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Class  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>>>>>> +\hline
>>>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>>>>>> +\hline
>>>>>> +2-63  & reserved    & - \\
>>>>>> +\hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>>>>>> +
>>>>>> +This class (opcode: 0) of commands is used to query generic device
>>>>>> +information. The following table describes the commands supported for
>>>>>> +this class:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Command  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>>>>>> +\hline
>>>>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>>>>>> +\hline
>>>>>> +2-255  & reserved    & - \\
>>>>>> +\hline
>>>>>> +\end{tabular}
>>>>> So there are several problems with this approach.
>>>>> One is that there is no two way negotiation.
>>>> you negotiate the adminq feature.
>>>>
>>>> then you can send admin commands and discover the supported commands.
>>>>
>>>>> No way for device to know what will driver use in the end.
>>>> device will be designed to support mandatory admin commands and some
>>>> optional.
>>>>
>>>> It doesn't need to care whether the driver will use it or not.
>>>>> This breaks things like e.g. accelerating some configurations
>>>>> but not others.
>>>> I don't understand what it breaks exactly.
>>> Long practice taught us that it is good for device to know
>>> what is driver going to use.
>>> For example, some features can be implemented in hardware
>>> and some in hypervisor software. If driver is going to use software
>>> features then you need to switch to a slower software
>>> implementation. Doing that dynamically at time of use is
>>> often much harder that up-front at negotiation time.
>> I don't think we should write specifications that should consider hypervisor
>> SW.
> Well considering hypervisors is clearly one of the purposes of virtio TC.
> Look it up in the charter, section 2. Statement of Purpose.
>
>
>> You might use virtio device without hypervisor at all.
> Yes and supporting that is also clearly an objective:
>
> 	With the 1.1, 1.2 and future revisions of the Specification, we aim to
> 	evolve the VIRTIO standard further, addressing such new requirements
> 	while both continuing to honor the goals of the 1.0 Specification and
> 	including new objectives.
>
>
There is nothing in admin queue that doesn't honor old spec.

Old driver will not be aware of it.


>
>>>>> Another is that everything is going through the admin vq.
>>>>> Hard for hypervisor to intervene and change just some aspects
>>>>> of the device.
>>>> we'll develop virtio-cli tool.
>>> will that be part of the spec somehow then?
>>> I'm not sure it can be ...
>> of course not.
>>
>> This is just a channel for the admin to send admin command to the virtio
>> device.
>>
>>>> I don't know what should be changed in the device by the hypervisor.
>>> Look at vdpa as an example, guest device can be different from
>>> host device.
>> VDPA is an acceleration. I can write different acceleration on to of virtio
>> device.
>>
>> Is shouldn't be considered in the specification.
>>
>> Nevertheless, there is nothing in this RFC that VDPA can't handle.
>>
>>
>>>>> This is also a problem for validation, tricky
>>>>> to find out during probe what does device support and whether
>>>>> driver can work with it.
>>>> device will support admin q with mandatory commands.
>>>>
>>>> Old driver shouldn't care and it won't negotiate this feature.
>>>>
>>>> New drivers will adjust and will be extended.
>>> Does not sound like I got the issue across.
>>>
>>> Operating VQs just to know whether to bind to device is
>>> problematic since there's an assumption that once
>>> DRIVER_OK is set driver is happy. Before DRIVER_OK
>>> device will not consume buffers.
>> Nothing is changed in DRIVER_OK negotiation rules.
> Right and that means you can't really use the admin vq to validate
> device during driver probe.
You said that before DRIVER_OK is set the device will not consume 
buffers, so what is the question/problem here ?

Do you think we need to change it ? I don't think it's needed.

>
>
>>>
>>>>> Can't we map this stuff into memory instead?
>>>>> Maybe have an admin config structure
>>>>> that is separate from regular config?
>>>>> Only issue we can't then make it too big.
>>>>> But so far I don't see a lot of info used, most is reserved.
>>>> This is initial commit to add infrastructure. I can't add 255 commands at
>>>> once.
>>>>
>>>> We need to add it incrementally.
>>> How about a list of future extensions?
>> Added a partial list in the commit message and in the answer to Cornelia.
>>
>>
>>>
>>>>>
>>>>>> +
>>>>>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>>>>>> +
>>>>>> +This mandatory command should return device identity in the following
>>>>>> +structure:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Bytes  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +03:00  & VIRTIO DEVICE ID    & M \\
>>>>>> +\hline
>>>>>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>>>>>> +\hline
>>>>>> +4095:06  & reserved    & - \\
>>>>>> + \hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
>>>>>> +
>>>>>> +This mandatory command should return device identity in the following
>>>>>> +structure:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Bytes  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
>>>>>> +\hline
>>>>>> +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
>>>>>> + \hline
>>>>>> +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
>>>>>> +\hline
>>>>>> +...  & ...    & M \\
>>>>>> +\hline
>>>>>> +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
>>>>>> +\hline
>>>>>> +4095:1024  & reserved    & - \\
>>>>>> +\hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +The above structure is divided to 256 sections of 4B each, that will
>>>>>> +describe whether a device supports a certain class code. The
>>>>>> +following 3072B are reserved. each of the non-reserved 4B sections will
>>>>>> +follow the following data structure:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Bits  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>>>>>> +\hline
>>>>>> +31:01  & reserved    & - \\
>>>>>> + \hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
>>>>>> +
>>>>>> +This class (opcode: 1) of commands is used to query supported commands
>>>>>> +for each supported device class.
>>>>>> The following table describes the commands
>>>>>> +supported for this class:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Command  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
>>>>>> +\hline
>>>>>> +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
>>>>>> +\hline
>>>>>> +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
>>>>>> +\hline
>>>>>> +...  & ...    & M \\
>>>>>> +\hline
>>>>>> +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
>>>>>> +\hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +Each command in this class, will return class identity in the following structure:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Bytes  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
>>>>>> +\hline
>>>>>> +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
>>>>>> + \hline
>>>>>> +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
>>>>>> +\hline
>>>>>> +...  & ...    & M \\
>>>>>> +\hline
>>>>>> +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
>>>>>> +\hline
>>>>>> +4095:1024  & reserved    & - \\
>>>>>> +\hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +The above structure is divided to 256 sections of 4B each, that will
>>>>>> +describe whether a class supports a certain command code. The
>>>>>> +following 3072B are reserved. each of the non-reserved 4B sections will
>>>>>> +follow the following data structure:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Bits  & Description    & M/O \\
>>>>>> +\hline \hline
>>>>>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>>>>>> +\hline
>>>>>> +31:01  & reserved    & - \\
>>>>>> + \hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
>>>>>> +
>>>>>> +The Transport command set is a group of classes and commands within
>>>>>> +each of these classes which are transport specific. Refer to each
>>>>>> +transport section for more information on the supported commands.
>>>>>> +
>>>>>> +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
>>>>>> +
>>>>>> +The Device command set is a group of classes and commands within
>>>>>> +each of these classes which are device specific. Refer to each
>>>>>> +device section for more information on the supported commands.
>>>>>> +
>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>>>>>> +
>>>>>> +The Vendor specific command set is a group of classes and commands
>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>> +each vendor specification for more information on the supported
>>>>>> +commands.
>>>>> Here's another example.
>>>>> It's important that there is a way to make a device completely
>>>>> generic without vendor specific expensions.
>>>>> Would be hard to do here.
>>>>>
>>>>> That's a generic comment.
>>>>>
>>>>> but specifically I am very reluctant to add vendor specific stuff like
>>>>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>>>>> and if that is not sufficient I would like to know why
>>>>> before we add more vendor specific stuff.
>>>> We are adding an option to add vendor specific commands. We're not adding it
>>>> to the spec since each vendor will have its own manual for that.
>>> You didn't actually answer the question though.
>>>
>>>> For example, we can use virtio-cli to pass command from command line to
>>>> device in pass-through manner without changing driver.
>>>>
>>> Opaque strings passed all the way from guest userspace to host device?
>>> Not sure why is that a good idea.
>> Where did I mentioned a guest ?
>>
>> Virtio-cli will control a device on the same host that it runs.
>>
>> If it's a bare metal host so it will manage the virtio attached device.
>>
>> If it's a guest it will manage the device attached to the guest.
> Opaqueness of all this is what worries at least me and Cornelia.

guest is not aware of host devices.

Sending raw command from Linux cmdline to a device is not something I 
invented.

If a user is aware of its HW he can use the virtio-cli tool to configure 
its unique features.

And if the user is not so smart, we can help him with adding vendor 
classes to virtio-cli management tool.


>
>>>>>> diff --git a/content.tex b/content.tex
>>>>>> index c266fd5..d350175 100644
>>>>>> --- a/content.tex
>>>>>> +++ b/content.tex
>>>>>> @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>>>>>     types. It is RECOMMENDED that devices generate version 4
>>>>>>     UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>>> +\input{admin-virtq.tex}
>>>>>> +
>>>>>>     \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>>>>>     We start with an overview of device initialization, then expand on the
>>>>>> @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>>>>>       data to be provided in driver notification and the delivery method is
>>>>>>       transport specific.
>>>>>>       For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
>>>>>> +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
>>>>>> +  the device supports administration virtqueue negotiation.
>>>>>>     \end{description}
>>>>>> -- 
>>>>>> 2.21.0

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Sun, Aug 01, 2021 at 01:53:10AM +0300, Max Gurtovoy wrote:
> > 
> > And yes, it also allows you to unload the generic driver and load
> > a vendor driver. Which can then go wild if it wants to - nothing
> > we can or want to do about it.
> 
> I don't get your point.

Talking about virtio admin interface (you are using a VQ now), it seems
to include or be planned to include several unrelated features, which
I'm not sure is appropriate and it sure makes review harder. I would
advise focusing on hypervisor management features such as live migration
etc, or some other subset. Also, there's overlap with existing
functionality such as feature negotiation.

Talking about vendor specific commands, I don't feel you demonstrated a
need for these beyond what already exists in the spec. In particular it
is unclear why there a need to layer vendor specific commands on top of
virtio. Using the existing capability would work at the PCI level which
seems more appropriate as vendor things are likely to be low level.

It might be a good idea to address these concerns in your next revision,
narrowing the scope as much as possible and explaining the motivation
for the new interfaces.

Thanks!

-- 
MST

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/1/2021 11:16 AM, Michael S. Tsirkin wrote:
> On Sun, Aug 01, 2021 at 01:53:10AM +0300, Max Gurtovoy wrote:
>>> And yes, it also allows you to unload the generic driver and load
>>> a vendor driver. Which can then go wild if it wants to - nothing
>>> we can or want to do about it.
>> I don't get your point.
> Talking about virtio admin interface (you are using a VQ now), it seems
> to include or be planned to include several unrelated features, which
> I'm not sure is appropriate and it sure makes review harder. I would
> advise focusing on hypervisor management features such as live migration
> etc, or some other subset. Also, there's overlap with existing
> functionality such as feature negotiation.

Only one feature is negotiated in the RFC and it's the admin virtq support.

You asked me to give examples for future features and I did. These 
features are not in the RFC.

So I don't understand what is hard to review. I've answered all the 
points you raised.

If I'll add Live Migration to the RFC we'll discuss it for months and 
then we'll discuss the admin queue interface again.

I don't want that. We already agreed to upstream first the Admin q 
interface.

Please don't change the order and make me run in cycles.

>
> Talking about vendor specific commands, I don't feel you demonstrated a
> need for these beyond what already exists in the spec. In particular it
> is unclear why there a need to layer vendor specific commands on top of
> virtio. Using the existing capability would work at the PCI level which
> seems more appropriate as vendor things are likely to be low level.

I did explain it.

Vendors might have their own special souse that might seems weird for 
the TC or maybe some internal management stuff that are not related to 
virtio, such as HW counters for example.

You're trying to address all the commands in one shut and then you ask 
the opposite to narrow it to minimal.

I did already all the needed work: sent an RFC for LM and send an RFC 
for new admin q with only 2 commands for us to have easy submission.

This RFC gives you all the flexibility to add Jason's stop/start queue 
commands in the future, Live migration RFC that I sent and all the 
features you'll ever think to add to virtio instead of creating a 
dedicated interface each time and run long review cycles.


>
> It might be a good idea to address these concerns in your next revision,
> narrowing the scope as much as possible and explaining the motivation
> for the new interfaces.

narrow to what ? we have only 2 classes here, come on.

>
> Thanks!
>

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 7/29/2021 11:48 AM, Stefan Hajnoczi wrote:
> On Wed, Jul 28, 2021 at 05:20:29PM +0300, Max Gurtovoy wrote:
>> On 7/28/2021 4:42 PM, Stefan Hajnoczi wrote:
>>> On Wed, Jul 28, 2021 at 01:59:26PM +0300, Max Gurtovoy wrote:
>>>> On 7/28/2021 11:52 AM, Stefan Hajnoczi wrote:
>>>>> On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
>>>>>> On 7/27/2021 5:28 PM, Cornelia Huck wrote:
>>>>>>> On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>> How many different (groups of) commands can we reasonably expect? Do we
>>>>>>> need a generic discovery command, or can we get away with a feature bit
>>>>>>> covering each new group of commands?
>>>>>> I can't predict the future but IMO we need a discovery command.
>>>>>>
>>>>>> We have many devices and more can be added in the future.
>>>>> A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin
>>>>> commands would not be included in VIRTIO Feature Bits but instead
>>>>> reported via a separate admin command that returns up to 8KB of data:
>>>>>
>>>>>      struct virtio_admin_report_cmds {
>>>>>          /* Bitmap of available admin commands [Device->Driver]
>>>>>           * bool command_present =
>>>>>           *        command_bits[class * 32 + command / 8] & (command % 8);
>>>>>           */
>>>>>          u8 command_bits[8192];
>>>>>      };
>>>> Yes, I divided it to multiple commands per class to cover the case we will
>>>> need more than 1 bit to describe a command.
>>>>
>>>> But I guess we can add it later on.
>>>>
>>>> I think the above should be:
>>>>
>>>> bool command_present = command_bits[class * 32 + command / 8] & (1 << (command % 8));
>>>>
>>>> isn't it ?
>>> You're right. I forgot to shift the bit :D.
>>>
>>>> Also what do you think about renaming <class, command> to <opcode, opmod> ?
>>> I need to understand how opcode and opmod values are used. I'm not sure.
>> Same as class and command just with different naming.
> Is opcode the same as command? What does opmod do?

no. opcode == class and opmod == command.


>
> Stefan

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>
> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>
>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic 
>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin command 
>>>>>>> set / Vendor specific command set}
>>>>>>> +
>>>>>>> +The Vendor specific command set is a group of classes and commands
>>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>>> +each vendor specification for more information on the supported
>>>>>>> +commands.
>>>>>> Here's another example.
>>>>>> It's important that there is a way to make a device completely
>>>>>> generic without vendor specific expensions.
>>>>>> Would be hard to do here.
>>>>>>
>>>>>> That's a generic comment.
>>>>>>
>>>>>> but specifically I am very reluctant to add vendor specific stuff 
>>>>>> like
>>>>>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>> and if that is not sufficient I would like to know why
>>>>>> before we add more vendor specific stuff.
>>>>> We are adding an option to add vendor specific commands. We're not
>>>>> adding it to the spec since each vendor will have its own manual for
>>>>> that.
>>>> IMHO, that way madness lies. I want to be able to look at the spec and
>>>> be able to implement a compliant device or a compliant driver. If a
>>>> vendor has some special feature they want to support, put it in the
>>>> spec, so that it is possible to actually implement it.
>>> You can implement a compliant device and a compliant. why do you 
>>> think you
>>> can't ?
>>>
>>> Some features are vendor/sub-vendor specific.


Please don't do this. This breaks the efforts for making virtio as a 
standard and generic device.


>>>
>>> And as mentioned, you already added it to  the spec so I guess it 
>>> was for a
>>> reason.
>> Well we basically just reserved a capability ID so if people add their
>> vendor specific ones they don't conflict with each other or capabilities
>> we'll add in the future.
>
> it shouldn't bother you as a driver maintainer. I don't think vendor 
> specific code should go to Linux kernel.
>
> No conflict will happen.
>
> There is no harm in having a virtio-cli tool that will pass-through 
> commands from command line to the device using the admin queue.
>
> All the HW devices I know have sw tool that can access it and virtio 
> shouldn't be different.
>
> Also, I can develop my own vendor specific user space driver for 
> virtio-blk for example (lets say in SPDK framework). And this driver 
> will be aware of this vendor specific capabilities.


That's even worse since the driver can only work for the vendor specific 
device which is in fact not a standard device.

For the vendor specific capability, the spec has already said that it 
will be only used for debugging/reporting purpose:

"""

The optional Vendor data capability allows the device to present
vendor-specific data to the driver, without
conflicts, for debugging and/or reporting purposes,
and without conflicting with standard functionality.

"""

It looks to me what you're trying to invent may violate the spec.

Thanks


>
>
>> And such a capability can prove useful for things like
>> identifying the device to enable workarounds, etc.
>>
>> That is a far cry from a fat interface which generic drivers will 
>> need to
>> support and who's sole purpose is vendor specific extensions.
>
> No vendor specific commands will be added to generic driver. I never 
> said this.
>
> I'll enable the option to create devices that support vendor specific 
> commands.
>
>
>>
>> And yes, it also allows you to unload the generic driver and load
>> a vendor driver. Which can then go wild if it wants to - nothing
>> we can or want to do about it.
>
> I don't get your point.
>
> If someone would like to do crazy stuff, you can't stop him.
>
>
>>
>>
>>> If someone would like to add its special souse to virtio device and 
>>> we have
>>> a flexible admin queue to manage it with a spec compliant manner, 
>>> why not ?
>>>
>>> The feature might not get support by the working group so do you 
>>> really want
>>> to limit vendor from implementing features that are not agreed here 
>>> on the
>>> mailing list ?
>>>
>>> Adding a vendor specific command set is trivial.
>> Supporting all this mess leter isn't. So we better make sure just how
>> is all this vendor stuff going to be limited.
>
> You define a range of commands for vendors and that's it.
>
> I don't see a scenario to add some vendor code to virtio generic drivers.
>
>
>>
>>>>> For example, we can use virtio-cli to pass command from command 
>>>>> line to
>>>>> device in pass-through manner without changing driver.
>>>> Things like that are part of the driver as in the spec sense. The spec
>>>> does not care how you actually split the implementation, or what
>>>> controls you are giving to whom. We need a defined interface.
>>> We have an interface. Its the admin queue.
>>>
>>> Virtio-cli will be a user space tool to send commands via this admin 
>>> queue
>>> and configure the device.
>>>
>>> The driver will just open a char dev to supply a channel to this 
>>> admin queue
>>> from user space.
>>>
>>> Examples:
>>>
>>> 1. format a virtio-blk device to add data integrity checks (T-10)
>>>
>>> 2. set 5 msix to VF_1, 12 msix to VF_2 and 2 msix to VF_3 before 
>>> enabling
>>> SRIOV
>>>
>>> 3. Get telemetry log
>>>
>>> 4. Get error log
>>>
>>> 5. Get vendor specific HW health report
>>>
>>> 6. Get generic virtio device health report
>>>
>>> 7. set 5 queues to VF_1, 12 queues to VF_2 and 2 queues to VF_3 before
>>> enabling SRIOV
>>>
>>>
>>> This knowledge is needed for the sys-admin and not the driver. The 
>>> driver
>>> will provide the channel to enable this configuration by an
>>> orchestrator/admin.
>>>
>>>
>>>
>

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/8/2 上午10:17, Jason Wang 写道:
>
> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>
>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>
>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic 
>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin 
>>>>>>>> command set / Vendor specific command set}
>>>>>>>> +
>>>>>>>> +The Vendor specific command set is a group of classes and 
>>>>>>>> commands
>>>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>>>> +each vendor specification for more information on the supported
>>>>>>>> +commands.
>>>>>>> Here's another example.
>>>>>>> It's important that there is a way to make a device completely
>>>>>>> generic without vendor specific expensions.
>>>>>>> Would be hard to do here.
>>>>>>>
>>>>>>> That's a generic comment.
>>>>>>>
>>>>>>> but specifically I am very reluctant to add vendor specific 
>>>>>>> stuff like
>>>>>>> this directly in the spec. We already have 
>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>> and if that is not sufficient I would like to know why
>>>>>>> before we add more vendor specific stuff.
>>>>>> We are adding an option to add vendor specific commands. We're not
>>>>>> adding it to the spec since each vendor will have its own manual for
>>>>>> that.
>>>>> IMHO, that way madness lies. I want to be able to look at the spec 
>>>>> and
>>>>> be able to implement a compliant device or a compliant driver. If a
>>>>> vendor has some special feature they want to support, put it in the
>>>>> spec, so that it is possible to actually implement it.
>>>> You can implement a compliant device and a compliant. why do you 
>>>> think you
>>>> can't ?
>>>>
>>>> Some features are vendor/sub-vendor specific.
>
>
> Please don't do this. This breaks the efforts for making virtio as a 
> standard and generic device.


The correct way is to generalize those features and introduce them in 
the spec instead.

Thanks


>
>
>>>>
>>>> And as mentioned, you already added it to  the spec so I guess it 
>>>> was for a
>>>> reason.
>>> Well we basically just reserved a capability ID so if people add their
>>> vendor specific ones they don't conflict with each other or 
>>> capabilities
>>> we'll add in the future.
>>
>> it shouldn't bother you as a driver maintainer. I don't think vendor 
>> specific code should go to Linux kernel.
>>
>> No conflict will happen.
>>
>> There is no harm in having a virtio-cli tool that will pass-through 
>> commands from command line to the device using the admin queue.
>>
>> All the HW devices I know have sw tool that can access it and virtio 
>> shouldn't be different.
>>
>> Also, I can develop my own vendor specific user space driver for 
>> virtio-blk for example (lets say in SPDK framework). And this driver 
>> will be aware of this vendor specific capabilities.
>
>
> That's even worse since the driver can only work for the vendor 
> specific device which is in fact not a standard device.
>
> For the vendor specific capability, the spec has already said that it 
> will be only used for debugging/reporting purpose:
>
> """
>
> The optional Vendor data capability allows the device to present
> vendor-specific data to the driver, without
> conflicts, for debugging and/or reporting purposes,
> and without conflicting with standard functionality.
>
> """
>
> It looks to me what you're trying to invent may violate the spec.
>
> Thanks
>
>
>>
>>
>>> And such a capability can prove useful for things like
>>> identifying the device to enable workarounds, etc.
>>>
>>> That is a far cry from a fat interface which generic drivers will 
>>> need to
>>> support and who's sole purpose is vendor specific extensions.
>>
>> No vendor specific commands will be added to generic driver. I never 
>> said this.
>>
>> I'll enable the option to create devices that support vendor specific 
>> commands.
>>
>>
>>>
>>> And yes, it also allows you to unload the generic driver and load
>>> a vendor driver. Which can then go wild if it wants to - nothing
>>> we can or want to do about it.
>>
>> I don't get your point.
>>
>> If someone would like to do crazy stuff, you can't stop him.
>>
>>
>>>
>>>
>>>> If someone would like to add its special souse to virtio device and 
>>>> we have
>>>> a flexible admin queue to manage it with a spec compliant manner, 
>>>> why not ?
>>>>
>>>> The feature might not get support by the working group so do you 
>>>> really want
>>>> to limit vendor from implementing features that are not agreed here 
>>>> on the
>>>> mailing list ?
>>>>
>>>> Adding a vendor specific command set is trivial.
>>> Supporting all this mess leter isn't. So we better make sure just how
>>> is all this vendor stuff going to be limited.
>>
>> You define a range of commands for vendors and that's it.
>>
>> I don't see a scenario to add some vendor code to virtio generic 
>> drivers.
>>
>>
>>>
>>>>>> For example, we can use virtio-cli to pass command from command 
>>>>>> line to
>>>>>> device in pass-through manner without changing driver.
>>>>> Things like that are part of the driver as in the spec sense. The 
>>>>> spec
>>>>> does not care how you actually split the implementation, or what
>>>>> controls you are giving to whom. We need a defined interface.
>>>> We have an interface. Its the admin queue.
>>>>
>>>> Virtio-cli will be a user space tool to send commands via this 
>>>> admin queue
>>>> and configure the device.
>>>>
>>>> The driver will just open a char dev to supply a channel to this 
>>>> admin queue
>>>> from user space.
>>>>
>>>> Examples:
>>>>
>>>> 1. format a virtio-blk device to add data integrity checks (T-10)
>>>>
>>>> 2. set 5 msix to VF_1, 12 msix to VF_2 and 2 msix to VF_3 before 
>>>> enabling
>>>> SRIOV
>>>>
>>>> 3. Get telemetry log
>>>>
>>>> 4. Get error log
>>>>
>>>> 5. Get vendor specific HW health report
>>>>
>>>> 6. Get generic virtio device health report
>>>>
>>>> 7. set 5 queues to VF_1, 12 queues to VF_2 and 2 queues to VF_3 before
>>>> enabling SRIOV
>>>>
>>>>
>>>> This knowledge is needed for the sys-admin and not the driver. The 
>>>> driver
>>>> will provide the channel to enable this configuration by an
>>>> orchestrator/admin.
>>>>
>>>>
>>>>
>>

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/7/28 下午8:48, Michael S. Tsirkin 写道:
> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>> Admin virtqueues will be used to send administrative commands to
>> manipulate various features of the device which would not easily map
>> into the configuration space.
>>
>> The same Admin command format will be used for all virtio devices. The
>> Admin command set will include 4 types of command classes:
>> 1. The generic common class
>> 2. The transport specific class
>> 3. The device specific class
>> 4. The vendor specific class
>>
>> The above mechanism will enable adding various features to the virtio
>> specification, e.g.:
>> 1. Format virtio-blk devices in various configurations (512B block size,
>>     512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>> 2. Live migration management.
>> 3. Encrypt/Decrypt descriptors.
>> 4. Virtualization management.
>> 5. Get device error logs.
>> 6. Implement advanced vendor/device/transport specific features.
>> 7. Run device health test.
>> 8. More.
>>
>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>> for the specification to become flexible and allow a wider feature set.
>> The corrent ctrl virtq that is defined for some of the virtio devices is
>> device specific and wasn't designed to be a generic virtq for
>> admininistration.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> Lots of things on this list seem to make sense when
> done from PF and affecting a VF.
> I think from this POV the generic structure should include
> a way to address one device from another.
>
>
>> ---
>>   admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++
>>   content.tex     |   4 +
>>   2 files changed, 245 insertions(+)
>>   create mode 100644 admin-virtq.tex
>>
>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>> new file mode 100644
>> index 0000000..ccec2ca
>> --- /dev/null
>> +++ b/admin-virtq.tex
>> @@ -0,0 +1,241 @@
>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>> +
>> +Admin virtqueues are used to send administrative commands to manipulate
>> +various features of the device which would not easily map into the
>> +configuration space.
>> +
>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>> +feature bit.
>> +
>> +Admin virtqueue index may vary among different device types.
>> +
>> +All commands are of the following form:
>> +
>> +\begin{lstlisting}
>> +struct virtio_admin_cmd {
>> +        /* Device-readable part */
>> +        u8 class;
>> +        u8 command;
>> +        u8 command-specific-data[];
>> +
>> +        /* Device-writable part */
>> +        u8 command-specific-result[];
>> +        u8 status_type : 4;
>> +        u8 reserved : 4;
>> +        u8 status;
>> +};
>> +
>> +/* Status type values */
>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>> +
>> +/* Generic status values */
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>> +\end{lstlisting}
>> +
>> +The \field{class}, \field{command} and \field{command-specific-data} are
>> +set by the driver, and the device sets the \field{status_type}, the
>> +\field{status} and  the \field{command-specific-result}, if needed.
>> +
>> +The virtio Admin command class codes are divided in the following form:
>> +
>> +\begin{lstlisting}
>> +/* class values that are transport, device and vendor independent */
>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>> +
>> +/* class values that are transport specific */
>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>> +
>> +/* class values that are device specific */
>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>> +
>> +/* class values that are vendor specific */
>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>> +\end{lstlisting}
>> +
>> +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set}
>> +
>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>> +support all the mandatory admin commands. A device MAY support also
>> +one or more optional admin commands.
>> +
>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set}
>> +
>> +The Common command set is a group of classes and commands within each
>> +of these classes which are transport, device and vendor independent.
>> +A mandatory class is a class that has at least one mandatory command.
>> +The Common command set is summarized in following table:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Class  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>> +\hline
>> +2-63  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
>> +
>> +This class (opcode: 0) of commands is used to query generic device
>> +information. The following table describes the commands supported for
>> +this class:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Command  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES    & M \\
>> +\hline
>> +2-255  & reserved    & - \\
>> +\hline
>> +\end{tabular}
> So there are several problems with this approach.
> One is that there is no two way negotiation.
> No way for device to know what will driver use in the end.
> This breaks things like e.g. accelerating some configurations
> but not others.
>
> Another is that everything is going through the admin vq.
> Hard for hypervisor to intervene and change just some aspects
> of the device.
>
> This is also a problem for validation, tricky
> to find out during probe what does device support and whether
> driver can work with it.


Yes, so I think it makes sense to make the admin virtqueue as a 
transport and then build other stuffs on top.


>
>
> Can't we map this stuff into memory instead?
> Maybe have an admin config structure
> that is separate from regular config?
> Only issue we can't then make it too big.
> But so far I don't see a lot of info used, most is reserved.


It should work (especially for PCI).

Thanks


>
>
>
>> +
>> +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}
>> +
>> +This mandatory command should return device identity in the following
>> +structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bytes  & Description    & M/O \\
>> +\hline \hline
>> +03:00  & VIRTIO DEVICE ID    & M \\
>> +\hline
>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>> +\hline
>> +4095:06  & reserved    & - \\
>> + \hline
>> +\end{tabular}
>> +
>> +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
>> +
>> +This mandatory command should return device identity in the following
>> +structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bytes  & Description    & M/O \\
>> +\hline \hline
>> +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
>> +\hline
>> +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
>> + \hline
>> +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
>> +\hline
>> +...  & ...    & M \\
>> +\hline
>> +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
>> +\hline
>> +4095:1024  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +The above structure is divided to 256 sections of 4B each, that will
>> +describe whether a device supports a certain class code. The
>> +following 3072B are reserved. each of the non-reserved 4B sections will
>> +follow the following data structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bits  & Description    & M/O \\
>> +\hline \hline
>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>> +\hline
>> +31:01  & reserved    & - \\
>> + \hline
>> +\end{tabular}
>> +
>> +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
>> +
>> +This class (opcode: 1) of commands is used to query supported commands
>> +for each supported device class.
>> The following table describes the commands
>> +supported for this class:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Command  & Description    & M/O \\
>> +\hline \hline
>> +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +...  & ...    & M \\
>> +\hline
>> +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
>> +\hline
>> +\end{tabular}
>> +
>> +Each command in this class, will return class identity in the following structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bytes  & Description    & M/O \\
>> +\hline \hline
>> +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
>> +\hline
>> +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
>> + \hline
>> +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
>> +\hline
>> +...  & ...    & M \\
>> +\hline
>> +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
>> +\hline
>> +4095:1024  & reserved    & - \\
>> +\hline
>> +\end{tabular}
>> +
>> +The above structure is divided to 256 sections of 4B each, that will
>> +describe whether a class supports a certain command code. The
>> +following 3072B are reserved. each of the non-reserved 4B sections will
>> +follow the following data structure:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Bits  & Description    & M/O \\
>> +\hline \hline
>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>> +\hline
>> +31:01  & reserved    & - \\
>> + \hline
>> +\end{tabular}
>> +
>> +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
>> +
>> +The Transport command set is a group of classes and commands within
>> +each of these classes which are transport specific. Refer to each
>> +transport section for more information on the supported commands.
>> +
>> +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set}
>> +
>> +The Device command set is a group of classes and commands within
>> +each of these classes which are device specific. Refer to each
>> +device section for more information on the supported commands.
>> +
>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>> +
>> +The Vendor specific command set is a group of classes and commands
>> +within each of these classes which are vendor specific. Refer to
>> +each vendor specification for more information on the supported
>> +commands.
> Here's another example.
> It's important that there is a way to make a device completely
> generic without vendor specific expensions.
> Would be hard to do here.
>
> That's a generic comment.
>
> but specifically I am very reluctant to add vendor specific stuff like
> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
> and if that is not sufficient I would like to know why
> before we add more vendor specific stuff.
>
>
>> diff --git a/content.tex b/content.tex
>> index c266fd5..d350175 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>>   types. It is RECOMMENDED that devices generate version 4
>>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>   
>> +\input{admin-virtq.tex}
>> +
>>   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>   
>>   We start with an overview of device initialization, then expand on the
>> @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>     data to be provided in driver notification and the delivery method is
>>     transport specific.
>>     For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
>> +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
>> +  the device supports administration virtqueue negotiation.
>>   
>>   \end{description}
>>   
>> -- 
>> 2.21.0

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/2/2021 5:25 AM, Jason Wang wrote:
>
> 在 2021/7/28 下午8:48, Michael S. Tsirkin 写道:
>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>> Admin virtqueues will be used to send administrative commands to
>>> manipulate various features of the device which would not easily map
>>> into the configuration space.
>>>
>>> The same Admin command format will be used for all virtio devices. The
>>> Admin command set will include 4 types of command classes:
>>> 1. The generic common class
>>> 2. The transport specific class
>>> 3. The device specific class
>>> 4. The vendor specific class
>>>
>>> The above mechanism will enable adding various features to the virtio
>>> specification, e.g.:
>>> 1. Format virtio-blk devices in various configurations (512B block 
>>> size,
>>>     512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>> 2. Live migration management.
>>> 3. Encrypt/Decrypt descriptors.
>>> 4. Virtualization management.
>>> 5. Get device error logs.
>>> 6. Implement advanced vendor/device/transport specific features.
>>> 7. Run device health test.
>>> 8. More.
>>>
>>> As virtio evolves beyond the para-virt/sw-emulated world, it's 
>>> mandatory
>>> for the specification to become flexible and allow a wider feature set.
>>> The corrent ctrl virtq that is defined for some of the virtio 
>>> devices is
>>> device specific and wasn't designed to be a generic virtq for
>>> admininistration.
>>>
>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> Lots of things on this list seem to make sense when
>> done from PF and affecting a VF.
>> I think from this POV the generic structure should include
>> a way to address one device from another.
>>
>>
>>> ---
>>>   admin-virtq.tex | 241 
>>> ++++++++++++++++++++++++++++++++++++++++++++++++
>>>   content.tex     |   4 +
>>>   2 files changed, 245 insertions(+)
>>>   create mode 100644 admin-virtq.tex
>>>
>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>> new file mode 100644
>>> index 0000000..ccec2ca
>>> --- /dev/null
>>> +++ b/admin-virtq.tex
>>> @@ -0,0 +1,241 @@
>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio 
>>> Device / Admin Virtqueues}
>>> +
>>> +Admin virtqueues are used to send administrative commands to 
>>> manipulate
>>> +various features of the device which would not easily map into the
>>> +configuration space.
>>> +
>>> +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ
>>> +feature bit.
>>> +
>>> +Admin virtqueue index may vary among different device types.
>>> +
>>> +All commands are of the following form:
>>> +
>>> +\begin{lstlisting}
>>> +struct virtio_admin_cmd {
>>> +        /* Device-readable part */
>>> +        u8 class;
>>> +        u8 command;
>>> +        u8 command-specific-data[];
>>> +
>>> +        /* Device-writable part */
>>> +        u8 command-specific-result[];
>>> +        u8 status_type : 4;
>>> +        u8 reserved : 4;
>>> +        u8 status;
>>> +};
>>> +
>>> +/* Status type values */
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC               0
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC        1
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC      2
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC    3
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC       4
>>> +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC       5
>>> +
>>> +/* Generic status values */
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_OK                     0
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR                    1
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS          2
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND        3
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR      4
>>> +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR    5
>>> +\end{lstlisting}
>>> +
>>> +The \field{class}, \field{command} and 
>>> \field{command-specific-data} are
>>> +set by the driver, and the device sets the \field{status_type}, the
>>> +\field{status} and  the \field{command-specific-result}, if needed.
>>> +
>>> +The virtio Admin command class codes are divided in the following 
>>> form:
>>> +
>>> +\begin{lstlisting}
>>> +/* class values that are transport, device and vendor independent */
>>> +#define VIRTIO_ADMIN_COMMON_CLASS_START    0
>>> +#define VIRTIO_ADMIN_COMMON_CLASS_END      63
>>> +
>>> +/* class values that are transport specific */
>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START  64
>>> +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END    127
>>> +
>>> +/* class values that are device specific */
>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_START     128
>>> +#define VIRTIO_ADMIN_DEVICE_CLASS_END       191
>>> +
>>> +/* class values that are vendor specific */
>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_START     192
>>> +#define VIRTIO_ADMIN_VENDOR_CLASS_END       255
>>> +\end{lstlisting}
>>> +
>>> +\subsection{Admin command set}\label{sec:Basic Facilities of a 
>>> Virtio Device / Admin Virtqueues / Admin command set}
>>> +
>>> +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
>>> +support all the mandatory admin commands. A device MAY support also
>>> +one or more optional admin commands.
>>> +
>>> +\subsubsection{Common command set}\label{sec:Basic Facilities of a 
>>> Virtio Device / Admin Virtqueues / Admin command set / Common 
>>> command set}
>>> +
>>> +The Common command set is a group of classes and commands within each
>>> +of these classes which are transport, device and vendor independent.
>>> +A mandatory class is a class that has at least one mandatory command.
>>> +The Common command set is summarized in following table:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Class  & Description    & M/O \\
>>> +\hline \hline
>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE    & M \\
>>> +\hline
>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS    & M \\
>>> +\hline
>>> +2-63  & reserved    & - \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +\paragraph{Discover device class}\label{sec:Basic Facilities of a 
>>> Virtio Device / Admin Virtqueues / Admin command set / Common 
>>> command set / Discover device class}
>>> +
>>> +This class (opcode: 0) of commands is used to query generic device
>>> +information. The following table describes the commands supported for
>>> +this class:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Command  & Description    & M/O \\
>>> +\hline \hline
>>> +0  & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY    & M \\
>>> +\hline
>>> +1  & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\
>>> +\hline
>>> +2-255  & reserved    & - \\
>>> +\hline
>>> +\end{tabular}
>> So there are several problems with this approach.
>> One is that there is no two way negotiation.
>> No way for device to know what will driver use in the end.
>> This breaks things like e.g. accelerating some configurations
>> but not others.
>>
>> Another is that everything is going through the admin vq.
>> Hard for hypervisor to intervene and change just some aspects
>> of the device.
>>
>> This is also a problem for validation, tricky
>> to find out during probe what does device support and whether
>> driver can work with it.
>
>
> Yes, so I think it makes sense to make the admin virtqueue as a 
> transport and then build other stuffs on top.


what transport exactly ?

it should be a virtq for devices to open.


>
>
>>
>>
>> Can't we map this stuff into memory instead?
>> Maybe have an admin config structure
>> that is separate from regular config?
>> Only issue we can't then make it too big.
>> But so far I don't see a lot of info used, most is reserved.
>
>
> It should work (especially for PCI).

you're trying to invent some mechanism that is not flexible and non trivial.

You have already a virtq - so lets use it.



>
> Thanks
>
>
>>
>>
>>
>>> +
>>> +\subparagraph{Device identity command}\label{sec:Basic Facilities 
>>> of a Virtio Device / Admin Virtqueues / Admin command set / Common 
>>> command set / Discover device class / Device identity command}
>>> +
>>> +This mandatory command should return device identity in the following
>>> +structure:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Bytes  & Description    & M/O \\
>>> +\hline \hline
>>> +03:00  & VIRTIO DEVICE ID    & M \\
>>> +\hline
>>> +05:04  & VIRTIO TRANSPORT ID    & M \\
>>> +\hline
>>> +4095:06  & reserved    & - \\
>>> + \hline
>>> +\end{tabular}
>>> +
>>> +\subparagraph{Device supported classes command}\label{sec:Basic 
>>> Facilities of a Virtio Device / Admin Virtqueues / Admin command set 
>>> / Common command set / Discover device class / Device supported 
>>> classes command}
>>> +
>>> +This mandatory command should return device identity in the following
>>> +structure:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Bytes  & Description    & M/O \\
>>> +\hline \hline
>>> +03:00  & VIRTIO_ADMIN_CLASS_0_PROPERTIES    & M \\
>>> +\hline
>>> +07:04  & VIRTIO_ADMIN_CLASS_1_PROPERTIES    & M \\
>>> + \hline
>>> +11:08  & VIRTIO_ADMIN_CLASS_2_PROPERTIES    & M \\
>>> +\hline
>>> +...  & ...    & M \\
>>> +\hline
>>> +1023:1020  & VIRTIO_ADMIN_CLASS_255_PROPERTIES    & M \\
>>> +\hline
>>> +4095:1024  & reserved    & - \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +The above structure is divided to 256 sections of 4B each, that will
>>> +describe whether a device supports a certain class code. The
>>> +following 3072B are reserved. each of the non-reserved 4B sections 
>>> will
>>> +follow the following data structure:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Bits  & Description    & M/O \\
>>> +\hline \hline
>>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>>> +\hline
>>> +31:01  & reserved    & - \\
>>> + \hline
>>> +\end{tabular}
>>> +
>>> +\paragraph{Discover device class commands class}\label{sec:Basic 
>>> Facilities of a Virtio Device / Admin Virtqueues / Admin command set 
>>> / Common command set / Discover device class commands class}
>>> +
>>> +This class (opcode: 1) of commands is used to query supported commands
>>> +for each supported device class.
>>> The following table describes the commands
>>> +supported for this class:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Command  & Description    & M/O \\
>>> +\hline \hline
>>> +0  & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY    & M \\
>>> +\hline
>>> +1  & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY    & M \\
>>> +\hline
>>> +2  & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY    & M \\
>>> +\hline
>>> +...  & ...    & M \\
>>> +\hline
>>> +255  & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY    & M \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +Each command in this class, will return class identity in the 
>>> following structure:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Bytes  & Description    & M/O \\
>>> +\hline \hline
>>> +03:00  & VIRTIO_ADMIN_COMMAND_0_PROPERTIES    & M \\
>>> +\hline
>>> +07:04  & VIRTIO_ADMIN_COMMAND_1_PROPERTIES    & M \\
>>> + \hline
>>> +11:08  & VIRTIO_ADMIN_COMMAND_2_PROPERTIES    & M \\
>>> +\hline
>>> +...  & ...    & M \\
>>> +\hline
>>> +1023:1020  & VIRTIO_ADMIN_COMMAND_255_PROPERTIES    & M \\
>>> +\hline
>>> +4095:1024  & reserved    & - \\
>>> +\hline
>>> +\end{tabular}
>>> +
>>> +The above structure is divided to 256 sections of 4B each, that will
>>> +describe whether a class supports a certain command code. The
>>> +following 3072B are reserved. each of the non-reserved 4B sections 
>>> will
>>> +follow the following data structure:
>>> +
>>> +\begin{tabular}{|l|l|l|}
>>> +\hline
>>> +Bits  & Description    & M/O \\
>>> +\hline \hline
>>> +0  & SUPPORTED (1: yes, 0:no)    & M \\
>>> +\hline
>>> +31:01  & reserved    & - \\
>>> + \hline
>>> +\end{tabular}
>>> +
>>> +\subsubsection{Transport command set}\label{sec:Basic Facilities of 
>>> a Virtio Device / Admin Virtqueues / Admin command set / Transport 
>>> command set}
>>> +
>>> +The Transport command set is a group of classes and commands within
>>> +each of these classes which are transport specific. Refer to each
>>> +transport section for more information on the supported commands.
>>> +
>>> +\subsubsection{Device command set}\label{sec:Basic Facilities of a 
>>> Virtio Device / Admin Virtqueues / Admin command set / Device 
>>> command set}
>>> +
>>> +The Device command set is a group of classes and commands within
>>> +each of these classes which are device specific. Refer to each
>>> +device section for more information on the supported commands.
>>> +
>>> +\subsubsection{Vendor specific command set}\label{sec:Basic 
>>> Facilities of a Virtio Device / Admin Virtqueues / Admin command set 
>>> / Vendor specific command set}
>>> +
>>> +The Vendor specific command set is a group of classes and commands
>>> +within each of these classes which are vendor specific. Refer to
>>> +each vendor specification for more information on the supported
>>> +commands.
>> Here's another example.
>> It's important that there is a way to make a device completely
>> generic without vendor specific expensions.
>> Would be hard to do here.
>>
>> That's a generic comment.
>>
>> but specifically I am very reluctant to add vendor specific stuff like
>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>> and if that is not sufficient I would like to know why
>> before we add more vendor specific stuff.
>>
>>
>>> diff --git a/content.tex b/content.tex
>>> index c266fd5..d350175 100644
>>> --- a/content.tex
>>> +++ b/content.tex
>>> @@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic 
>>> Facilities of a Virtio Device / Expo
>>>   types. It is RECOMMENDED that devices generate version 4
>>>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>   +\input{admin-virtq.tex}
>>> +
>>>   \chapter{General Initialization And Device 
>>> Operation}\label{sec:General Initialization And Device Operation}
>>>     We start with an overview of device initialization, then expand 
>>> on the
>>> @@ -6671,6 +6673,8 @@ \chapter{Reserved Feature 
>>> Bits}\label{sec:Reserved Feature Bits}
>>>     data to be provided in driver notification and the delivery 
>>> method is
>>>     transport specific.
>>>     For more details about driver notifications over PCI see 
>>> \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / 
>>> PCI-specific Initialization And Device Operation / Available Buffer 
>>> Notifications}.
>>> +  \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
>>> +  the device supports administration virtqueue negotiation.
>>>     \end{description}
>>>   --
>>> 2.21.0
>

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/2/2021 5:17 AM, Jason Wang wrote:
>
> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>
>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>
>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic 
>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin 
>>>>>>>> command set / Vendor specific command set}
>>>>>>>> +
>>>>>>>> +The Vendor specific command set is a group of classes and 
>>>>>>>> commands
>>>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>>>> +each vendor specification for more information on the supported
>>>>>>>> +commands.
>>>>>>> Here's another example.
>>>>>>> It's important that there is a way to make a device completely
>>>>>>> generic without vendor specific expensions.
>>>>>>> Would be hard to do here.
>>>>>>>
>>>>>>> That's a generic comment.
>>>>>>>
>>>>>>> but specifically I am very reluctant to add vendor specific 
>>>>>>> stuff like
>>>>>>> this directly in the spec. We already have 
>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>> and if that is not sufficient I would like to know why
>>>>>>> before we add more vendor specific stuff.
>>>>>> We are adding an option to add vendor specific commands. We're not
>>>>>> adding it to the spec since each vendor will have its own manual for
>>>>>> that.
>>>>> IMHO, that way madness lies. I want to be able to look at the spec 
>>>>> and
>>>>> be able to implement a compliant device or a compliant driver. If a
>>>>> vendor has some special feature they want to support, put it in the
>>>>> spec, so that it is possible to actually implement it.
>>>> You can implement a compliant device and a compliant. why do you 
>>>> think you
>>>> can't ?
>>>>
>>>> Some features are vendor/sub-vendor specific.
>
>
> Please don't do this. This breaks the efforts for making virtio as a 
> standard and generic device.
>
no it doesn't.

virtio already has vendor specific section.


>
>>>>
>>>> And as mentioned, you already added it to  the spec so I guess it 
>>>> was for a
>>>> reason.
>>> Well we basically just reserved a capability ID so if people add their
>>> vendor specific ones they don't conflict with each other or 
>>> capabilities
>>> we'll add in the future.
>>
>> it shouldn't bother you as a driver maintainer. I don't think vendor 
>> specific code should go to Linux kernel.
>>
>> No conflict will happen.
>>
>> There is no harm in having a virtio-cli tool that will pass-through 
>> commands from command line to the device using the admin queue.
>>
>> All the HW devices I know have sw tool that can access it and virtio 
>> shouldn't be different.
>>
>> Also, I can develop my own vendor specific user space driver for 
>> virtio-blk for example (lets say in SPDK framework). And this driver 
>> will be aware of this vendor specific capabilities.
>
>
> That's even worse since the driver can only work for the vendor 
> specific device which is in fact not a standard device.

this is not true.


>
> For the vendor specific capability, the spec has already said that it 
> will be only used for debugging/reporting purpose:
>
> """
>
> The optional Vendor data capability allows the device to present
> vendor-specific data to the driver, without
> conflicts, for debugging and/or reporting purposes,
> and without conflicting with standard functionality.
>
> """
>
> It looks to me what you're trying to invent may violate the spec.
>
this is also not true.

> Thanks
>
>
>>
>>
>>> And such a capability can prove useful for things like
>>> identifying the device to enable workarounds, etc.
>>>
>>> That is a far cry from a fat interface which generic drivers will 
>>> need to
>>> support and who's sole purpose is vendor specific extensions.
>>
>> No vendor specific commands will be added to generic driver. I never 
>> said this.
>>
>> I'll enable the option to create devices that support vendor specific 
>> commands.
>>
>>
>>>
>>> And yes, it also allows you to unload the generic driver and load
>>> a vendor driver. Which can then go wild if it wants to - nothing
>>> we can or want to do about it.
>>
>> I don't get your point.
>>
>> If someone would like to do crazy stuff, you can't stop him.
>>
>>
>>>
>>>
>>>> If someone would like to add its special souse to virtio device and 
>>>> we have
>>>> a flexible admin queue to manage it with a spec compliant manner, 
>>>> why not ?
>>>>
>>>> The feature might not get support by the working group so do you 
>>>> really want
>>>> to limit vendor from implementing features that are not agreed here 
>>>> on the
>>>> mailing list ?
>>>>
>>>> Adding a vendor specific command set is trivial.
>>> Supporting all this mess leter isn't. So we better make sure just how
>>> is all this vendor stuff going to be limited.
>>
>> You define a range of commands for vendors and that's it.
>>
>> I don't see a scenario to add some vendor code to virtio generic 
>> drivers.
>>
>>
>>>
>>>>>> For example, we can use virtio-cli to pass command from command 
>>>>>> line to
>>>>>> device in pass-through manner without changing driver.
>>>>> Things like that are part of the driver as in the spec sense. The 
>>>>> spec
>>>>> does not care how you actually split the implementation, or what
>>>>> controls you are giving to whom. We need a defined interface.
>>>> We have an interface. Its the admin queue.
>>>>
>>>> Virtio-cli will be a user space tool to send commands via this 
>>>> admin queue
>>>> and configure the device.
>>>>
>>>> The driver will just open a char dev to supply a channel to this 
>>>> admin queue
>>>> from user space.
>>>>
>>>> Examples:
>>>>
>>>> 1. format a virtio-blk device to add data integrity checks (T-10)
>>>>
>>>> 2. set 5 msix to VF_1, 12 msix to VF_2 and 2 msix to VF_3 before 
>>>> enabling
>>>> SRIOV
>>>>
>>>> 3. Get telemetry log
>>>>
>>>> 4. Get error log
>>>>
>>>> 5. Get vendor specific HW health report
>>>>
>>>> 6. Get generic virtio device health report
>>>>
>>>> 7. set 5 queues to VF_1, 12 queues to VF_2 and 2 queues to VF_3 before
>>>> enabling SRIOV
>>>>
>>>>
>>>> This knowledge is needed for the sys-admin and not the driver. The 
>>>> driver
>>>> will provide the channel to enable this configuration by an
>>>> orchestrator/admin.
>>>>
>>>>
>>>>
>>
>

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2561 bytes --]

On Sun, Aug 01, 2021 at 01:46:03PM +0300, Max Gurtovoy wrote:
> 
> On 7/29/2021 11:48 AM, Stefan Hajnoczi wrote:
> > On Wed, Jul 28, 2021 at 05:20:29PM +0300, Max Gurtovoy wrote:
> > > On 7/28/2021 4:42 PM, Stefan Hajnoczi wrote:
> > > > On Wed, Jul 28, 2021 at 01:59:26PM +0300, Max Gurtovoy wrote:
> > > > > On 7/28/2021 11:52 AM, Stefan Hajnoczi wrote:
> > > > > > On Tue, Jul 27, 2021 at 06:29:49PM +0300, Max Gurtovoy wrote:
> > > > > > > On 7/27/2021 5:28 PM, Cornelia Huck wrote:
> > > > > > > > On Tue, Jul 27 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > > > > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > > > > > How many different (groups of) commands can we reasonably expect? Do we
> > > > > > > > need a generic discovery command, or can we get away with a feature bit
> > > > > > > > covering each new group of commands?
> > > > > > > I can't predict the future but IMO we need a discovery command.
> > > > > > > 
> > > > > > > We have many devices and more can be added in the future.
> > > > > > A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin
> > > > > > commands would not be included in VIRTIO Feature Bits but instead
> > > > > > reported via a separate admin command that returns up to 8KB of data:
> > > > > > 
> > > > > >      struct virtio_admin_report_cmds {
> > > > > >          /* Bitmap of available admin commands [Device->Driver]
> > > > > >           * bool command_present =
> > > > > >           *        command_bits[class * 32 + command / 8] & (command % 8);
> > > > > >           */
> > > > > >          u8 command_bits[8192];
> > > > > >      };
> > > > > Yes, I divided it to multiple commands per class to cover the case we will
> > > > > need more than 1 bit to describe a command.
> > > > > 
> > > > > But I guess we can add it later on.
> > > > > 
> > > > > I think the above should be:
> > > > > 
> > > > > bool command_present = command_bits[class * 32 + command / 8] & (1 << (command % 8));
> > > > > 
> > > > > isn't it ?
> > > > You're right. I forgot to shift the bit :D.
> > > > 
> > > > > Also what do you think about renaming <class, command> to <opcode, opmod> ?
> > > > I need to understand how opcode and opmod values are used. I'm not sure.
> > > Same as class and command just with different naming.
> > Is opcode the same as command? What does opmod do?
> 
> no. opcode == class and opmod == command.

I find those terms confusing. "class" and "command" was clearer to me,
at least.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2022 bytes --]

On Sun, Aug 01, 2021 at 02:46:13AM +0300, Max Gurtovoy wrote:
> 
> On 8/1/2021 1:17 AM, Michael S. Tsirkin wrote:
> > On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
> > > On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
> > > > On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
> > > > > On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
> > > > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > > > > Admin virtqueues will be used to send administrative commands to
> > > > > > > manipulate various features of the device which would not easily map
> > > > > > > into the configuration space.
> > > > > > > 
> > > > > > > The same Admin command format will be used for all virtio devices. The
> > > > > > > Admin command set will include 4 types of command classes:
> > > > > > > 1. The generic common class
> > > > > > > 2. The transport specific class
> > > > > > > 3. The device specific class
> > > > > > > 4. The vendor specific class
> > > > > > > 
> > > > > > > The above mechanism will enable adding various features to the virtio
> > > > > > > specification, e.g.:
> > > > > > > 1. Format virtio-blk devices in various configurations (512B block size,
> > > > > > >       512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> > > > > > > 2. Live migration management.
> > > > > > > 3. Encrypt/Decrypt descriptors.
> > > > > > > 4. Virtualization management.
> > > > > > > 5. Get device error logs.
> > > > > > > 6. Implement advanced vendor/device/transport specific features.
> > > > > > > 7. Run device health test.
> > > > > > > 8. More.
> > I still don't really see what do all these things have in common?
> 
> I don't think you need to look on this in that direction.
> 
> This is a queue for administration.
> 
> Cornelia and Stefan already agreed on this approach.

We're discussing these patches and I'm trying to understand what you're
proposing. It's a bit early to say I've agreed to anything.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Mon, Aug 02 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Sun, Aug 01, 2021 at 02:46:13AM +0300, Max Gurtovoy wrote:
>> 
>> On 8/1/2021 1:17 AM, Michael S. Tsirkin wrote:
>> > On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
>> > > On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
>> > > > On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
>> > > > > On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>> > > > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>> > > > > > > Admin virtqueues will be used to send administrative commands to
>> > > > > > > manipulate various features of the device which would not easily map
>> > > > > > > into the configuration space.
>> > > > > > > 
>> > > > > > > The same Admin command format will be used for all virtio devices. The
>> > > > > > > Admin command set will include 4 types of command classes:
>> > > > > > > 1. The generic common class
>> > > > > > > 2. The transport specific class
>> > > > > > > 3. The device specific class
>> > > > > > > 4. The vendor specific class
>> > > > > > > 
>> > > > > > > The above mechanism will enable adding various features to the virtio
>> > > > > > > specification, e.g.:
>> > > > > > > 1. Format virtio-blk devices in various configurations (512B block size,
>> > > > > > >       512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>> > > > > > > 2. Live migration management.
>> > > > > > > 3. Encrypt/Decrypt descriptors.
>> > > > > > > 4. Virtualization management.
>> > > > > > > 5. Get device error logs.
>> > > > > > > 6. Implement advanced vendor/device/transport specific features.
>> > > > > > > 7. Run device health test.
>> > > > > > > 8. More.
>> > I still don't really see what do all these things have in common?
>> 
>> I don't think you need to look on this in that direction.
>> 
>> This is a queue for administration.
>> 
>> Cornelia and Stefan already agreed on this approach.
>
> We're discussing these patches and I'm trying to understand what you're
> proposing. It's a bit early to say I've agreed to anything.

For the record, same for me.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Mon, Aug 02 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 8/2/2021 5:17 AM, Jason Wang wrote:
>>
>> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>>
>>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>
>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic 
>>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin 
>>>>>>>>> command set / Vendor specific command set}
>>>>>>>>> +
>>>>>>>>> +The Vendor specific command set is a group of classes and 
>>>>>>>>> commands
>>>>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>>>>> +each vendor specification for more information on the supported
>>>>>>>>> +commands.
>>>>>>>> Here's another example.
>>>>>>>> It's important that there is a way to make a device completely
>>>>>>>> generic without vendor specific expensions.
>>>>>>>> Would be hard to do here.
>>>>>>>>
>>>>>>>> That's a generic comment.
>>>>>>>>
>>>>>>>> but specifically I am very reluctant to add vendor specific 
>>>>>>>> stuff like
>>>>>>>> this directly in the spec. We already have 
>>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>>> and if that is not sufficient I would like to know why
>>>>>>>> before we add more vendor specific stuff.
>>>>>>> We are adding an option to add vendor specific commands. We're not
>>>>>>> adding it to the spec since each vendor will have its own manual for
>>>>>>> that.
>>>>>> IMHO, that way madness lies. I want to be able to look at the spec 
>>>>>> and
>>>>>> be able to implement a compliant device or a compliant driver. If a
>>>>>> vendor has some special feature they want to support, put it in the
>>>>>> spec, so that it is possible to actually implement it.
>>>>> You can implement a compliant device and a compliant. why do you 
>>>>> think you
>>>>> can't ?

If I want to implement a device/driver, I need a standard to refer
to. How can something vendor-specific be standard? If it is useful, add
it to the virtio spec.

>>>>>
>>>>> Some features are vendor/sub-vendor specific.
>>
>>
>> Please don't do this. This breaks the efforts for making virtio as a 
>> standard and generic device.
>>
> no it doesn't.
>
> virtio already has vendor specific section.

But only as a pci-specific capability, with a very narrow scope and
constraints; that is a very far cry from what you're proposing!

>
>
>>
>>>>>
>>>>> And as mentioned, you already added it to  the spec so I guess it 
>>>>> was for a
>>>>> reason.
>>>> Well we basically just reserved a capability ID so if people add their
>>>> vendor specific ones they don't conflict with each other or 
>>>> capabilities
>>>> we'll add in the future.
>>>
>>> it shouldn't bother you as a driver maintainer. I don't think vendor 
>>> specific code should go to Linux kernel.
>>>
>>> No conflict will happen.
>>>
>>> There is no harm in having a virtio-cli tool that will pass-through 
>>> commands from command line to the device using the admin queue.
>>>
>>> All the HW devices I know have sw tool that can access it and virtio 
>>> shouldn't be different.
>>>
>>> Also, I can develop my own vendor specific user space driver for 
>>> virtio-blk for example (lets say in SPDK framework). And this driver 
>>> will be aware of this vendor specific capabilities.
>>
>>
>> That's even worse since the driver can only work for the vendor 
>> specific device which is in fact not a standard device.
>
> this is not true.

What is not true? Adding some magic vendor-specific stuff that is needed
for the device to function does make it de facto non-standard.

>
>
>>
>> For the vendor specific capability, the spec has already said that it 
>> will be only used for debugging/reporting purpose:
>>
>> """
>>
>> The optional Vendor data capability allows the device to present
>> vendor-specific data to the driver, without
>> conflicts, for debugging and/or reporting purposes,
>> and without conflicting with standard functionality.
>>
>> """
>>
>> It looks to me what you're trying to invent may violate the spec.
>>
> this is also not true.

What is not true about that? That capability has even more restrictions
than what is cited here.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/2/2021 5:34 PM, Cornelia Huck wrote:
> On Mon, Aug 02 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
>> On Sun, Aug 01, 2021 at 02:46:13AM +0300, Max Gurtovoy wrote:
>>> On 8/1/2021 1:17 AM, Michael S. Tsirkin wrote:
>>>> On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
>>>>> On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
>>>>>> On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>>> Admin virtqueues will be used to send administrative commands to
>>>>>>>>> manipulate various features of the device which would not easily map
>>>>>>>>> into the configuration space.
>>>>>>>>>
>>>>>>>>> The same Admin command format will be used for all virtio devices. The
>>>>>>>>> Admin command set will include 4 types of command classes:
>>>>>>>>> 1. The generic common class
>>>>>>>>> 2. The transport specific class
>>>>>>>>> 3. The device specific class
>>>>>>>>> 4. The vendor specific class
>>>>>>>>>
>>>>>>>>> The above mechanism will enable adding various features to the virtio
>>>>>>>>> specification, e.g.:
>>>>>>>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>>>>>>>        512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>>>>>>>> 2. Live migration management.
>>>>>>>>> 3. Encrypt/Decrypt descriptors.
>>>>>>>>> 4. Virtualization management.
>>>>>>>>> 5. Get device error logs.
>>>>>>>>> 6. Implement advanced vendor/device/transport specific features.
>>>>>>>>> 7. Run device health test.
>>>>>>>>> 8. More.
>>>> I still don't really see what do all these things have in common?
>>> I don't think you need to look on this in that direction.
>>>
>>> This is a queue for administration.
>>>
>>> Cornelia and Stefan already agreed on this approach.
>> We're discussing these patches and I'm trying to understand what you're
>> proposing. It's a bit early to say I've agreed to anything.
> For the record, same for me.

didn't you agree on admin q approach ?

I'm not talking about the command set.


>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Sun, Aug 01 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 8/1/2021 1:17 AM, Michael S. Tsirkin wrote:
>> On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
>>> On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
>>>> On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>> Admin virtqueues will be used to send administrative commands to
>>>>>>> manipulate various features of the device which would not easily map
>>>>>>> into the configuration space.
>>>>>>>
>>>>>>> The same Admin command format will be used for all virtio devices. The
>>>>>>> Admin command set will include 4 types of command classes:
>>>>>>> 1. The generic common class
>>>>>>> 2. The transport specific class
>>>>>>> 3. The device specific class
>>>>>>> 4. The vendor specific class
>>>>>>>
>>>>>>> The above mechanism will enable adding various features to the virtio
>>>>>>> specification, e.g.:
>>>>>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>>>>>       512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>>>>>> 2. Live migration management.
>>>>>>> 3. Encrypt/Decrypt descriptors.
>>>>>>> 4. Virtualization management.
>>>>>>> 5. Get device error logs.
>>>>>>> 6. Implement advanced vendor/device/transport specific features.
>>>>>>> 7. Run device health test.
>>>>>>> 8. More.
>> I still don't really see what do all these things have in common?
>
> I don't think you need to look on this in that direction.
>
> This is a queue for administration.
>
> Cornelia and Stefan already agreed on this approach.
>
> Please lets progress and not go back to the beginning.
>
>> Why are they grouped behind the same feature bit? Share same VQ?
>
> They are grouped behind an admin q.

It's fine for a variety of things to use the admin q. But I think each
feature should come with its own feature bit that depends on the admin
vq.

What actually makes sense to use the admin vq is also worth further
discussion.

>
>
>>
>>>>>>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>>>>>>> for the specification to become flexible and allow a wider feature set.
>>>>>>> The corrent ctrl virtq that is defined for some of the virtio devices is
>>>>>>> device specific and wasn't designed to be a generic virtq for
>>>>>>> admininistration.
>>>>>>>
>>>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>> Lots of things on this list seem to make sense when
>>>>>> done from PF and affecting a VF.
>>>>>> I think from this POV the generic structure should include
>>>>>> a way to address one device from another.
>>>>> This will be defined per command.
>>>>>
>>>>> For example, funcion_id will be given as command data.
>>>> Why? Sounds like a generic thing to me.
>>> Generic to a command that handles virtualization management.
>>
>> It could be that mixing up virtualization management and arbitrary
>> other management commands in the same interface is a mistake.
>
> It's not a mistake.
>
> This is the right design.

Well, we're clearly not all in agreement what the "right design"
is. Figuring that out is the whole point of this discussion.

>
> Creating a new interface for each feature is madness.
>
>
>> Or maybe not - do we want host to have ability to run health tests
>> for a VF without loading VF driver? Get error logs?
>
> This can be an optional feature.
>
> We need to define it in the future. We can't define the entire command 
> set now.
>
> We need to define the infrastructure.
>
>>
>> In fact besides migration and virtualization management the rest of
>> examples that you give all seem to be more or less device specific, with
>> the possible exception of 3. Encrypt/Decrypt descriptors.  what does
>> this one imply, exactly?
>
> For storage devices there is an option to have a self encrypted drive.
>
> Maybe we can develop encryption/decryption also  for net devices.
>
> This will be developed in the future.
>
> But the infrastructure will allow it. This is the beauty if it, you 
> create infrastructure today and add optional commands tomorrow.
>
> Nobody can think of thousands of features and commands today, and we 
> also can't push thousands of pages to the spec.
>
> Lets push 2-3 mandatory commands with the infrastructure and build new 
> features incrementally.

Why "mandatory commands"? Just make them covered by a feature bit.

>>>>>> So there are several problems with this approach.
>>>>>> One is that there is no two way negotiation.
>>>>> you negotiate the adminq feature.
>>>>>
>>>>> then you can send admin commands and discover the supported commands.
>>>>>
>>>>>> No way for device to know what will driver use in the end.
>>>>> device will be designed to support mandatory admin commands and some
>>>>> optional.
>>>>>
>>>>> It doesn't need to care whether the driver will use it or not.
>>>>>> This breaks things like e.g. accelerating some configurations
>>>>>> but not others.
>>>>> I don't understand what it breaks exactly.
>>>> Long practice taught us that it is good for device to know
>>>> what is driver going to use.
>>>> For example, some features can be implemented in hardware
>>>> and some in hypervisor software. If driver is going to use software
>>>> features then you need to switch to a slower software
>>>> implementation. Doing that dynamically at time of use is
>>>> often much harder that up-front at negotiation time.
>>> I don't think we should write specifications that should consider hypervisor
>>> SW.
>> Well considering hypervisors is clearly one of the purposes of virtio TC.
>> Look it up in the charter, section 2. Statement of Purpose.
>>
>>
>>> You might use virtio device without hypervisor at all.
>> Yes and supporting that is also clearly an objective:
>>
>> 	With the 1.1, 1.2 and future revisions of the Specification, we aim to
>> 	evolve the VIRTIO standard further, addressing such new requirements
>> 	while both continuing to honor the goals of the 1.0 Specification and
>> 	including new objectives.
>>
>>
> There is nothing in admin queue that doesn't honor old spec.
>
> Old driver will not be aware of it.

I don't see how this helps; negotiating the admin vq only tells the
device that the driver wants to use the admin vq, but not what it
actually wants to use. This is a departure from the feature negotiation
method used up to now.

>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>>>>>>> +
>>>>>>> +The Vendor specific command set is a group of classes and commands
>>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>>> +each vendor specification for more information on the supported
>>>>>>> +commands.
>>>>>> Here's another example.
>>>>>> It's important that there is a way to make a device completely
>>>>>> generic without vendor specific expensions.
>>>>>> Would be hard to do here.
>>>>>>
>>>>>> That's a generic comment.
>>>>>>
>>>>>> but specifically I am very reluctant to add vendor specific stuff like
>>>>>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>> and if that is not sufficient I would like to know why
>>>>>> before we add more vendor specific stuff.
>>>>> We are adding an option to add vendor specific commands. We're not adding it
>>>>> to the spec since each vendor will have its own manual for that.
>>>> You didn't actually answer the question though.
>>>>
>>>>> For example, we can use virtio-cli to pass command from command line to
>>>>> device in pass-through manner without changing driver.
>>>>>
>>>> Opaque strings passed all the way from guest userspace to host device?
>>>> Not sure why is that a good idea.
>>> Where did I mentioned a guest ?
>>>
>>> Virtio-cli will control a device on the same host that it runs.
>>>
>>> If it's a bare metal host so it will manage the virtio attached device.
>>>
>>> If it's a guest it will manage the device attached to the guest.
>> Opaqueness of all this is what worries at least me and Cornelia.
>
> guest is not aware of host devices.
>
> Sending raw command from Linux cmdline to a device is not something I 
> invented.
>
> If a user is aware of its HW he can use the virtio-cli tool to configure 
> its unique features.
>
> And if the user is not so smart, we can help him with adding vendor 
> classes to virtio-cli management tool.

There's something very wrong in relying on an external tool to configure
a supposedly standardized device. This spec is supposed to be
platform-agnostic. Everything must be implementable by a random OS or HW
maker, and for that, it needs to be properly specified.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/2/2021 5:51 PM, Cornelia Huck wrote:
> On Mon, Aug 02 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>
>> On 8/2/2021 5:17 AM, Jason Wang wrote:
>>> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>
>>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic
>>>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin
>>>>>>>>>> command set / Vendor specific command set}
>>>>>>>>>> +
>>>>>>>>>> +The Vendor specific command set is a group of classes and
>>>>>>>>>> commands
>>>>>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>>>>>> +each vendor specification for more information on the supported
>>>>>>>>>> +commands.
>>>>>>>>> Here's another example.
>>>>>>>>> It's important that there is a way to make a device completely
>>>>>>>>> generic without vendor specific expensions.
>>>>>>>>> Would be hard to do here.
>>>>>>>>>
>>>>>>>>> That's a generic comment.
>>>>>>>>>
>>>>>>>>> but specifically I am very reluctant to add vendor specific
>>>>>>>>> stuff like
>>>>>>>>> this directly in the spec. We already have
>>>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>>>> and if that is not sufficient I would like to know why
>>>>>>>>> before we add more vendor specific stuff.
>>>>>>>> We are adding an option to add vendor specific commands. We're not
>>>>>>>> adding it to the spec since each vendor will have its own manual for
>>>>>>>> that.
>>>>>>> IMHO, that way madness lies. I want to be able to look at the spec
>>>>>>> and
>>>>>>> be able to implement a compliant device or a compliant driver. If a
>>>>>>> vendor has some special feature they want to support, put it in the
>>>>>>> spec, so that it is possible to actually implement it.
>>>>>> You can implement a compliant device and a compliant. why do you
>>>>>> think you
>>>>>> can't ?
> If I want to implement a device/driver, I need a standard to refer
> to. How can something vendor-specific be standard? If it is useful, add
> it to the virtio spec.

again, Linux driver will be generic.

It will include a char dev to user space virtio-cli that will enable 
sending admin commands to do administration.

Nobody asked to add vendor specific code to the driver.

The spec allows vendor specific stuff today that are not usable and I'm 
trying to create a usable API without adding one line of vendor code to 
driver.

You just need to open a window for few commands like others do and like 
you already did in the spec.

NVIDIA HW internals are different from VENDOR_A and VENDOR_B.

I want to debug my HW or enable some accelerations in it.

This is the way to do it.

>>>>>> Some features are vendor/sub-vendor specific.
>>>
>>> Please don't do this. This breaks the efforts for making virtio as a
>>> standard and generic device.
>>>
>> no it doesn't.
>>
>> virtio already has vendor specific section.
> But only as a pci-specific capability, with a very narrow scope and
> constraints; that is a very far cry from what you're proposing!

why creating some pci vendor specific channel is ok but creating few 
admin command isn't ?

>
>>
>>>>>> And as mentioned, you already added it to  the spec so I guess it
>>>>>> was for a
>>>>>> reason.
>>>>> Well we basically just reserved a capability ID so if people add their
>>>>> vendor specific ones they don't conflict with each other or
>>>>> capabilities
>>>>> we'll add in the future.
>>>> it shouldn't bother you as a driver maintainer. I don't think vendor
>>>> specific code should go to Linux kernel.
>>>>
>>>> No conflict will happen.
>>>>
>>>> There is no harm in having a virtio-cli tool that will pass-through
>>>> commands from command line to the device using the admin queue.
>>>>
>>>> All the HW devices I know have sw tool that can access it and virtio
>>>> shouldn't be different.
>>>>
>>>> Also, I can develop my own vendor specific user space driver for
>>>> virtio-blk for example (lets say in SPDK framework). And this driver
>>>> will be aware of this vendor specific capabilities.
>>>
>>> That's even worse since the driver can only work for the vendor
>>> specific device which is in fact not a standard device.
>> this is not true.
> What is not true? Adding some magic vendor-specific stuff that is needed
> for the device to function does make it de facto non-standard.

This is not true because I can write a user space driver any way I want.

This discussion is repeating in cycles.

You already added vendor specific channel to the spec so why you ask me 
the need for it ?

lets progress to other sections.

>>
>>> For the vendor specific capability, the spec has already said that it
>>> will be only used for debugging/reporting purpose:
>>>
>>> """
>>>
>>> The optional Vendor data capability allows the device to present
>>> vendor-specific data to the driver, without
>>> conflicts, for debugging and/or reporting purposes,
>>> and without conflicting with standard functionality.
>>>
>>> """
>>>
>>> It looks to me what you're trying to invent may violate the spec.
>>>
>> this is also not true.
> What is not true about that? That capability has even more restrictions
> than what is cited here.
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/2/2021 6:21 PM, Cornelia Huck wrote:
> On Sun, Aug 01 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>
>> On 8/1/2021 1:17 AM, Michael S. Tsirkin wrote:
>>> On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
>>>> On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
>>>>> On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>> Admin virtqueues will be used to send administrative commands to
>>>>>>>> manipulate various features of the device which would not easily map
>>>>>>>> into the configuration space.
>>>>>>>>
>>>>>>>> The same Admin command format will be used for all virtio devices. The
>>>>>>>> Admin command set will include 4 types of command classes:
>>>>>>>> 1. The generic common class
>>>>>>>> 2. The transport specific class
>>>>>>>> 3. The device specific class
>>>>>>>> 4. The vendor specific class
>>>>>>>>
>>>>>>>> The above mechanism will enable adding various features to the virtio
>>>>>>>> specification, e.g.:
>>>>>>>> 1. Format virtio-blk devices in various configurations (512B block size,
>>>>>>>>        512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
>>>>>>>> 2. Live migration management.
>>>>>>>> 3. Encrypt/Decrypt descriptors.
>>>>>>>> 4. Virtualization management.
>>>>>>>> 5. Get device error logs.
>>>>>>>> 6. Implement advanced vendor/device/transport specific features.
>>>>>>>> 7. Run device health test.
>>>>>>>> 8. More.
>>> I still don't really see what do all these things have in common?
>> I don't think you need to look on this in that direction.
>>
>> This is a queue for administration.
>>
>> Cornelia and Stefan already agreed on this approach.
>>
>> Please lets progress and not go back to the beginning.
>>
>>> Why are they grouped behind the same feature bit? Share same VQ?
>> They are grouped behind an admin q.
> It's fine for a variety of things to use the admin q. But I think each
> feature should come with its own feature bit that depends on the admin
> vq.

There is a specific command in the admin command set that query all 
these feature bits.

There is no much bits left in the generic feature field for all the 
features we would like to add.

I mentioned only 5-6 in the above example and it will bring us to bit 46 
already.

please think of 5-10 years from today.


>
> What actually makes sense to use the admin vq is also worth further
> discussion.
>
>>
>>>>>>>> As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
>>>>>>>> for the specification to become flexible and allow a wider feature set.
>>>>>>>> The corrent ctrl virtq that is defined for some of the virtio devices is
>>>>>>>> device specific and wasn't designed to be a generic virtq for
>>>>>>>> admininistration.
>>>>>>>>
>>>>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>>> Lots of things on this list seem to make sense when
>>>>>>> done from PF and affecting a VF.
>>>>>>> I think from this POV the generic structure should include
>>>>>>> a way to address one device from another.
>>>>>> This will be defined per command.
>>>>>>
>>>>>> For example, funcion_id will be given as command data.
>>>>> Why? Sounds like a generic thing to me.
>>>> Generic to a command that handles virtualization management.
>>> It could be that mixing up virtualization management and arbitrary
>>> other management commands in the same interface is a mistake.
>> It's not a mistake.
>>
>> This is the right design.
> Well, we're clearly not all in agreement what the "right design"
> is. Figuring that out is the whole point of this discussion.

I suggested a solid infrastructure for adding any feature easily in the 
future without having half a year waste on debates.

I didn't see any other suggestion so far.

Only some half baked phrases and answers.

>
>> Creating a new interface for each feature is madness.
>>
>>
>>> Or maybe not - do we want host to have ability to run health tests
>>> for a VF without loading VF driver? Get error logs?
>> This can be an optional feature.
>>
>> We need to define it in the future. We can't define the entire command
>> set now.
>>
>> We need to define the infrastructure.
>>
>>> In fact besides migration and virtualization management the rest of
>>> examples that you give all seem to be more or less device specific, with
>>> the possible exception of 3. Encrypt/Decrypt descriptors.  what does
>>> this one imply, exactly?
>> For storage devices there is an option to have a self encrypted drive.
>>
>> Maybe we can develop encryption/decryption also  for net devices.
>>
>> This will be developed in the future.
>>
>> But the infrastructure will allow it. This is the beauty if it, you
>> create infrastructure today and add optional commands tomorrow.
>>
>> Nobody can think of thousands of features and commands today, and we
>> also can't push thousands of pages to the spec.
>>
>> Lets push 2-3 mandatory commands with the infrastructure and build new
>> features incrementally.
> Why "mandatory commands"? Just make them covered by a feature bit.

mandatory for anyone that supports admin q.

They are covered by admin q bit.


>
>>>>>>> So there are several problems with this approach.
>>>>>>> One is that there is no two way negotiation.
>>>>>> you negotiate the adminq feature.
>>>>>>
>>>>>> then you can send admin commands and discover the supported commands.
>>>>>>
>>>>>>> No way for device to know what will driver use in the end.
>>>>>> device will be designed to support mandatory admin commands and some
>>>>>> optional.
>>>>>>
>>>>>> It doesn't need to care whether the driver will use it or not.
>>>>>>> This breaks things like e.g. accelerating some configurations
>>>>>>> but not others.
>>>>>> I don't understand what it breaks exactly.
>>>>> Long practice taught us that it is good for device to know
>>>>> what is driver going to use.
>>>>> For example, some features can be implemented in hardware
>>>>> and some in hypervisor software. If driver is going to use software
>>>>> features then you need to switch to a slower software
>>>>> implementation. Doing that dynamically at time of use is
>>>>> often much harder that up-front at negotiation time.
>>>> I don't think we should write specifications that should consider hypervisor
>>>> SW.
>>> Well considering hypervisors is clearly one of the purposes of virtio TC.
>>> Look it up in the charter, section 2. Statement of Purpose.
>>>
>>>
>>>> You might use virtio device without hypervisor at all.
>>> Yes and supporting that is also clearly an objective:
>>>
>>> 	With the 1.1, 1.2 and future revisions of the Specification, we aim to
>>> 	evolve the VIRTIO standard further, addressing such new requirements
>>> 	while both continuing to honor the goals of the 1.0 Specification and
>>> 	including new objectives.
>>>
>>>
>> There is nothing in admin queue that doesn't honor old spec.
>>
>> Old driver will not be aware of it.
> I don't see how this helps; negotiating the admin vq only tells the
> device that the driver wants to use the admin vq, but not what it
> actually wants to use. This is a departure from the feature negotiation
> method used up to now.

The driver can use all the admin q commands the device is capable to 
support.

The HW tells the cap of admin queue and this is negotiated.

Then all the supported commands should be usable and supported by the 
device.

>
>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
>>>>>>>> +
>>>>>>>> +The Vendor specific command set is a group of classes and commands
>>>>>>>> +within each of these classes which are vendor specific. Refer to
>>>>>>>> +each vendor specification for more information on the supported
>>>>>>>> +commands.
>>>>>>> Here's another example.
>>>>>>> It's important that there is a way to make a device completely
>>>>>>> generic without vendor specific expensions.
>>>>>>> Would be hard to do here.
>>>>>>>
>>>>>>> That's a generic comment.
>>>>>>>
>>>>>>> but specifically I am very reluctant to add vendor specific stuff like
>>>>>>> this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>> and if that is not sufficient I would like to know why
>>>>>>> before we add more vendor specific stuff.
>>>>>> We are adding an option to add vendor specific commands. We're not adding it
>>>>>> to the spec since each vendor will have its own manual for that.
>>>>> You didn't actually answer the question though.
>>>>>
>>>>>> For example, we can use virtio-cli to pass command from command line to
>>>>>> device in pass-through manner without changing driver.
>>>>>>
>>>>> Opaque strings passed all the way from guest userspace to host device?
>>>>> Not sure why is that a good idea.
>>>> Where did I mentioned a guest ?
>>>>
>>>> Virtio-cli will control a device on the same host that it runs.
>>>>
>>>> If it's a bare metal host so it will manage the virtio attached device.
>>>>
>>>> If it's a guest it will manage the device attached to the guest.
>>> Opaqueness of all this is what worries at least me and Cornelia.
>> guest is not aware of host devices.
>>
>> Sending raw command from Linux cmdline to a device is not something I
>> invented.
>>
>> If a user is aware of its HW he can use the virtio-cli tool to configure
>> its unique features.
>>
>> And if the user is not so smart, we can help him with adding vendor
>> classes to virtio-cli management tool.
> There's something very wrong in relying on an external tool to configure
> a supposedly standardized device. This spec is supposed to be
> platform-agnostic. Everything must be implementable by a random OS or HW
> maker, and for that, it needs to be properly specified.

Where do I rely on it ? I said it's an option.

How would you like to format a virtio block device without tools ? the 
driver can't do this job out of the blue.

It is platform agnostic.

And it is a proper spec.

Opening a vendor channel is already there.

Nothing new here.

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2794 bytes --]

On Mon, Aug 02, 2021 at 05:58:19PM +0300, Max Gurtovoy wrote:
> 
> On 8/2/2021 5:34 PM, Cornelia Huck wrote:
> > On Mon, Aug 02 2021, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > 
> > > On Sun, Aug 01, 2021 at 02:46:13AM +0300, Max Gurtovoy wrote:
> > > > On 8/1/2021 1:17 AM, Michael S. Tsirkin wrote:
> > > > > On Sat, Jul 31, 2021 at 02:53:45PM +0300, Max Gurtovoy wrote:
> > > > > > On 7/30/2021 10:36 AM, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Jul 29, 2021 at 05:51:07PM +0300, Max Gurtovoy wrote:
> > > > > > > > On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
> > > > > > > > > On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
> > > > > > > > > > Admin virtqueues will be used to send administrative commands to
> > > > > > > > > > manipulate various features of the device which would not easily map
> > > > > > > > > > into the configuration space.
> > > > > > > > > > 
> > > > > > > > > > The same Admin command format will be used for all virtio devices. The
> > > > > > > > > > Admin command set will include 4 types of command classes:
> > > > > > > > > > 1. The generic common class
> > > > > > > > > > 2. The transport specific class
> > > > > > > > > > 3. The device specific class
> > > > > > > > > > 4. The vendor specific class
> > > > > > > > > > 
> > > > > > > > > > The above mechanism will enable adding various features to the virtio
> > > > > > > > > > specification, e.g.:
> > > > > > > > > > 1. Format virtio-blk devices in various configurations (512B block size,
> > > > > > > > > >        512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
> > > > > > > > > > 2. Live migration management.
> > > > > > > > > > 3. Encrypt/Decrypt descriptors.
> > > > > > > > > > 4. Virtualization management.
> > > > > > > > > > 5. Get device error logs.
> > > > > > > > > > 6. Implement advanced vendor/device/transport specific features.
> > > > > > > > > > 7. Run device health test.
> > > > > > > > > > 8. More.
> > > > > I still don't really see what do all these things have in common?
> > > > I don't think you need to look on this in that direction.
> > > > 
> > > > This is a queue for administration.
> > > > 
> > > > Cornelia and Stefan already agreed on this approach.
> > > We're discussing these patches and I'm trying to understand what you're
> > > proposing. It's a bit early to say I've agreed to anything.
> > For the record, same for me.
> 
> didn't you agree on admin q approach ?
> 
> I'm not talking about the command set.

I like the idea of a common admin virtqueue but it's still necessary to
answer questions that have been raised. If someone identifies a
significant issue or suggests an alternative then that needs to be dealt
with on its technical merit.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Mon, Aug 02, 2021 at 07:03:11PM +0300, Max Gurtovoy wrote:
> There is no much bits left in the generic feature field for all the features
> we would like to add.
> I mentioned only 5-6 in the above example and it will bring us to bit 46
> already.
> 
> please think of 5-10 years from today.
> 

IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually
pretty important to make sure the feature negotiation works well
and covers relevant usecases. If we have limitations preventing that
I'd like to at least try to fix that not replacing feature negotiation with
something else.

> 
> > 
> > What actually makes sense to use the admin vq is also worth further
> > discussion.
> > 
> > > 
> > > > > > > > > As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory
> > > > > > > > > for the specification to become flexible and allow a wider feature set.
> > > > > > > > > The corrent ctrl virtq that is defined for some of the virtio devices is
> > > > > > > > > device specific and wasn't designed to be a generic virtq for
> > > > > > > > > admininistration.
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > > > > > Lots of things on this list seem to make sense when
> > > > > > > > done from PF and affecting a VF.
> > > > > > > > I think from this POV the generic structure should include
> > > > > > > > a way to address one device from another.
> > > > > > > This will be defined per command.
> > > > > > > 
> > > > > > > For example, funcion_id will be given as command data.
> > > > > > Why? Sounds like a generic thing to me.
> > > > > Generic to a command that handles virtualization management.
> > > > It could be that mixing up virtualization management and arbitrary
> > > > other management commands in the same interface is a mistake.
> > > It's not a mistake.
> > > 
> > > This is the right design.
> > Well, we're clearly not all in agreement what the "right design"
> > is. Figuring that out is the whole point of this discussion.
> 
> I suggested a solid infrastructure for adding any feature easily in the
> future without having half a year waste on debates.

Well we don't really normally waste time on debating adding feature bits.
We do that a lot without a lot of fuss.

-- 
MST

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Mon, Aug 02, 2021 at 10:25:33AM +0800, Jason Wang wrote:
> Yes, so I think it makes sense to make the admin virtqueue as a transport
> and then build other stuffs on top.

As an alternative to pci/mmio etc? We'd need to better define what
we are trying to achieve here ...

-- 
MST

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Michael S. Tsirkin]

On Mon, Aug 02, 2021 at 06:27:58PM +0300, Max Gurtovoy wrote:
> The spec allows vendor specific stuff today that are not usable

Maybe start there and explain what are the issues you are
trying to address. I think there's a wide consensus in the
TC that significant features should be part of the spec
not a vendor specific thing. Existing vendor specific
capability is thus geared towards enabling minor extensions
such as bug work arounds.

> trying to create a usable API without adding one line of vendor code to
> driver.

Driver in the virtio terminology is taken widely to mean all software
interacting with the device through the virtio interface. Your
virtio-cli tool would be part of the driver.
-- 
MST

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/8/3 上午1:07, Michael S. Tsirkin 写道:
> On Mon, Aug 02, 2021 at 10:25:33AM +0800, Jason Wang wrote:
>> Yes, so I think it makes sense to make the admin virtqueue as a transport
>> and then build other stuffs on top.
> As an alternative to pci/mmio etc? We'd need to better define what
> we are trying to achieve here ...
>

I've posted an RFC for this.

The motivation is to slice the device at the virtio level (transport 
independent).

But it could be extended for other usage.

Thanks

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/8/2 下午11:27, Max Gurtovoy 写道:
>
> On 8/2/2021 5:51 PM, Cornelia Huck wrote:
>> On Mon, Aug 02 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>
>>> On 8/2/2021 5:17 AM, Jason Wang wrote:
>>>> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>>>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>>>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>>
>>>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic
>>>>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin
>>>>>>>>>>> command set / Vendor specific command set}
>>>>>>>>>>> +
>>>>>>>>>>> +The Vendor specific command set is a group of classes and
>>>>>>>>>>> commands
>>>>>>>>>>> +within each of these classes which are vendor specific. 
>>>>>>>>>>> Refer to
>>>>>>>>>>> +each vendor specification for more information on the 
>>>>>>>>>>> supported
>>>>>>>>>>> +commands.
>>>>>>>>>> Here's another example.
>>>>>>>>>> It's important that there is a way to make a device completely
>>>>>>>>>> generic without vendor specific expensions.
>>>>>>>>>> Would be hard to do here.
>>>>>>>>>>
>>>>>>>>>> That's a generic comment.
>>>>>>>>>>
>>>>>>>>>> but specifically I am very reluctant to add vendor specific
>>>>>>>>>> stuff like
>>>>>>>>>> this directly in the spec. We already have
>>>>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>>>>> and if that is not sufficient I would like to know why
>>>>>>>>>> before we add more vendor specific stuff.
>>>>>>>>> We are adding an option to add vendor specific commands. We're 
>>>>>>>>> not
>>>>>>>>> adding it to the spec since each vendor will have its own 
>>>>>>>>> manual for
>>>>>>>>> that.
>>>>>>>> IMHO, that way madness lies. I want to be able to look at the spec
>>>>>>>> and
>>>>>>>> be able to implement a compliant device or a compliant driver. 
>>>>>>>> If a
>>>>>>>> vendor has some special feature they want to support, put it in 
>>>>>>>> the
>>>>>>>> spec, so that it is possible to actually implement it.
>>>>>>> You can implement a compliant device and a compliant. why do you
>>>>>>> think you
>>>>>>> can't ?
>> If I want to implement a device/driver, I need a standard to refer
>> to. How can something vendor-specific be standard? If it is useful, add
>> it to the virtio spec.
>
> again, Linux driver will be generic.
>
> It will include a char dev to user space virtio-cli that will enable 
> sending admin commands to do administration.
>
> Nobody asked to add vendor specific code to the driver.


It's not about the general driver but about the general device. We want 
to present a general standard device which means we can migrate virtio 
devices among different vendors.

If each vendor choose to implement their own vendor specific features 
via vendor specific commands that are not ruled by the spec, there will 
be little value of the spec.


>
> The spec allows vendor specific stuff today that are not usable and 
> I'm trying to create a usable API without adding one line of vendor 
> code to driver.
>
> You just need to open a window for few commands like others do and 
> like you already did in the spec.
>
> NVIDIA HW internals are different from VENDOR_A and VENDOR_B.
>
> I want to debug my HW or enable some accelerations in it.


A big question is that, on what layer do you want to provide those 
functions?

1) virtio level
2) vendor level

For 1), it needs to be done in the spec instead of introducing backdoor 
for vendor specific command

For 2), virtio keeps simple as is, functionality is provided in a vendor 
specific device interface. In this case, you can't use virtio-pci 
directly, then you need present virtio on the upper layer (with some 
software meditation)


>
> This is the way to do it.


Again, for debug or acceleration, if you want to do it at the virtio 
level, you need generalize those features and define them in the spec.

Maybe we can start the work on generalizing the debug facility first. 
(adding Laurent who is working on this).

Tahnks

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Mon, Aug 02 2021, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Aug 02, 2021 at 07:03:11PM +0300, Max Gurtovoy wrote:
>> There is no much bits left in the generic feature field for all the features
>> we would like to add.
>> I mentioned only 5-6 in the above example and it will bring us to bit 46
>> already.
>> 
>> please think of 5-10 years from today.
>> 
>
> IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually
> pretty important to make sure the feature negotiation works well
> and covers relevant usecases. If we have limitations preventing that
> I'd like to at least try to fix that not replacing feature negotiation with
> something else.

I recall that we had a discussion about that years ago when we
introduced VERSION_1; we explicitly agreed that we can extend features
beyond 64 bit once we need it. (A quick search did not turn up that mail
exchange, though.)


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/8/3 下午2:28, Cornelia Huck 写道:
> On Mon, Aug 02 2021, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
>> On Mon, Aug 02, 2021 at 07:03:11PM +0300, Max Gurtovoy wrote:
>>> There is no much bits left in the generic feature field for all the features
>>> we would like to add.
>>> I mentioned only 5-6 in the above example and it will bring us to bit 46
>>> already.
>>>
>>> please think of 5-10 years from today.
>>>
>> IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually
>> pretty important to make sure the feature negotiation works well
>> and covers relevant usecases. If we have limitations preventing that
>> I'd like to at least try to fix that not replacing feature negotiation with
>> something else.
> I recall that we had a discussion about that years ago when we
> introduced VERSION_1; we explicitly agreed that we can extend features
> beyond 64 bit once we need it. (A quick search did not turn up that mail
> exchange, though.)
>

E.g PCI transport has feature_select.

         le32 device_feature_select;     /* read-write */
         le32 device_feature;            /* read-only for driver */
         le32 driver_feature_select;     /* read-write */
         le32 driver_feature;            /* read-write */

Technically it can support 32*32 different features.

Thanks

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Tue, Aug 03 2021, Jason Wang <jasowang@redhat.com> wrote:

> 在 2021/8/3 下午2:28, Cornelia Huck 写道:
>> On Mon, Aug 02 2021, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>
>>> On Mon, Aug 02, 2021 at 07:03:11PM +0300, Max Gurtovoy wrote:
>>>> There is no much bits left in the generic feature field for all the features
>>>> we would like to add.
>>>> I mentioned only 5-6 in the above example and it will bring us to bit 46
>>>> already.
>>>>
>>>> please think of 5-10 years from today.
>>>>
>>> IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually
>>> pretty important to make sure the feature negotiation works well
>>> and covers relevant usecases. If we have limitations preventing that
>>> I'd like to at least try to fix that not replacing feature negotiation with
>>> something else.
>> I recall that we had a discussion about that years ago when we
>> introduced VERSION_1; we explicitly agreed that we can extend features
>> beyond 64 bit once we need it. (A quick search did not turn up that mail
>> exchange, though.)
>>
>
> E.g PCI transport has feature_select.
>
>          le32 device_feature_select;     /* read-write */
>          le32 device_feature;            /* read-only for driver */
>          le32 driver_feature_select;     /* read-write */
>          le32 driver_feature;            /* read-write */
>
> Technically it can support 32*32 different features.

MMIO uses a similar scheme; CCW uses

struct virtio_feature_desc {
       le32 features;
       u8 index;
};

so there's plenty of room to spare.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/3/2021 9:51 AM, Cornelia Huck wrote:
> On Tue, Aug 03 2021, Jason Wang <jasowang@redhat.com> wrote:
>
>> 在 2021/8/3 下午2:28, Cornelia Huck 写道:
>>> On Mon, Aug 02 2021, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>
>>>> On Mon, Aug 02, 2021 at 07:03:11PM +0300, Max Gurtovoy wrote:
>>>>> There is no much bits left in the generic feature field for all the features
>>>>> we would like to add.
>>>>> I mentioned only 5-6 in the above example and it will bring us to bit 46
>>>>> already.
>>>>>
>>>>> please think of 5-10 years from today.
>>>>>
>>>> IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually
>>>> pretty important to make sure the feature negotiation works well
>>>> and covers relevant usecases. If we have limitations preventing that
>>>> I'd like to at least try to fix that not replacing feature negotiation with
>>>> something else.
>>> I recall that we had a discussion about that years ago when we
>>> introduced VERSION_1; we explicitly agreed that we can extend features
>>> beyond 64 bit once we need it. (A quick search did not turn up that mail
>>> exchange, though.)
>>>
>> E.g PCI transport has feature_select.
>>
>>           le32 device_feature_select;     /* read-write */
>>           le32 device_feature;            /* read-only for driver */
>>           le32 driver_feature_select;     /* read-write */
>>           le32 driver_feature;            /* read-write */
>>
>> Technically it can support 32*32 different features.
> MMIO uses a similar scheme; CCW uses
>
> struct virtio_feature_desc {
>         le32 features;
>         u8 index;
> };
>
> so there's plenty of room to spare.

Feature negotiation can be done in admin q level. You don't want this 
branches in the spec per each transport or device type.

We have an admin command to get all device admin q features and we can 
create a command to set all the driver supported.

Again, I don't see the point of doing that. If a device is supporting 
command_A, the driver should be able to use it. Telling the device "hi, 
I'm going to use command_A in the future" is redundant.

The device knows already how to handle it.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/3/2021 6:39 AM, Jason Wang wrote:
>
> 在 2021/8/2 下午11:27, Max Gurtovoy 写道:
>>
>> On 8/2/2021 5:51 PM, Cornelia Huck wrote:
>>> On Mon, Aug 02 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>
>>>> On 8/2/2021 5:17 AM, Jason Wang wrote:
>>>>> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>>>>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>>>>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>>>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>>>
>>>>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic
>>>>>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin
>>>>>>>>>>>> command set / Vendor specific command set}
>>>>>>>>>>>> +
>>>>>>>>>>>> +The Vendor specific command set is a group of classes and
>>>>>>>>>>>> commands
>>>>>>>>>>>> +within each of these classes which are vendor specific. 
>>>>>>>>>>>> Refer to
>>>>>>>>>>>> +each vendor specification for more information on the 
>>>>>>>>>>>> supported
>>>>>>>>>>>> +commands.
>>>>>>>>>>> Here's another example.
>>>>>>>>>>> It's important that there is a way to make a device completely
>>>>>>>>>>> generic without vendor specific expensions.
>>>>>>>>>>> Would be hard to do here.
>>>>>>>>>>>
>>>>>>>>>>> That's a generic comment.
>>>>>>>>>>>
>>>>>>>>>>> but specifically I am very reluctant to add vendor specific
>>>>>>>>>>> stuff like
>>>>>>>>>>> this directly in the spec. We already have
>>>>>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>>>>>> and if that is not sufficient I would like to know why
>>>>>>>>>>> before we add more vendor specific stuff.
>>>>>>>>>> We are adding an option to add vendor specific commands. 
>>>>>>>>>> We're not
>>>>>>>>>> adding it to the spec since each vendor will have its own 
>>>>>>>>>> manual for
>>>>>>>>>> that.
>>>>>>>>> IMHO, that way madness lies. I want to be able to look at the 
>>>>>>>>> spec
>>>>>>>>> and
>>>>>>>>> be able to implement a compliant device or a compliant driver. 
>>>>>>>>> If a
>>>>>>>>> vendor has some special feature they want to support, put it 
>>>>>>>>> in the
>>>>>>>>> spec, so that it is possible to actually implement it.
>>>>>>>> You can implement a compliant device and a compliant. why do you
>>>>>>>> think you
>>>>>>>> can't ?
>>> If I want to implement a device/driver, I need a standard to refer
>>> to. How can something vendor-specific be standard? If it is useful, add
>>> it to the virtio spec.
>>
>> again, Linux driver will be generic.
>>
>> It will include a char dev to user space virtio-cli that will enable 
>> sending admin commands to do administration.
>>
>> Nobody asked to add vendor specific code to the driver.
>
>
> It's not about the general driver but about the general device. We 
> want to present a general standard device which means we can migrate 
> virtio devices among different vendors.

I don't know if it's reasonable.

You're looking on a device as a SW component since it was like this from 
day 1. But it's not. It's HW, and HW has its own internal state and 
implementation that is known only to the vendor.

>
>
> If each vendor choose to implement their own vendor specific features 
> via vendor specific commands that are not ruled by the spec, there 
> will be little value of the spec.

Not at all.

I'm very surprised that you are negative to this RFC instead of working 
on defining admin commands to stop/start queues and other commands that 
can be helpful for vDPA migration.

>
>
>>
>> The spec allows vendor specific stuff today that are not usable and 
>> I'm trying to create a usable API without adding one line of vendor 
>> code to driver.
>>
>> You just need to open a window for few commands like others do and 
>> like you already did in the spec.
>>
>> NVIDIA HW internals are different from VENDOR_A and VENDOR_B.
>>
>> I want to debug my HW or enable some accelerations in it.
>
>
> A big question is that, on what layer do you want to provide those 
> functions?
>
> 1) virtio level
> 2) vendor level
>
> For 1), it needs to be done in the spec instead of introducing 
> backdoor for vendor specific command

I'm repeating myself. You already have this backdoor for workarounding bugs.

I don't ask for workarounds. I want to create a clean spec definition 
for vendor to debug/control their special souse.

>
> For 2), virtio keeps simple as is, functionality is provided in a 
> vendor specific device interface. In this case, you can't use 
> virtio-pci directly, then you need present virtio on the upper layer 
> (with some software meditation)

you mean VDPA ?

>
>
>>
>> This is the way to do it.
>
>
> Again, for debug or acceleration, if you want to do it at the virtio 
> level, you need generalize those features and define them in the spec.
>
> Maybe we can start the work on generalizing the debug facility first. 
> (adding Laurent who is working on this).

Virtio debug stuff can be generalized.

Vendor debug stuff should use vendor specific commands. A HW device has 
more than virtio state. It might have FW that controls it.  It might be 
an SoC with a lot of components that are used to accelerate this device.

It's not always a QEMU code that implements virtio device.

>
> Tahnks
>

Re: [virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Cornelia Huck]

On Tue, Aug 03 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> On 8/3/2021 9:51 AM, Cornelia Huck wrote:
>> On Tue, Aug 03 2021, Jason Wang <jasowang@redhat.com> wrote:
>>
>>> 在 2021/8/3 下午2:28, Cornelia Huck 写道:
>>>> On Mon, Aug 02 2021, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>
>>>>> On Mon, Aug 02, 2021 at 07:03:11PM +0300, Max Gurtovoy wrote:
>>>>>> There is no much bits left in the generic feature field for all the features
>>>>>> we would like to add.
>>>>>> I mentioned only 5-6 in the above example and it will bring us to bit 46
>>>>>> already.
>>>>>>
>>>>>> please think of 5-10 years from today.
>>>>>>
>>>>> IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually
>>>>> pretty important to make sure the feature negotiation works well
>>>>> and covers relevant usecases. If we have limitations preventing that
>>>>> I'd like to at least try to fix that not replacing feature negotiation with
>>>>> something else.
>>>> I recall that we had a discussion about that years ago when we
>>>> introduced VERSION_1; we explicitly agreed that we can extend features
>>>> beyond 64 bit once we need it. (A quick search did not turn up that mail
>>>> exchange, though.)
>>>>
>>> E.g PCI transport has feature_select.
>>>
>>>           le32 device_feature_select;     /* read-write */
>>>           le32 device_feature;            /* read-only for driver */
>>>           le32 driver_feature_select;     /* read-write */
>>>           le32 driver_feature;            /* read-write */
>>>
>>> Technically it can support 32*32 different features.
>> MMIO uses a similar scheme; CCW uses
>>
>> struct virtio_feature_desc {
>>         le32 features;
>>         u8 index;
>> };
>>
>> so there's plenty of room to spare.
>
> Feature negotiation can be done in admin q level. You don't want this 
> branches in the spec per each transport or device type.

This has nothing to do with transports. We only wanted to demonstrate
that all of the transports can be extended, so feature bits are not a
scarce resource.

>
> We have an admin command to get all device admin q features and we can 
> create a command to set all the driver supported.
>
> Again, I don't see the point of doing that. If a device is supporting 
> command_A, the driver should be able to use it. Telling the device "hi, 
> I'm going to use command_A in the future" is redundant.
>
> The device knows already how to handle it.

If the driver negotiates a certain feature, it can mean that the device
can make certain optimizations.

That's not a particularily exotic pattern.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/8/3 下午4:32, Max Gurtovoy 写道:
>
> On 8/3/2021 6:39 AM, Jason Wang wrote:
>>
>> 在 2021/8/2 下午11:27, Max Gurtovoy 写道:
>>>
>>> On 8/2/2021 5:51 PM, Cornelia Huck wrote:
>>>> On Mon, Aug 02 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>
>>>>> On 8/2/2021 5:17 AM, Jason Wang wrote:
>>>>>> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>>>>>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>>>>>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>>>>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>>>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic
>>>>>>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin
>>>>>>>>>>>>> command set / Vendor specific command set}
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +The Vendor specific command set is a group of classes and
>>>>>>>>>>>>> commands
>>>>>>>>>>>>> +within each of these classes which are vendor specific. 
>>>>>>>>>>>>> Refer to
>>>>>>>>>>>>> +each vendor specification for more information on the 
>>>>>>>>>>>>> supported
>>>>>>>>>>>>> +commands.
>>>>>>>>>>>> Here's another example.
>>>>>>>>>>>> It's important that there is a way to make a device completely
>>>>>>>>>>>> generic without vendor specific expensions.
>>>>>>>>>>>> Would be hard to do here.
>>>>>>>>>>>>
>>>>>>>>>>>> That's a generic comment.
>>>>>>>>>>>>
>>>>>>>>>>>> but specifically I am very reluctant to add vendor specific
>>>>>>>>>>>> stuff like
>>>>>>>>>>>> this directly in the spec. We already have
>>>>>>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>>>>>>> and if that is not sufficient I would like to know why
>>>>>>>>>>>> before we add more vendor specific stuff.
>>>>>>>>>>> We are adding an option to add vendor specific commands. 
>>>>>>>>>>> We're not
>>>>>>>>>>> adding it to the spec since each vendor will have its own 
>>>>>>>>>>> manual for
>>>>>>>>>>> that.
>>>>>>>>>> IMHO, that way madness lies. I want to be able to look at the 
>>>>>>>>>> spec
>>>>>>>>>> and
>>>>>>>>>> be able to implement a compliant device or a compliant 
>>>>>>>>>> driver. If a
>>>>>>>>>> vendor has some special feature they want to support, put it 
>>>>>>>>>> in the
>>>>>>>>>> spec, so that it is possible to actually implement it.
>>>>>>>>> You can implement a compliant device and a compliant. why do you
>>>>>>>>> think you
>>>>>>>>> can't ?
>>>> If I want to implement a device/driver, I need a standard to refer
>>>> to. How can something vendor-specific be standard? If it is useful, 
>>>> add
>>>> it to the virtio spec.
>>>
>>> again, Linux driver will be generic.
>>>
>>> It will include a char dev to user space virtio-cli that will enable 
>>> sending admin commands to do administration.
>>>
>>> Nobody asked to add vendor specific code to the driver.
>>
>>
>> It's not about the general driver but about the general device. We 
>> want to present a general standard device which means we can migrate 
>> virtio devices among different vendors.
>
> I don't know if it's reasonable.
>
> You're looking on a device as a SW component since it was like this 
> from day 1. But it's not. It's HW, and HW has its own internal state 
> and implementation that is known only to the vendor.


Spec doesn't forbid device internal state. But it cares the state that 
is visible to the driver. If we had vendor specific state, it would be 
very hard to claim to be a standard virtio device.


>
>>
>>
>> If each vendor choose to implement their own vendor specific features 
>> via vendor specific commands that are not ruled by the spec, there 
>> will be little value of the spec.
>
> Not at all.
>
> I'm very surprised that you are negative to this RFC instead of 
> working on defining admin commands to stop/start queues and other 
> commands that can be helpful for vDPA migration.


I think there are some misunderstanding here. I'm not negative to admin 
virtqueue, actually such concept has been discussed by the end of last 
year. If you search the archive, I've posted RFC for net specific admin 
virtqueue sometime early this year.

But what I want to be negative is the vendor specific commands/features 
through the admin virtqueue. The admin virtqueue should be used for 
general facility like (device stop and migration as you said above).


>
>>
>>
>>>
>>> The spec allows vendor specific stuff today that are not usable and 
>>> I'm trying to create a usable API without adding one line of vendor 
>>> code to driver.
>>>
>>> You just need to open a window for few commands like others do and 
>>> like you already did in the spec.
>>>
>>> NVIDIA HW internals are different from VENDOR_A and VENDOR_B.
>>>
>>> I want to debug my HW or enable some accelerations in it.
>>
>>
>> A big question is that, on what layer do you want to provide those 
>> functions?
>>
>> 1) virtio level
>> 2) vendor level
>>
>> For 1), it needs to be done in the spec instead of introducing 
>> backdoor for vendor specific command
>
> I'm repeating myself. You already have this backdoor for workarounding 
> bugs.


To say the truth, I don't like that. And it was probably too late to 
fix. And I am not aware of any vendor that implements the VENDOR_CFG 
capability.

But we can do better now by not introducing any new vendor specific stuffs.


>
> I don't ask for workarounds. I want to create a clean spec definition 
> for vendor to debug/control their special souse.


That's fine, but if it can't be generalized, it's better to be done at 
the layer under virtio.


>
>>
>> For 2), virtio keeps simple as is, functionality is provided in a 
>> vendor specific device interface. In this case, you can't use 
>> virtio-pci directly, then you need present virtio on the upper layer 
>> (with some software meditation)
>
> you mean VDPA ?


vDPA is one approach but not all.

Another example is the IFCVF, it has a dedicated bar for live migration 
which is not a part of the virtio-pci capability. (I'm not saying it's a 
good design, but it's indeed another approach).


>
>>
>>
>>>
>>> This is the way to do it.
>>
>>
>> Again, for debug or acceleration, if you want to do it at the virtio 
>> level, you need generalize those features and define them in the spec.
>>
>> Maybe we can start the work on generalizing the debug facility first. 
>> (adding Laurent who is working on this).
>
> Virtio debug stuff can be generalized.
>
> Vendor debug stuff should use vendor specific commands. A HW device 
> has more than virtio state. It might have FW that controls it.  It 
> might be an SoC with a lot of components that are used to accelerate 
> this device.


If the states/statistics/counters are highly vendor specific, let's do 
that under virtio(pci). Such method has been used by other vendors.


>
> It's not always a QEMU code that implements virtio device.


It's unrelated to whether or not it's a software device. It's about the 
effort of the spec to present a standard device.

What I don't understand is that, the mlx5-vdpa approach has been proved 
to succeed in all points you raised above (I meant the vendor specific 
part). Why need to bother them in the virtio layer if the feature can't 
be generalized?

Thanks


>
>>
>> Tahnks
>>
>

Re: [virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/3/2021 11:55 AM, Cornelia Huck wrote:
> On Tue, Aug 03 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>
>> On 8/3/2021 9:51 AM, Cornelia Huck wrote:
>>> On Tue, Aug 03 2021, Jason Wang <jasowang@redhat.com> wrote:
>>>
>>>> 在 2021/8/3 下午2:28, Cornelia Huck 写道:
>>>>> On Mon, Aug 02 2021, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>
>>>>>> On Mon, Aug 02, 2021 at 07:03:11PM +0300, Max Gurtovoy wrote:
>>>>>>> There is no much bits left in the generic feature field for all the features
>>>>>>> we would like to add.
>>>>>>> I mentioned only 5-6 in the above example and it will bring us to bit 46
>>>>>>> already.
>>>>>>>
>>>>>>> please think of 5-10 years from today.
>>>>>>>
>>>>>> IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually
>>>>>> pretty important to make sure the feature negotiation works well
>>>>>> and covers relevant usecases. If we have limitations preventing that
>>>>>> I'd like to at least try to fix that not replacing feature negotiation with
>>>>>> something else.
>>>>> I recall that we had a discussion about that years ago when we
>>>>> introduced VERSION_1; we explicitly agreed that we can extend features
>>>>> beyond 64 bit once we need it. (A quick search did not turn up that mail
>>>>> exchange, though.)
>>>>>
>>>> E.g PCI transport has feature_select.
>>>>
>>>>            le32 device_feature_select;     /* read-write */
>>>>            le32 device_feature;            /* read-only for driver */
>>>>            le32 driver_feature_select;     /* read-write */
>>>>            le32 driver_feature;            /* read-write */
>>>>
>>>> Technically it can support 32*32 different features.
>>> MMIO uses a similar scheme; CCW uses
>>>
>>> struct virtio_feature_desc {
>>>          le32 features;
>>>          u8 index;
>>> };
>>>
>>> so there's plenty of room to spare.
>> Feature negotiation can be done in admin q level. You don't want this
>> branches in the spec per each transport or device type.
> This has nothing to do with transports. We only wanted to demonstrate
> that all of the transports can be extended, so feature bits are not a
> scarce resource.
>
>> We have an admin command to get all device admin q features and we can
>> create a command to set all the driver supported.
>>
>> Again, I don't see the point of doing that. If a device is supporting
>> command_A, the driver should be able to use it. Telling the device "hi,
>> I'm going to use command_A in the future" is redundant.
>>
>> The device knows already how to handle it.
> If the driver negotiates a certain feature, it can mean that the device
> can make certain optimizations.
>
> That's not a particularily exotic pattern.

It's also doesn't allow forward compatibility.

Do you really want to add SW for each new HW cap ?

We're having difficulties because of this non fwd-compatible handshake.

Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Max Gurtovoy]

On 8/3/2021 12:01 PM, Jason Wang wrote:
>
> 在 2021/8/3 下午4:32, Max Gurtovoy 写道:
>>
>> On 8/3/2021 6:39 AM, Jason Wang wrote:
>>>
>>> 在 2021/8/2 下午11:27, Max Gurtovoy 写道:
>>>>
>>>> On 8/2/2021 5:51 PM, Cornelia Huck wrote:
>>>>> On Mon, Aug 02 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>
>>>>>> On 8/2/2021 5:17 AM, Jason Wang wrote:
>>>>>>> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>>>>>>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>>>>>>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>>>>>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>>>>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
>>>>>>>>>>>>>> +\subsubsection{Vendor specific command set}\label{sec:Basic
>>>>>>>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin
>>>>>>>>>>>>>> command set / Vendor specific command set}
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +The Vendor specific command set is a group of classes and
>>>>>>>>>>>>>> commands
>>>>>>>>>>>>>> +within each of these classes which are vendor specific. 
>>>>>>>>>>>>>> Refer to
>>>>>>>>>>>>>> +each vendor specification for more information on the 
>>>>>>>>>>>>>> supported
>>>>>>>>>>>>>> +commands.
>>>>>>>>>>>>> Here's another example.
>>>>>>>>>>>>> It's important that there is a way to make a device 
>>>>>>>>>>>>> completely
>>>>>>>>>>>>> generic without vendor specific expensions.
>>>>>>>>>>>>> Would be hard to do here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's a generic comment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> but specifically I am very reluctant to add vendor specific
>>>>>>>>>>>>> stuff like
>>>>>>>>>>>>> this directly in the spec. We already have
>>>>>>>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>>>>>>>> and if that is not sufficient I would like to know why
>>>>>>>>>>>>> before we add more vendor specific stuff.
>>>>>>>>>>>> We are adding an option to add vendor specific commands. 
>>>>>>>>>>>> We're not
>>>>>>>>>>>> adding it to the spec since each vendor will have its own 
>>>>>>>>>>>> manual for
>>>>>>>>>>>> that.
>>>>>>>>>>> IMHO, that way madness lies. I want to be able to look at 
>>>>>>>>>>> the spec
>>>>>>>>>>> and
>>>>>>>>>>> be able to implement a compliant device or a compliant 
>>>>>>>>>>> driver. If a
>>>>>>>>>>> vendor has some special feature they want to support, put it 
>>>>>>>>>>> in the
>>>>>>>>>>> spec, so that it is possible to actually implement it.
>>>>>>>>>> You can implement a compliant device and a compliant. why do you
>>>>>>>>>> think you
>>>>>>>>>> can't ?
>>>>> If I want to implement a device/driver, I need a standard to refer
>>>>> to. How can something vendor-specific be standard? If it is 
>>>>> useful, add
>>>>> it to the virtio spec.
>>>>
>>>> again, Linux driver will be generic.
>>>>
>>>> It will include a char dev to user space virtio-cli that will 
>>>> enable sending admin commands to do administration.
>>>>
>>>> Nobody asked to add vendor specific code to the driver.
>>>
>>>
>>> It's not about the general driver but about the general device. We 
>>> want to present a general standard device which means we can migrate 
>>> virtio devices among different vendors.
>>
>> I don't know if it's reasonable.
>>
>> You're looking on a device as a SW component since it was like this 
>> from day 1. But it's not. It's HW, and HW has its own internal state 
>> and implementation that is known only to the vendor.
>
>
> Spec doesn't forbid device internal state. But it cares the state that 
> is visible to the driver. If we had vendor specific state, it would be 
> very hard to claim to be a standard virtio device.
>
>
>>
>>>
>>>
>>> If each vendor choose to implement their own vendor specific 
>>> features via vendor specific commands that are not ruled by the 
>>> spec, there will be little value of the spec.
>>
>> Not at all.
>>
>> I'm very surprised that you are negative to this RFC instead of 
>> working on defining admin commands to stop/start queues and other 
>> commands that can be helpful for vDPA migration.
>
>
> I think there are some misunderstanding here. I'm not negative to 
> admin virtqueue, actually such concept has been discussed by the end 
> of last year. If you search the archive, I've posted RFC for net 
> specific admin virtqueue sometime early this year.
>
> But what I want to be negative is the vendor specific 
> commands/features through the admin virtqueue. The admin virtqueue 
> should be used for general facility like (device stop and migration as 
> you said above).
>
>
>>
>>>
>>>
>>>>
>>>> The spec allows vendor specific stuff today that are not usable and 
>>>> I'm trying to create a usable API without adding one line of vendor 
>>>> code to driver.
>>>>
>>>> You just need to open a window for few commands like others do and 
>>>> like you already did in the spec.
>>>>
>>>> NVIDIA HW internals are different from VENDOR_A and VENDOR_B.
>>>>
>>>> I want to debug my HW or enable some accelerations in it.
>>>
>>>
>>> A big question is that, on what layer do you want to provide those 
>>> functions?
>>>
>>> 1) virtio level
>>> 2) vendor level
>>>
>>> For 1), it needs to be done in the spec instead of introducing 
>>> backdoor for vendor specific command
>>
>> I'm repeating myself. You already have this backdoor for 
>> workarounding bugs.
>
>
> To say the truth, I don't like that. And it was probably too late to 
> fix. And I am not aware of any vendor that implements the VENDOR_CFG 
> capability.
>
> But we can do better now by not introducing any new vendor specific 
> stuffs.
>
>
>>
>> I don't ask for workarounds. I want to create a clean spec definition 
>> for vendor to debug/control their special souse.
>
>
> That's fine, but if it can't be generalized, it's better to be done at 
> the layer under virtio.
>
what layer are you talking about ?

There is the virtio layer and the transport layer. That's it.

The transport layer has a small window for configuration and handshake.

The virtio layer has ability to transfer data to/from driver from/to device.

>
>>
>>>
>>> For 2), virtio keeps simple as is, functionality is provided in a 
>>> vendor specific device interface. In this case, you can't use 
>>> virtio-pci directly, then you need present virtio on the upper layer 
>>> (with some software meditation)
>>
>> you mean VDPA ?
>
>
> vDPA is one approach but not all.
>
> Another example is the IFCVF, it has a dedicated bar for live 
> migration which is not a part of the virtio-pci capability. (I'm not 
> saying it's a good design, but it's indeed another approach).

Sorry, I don't this virtio spec should care about vDPA and IFCVF. This 
should be the other way around.

And you try to enforce the opposite.

>
>
>>
>>>
>>>
>>>>
>>>> This is the way to do it.
>>>
>>>
>>> Again, for debug or acceleration, if you want to do it at the virtio 
>>> level, you need generalize those features and define them in the spec.
>>>
>>> Maybe we can start the work on generalizing the debug facility 
>>> first. (adding Laurent who is working on this).
>>
>> Virtio debug stuff can be generalized.
>>
>> Vendor debug stuff should use vendor specific commands. A HW device 
>> has more than virtio state. It might have FW that controls it.  It 
>> might be an SoC with a lot of components that are used to accelerate 
>> this device.
>
>
> If the states/statistics/counters are highly vendor specific, let's do 
> that under virtio(pci). Such method has been used by other vendors.

No other layers.

Lets stop mentioning this.

>
>
>>
>> It's not always a QEMU code that implements virtio device.
>
>
> It's unrelated to whether or not it's a software device. It's about 
> the effort of the spec to present a standard device.
>
> What I don't understand is that, the mlx5-vdpa approach has been 
> proved to succeed in all points you raised above (I meant the vendor 
> specific part). Why need to bother them in the virtio layer if the 
> feature can't be generalized?

vDPA code shouldn't change at all.

Let's keep vDPA out of this discussion.

I can invent another my_virtio_data_path_acceleration and when I'll do 
that I don't expect the TC to change the spec to fit it.

>
> Thanks
>
>
>>
>>>
>>> Tahnks
>>>
>>
>

Re: [virtio-comment] Re: [RFC PATCH v2 1/2] Add virtio Admin Virtqueue specification

[Jason Wang]

在 2021/8/3 下午5:21, Max Gurtovoy 写道:
>
> On 8/3/2021 12:01 PM, Jason Wang wrote:
>>
>> 在 2021/8/3 下午4:32, Max Gurtovoy 写道:
>>>
>>> On 8/3/2021 6:39 AM, Jason Wang wrote:
>>>>
>>>> 在 2021/8/2 下午11:27, Max Gurtovoy 写道:
>>>>>
>>>>> On 8/2/2021 5:51 PM, Cornelia Huck wrote:
>>>>>> On Mon, Aug 02 2021, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>
>>>>>>> On 8/2/2021 5:17 AM, Jason Wang wrote:
>>>>>>>> 在 2021/8/1 上午6:53, Max Gurtovoy 写道:
>>>>>>>>> On 8/1/2021 1:26 AM, Michael S. Tsirkin wrote:
>>>>>>>>>> On Sat, Jul 31, 2021 at 02:34:25PM +0300, Max Gurtovoy wrote:
>>>>>>>>>>> On 7/30/2021 10:05 AM, Cornelia Huck wrote:
>>>>>>>>>>>> On Thu, Jul 29 2021, Max Gurtovoy <mgurtovoy@nvidia.com> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
>>>>>>>>>>>>>> On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> +\subsubsection{Vendor specific command 
>>>>>>>>>>>>>>> set}\label{sec:Basic
>>>>>>>>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / Admin
>>>>>>>>>>>>>>> command set / Vendor specific command set}
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +The Vendor specific command set is a group of classes and
>>>>>>>>>>>>>>> commands
>>>>>>>>>>>>>>> +within each of these classes which are vendor specific. 
>>>>>>>>>>>>>>> Refer to
>>>>>>>>>>>>>>> +each vendor specification for more information on the 
>>>>>>>>>>>>>>> supported
>>>>>>>>>>>>>>> +commands.
>>>>>>>>>>>>>> Here's another example.
>>>>>>>>>>>>>> It's important that there is a way to make a device 
>>>>>>>>>>>>>> completely
>>>>>>>>>>>>>> generic without vendor specific expensions.
>>>>>>>>>>>>>> Would be hard to do here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's a generic comment.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> but specifically I am very reluctant to add vendor specific
>>>>>>>>>>>>>> stuff like
>>>>>>>>>>>>>> this directly in the spec. We already have
>>>>>>>>>>>>>> VIRTIO_PCI_CAP_VENDOR_CFG
>>>>>>>>>>>>>> and if that is not sufficient I would like to know why
>>>>>>>>>>>>>> before we add more vendor specific stuff.
>>>>>>>>>>>>> We are adding an option to add vendor specific commands. 
>>>>>>>>>>>>> We're not
>>>>>>>>>>>>> adding it to the spec since each vendor will have its own 
>>>>>>>>>>>>> manual for
>>>>>>>>>>>>> that.
>>>>>>>>>>>> IMHO, that way madness lies. I want to be able to look at 
>>>>>>>>>>>> the spec
>>>>>>>>>>>> and
>>>>>>>>>>>> be able to implement a compliant device or a compliant 
>>>>>>>>>>>> driver. If a
>>>>>>>>>>>> vendor has some special feature they want to support, put 
>>>>>>>>>>>> it in the
>>>>>>>>>>>> spec, so that it is possible to actually implement it.
>>>>>>>>>>> You can implement a compliant device and a compliant. why do 
>>>>>>>>>>> you
>>>>>>>>>>> think you
>>>>>>>>>>> can't ?
>>>>>> If I want to implement a device/driver, I need a standard to refer
>>>>>> to. How can something vendor-specific be standard? If it is 
>>>>>> useful, add
>>>>>> it to the virtio spec.
>>>>>
>>>>> again, Linux driver will be generic.
>>>>>
>>>>> It will include a char dev to user space virtio-cli that will 
>>>>> enable sending admin commands to do administration.
>>>>>
>>>>> Nobody asked to add vendor specific code to the driver.
>>>>
>>>>
>>>> It's not about the general driver but about the general device. We 
>>>> want to present a general standard device which means we can 
>>>> migrate virtio devices among different vendors.
>>>
>>> I don't know if it's reasonable.
>>>
>>> You're looking on a device as a SW component since it was like this 
>>> from day 1. But it's not. It's HW, and HW has its own internal state 
>>> and implementation that is known only to the vendor.
>>
>>
>> Spec doesn't forbid device internal state. But it cares the state 
>> that is visible to the driver. If we had vendor specific state, it 
>> would be very hard to claim to be a standard virtio device.
>>
>>
>>>
>>>>
>>>>
>>>> If each vendor choose to implement their own vendor specific 
>>>> features via vendor specific commands that are not ruled by the 
>>>> spec, there will be little value of the spec.
>>>
>>> Not at all.
>>>
>>> I'm very surprised that you are negative to this RFC instead of 
>>> working on defining admin commands to stop/start queues and other 
>>> commands that can be helpful for vDPA migration.
>>
>>
>> I think there are some misunderstanding here. I'm not negative to 
>> admin virtqueue, actually such concept has been discussed by the end 
>> of last year. If you search the archive, I've posted RFC for net 
>> specific admin virtqueue sometime early this year.
>>
>> But what I want to be negative is the vendor specific 
>> commands/features through the admin virtqueue. The admin virtqueue 
>> should be used for general facility like (device stop and migration 
>> as you said above).
>>
>>
>>>
>>>>
>>>>
>>>>>
>>>>> The spec allows vendor specific stuff today that are not usable 
>>>>> and I'm trying to create a usable API without adding one line of 
>>>>> vendor code to driver.
>>>>>
>>>>> You just need to open a window for few commands like others do and 
>>>>> like you already did in the spec.
>>>>>
>>>>> NVIDIA HW internals are different from VENDOR_A and VENDOR_B.
>>>>>
>>>>> I want to debug my HW or enable some accelerations in it.
>>>>
>>>>
>>>> A big question is that, on what layer do you want to provide those 
>>>> functions?
>>>>
>>>> 1) virtio level
>>>> 2) vendor level
>>>>
>>>> For 1), it needs to be done in the spec instead of introducing 
>>>> backdoor for vendor specific command
>>>
>>> I'm repeating myself. You already have this backdoor for 
>>> workarounding bugs.
>>
>>
>> To say the truth, I don't like that. And it was probably too late to 
>> fix. And I am not aware of any vendor that implements the VENDOR_CFG 
>> capability.
>>
>> But we can do better now by not introducing any new vendor specific 
>> stuffs.
>>
>>
>>>
>>> I don't ask for workarounds. I want to create a clean spec 
>>> definition for vendor to debug/control their special souse.
>>
>>
>> That's fine, but if it can't be generalized, it's better to be done 
>> at the layer under virtio.
>>
> what layer are you talking about ?
>
> There is the virtio layer and the transport layer. That's it.
>
> The transport layer has a small window for configuration and handshake.


Nope. The spec only cares about virtio related features. It doesn't 
prevent you from adding vendor specific features in the transport layer.


> The virtio layer has ability to transfer data to/from driver from/to 
> device.
>
>>
>>>
>>>>
>>>> For 2), virtio keeps simple as is, functionality is provided in a 
>>>> vendor specific device interface. In this case, you can't use 
>>>> virtio-pci directly, then you need present virtio on the upper 
>>>> layer (with some software meditation)
>>>
>>> you mean VDPA ?
>>
>>
>> vDPA is one approach but not all.
>>
>> Another example is the IFCVF, it has a dedicated bar for live 
>> migration which is not a part of the virtio-pci capability. (I'm not 
>> saying it's a good design, but it's indeed another approach).
>
> Sorry, I don't this virtio spec should care about vDPA and IFCVF. This 
> should be the other way around.
>
> And you try to enforce the opposite.


No. It's not related to vDPA but just an example of adding vendor 
specific features.

If you look at the IFCVF driver codes, it's easy to see that the device 
could be probed by the standard virtio-pci driver.


>
>>
>>
>>>
>>>>
>>>>
>>>>>
>>>>> This is the way to do it.
>>>>
>>>>
>>>> Again, for debug or acceleration, if you want to do it at the 
>>>> virtio level, you need generalize those features and define them in 
>>>> the spec.
>>>>
>>>> Maybe we can start the work on generalizing the debug facility 
>>>> first. (adding Laurent who is working on this).
>>>
>>> Virtio debug stuff can be generalized.
>>>
>>> Vendor debug stuff should use vendor specific commands. A HW device 
>>> has more than virtio state. It might have FW that controls it.  It 
>>> might be an SoC with a lot of components that are used to accelerate 
>>> this device.
>>
>>
>> If the states/statistics/counters are highly vendor specific, let's 
>> do that under virtio(pci). Such method has been used by other vendors.
>
> No other layers.


Virtio can't work by itself.


>
> Lets stop mentioning this.


Let's keep the discussion constructive. I'm not obliged to design 
hardware for you, but obliged to figure out the design that violates the 
goal of the virtio spec. What I mentioned has been used by other vendor 
and it works well without bothering the virtio spec at all.

I think I've stated many times that adding vendor specific features in 
the spec is wrong, we want to have general features within a standard 
device. And you don't even explain why it must be done at virtio spec, 
(actually, not even defined in the spec but just leave a door for vendor 
to implement anything). Technically, you can have non virtio PF with 
virtio VF.


>
>>
>>
>>>
>>> It's not always a QEMU code that implements virtio device.
>>
>>
>> It's unrelated to whether or not it's a software device. It's about 
>> the effort of the spec to present a standard device.
>>
>> What I don't understand is that, the mlx5-vdpa approach has been 
>> proved to succeed in all points you raised above (I meant the vendor 
>> specific part). Why need to bother them in the virtio layer if the 
>> feature can't be generalized?
>
> vDPA code shouldn't change at all.
>
> Let's keep vDPA out of this discussion.
>
> I can invent another my_virtio_data_path_acceleration and when I'll do 
> that I don't expect the TC to change the spec to fit it.


I think we need some clarification on the concept "vDPA". To me it means 
virtio + vendor specific features (out of the spec).

Thanks


>
>>
>> Thanks
>>
>>
>>>
>>>>
>>>> Tahnks
>>>>
>>>
>>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: 
> https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>