[PATCH V2 0/5] Introduce virtio transport virtqueue

[PATCH V2 0/5] Introduce virtio transport virtqueue

[Zhu Lingshan]

This series introduces virtio transport virtqueue as a new
transport layer for virtio devices.

This series defines the concepts of virtio management device
and virtio managed devices.

This series also defines a commands set over the transport virtqueue,
including:
The command to query available resources of the management device
The commands to create / destroy the managed devices.
The commands to config the managed devices.
The commands to config virtqueues of the managed devices.

Changes form V1:
1)Add a command to set per-vq asid(Jason)
2)Add a command to reset a virtqueue(Jason)
3)Add support for packed virtqueue state(Jason)
4)Allow nested managed devices(Jason)
5)Provides device feature bits in the provisioning(Jason)
6)Removed most of the error codes, only left OK and ERR(Jason)
7)“virtual devices” is renamed to “managed devices”, to be more general(Jason)

Zhu Lingshan (5):
  Introduce virito transport virtqueue
  Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV
  Introduce the commands set of the transport vq
  Describe the process to present a managed device
  Add transport vq number for virtio blk and net

 content.tex      | 784 +++++++++++++++++++++++++++++++++++++++++++++++
 introduction.tex |   3 +
 2 files changed, 787 insertions(+)

-- 
2.35.3

[PATCH V2 1/5] Introduce virito transport virtqueue

[Zhu Lingshan]

This commit introduces transport virtqueue as a new transport
layer for virtio devices. And the format of the commands
through the transport virtqueue is defined as well.

We also give examples for the management devices and the
managed devices.

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex      | 73 ++++++++++++++++++++++++++++++++++++++++++++++++
 introduction.tex |  3 ++
 2 files changed, 76 insertions(+)

diff --git a/content.tex b/content.tex
index e863709..969ca46 100644
--- a/content.tex
+++ b/content.tex
@@ -2895,6 +2895,79 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
 MAY also choose to verify reset completion by reading \field{device status} via
 CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
 
+\section{Virtio Over Transport Virtqueue}\label{sec:Virtio Transport Options Virtio Over Transport Virtqueue}
+
+In some cases, it is challenging to implement a virtio device in a transport specific method.
+
+One example is that a physical device may try to present multiple managed (usually virtual) devices
+with limited transport specific resources.
+
+Another example is to implement managed devices which are transport independent.
+
+In those cases, a transport virtqueue could be used as the transport layer to
+implement virtio managed device.
+
+Feature bit VIRTIO_F_TRANSPT_VQ indicates that a device offers a transport virtqueue.
+
+\subsection{Basic Concepts}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts}
+
+All transport virtqueue commands are of the following form:
+
+\begin{lstlisting}
+struct virtio_transportq_ctrl {
+        u64 device_id;
+        u16 class;
+        u16 command;
+        u8 command-out-data[];
+        u32 ack;
+        u8 command-in-data[];
+};
+
+/* ack values */
+#define VIRTIO_TRANSPTQ_OK     0
+#define VIRTIO_TRANSPTQ_ERR    255
+\end{lstlisting}
+
+The \field{device_id}, \field{class}, \field{command} and
+\field{command-out-data} are set by the management driver,
+and the device sets \field{ack} and \field{command-in-data}.
+
+\field{device_id} is an UUID to identify a virtio managed device,
+The \field{device_id} value 0 is used for identify the management device itself
+
+\subsubsection{The Management Devices}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Management Devices}
+
+A device that offers a transport virtqueue (via feature VIRTIO_F_TRANSPT_VQ)
+is a management device. It processes the commands through the transport virtqueue commands.
+
+For example, a PCIe PF with a transport virtqueue is a management device.
+
+\subsubsection{The Managed Devices}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Managed Devices}
+
+A managed device is a kind of device that is created through a transport virtqueue,
+and utilizes the transport virtqueue as its transport layer.
+
+An example of managed devices is a Scalable I/O Virtualization \hyperref[intro:SIOV]{[SIOV]} device which is created and managed
+through a transport virtqueue of a management device.
+
+\devicenormative{\subsubsection}{Basic Concepts}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts}
+
+The management device MUST offer feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue as a device specific virtqueue.
+
+The management device MUST fail all commands against the transport virtqueue if VIRTIO_F_TRANSPT_VQ is not negotiated.
+
+\drivernormative{\subsubsection}{Basic Concepts}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts}
+
+The management driver MUST not send any commands against the transport virtqueue if VIRTIO_F_TRANSPT_VQ is not negotiated.
+
+\subsection{Management Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Management Devices Discovery}
+
+Management devices are discovered through their own transport and device-specific method.
+
+\subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Discovery}
+
+Managed devices are created and discovered through a transport virtqueue.
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
diff --git a/introduction.tex b/introduction.tex
index a9491cf..94a1b51 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -80,6 +80,9 @@ \section{Normative References}\label{sec:Normative References}
 	\phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
 	Arm System Control and Management Interface, DEN0056,
 	\newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
+	\phantomsection\label{intro:SIOV}\textbf{[SIOV]} &
+	Scalable I/O Virtualization,
+	\newline\url{https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf}\\
 
 \end{longtable}
 
-- 
2.35.3

[PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Zhu Lingshan]

This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
which depends on VIRTIO_F_TRANSPT_VQ.

Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
devices are created, configured and destroyed through the
transport virtqueue, the transport virtqueue is the
transport layer for the managed devices.

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/content.tex b/content.tex
index 969ca46..143949e 100644
--- a/content.tex
+++ b/content.tex
@@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
 
 Managed devices are created and discovered through a transport virtqueue.
 
+\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
+
+Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
+and destroyed through the transport virtqueue.
+This means the transport virtqueue is the transport layer for the managed devices.
+
+Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
+
+\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
+
+The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
-- 
2.35.3

[PATCH V2 3/5] Introduce the commands set of the transport vq

[Zhu Lingshan]

This commit introduces the commands set of the
transport virtqueue, including:

The command to query available resources of the management device
The commands to create / destroy the managed devices.
The commands to config the managed devices.
The commands to config virtqueues of the managed devices.

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex | 656 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 656 insertions(+)

diff --git a/content.tex b/content.tex
index 143949e..e625faf 100644
--- a/content.tex
+++ b/content.tex
@@ -2980,6 +2980,662 @@ \subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Vi
 
 The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
 
+\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+statistical information of available resources of the management device could be fetched
+by the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
+ #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
+\end{lstlisting}
+
+VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
+resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
+command. The management device fills command-in-data in the following format:
+
+\begin{lstlisting}
+struct mgmt_dev_avail_res{
+        /* The number of total remaining virtqueues for the managed devices */
+        u16 num_avail_vqs;
+        /* The minimal number of virtqueues for a managed device */
+        u16 min_mdev_vqs;
+        /* The maximum number of virtqueues for a managed device */
+        u16 max_mdev_vqs;
+        /* The number of managed devices that can be created with min_mdev_vqs virtqueues.
+         * This number may not equal to num_avail_vqs divided by min_mdev_vqs due to
+         * the limitations of other resources
+         */
+        u16 num_avail_mdev;
+};
+\end{lstlisting}
+
+If \field{min_mdev_vqs} equals to \field{min_mdev_vqs}, that means the management device
+can only create managed devices with a fixed number of virtqueues, then the managed devices
+can be pre-allocated.
+
+\devicenormative{\subsubsection}{Available resource of the management device}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Available resource of the management device}
+
+The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
+
+\drivernormative{\subsubsection}{Available resource of the management device}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Available resource of the management device}
+
+The management driver MUST use 0 as \field{device_id} for
+VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
+
+\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+managed devices must be created and destroyed through the transport virtqueue.
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPORTQ_CTRL_MDEV    2
+ #define VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE        0
+ #define VIRTIO_TRANSPORTQ_CTRL_MDEV_DESTROY       1
+
+struct transportq_ctrl_mdev_attribute {
+       u32 virtio_device_id;
+       u64 device_features;
+       u8 dev_config[];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE command is used to create a managed device.
+The command-out-data for VIRTIO_TRANSPTQ_CTRL_MDEV_CREATE is the virtio device id (\field{virtio_device_id}),
+feature bits and device type  specific configuration (\field{dev_config}) for creating the managed device.
+(as described in struct transportq_ctrl_mdev_attribute)
+
+The field \field{dev_config} is device type specific configurations. E.g., for a virtio-net device,
+it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
+for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
+
+The field \field{device_features} is the device feature bits for the managed device.
+
+When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
+
+The VIRTIO_TRANSPORTQ_CTRL_MDEV_DESTROY command is used to destroy a
+managed device which is identified by its 64 bits UUID
+\field{device_id}. There's no command-in-data nor command-out-data for
+VIRTIO_TRANSPTQ_CTRL_MDEV_DESTROY command.
+
+\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
+
+The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE command
+if \field{device_id} is not 0.
+
+The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_MDEV CREATE command
+if \field{device_features} exceeds the feature bits of the management device.
+
+The management device MUST fail VIRTO_TRANSPORTQ_CTRL_MDEV_DESTROY command
+if \field{device_id} is 0.
+
+\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
+
+The management driver MUST use 0 as \field{device_id} for
+TRANSPORTQ_CTRL_MDEV_CREATE command.
+
+\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Features Negotiation}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the features negotiation of managed devices is done by the
+following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
+ #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
+ #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
+ #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
+by a managed device.
+
+The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
+bits offered by the managed device.
+
+The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
+by both the virtual driver and the device.
+
+The features is a 64 bits mask of the virtio features bit.
+For VIRTIO_TRANSPTQ_CTRL_DRIVER_SET, the feature bits are passed to the device
+through command-out-data. For VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET and
+VIRTIO_TRANSPTQ_CTRL_DRIVER_GET the feature is returned for the device
+through command-in-data.
+
+\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Features Negotiation}
+
+The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
+\field{device_id} is 0.
+
+\drivernormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Features Negotiation}
+
+The management driver MAY mediate between the feature negotiation
+request of the managed devices and the transport virtqueue. E.g when
+offering features to the managed device, the management driver MAY
+exclude some features in order to limit the behaviors of the managed
+device.
+
+\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Device Status}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the device status of a managed device can be accessed by the following
+commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
+ #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        0
+ #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        1
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
+a managed device. The command-out-data is the one byte status
+to set to the device. There's no command-in-data for this command.
+
+The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
+a managed device. The command-in-data is the one byte status
+returned from the device. There's no command-out-data for this
+command.
+
+\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Device Status}
+
+The management device MUST reset the managed device when 0
+is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
+command demonstrates the success of the reset.
+
+The management device MUST present 0 through
+VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
+
+The management device MUST fail the device status access if
+\field{device_id} is 0.
+
+\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Device Status}
+
+After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
+for the success of the command before re-initializing the device.
+
+\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the device generation count can be read from the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
+ #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
+of the managed device. The command-in-data is the one byte device
+generation returned from the device. There's no command-out-data for
+this command.
+
+\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
+
+The management device MUST present a changed generation count after the driver
+has read a device-specific configuration value which has changed since
+any part of the device-specific configuration was last read.
+
+The management device MUST fail the device generation access if \field{device_id} is 0.
+
+\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
+Virtio Over Transport Virtqueue / Device Specific Configuration}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the device config space contents of a managed device can be accessed through
+the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
+  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
+  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
+
+struct transportq_ctrl_mdev_config_get {
+       u32 offset;
+       u32 size;
+};
+
+struct transportq_ctrl_mdev_config_set {
+       u32 offset;
+       u32 size;
+       u8  data[];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
+device configuration space.  The command-out-data is the \field{offset}
+from the start of the config space and the \field{size}
+of the data (as described in struct transportq_ctrl_mdev_config_get).
+The command-in-data is an array of the data that read from the config space.
+
+The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
+configuration space. The command-out-data contains the
+\field{offset} from the start of the config space, the \field{size} of the data and
+the \field{data} to be written (as described in struct transportq_ctrl_mdev_config_set).
+There's no command-in-data for this command.
+
+\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
+Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
+
+The management device MUST fail the device configuration space access
+if the driver wants to access the range which is outside the config space.
+
+The management device MUST fail the device configuration space access
+if \field{device_id} is 0.
+
+\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / MSI Configuration}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the MSI entries of a managed device are set through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI    7
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        0
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK       2
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET    3
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 4
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_MASK   5
+
+struct transportq_ctrl_mdev_msi_vq_config {
+       u16 queue_index;
+       u64 address;
+       u32 data;
+};
+
+struct transportq_ctrl_mdev_msi_vq_enable {
+       u16 queue_index;
+       u8 enable;
+};
+
+struct transportq_ctrl_mdev_msi_vq_mask {
+       u16 queue_index;
+       u8 mask;
+};
+
+struct transportq_ctrl_mdev_msi_config {
+       u64 address;
+       u32 data;
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
+specific virtqueue. The command-out-data is the \field{queue_index} and
+the MSI \field{address} and \field{data} (as described in struct transportq_ctrl_mdev_msi_vq_config).
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
+the MSI interrupt for a specific virtqueue. The command-out-data is the
+\field{queue_index} and an \field{enable} which representing ENABLE or DISABLE the MSI.
+(as described in struct transportq_ctrl_mdev_msi_vq_enable).
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
+#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK command is used to mask or unmask the
+MSI interrupt for a specific virtqueue. The command-out-data is the
+\field{queue_index} and the \field{mask} status
+(as described in struct transportq_ctrl_mdev_msi_vq_mask).
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK       0
+#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_UNMASK     1
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
+for the config interrupt. The command-out-data is the MSI \field{address} and
+\field{data} (as described in struct transportq_ctrl_mdev_msi_config).
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable and disable
+MSI for config space. The command-out-data is an u8 byte which representing
+ENABLE or DISABLE the config MSI.
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
+#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_MASK command is used to mask and unmask the
+MSI interrupt for config space. The command-out-data is an u8 byte which representing
+the mask status.
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_MASK      0
+#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_UNMASK    1
+\end{lstlisting}
+
+There's no command-in-data for all the above MSI commands.
+
+\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
+ver Transport Virtqueue / MSI Configuration}
+
+The managed device MUST record the pending MSI interrupts when masking, and
+re-generate the pending MSI interrupts when unmasking.
+
+The managed device MUST disable the MSI for both virtqueues and config space
+upon reset.
+
+\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / MSI Configuration}
+
+The driver MUST allocate transport or platform specific MSI entries
+for both virtqueues and config space if it wants to use interrupts.
+
+The driver MAY choose to disable the MSI if polling mode is used.
+
+\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Address}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the addresses of a specific virtqueue are accessed through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       0
+
+struct transportq_ctrl_mdev_vq_addr {
+       u16 queue_index;
+       u64 descriptor_area;
+       u64 device_area;
+       u64 driver_area;
+};
+\end{lstlisting}
+
+The command-out-data is the \field{queue_index}, the addresses of \field{device_area},
+\field{descriptor_area} and \field{driver_area} (as described in struct
+transportq_ctrl_mdev_vq_addr. There's no command-in-data.
+
+\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueeu Address}
+
+The management device MUST fail the commands of class
+VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
+
+The management device MUST fail the commands of class
+VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is invalid.
+
+\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Status}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+virtqueue status is accessed through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       0
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       1
+
+struct transportq_ctrl_mdev_vq_status_set {
+  u16 queue_index;
+  u8 status;
+
+#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
+#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
+
+};
+
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
+specific virtqueue. The command-out-data is the \field{queue_index} and the
+\field{status} representing ENABLE or DISABLE that is set to the virtqueue
+(as described in struct virtio_transportq_ctrl_vq_status_set).
+There's no command-in-data.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
+specific virtqueue. The command-out-data is the queue
+index. The command-in-data is the virtqueue status.
+
+\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Status}
+
+When disabled, the managed device MUST stop processing requests from
+this virtqueue.
+
+The management device MUST present a 0 via
+VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
+
+The management device MUST fail the virtqueue status access if
+\field{device_id} is 0.
+
+The management device MUST fail the virtqueue status access if
+the queue_index is invalid.
+
+\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Status}
+
+The driver MUST configure other virtqueue fields before enabling
+the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
+
+\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Size}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+virtqueue size is accessed through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       0
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       1
+
+struct transportq_ctrl_mdev_vq_size_set {
+       u16 queue_index;
+       u16 size;
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
+size. The command-out-data is the \field{queue_index} and the \field{size}
+of the virtqueue (as described in struct transportq_ctrl_mdev_vq_size_set).
+There's no command-in-data.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
+size. On reset, the maximum queue size supported by the device is
+returned. The command-out-data is an u16 of the queue index. The
+command-in-data is an u16 of the queue size of the virtqueue.
+
+\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Size}
+
+The management device MUST fail the virtqueue size access if
+\field{device_id} is 0.
+
+The management device MUST fail the virtqueue size access if
+the queue index is invalid.
+
+\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Notification}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the virtqueue notification area information can be get through the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          1
+
+struct transportq_ctrl_mdev_vq_notification_area {
+       u64 address;
+       u64 size;
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
+specific address area that can be used to notify a virtqueue. The
+command-out-data is an u16 of the queue index. The command-in-data
+contains the \field{address} and the \field{size}
+of the notification area (as described in struct transportq_ctrl_mdev_vq_notification_area).
+
+\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Notification}
+
+The management device MUST fail the virtqueue notification area information
+access if \field{device_id} is 0.
+
+The management device MUST fail the virtqueue notification area information
+access if the queue index is invalid.
+
+\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Notification}
+
+The driver MAY choose to notify the virtqueue by writing the queue
+index at address \field{address} which is fetched from the
+VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
+
+\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue State}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
+the virtqueue state is accessed through the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       0
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       1
+\end{lstlisting}
+
+\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
+
+\begin{lstlisting}
+struct transportq_ctrl_mdev_split_vq_state_set {
+    u16 queue_index;
+    u16 avail_index;
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
+split virtqueue. The command-out-data contains the \field{queue_index} and the
+available index for the virtqueue (as described in struct transportq_ctrl_mdev_split_vq_state_set).
+There is no command-in-data.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
+split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
+
+\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
+
+\begin{lstlisting}
+struct transportq_ctrl_mdev_packed_vq_state_set {
+    u16 queue_index;
+    u16 avail_counter;
+    u16 avail_index;
+    u16 used_counter;
+    u16 used_index;
+};
+
+struct transportq_ctrl_mdev_packed_vq_state_get {
+    u16 avail_counter;
+    u16 avail_index;
+    u16 used_counter;
+    u16 used_index;
+};
+\end{lstlisting}
+
+The state of a packed virtqueue includes :\\
+\field{avail_counter}: last driver ring wrap counter observed by device.\\
+\field{avail_index}: virtqueue available index.\\
+\field{used_counter}: device ring wrap counter.\\
+\field{used_index}: virtqueue used index.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
+The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
+\field{used_counter} and \field{used_index} for the virtqueue
+(as described in transportq_ctrl_mdev_packed_vq_state_set).
+There is no command-in-data.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
+The command-out-data is the queue index, the command-in-data contains \field{avail_counter}, \field{avail_index},
+\field{used_counter} and \field{used_index} of the virtqueue
+(as described in transportq_ctrl_mdev_packed_vq_state_set).
+
+\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue State}
+
+The management device MUST fail the virtqueue index access if \field{device_id} is 0.
+
+The management device MUST fail the virtqueue index access if \field{queue_index} is invalid.
+
+\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue ASID}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated, the address space id of a virtqueue
+could be set through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
+
+struct transportq_ctrl_mdev_vq_asid_set {
+	u16 queue_index;
+	u32 asid;
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
+of a virtqueue.
+
+The address space id is an identifier of a memory space which is used to convey the address space targeted by the
+memory accesses, and to distinguish memory accesses performed by different virtqueues
+
+One example of the address space id is PASID (Process Address Space Identifier) which is
+defined in \hyperref[intro:PCIe]{[PCIe]} specification.
+
+The command-out-data of VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is the \field{queue_index}
+and the address space id \field{asid}
+(as described in struct transportq_ctrl_mdev_vq_asid_set).
+
+\devicenormative{\subsubsection}{Virtqueue ASID}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue ASID}
+
+Once a virtqueue has been set an asid, it MUST perform any memory accesses with the asid.
+
+The management device MUST fail the virtqueue index access if \field{device_id} is 0.
+
+The management device MUST fail the virtqueue index access if \field{queue_index} is invalid.
+
+\subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Reset}
+
+When VIRTIO_F_TRANSPT_VQ_MDEV and VIRTIO_F_RING_RESET are negotiated,
+a virtqueue can be reset through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_RESET        14
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET    1
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET command is used to reset a virtqueue.
+The command-out-data is the virtqueue index, there is no command-in-data.
+
+\devicenormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Reset}
+
+The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
+\field{device_id} is 0.
+
+The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
+the virtqueue index is invalid.
+
+The managed device MUST stop consuming the descriptors in the virtqueue once reset it.
+
+The managed device MUST present default states of a virtqueue after reset it.
+
+\drivernormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Reset}
+
+After reseting a virtqueue, the driver MUST wait until the reset is successfully
+done (by verifying the virtqueue state has been reset to default)
+before re-enabling it.
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
-- 
2.35.3

[PATCH V2 4/5] Describe the process to present a managed device

[Zhu Lingshan]

This commit describes the process to present a manage
device, including device creation, config and virtqueue config

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/content.tex b/content.tex
index e625faf..2d4816d 100644
--- a/content.tex
+++ b/content.tex
@@ -3636,6 +3636,43 @@ \subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
 done (by verifying the virtqueue state has been reset to default)
 before re-enabling it.
 
+\subsection{Presenting Managed Devices}\label{sec:Virtio Transport Options /
+Virtio Over Transport Virtqueue / Presenting Managed Device}
+
+If VIRTIO_F_TRANSPT_VQ_MDEV is negotiated. The presenting of
+the managed devices requires cooperation between the management
+driver, the transport virtqueue and the managed device driver.
+
+The management driver typically takes the following steps for creating a
+managed device:
+
+\begin{enumerate}
+\item Determine the virtio device id and device specific configuration.
+\item Create the managed devices by VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE command.
+\item Optionally, configure the MSI entries by VIRTIO_TRANSPTQ_CTRL_MSI class commands
+\item The managed device driver probes the managed device.
+\end{enumerate}
+
+The management driver will then use the transport virtqueue to
+process the requests from the managed device driver.
+
+Managed device setup performed by the managde device driver:
+\begin{itemize*}
+\item Features negotiation by VIRTIO_TRANSPTQ_CTRL_FEAT class commands.
+\item Device status management by VIRTIO_TRANSPTQ_CTRL_STATUS class commands.
+\item Device configuration space access by VIRTIO_TRANSPTQ_CTRL_CONFIG class commands.
+\end{itemize*}
+
+Virtqueues setup performed by the managed driver:
+\begin{itemize*}
+\item Set the virtqueue address by VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command.
+\item Set the virtqueue size by VIRTIO_TRANSPTQ_CTRL_VQ_SIZE class commands.
+\item Get the virtqueue notification area by VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
+\item Set the virtqueue state by VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command.
+\item Optionally, set virtqueue asid by VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command.
+\item Enable the virtqueue by VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command.
+\end{itemize*}
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
-- 
2.35.3

[PATCH V2 5/5] Add transport vq number for virtio blk and net

[Zhu Lingshan]

This commit add the queue number of the transport virtqueue
for virtio_blk and virtio_net

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/content.tex b/content.tex
index 2d4816d..2536d2a 100644
--- a/content.tex
+++ b/content.tex
@@ -3801,6 +3801,7 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
 \item[2(N-1)] receiveqN
 \item[2(N-1)+1] transmitqN
 \item[2N] controlq
+\item[2N+1] transportq
 \end{description}
 
  N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by
@@ -3808,6 +3809,8 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
 
  controlq only exists if VIRTIO_NET_F_CTRL_VQ set.
 
+ transportq only exists if VIRTIO_F_TRANSPT_VQ set
+
 \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits}
 
 \begin{description}
@@ -5387,11 +5390,14 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
 \item[0] requestq1
 \item[\ldots]
 \item[N-1] requestqN
+\item[N] transportq
 \end{description}
 
  N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
  \field{num_queues}.
 
+ transportq only exists if VIRTIO_F_TRANSPT_VQ set.
+
 \subsection{Feature bits}\label{sec:Device Types / Block Device / Feature bits}
 
 \begin{description}
-- 
2.35.3

Re: [PATCH V2 0/5] Introduce virtio transport virtqueue

[Michael S. Tsirkin]

On Mon, Aug 01, 2022 at 05:32:11PM +0800, Zhu Lingshan wrote:
> This series introduces virtio transport virtqueue as a new
> transport layer for virtio devices.
> 
> This series defines the concepts of virtio management device
> and virtio managed devices.
> 
> This series also defines a commands set over the transport virtqueue,
> including:
> The command to query available resources of the management device
> The commands to create / destroy the managed devices.
> The commands to config the managed devices.
> The commands to config virtqueues of the managed devices.


Is there interest in at least analysing how this interacts with the
recent admin queue proposals?


I see some similarity in what these are trying to address - for example,
feature bit provisioning - any analysis on how the two proposals
compare?


> Changes form V1:
> 1)Add a command to set per-vq asid(Jason)
> 2)Add a command to reset a virtqueue(Jason)
> 3)Add support for packed virtqueue state(Jason)
> 4)Allow nested managed devices(Jason)
> 5)Provides device feature bits in the provisioning(Jason)
> 6)Removed most of the error codes, only left OK and ERR(Jason)
> 7)“virtual devices” is renamed to “managed devices”, to be more general(Jason)
> 
> Zhu Lingshan (5):
>   Introduce virito transport virtqueue
>   Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV
>   Introduce the commands set of the transport vq
>   Describe the process to present a managed device
>   Add transport vq number for virtio blk and net
> 
>  content.tex      | 784 +++++++++++++++++++++++++++++++++++++++++++++++
>  introduction.tex |   3 +
>  2 files changed, 787 insertions(+)
> 
> -- 
> 2.35.3

Re: [PATCH V2 1/5] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Mon, Aug 01, 2022 at 05:32:12PM +0800, Zhu Lingshan wrote:
> This commit introduces transport virtqueue as a new transport
> layer for virtio devices. And the format of the commands
> through the transport virtqueue is defined as well.
> 
> We also give examples for the management devices and the
> managed devices.
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>  content.tex      | 73 ++++++++++++++++++++++++++++++++++++++++++++++++
>  introduction.tex |  3 ++
>  2 files changed, 76 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index e863709..969ca46 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2895,6 +2895,79 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
>  MAY also choose to verify reset completion by reading \field{device status} via
>  CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
>  
> +\section{Virtio Over Transport Virtqueue}\label{sec:Virtio Transport Options Virtio Over Transport Virtqueue}
> +
> +In some cases, it is challenging to implement a virtio device in a transport specific method.
> +
> +One example is that a physical device may try to present multiple managed (usually virtual) devices
> +with limited transport specific resources.
> +
> +Another example is to implement managed devices which are transport independent.
> +
> +In those cases, a transport virtqueue could be used as the transport layer to
> +implement virtio managed device.
> +
> +Feature bit VIRTIO_F_TRANSPT_VQ indicates that a device offers a transport virtqueue.
> +
> +\subsection{Basic Concepts}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts}
> +
> +All transport virtqueue commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_transportq_ctrl {
> +        u64 device_id;
> +        u16 class;
> +        u16 command;
> +        u8 command-out-data[];
> +        u32 ack;
> +        u8 command-in-data[];
> +};
> +
> +/* ack values */
> +#define VIRTIO_TRANSPTQ_OK     0
> +#define VIRTIO_TRANSPTQ_ERR    255
> +\end{lstlisting}
> +
> +The \field{device_id}, \field{class}, \field{command} and


class seems to not be defined.

> +\field{command-out-data} are set by the management driver,
> +and the device sets \field{ack} and \field{command-in-data}.
> +
> +\field{device_id} is an UUID to identify a virtio managed device,

Using UUIDs is very iffy IMHO. Do we really need them unique and in
what context?

> +The \field{device_id} value 0 is used for identify the management device itself


I don't really understand when is device itself applicable.


> +
> +\subsubsection{The Management Devices}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Management Devices}
> +
> +A device that offers a transport virtqueue (via feature VIRTIO_F_TRANSPT_VQ)
> +is a management device. It processes the commands through the transport virtqueue commands.
> +
> +For example, a PCIe PF with a transport virtqueue is a management device.

Why a pf specifically?

> +
> +\subsubsection{The Managed Devices}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Managed Devices}
> +
> +A managed device is a kind of device that is created through a transport virtqueue,
> +and utilizes the transport virtqueue as its transport layer.
> +
> +An example of managed devices is a Scalable I/O Virtualization \hyperref[intro:SIOV]{[SIOV]} device which is created and managed
> +through a transport virtqueue of a management device.
> +
> +\devicenormative{\subsubsection}{Basic Concepts}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts}
> +
> +The management device MUST offer feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue as a device specific virtqueue.


I think I like the idea of exposing the vq index in a register better.

> +
> +The management device MUST fail all commands against the transport virtqueue if VIRTIO_F_TRANSPT_VQ is not negotiated.

Fail how? The vq can't be configured even. Are you saying is MUST
process the vq just to fail commands even though the feature bit is not there?
That's .... unusual.

> +
> +\drivernormative{\subsubsection}{Basic Concepts}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts}
> +
> +The management driver MUST not send any commands against the transport virtqueue if VIRTIO_F_TRANSPT_VQ is not negotiated.


what is the management driver? Do we need this term even?

> +
> +\subsection{Management Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Management Devices Discovery}
> +
> +Management devices are discovered through their own transport and device-specific method.

this does not seem to mean anything reader won't already know.

> +
> +\subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Discovery}
> +
> +Managed devices are created and discovered through a transport virtqueue.
> +
>  \chapter{Device Types}\label{sec:Device Types}
>  
>  On top of the queues, config space and feature negotiation facilities
> diff --git a/introduction.tex b/introduction.tex
> index a9491cf..94a1b51 100644
> --- a/introduction.tex
> +++ b/introduction.tex
> @@ -80,6 +80,9 @@ \section{Normative References}\label{sec:Normative References}
>  	\phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
>  	Arm System Control and Management Interface, DEN0056,
>  	\newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
> +	\phantomsection\label{intro:SIOV}\textbf{[SIOV]} &
> +	Scalable I/O Virtualization,
> +	\newline\url{https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf}\\
>  
>  \end{longtable}
>  
> -- 
> 2.35.3

[virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Michael S. Tsirkin]

On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
> This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
> which depends on VIRTIO_F_TRANSPT_VQ.
> 
> Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
> devices are created, configured and destroyed through the
> transport virtqueue, the transport virtqueue is the
> transport layer for the managed devices.

I'd avoid the term MDEV if possible, it's already used in virt contexts.


> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>  content.tex | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 969ca46..143949e 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
>  
>  Managed devices are created and discovered through a transport virtqueue.
>  
> +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> +
> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
> +and destroyed through the transport virtqueue.

Feature bit being present where? On the management device?

> +This means the transport virtqueue is the transport layer for the managed devices.

confused. how does this differ from VIRTIO_F_TRANSPT_VQ?

> +
> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
> +
> +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> +
> +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
> +
>  \chapter{Device Types}\label{sec:Device Types}
>  
>  On top of the queues, config space and feature negotiation facilities
> -- 
> 2.35.3


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH V2 3/5] Introduce the commands set of the transport vq

[Michael S. Tsirkin]

On Mon, Aug 01, 2022 at 05:32:14PM +0800, Zhu Lingshan wrote:
> This commit introduces the commands set of the
> transport virtqueue, including:
> 
> The command to query available resources of the management device
> The commands to create / destroy the managed devices.
> The commands to config the managed devices.
> The commands to config virtqueues of the managed devices.
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>  content.tex | 656 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 656 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 143949e..e625faf 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2980,6 +2980,662 @@ \subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Vi
>  
>  The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
>  
> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +statistical information of available resources of the management device could be fetched
> +by the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
> +\end{lstlisting}
> +
> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
> +command. The management device fills command-in-data in the following format:
> +
> +\begin{lstlisting}
> +struct mgmt_dev_avail_res{

space before {. And pls prefix things with virtio_ .

> +        /* The number of total remaining virtqueues for the managed devices */

remaining after what?

> +        u16 num_avail_vqs;
> +        /* The minimal number of virtqueues for a managed device */
> +        u16 min_mdev_vqs;
> +        /* The maximum number of virtqueues for a managed device */
> +        u16 max_mdev_vqs;
> +        /* The number of managed devices that can be created with min_mdev_vqs virtqueues.

why with min_mdev_vqs specifically?

can be created generally? or additionally to ones already created?

> +         * This number may not equal to num_avail_vqs divided by min_mdev_vqs due to
> +         * the limitations of other resources
> +         */
> +        u16 num_avail_mdev;
> +};
> +\end{lstlisting}
> +
> +If \field{min_mdev_vqs} equals to \field{min_mdev_vqs},

I don't understand what this says

> that means the management device
> +can only create managed devices with a fixed number of virtqueues, then the managed devices
> +can be pre-allocated.

what is this saying? what does pre-allocated mean? is it worth saying
here?

> +
> +\devicenormative{\subsubsection}{Available resource of the management device}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Available resource of the management device}
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
> +
> +\drivernormative{\subsubsection}{Available resource of the management device}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Available resource of the management device}
> +
> +The management driver MUST use 0 as \field{device_id} for
> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.

Now I am tempted to ask why not put it all in config space.
Why prefer it there you ask?
Look how it is handy since it allows using config struct to create
devices.
Worth considering.


> +
> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +managed devices must be created and destroyed through the transport virtqueue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPORTQ_CTRL_MDEV    2
> + #define VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE        0
> + #define VIRTIO_TRANSPORTQ_CTRL_MDEV_DESTROY       1
> +
> +struct transportq_ctrl_mdev_attribute {
> +       u32 virtio_device_id;
> +       u64 device_features;


alignment issues.


> +       u8 dev_config[];
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE command is used to create a managed device.
> +The command-out-data for VIRTIO_TRANSPTQ_CTRL_MDEV_CREATE is the virtio device id (\field{virtio_device_id}),


yes but you do not actually say what this field is.

> +feature bits and device type  specific configuration (\field{dev_config}) for creating the managed device.
> +(as described in struct transportq_ctrl_mdev_attribute)
> +
> +The field \field{dev_config} is device type specific configurations. E.g., for a virtio-net device,
> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};




> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
> +
> +The field \field{device_features} is the device feature bits for the managed device.
> +
> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
> +
> +The VIRTIO_TRANSPORTQ_CTRL_MDEV_DESTROY command is used to destroy a
> +managed device which is identified by its 64 bits UUID
> +\field{device_id}. There's no command-in-data nor command-out-data for
> +VIRTIO_TRANSPTQ_CTRL_MDEV_DESTROY command.
> +
> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> +
> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE command
> +if \field{device_id} is not 0.
> +
> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_MDEV CREATE command
> +if \field{device_features} exceeds the feature bits of the management device.

this is a bit vague.

> +
> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_MDEV_DESTROY command
> +if \field{device_id} is 0.
> +
> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> +
> +The management driver MUST use 0 as \field{device_id} for
> +TRANSPORTQ_CTRL_MDEV_CREATE command.
> +
> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Features Negotiation}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,

this is very confusing. I assume this refers to the management device?
we need to somehow move discussion of management device out of this
section, just focus on commands.


> +the features negotiation of managed devices is done by the
> +following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
> +by a managed device.
> +
> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
> +bits offered by the managed device.
> +
> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
> +by both the virtual driver and the device.

what is virtual driver?

> +
> +The features is a 64 bits mask of the virtio features bit.

let's find a way not to limit this to 64 bit please.
why should we? why not let driver specify how many driver wants.

> +For VIRTIO_TRANSPTQ_CTRL_DRIVER_SET, the feature bits are passed to the device
> +through command-out-data. For VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET and
> +VIRTIO_TRANSPTQ_CTRL_DRIVER_GET the feature is returned for the device
> +through command-in-data.
> +
> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Features Negotiation}
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
> +\field{device_id} is 0.
> +
> +\drivernormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Features Negotiation}
> +
> +The management driver MAY mediate between the feature negotiation
> +request of the managed devices and the transport virtqueue. E.g when
> +offering features to the managed device, the management driver MAY
> +exclude some features in order to limit the behaviors of the managed
> +device.

I don't understand what you are saying here. how is this different
from any other transport?

> +
> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Device Status}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +the device status of a managed device can be accessed by the following
> +commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        0
> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        1
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
> +a managed device. The command-out-data is the one byte status
> +to set to the device. There's no command-in-data for this command.
> +
> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
> +a managed device. The command-in-data is the one byte status
> +returned from the device. There's no command-out-data for this
> +command.
> +
> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Device Status}
> +
> +The management device MUST reset the managed device when 0
> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
> +command demonstrates the success of the reset.
> +
> +The management device MUST present 0 through
> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
> +
> +The management device MUST fail the device status access if
> +\field{device_id} is 0.
> +
> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Device Status}
> +
> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
> +for the success of the command before re-initializing the device.
> +
> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +the device generation count can be read from the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
> +of the managed device. The command-in-data is the one byte device
> +generation returned from the device. There's no command-out-data for
> +this command.
> +
> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> +
> +The management device MUST present a changed generation count after the driver
> +has read a device-specific configuration value which has changed since
> +any part of the device-specific configuration was last read.

I know PCI says this too but I was never sure what this means ;)

> +
> +The management device MUST fail the device generation access if \field{device_id} is 0.
> +
> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Device Specific Configuration}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +the device config space contents of a managed device can be accessed through
> +the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
> +
> +struct transportq_ctrl_mdev_config_get {
> +       u32 offset;
> +       u32 size;
> +};
> +
> +struct transportq_ctrl_mdev_config_set {
> +       u32 offset;
> +       u32 size;
> +       u8  data[];
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
> +device configuration space.  The command-out-data is the \field{offset}
> +from the start of the config space and the \field{size}
> +of the data (as described in struct transportq_ctrl_mdev_config_get).
> +The command-in-data is an array of the data that read from the config space.
> +
> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
> +configuration space. The command-out-data contains the
> +\field{offset} from the start of the config space, the \field{size} of the data and
> +the \field{data} to be written (as described in struct transportq_ctrl_mdev_config_set).
> +There's no command-in-data for this command.

alignment limitations? I think what pci does here is good ...

> +
> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
> +
> +The management device MUST fail the device configuration space access
> +if the driver wants to access the range which is outside the config space.

wants? pls be more specific.

> +
> +The management device MUST fail the device configuration space access
> +if \field{device_id} is 0.


0 seems to just be for the get command. Let's get rid of it
and then we don't need it?

> +
> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / MSI Configuration}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +the MSI entries of a managed device are set through the following command:

wait what? there are msi entries? what are they? we need a chapter
about signalling.

> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        0
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK       2
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET    3
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 4
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_MASK   5


I think all these SET things really need GET commands too,
so driver can query device for how it's been configured.


> +
> +struct transportq_ctrl_mdev_msi_vq_config {
> +       u16 queue_index;
> +       u64 address;
> +       u32 data;
> +};
> +
> +struct transportq_ctrl_mdev_msi_vq_enable {
> +       u16 queue_index;
> +       u8 enable;
> +};
> +
> +struct transportq_ctrl_mdev_msi_vq_mask {
> +       u16 queue_index;
> +       u8 mask;
> +};
> +
> +struct transportq_ctrl_mdev_msi_config {
> +       u64 address;
> +       u32 data;
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
> +specific virtqueue. The command-out-data is the \field{queue_index} and
> +the MSI \field{address} and \field{data} (as described in struct transportq_ctrl_mdev_msi_vq_config).
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
> +the MSI interrupt for a specific virtqueue. The command-out-data is the
> +\field{queue_index} and an \field{enable} which representing ENABLE or DISABLE the MSI.
> +(as described in struct transportq_ctrl_mdev_msi_vq_enable).

and what happens if it's disabled? is the interrupt remembered until
reenabled? or discarded?

> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK command is used to mask or unmask the
> +MSI interrupt for a specific virtqueue. The command-out-data is the
> +\field{queue_index} and the \field{mask} status
> +(as described in struct transportq_ctrl_mdev_msi_vq_mask).

how is mask/unmask different from enable/disable?

> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK       0
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_UNMASK     1
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
> +for the config interrupt. The command-out-data is the MSI \field{address} and
> +\field{data} (as described in struct transportq_ctrl_mdev_msi_config).
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable and disable
> +MSI for config space. The command-out-data is an u8 byte which representing
> +ENABLE or DISABLE the config MSI.
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_MASK command is used to mask and unmask the
> +MSI interrupt for config space. The command-out-data is an u8 byte which representing
> +the mask status.
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_MASK      0
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_UNMASK    1
> +\end{lstlisting}
> +
> +There's no command-in-data for all the above MSI commands.
> +
> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> +ver Transport Virtqueue / MSI Configuration}
> +
> +The managed device MUST record the pending MSI interrupts when masking, and
> +re-generate the pending MSI interrupts when unmasking.
> +
> +The managed device MUST disable the MSI for both virtqueues and config space
> +upon reset.
> +
> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / MSI Configuration}
> +
> +The driver MUST allocate transport or platform specific MSI entries
> +for both virtqueues and config space if it wants to use interrupts.
> +
> +The driver MAY choose to disable the MSI if polling mode is used.
> +
> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Address}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +the addresses of a specific virtqueue are accessed through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       0
> +
> +struct transportq_ctrl_mdev_vq_addr {
> +       u16 queue_index;
> +       u64 descriptor_area;
> +       u64 device_area;
> +       u64 driver_area;
> +};
> +\end{lstlisting}
> +
> +The command-out-data is the \field{queue_index}, the addresses of \field{device_area},
> +\field{descriptor_area} and \field{driver_area} (as described in struct
> +transportq_ctrl_mdev_vq_addr. There's no command-in-data.
> +
> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueeu Address}
> +
> +The management device MUST fail the commands of class
> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
> +
> +The management device MUST fail the commands of class
> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is invalid.
> +
> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Status}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +virtqueue status is accessed through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       0
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       1
> +
> +struct transportq_ctrl_mdev_vq_status_set {
> +  u16 queue_index;
> +  u8 status;
> +
> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
> +


padding to align things to 64 bit pls.

> +};
> +
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
> +specific virtqueue. The command-out-data is the \field{queue_index} and the
> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
> +(as described in struct virtio_transportq_ctrl_vq_status_set).
> +There's no command-in-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
> +specific virtqueue. The command-out-data is the queue
> +index. The command-in-data is the virtqueue status.
> +
> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Status}
> +
> +When disabled, the managed device MUST stop processing requests from
> +this virtqueue.
> +
> +The management device MUST present a 0 via
> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
> +
> +The management device MUST fail the virtqueue status access if
> +\field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue status access if
> +the queue_index is invalid.
> +

invalid how?

> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Status}
> +
> +The driver MUST configure other virtqueue fields before enabling
> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
> +
> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Size}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +virtqueue size is accessed through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       0
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       1
> +
> +struct transportq_ctrl_mdev_vq_size_set {
> +       u16 queue_index;
> +       u16 size;
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
> +size. The command-out-data is the \field{queue_index} and the \field{size}
> +of the virtqueue (as described in struct transportq_ctrl_mdev_vq_size_set).
> +There's no command-in-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
> +size. On reset, the maximum queue size supported by the device is
> +returned. The command-out-data is an u16 of the queue index. The
> +command-in-data is an u16 of the queue size of the virtqueue.
> +
> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Size}
> +
> +The management device MUST fail the virtqueue size access if
> +\field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue size access if
> +the queue index is invalid.
> +
> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Notification}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +the virtqueue notification area information can be get through the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          1
> +
> +struct transportq_ctrl_mdev_vq_notification_area {
> +       u64 address;
> +       u64 size;
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
> +specific address area that can be used to notify a virtqueue. The
> +command-out-data is an u16 of the queue index. The command-in-data
> +contains the \field{address} and the \field{size}
> +of the notification area (as described in struct transportq_ctrl_mdev_vq_notification_area).
> +
> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> +
> +The management device MUST fail the virtqueue notification area information
> +access if \field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue notification area information
> +access if the queue index is invalid.
> +
> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> +
> +The driver MAY choose to notify the virtqueue by writing the queue
> +index at address \field{address} which is fetched from the
> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
> +
> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue State}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> +the virtqueue state is accessed through the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       0
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       1
> +\end{lstlisting}
> +
> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
> +
> +\begin{lstlisting}
> +struct transportq_ctrl_mdev_split_vq_state_set {
> +    u16 queue_index;
> +    u16 avail_index;
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
> +split virtqueue. The command-out-data contains the \field{queue_index} and the
> +available index for the virtqueue (as described in struct transportq_ctrl_mdev_split_vq_state_set).
> +There is no command-in-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
> +
> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
> +
> +\begin{lstlisting}
> +struct transportq_ctrl_mdev_packed_vq_state_set {
> +    u16 queue_index;
> +    u16 avail_counter;
> +    u16 avail_index;
> +    u16 used_counter;
> +    u16 used_index;
> +};
> +
> +struct transportq_ctrl_mdev_packed_vq_state_get {
> +    u16 avail_counter;
> +    u16 avail_index;
> +    u16 used_counter;
> +    u16 used_index;
> +};

reuse the structure here to avoid duplication?


> +\end{lstlisting}
> +
> +The state of a packed virtqueue includes :\\
> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
> +\field{avail_index}: virtqueue available index.\\
> +\field{used_counter}: device ring wrap counter.\\
> +\field{used_index}: virtqueue used index.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
> +\field{used_counter} and \field{used_index} for the virtqueue
> +(as described in transportq_ctrl_mdev_packed_vq_state_set).
> +There is no command-in-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
> +The command-out-data is the queue index, the command-in-data contains \field{avail_counter}, \field{avail_index},
> +\field{used_counter} and \field{used_index} of the virtqueue
> +(as described in transportq_ctrl_mdev_packed_vq_state_set).
> +
> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue State}
> +
> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue index access if \field{queue_index} is invalid.
> +
> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue ASID}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated, the address space id of a virtqueue
> +could be set through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
> +
> +struct transportq_ctrl_mdev_vq_asid_set {
> +	u16 queue_index;
> +	u32 asid;
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
> +of a virtqueue.
> +
> +The address space id is an identifier of a memory space which is used to convey the address space targeted by the
> +memory accesses, and to distinguish memory accesses performed by different virtqueues
> +
> +One example of the address space id is PASID (Process Address Space Identifier) which is
> +defined in \hyperref[intro:PCIe]{[PCIe]} specification.
> +
> +The command-out-data of VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is the \field{queue_index}
> +and the address space id \field{asid}
> +(as described in struct transportq_ctrl_mdev_vq_asid_set).
> +
> +\devicenormative{\subsubsection}{Virtqueue ASID}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue ASID}
> +
> +Once a virtqueue has been set an asid, it MUST perform any memory accesses with the asid.
> +
> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue index access if \field{queue_index} is invalid.
> +
> +\subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Reset}
> +
> +When VIRTIO_F_TRANSPT_VQ_MDEV and VIRTIO_F_RING_RESET are negotiated,


wait this is a mess. VIRTIO_F_TRANSPT_VQ_MDEV is on management device.
VIRTIO_F_RING_RESET is I assume on the managed one?


> +a virtqueue can be reset through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_RESET        14
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET    1
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET command is used to reset a virtqueue.
> +The command-out-data is the virtqueue index, there is no command-in-data.
> +
> +\devicenormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Reset}
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
> +\field{device_id} is 0.
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
> +the virtqueue index is invalid.
> +
> +The managed device MUST stop consuming the descriptors in the virtqueue once reset it.
> +
> +The managed device MUST present default states of a virtqueue after reset it.
> +
> +\drivernormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Reset}
> +
> +After reseting a virtqueue, the driver MUST wait until the reset is successfully
> +done (by verifying the virtqueue state has been reset to default)

verifying how? why isn't completion of the command sufficient?

> +before re-enabling it.
> +
>  \chapter{Device Types}\label{sec:Device Types}
>  
>  On top of the queues, config space and feature negotiation facilities
> -- 
> 2.35.3

Re: [PATCH V2 5/5] Add transport vq number for virtio blk and net

[Michael S. Tsirkin]

On Mon, Aug 01, 2022 at 05:32:16PM +0800, Zhu Lingshan wrote:
> This commit add the queue number of the transport virtqueue
> for virtio_blk and virtio_net
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>

I think I prefer a transport register for this.
Otherwise, you most document that other devices must not offer
the feature bit.


> ---
>  content.tex | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 2d4816d..2536d2a 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -3801,6 +3801,7 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
>  \item[2(N-1)] receiveqN
>  \item[2(N-1)+1] transmitqN
>  \item[2N] controlq
> +\item[2N+1] transportq
>  \end{description}
>  
>   N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by
> @@ -3808,6 +3809,8 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
>  
>   controlq only exists if VIRTIO_NET_F_CTRL_VQ set.
>  
> + transportq only exists if VIRTIO_F_TRANSPT_VQ set
> +
>  \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits}
>  
>  \begin{description}
> @@ -5387,11 +5390,14 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
>  \item[0] requestq1
>  \item[\ldots]
>  \item[N-1] requestqN
> +\item[N] transportq
>  \end{description}
>  
>   N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
>   \field{num_queues}.
>  
> + transportq only exists if VIRTIO_F_TRANSPT_VQ set.
> +
>  \subsection{Feature bits}\label{sec:Device Types / Block Device / Feature bits}
>  
>  \begin{description}
> -- 
> 2.35.3

Re: [PATCH V2 0/5] Introduce virtio transport virtqueue

[Michael S. Tsirkin]

On Mon, Aug 01, 2022 at 05:32:11PM +0800, Zhu Lingshan wrote:
> This series introduces virtio transport virtqueue as a new
> transport layer for virtio devices.
> 
> This series defines the concepts of virtio management device
> and virtio managed devices.


Overall I'm happy with the progress made here. Pls keep iterating.
I would like to see whether we can unify this with the admin
queue work somehow. The technology is different (SIOV vs SRIOV)
but there are also similarities.

Do you care to review some patches from the admin queue proposals maybe?


> This series also defines a commands set over the transport virtqueue,
> including:
> The command to query available resources of the management device
> The commands to create / destroy the managed devices.
> The commands to config the managed devices.
> The commands to config virtqueues of the managed devices.
> 
> Changes form V1:
> 1)Add a command to set per-vq asid(Jason)
> 2)Add a command to reset a virtqueue(Jason)
> 3)Add support for packed virtqueue state(Jason)
> 4)Allow nested managed devices(Jason)
> 5)Provides device feature bits in the provisioning(Jason)
> 6)Removed most of the error codes, only left OK and ERR(Jason)
> 7)“virtual devices” is renamed to “managed devices”, to be more general(Jason)
> 
> Zhu Lingshan (5):
>   Introduce virito transport virtqueue
>   Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV
>   Introduce the commands set of the transport vq
>   Describe the process to present a managed device
>   Add transport vq number for virtio blk and net
> 
>  content.tex      | 784 +++++++++++++++++++++++++++++++++++++++++++++++
>  introduction.tex |   3 +
>  2 files changed, 787 insertions(+)
> 
> -- 
> 2.35.3

Re: [PATCH V2 1/5] Introduce virito transport virtqueue

[Zhu, Lingshan]

On 8/2/2022 4:25 AM, Michael S. Tsirkin wrote:
> On Mon, Aug 01, 2022 at 05:32:12PM +0800, Zhu Lingshan wrote:
>> This commit introduces transport virtqueue as a new transport
>> layer for virtio devices. And the format of the commands
>> through the transport virtqueue is defined as well.
>>
>> We also give examples for the management devices and the
>> managed devices.
>>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>> ---
>>   content.tex      | 73 ++++++++++++++++++++++++++++++++++++++++++++++++
>>   introduction.tex |  3 ++
>>   2 files changed, 76 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index e863709..969ca46 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -2895,6 +2895,79 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
>>   MAY also choose to verify reset completion by reading \field{device status} via
>>   CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
>>   
>> +\section{Virtio Over Transport Virtqueue}\label{sec:Virtio Transport Options Virtio Over Transport Virtqueue}
>> +
>> +In some cases, it is challenging to implement a virtio device in a transport specific method.
>> +
>> +One example is that a physical device may try to present multiple managed (usually virtual) devices
>> +with limited transport specific resources.
>> +
>> +Another example is to implement managed devices which are transport independent.
>> +
>> +In those cases, a transport virtqueue could be used as the transport layer to
>> +implement virtio managed device.
>> +
>> +Feature bit VIRTIO_F_TRANSPT_VQ indicates that a device offers a transport virtqueue.
>> +
>> +\subsection{Basic Concepts}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts}
>> +
>> +All transport virtqueue commands are of the following form:
>> +
>> +\begin{lstlisting}
>> +struct virtio_transportq_ctrl {
>> +        u64 device_id;
>> +        u16 class;
>> +        u16 command;
>> +        u8 command-out-data[];
>> +        u32 ack;
>> +        u8 command-in-data[];
>> +};
>> +
>> +/* ack values */
>> +#define VIRTIO_TRANSPTQ_OK     0
>> +#define VIRTIO_TRANSPTQ_ERR    255
>> +\end{lstlisting}
>> +
>> +The \field{device_id}, \field{class}, \field{command} and
>
> class seems to not be defined.
I will add a description here, a class is a set of commands with similar 
purpose.
>
>> +\field{command-out-data} are set by the management driver,
>> +and the device sets \field{ack} and \field{command-in-data}.
>> +
>> +\field{device_id} is an UUID to identify a virtio managed device,
> Using UUIDs is very iffy IMHO. Do we really need them unique and in
> what context?
We need an identifier to address a managed device, and the ID should be
allocated by the management device(not the system wide), and be unique 
in the management device context.
So I think UUID can meet the requirement.
>> +The \field{device_id} value 0 is used for identify the management device itself
>
> I don't really understand when is device itself applicable.
Sometimes we may need to send commands against the management device itself.
ike querying the reaming available resource (e.g., how many available 
queues),
to determine what kind and how many managed device(like SIOV devices) 
can be created on the management device.

This also leave flexibility for further extensions.
>
>
>> +
>> +\subsubsection{The Management Devices}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Management Devices}
>> +
>> +A device that offers a transport virtqueue (via feature VIRTIO_F_TRANSPT_VQ)
>> +is a management device. It processes the commands through the transport virtqueue commands.
>> +
>> +For example, a PCIe PF with a transport virtqueue is a management device.
> Why a pf specifically?
It is just an example, it can be a VF or other devices, even nested
>
>> +
>> +\subsubsection{The Managed Devices}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Managed Devices}
>> +
>> +A managed device is a kind of device that is created through a transport virtqueue,
>> +and utilizes the transport virtqueue as its transport layer.
>> +
>> +An example of managed devices is a Scalable I/O Virtualization \hyperref[intro:SIOV]{[SIOV]} device which is created and managed
>> +through a transport virtqueue of a management device.
>> +
>> +\devicenormative{\subsubsection}{Basic Concepts}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts}
>> +
>> +The management device MUST offer feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue as a device specific virtqueue.
>
> I think I like the idea of exposing the vq index in a register better.
Is it add a new filed "transport_vq_idx" in the device config space? I 
am not sure, if we expose the queue_index of the transport virtqueue in 
the config sapce,
it looks we have limited the management device transport interface to be 
PCI, then the transport vq can not be nested and self-contained.
I see the control vq is just offered through the CVQ feature bit, and in 
the chapter "Device Types", we have defined the queue index of cvq.
>
>> +
>> +The management device MUST fail all commands against the transport virtqueue if VIRTIO_F_TRANSPT_VQ is not negotiated.
> Fail how? The vq can't be configured even. Are you saying is MUST
> process the vq just to fail commands even though the feature bit is not there?
> That's .... unusual.
you are right, I should remove this section since if VIRTIO_F_TRANSPT_VQ 
is not negotiated, the transport vq can not be initialized,
then no commands through the transport vq.
>
>> +
>> +\drivernormative{\subsubsection}{Basic Concepts}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts}
>> +
>> +The management driver MUST not send any commands against the transport virtqueue if VIRTIO_F_TRANSPT_VQ is not negotiated.
>
> what is the management driver? Do we need this term even?
I can add a description here, the management driver is the driver of the 
management device.
>
>> +
>> +\subsection{Management Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Management Devices Discovery}
>> +
>> +Management devices are discovered through their own transport and device-specific method.
> this does not seem to mean anything reader won't already know.
Yes, I can remove this.
>
>> +
>> +\subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Discovery}
>> +
>> +Managed devices are created and discovered through a transport virtqueue.
>> +
>>   \chapter{Device Types}\label{sec:Device Types}
>>   
>>   On top of the queues, config space and feature negotiation facilities
>> diff --git a/introduction.tex b/introduction.tex
>> index a9491cf..94a1b51 100644
>> --- a/introduction.tex
>> +++ b/introduction.tex
>> @@ -80,6 +80,9 @@ \section{Normative References}\label{sec:Normative References}
>>   	\phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
>>   	Arm System Control and Management Interface, DEN0056,
>>   	\newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
>> +	\phantomsection\label{intro:SIOV}\textbf{[SIOV]} &
>> +	Scalable I/O Virtualization,
>> +	\newline\url{https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf}\\
>>   
>>   \end{longtable}
>>   
>> -- 
>> 2.35.3

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Zhu, Lingshan]

On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
> On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
>> This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
>> which depends on VIRTIO_F_TRANSPT_VQ.
>>
>> Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
>> devices are created, configured and destroyed through the
>> transport virtqueue, the transport virtqueue is the
>> transport layer for the managed devices.
> I'd avoid the term MDEV if possible, it's already used in virt contexts.
yes, let me work out a better name, it was vdev (virtual device) before, 
and maybe
sdev(sub device) or a better one.
>
>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>> ---
>>   content.tex | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index 969ca46..143949e 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
>>   
>>   Managed devices are created and discovered through a transport virtqueue.
>>   
>> +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>> +
>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
>> +and destroyed through the transport virtqueue.
> Feature bit being present where? On the management device?
Yes, the management device, I will add this.
>
>> +This means the transport virtqueue is the transport layer for the managed devices.
> confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the 
management device create / destroy and configure the managed device via 
the transport vq.

This reminds me that maybe I should use the term vdev(virtual device), 
so that other kinds of devices can add their own feature bits under 
VIRTIO_F_TRANSPT_VQ

Thanks,
Zhu Lingshan
>
>> +
>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
>> +
>> +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>> +
>> +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
>> +
>>   \chapter{Device Types}\label{sec:Device Types}
>>   
>>   On top of the queues, config space and feature negotiation facilities
>> -- 
>> 2.35.3
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Zhu, Lingshan]

On 8/2/2022 11:17 AM, Zhu, Lingshan wrote:
>
>
> On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
>> On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
>>> This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
>>> which depends on VIRTIO_F_TRANSPT_VQ.
>>>
>>> Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
>>> devices are created, configured and destroyed through the
>>> transport virtqueue, the transport virtqueue is the
>>> transport layer for the managed devices.
>> I'd avoid the term MDEV if possible, it's already used in virt contexts.
> yes, let me work out a better name, it was vdev (virtual device) 
> before, and maybe
> sdev(sub device) or a better one.
>>
>>
>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>> ---
>>>   content.tex | 12 ++++++++++++
>>>   1 file changed, 12 insertions(+)
>>>
>>> diff --git a/content.tex b/content.tex
>>> index 969ca46..143949e 100644
>>> --- a/content.tex
>>> +++ b/content.tex
>>> @@ -2968,6 +2968,18 @@ \subsection{Managed Devices 
>>> Dscovery}\label{sec:Virtio Transport Options /Virtio
>>>     Managed devices are created and discovered through a transport 
>>> virtqueue.
>>>   +\subsection{Transport Virtqueue Features}\label{sec:Virtio 
>>> Transport Options /Virtio Over Transport Virtqueue / Transport 
>>> Virtqueue Features}
>>> +
>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed 
>>> devices are created, configured
>>> +and destroyed through the transport virtqueue.
>> Feature bit being present where? On the management device?
> Yes, the management device, I will add this.
>>
>>> +This means the transport virtqueue is the transport layer for the 
>>> managed devices.
>> confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
> VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
> VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the 
> management device create / destroy and configure the managed device 
> via the transport vq.
>
> This reminds me that maybe I should use the term vdev(virtual device), 
> so that other kinds of devices can add their own feature bits under 
> VIRTIO_F_TRANSPT_VQ
>
> Thanks,
> Zhu Lingshan
Hi Michael,

After a re-consideration, I think I should remove 
VIRTIO_F_TRANSPT_VQ_MDEV, and remove _mdev_ in the structures.

Thanks,
Zhu Lingshan
>>
>>> +
>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
>>> +
>>> +\devicenormative{\subsubsection}{Transport Virtqueue 
>>> Features}{Virtio Transport Options / Virtio Over Transport Virtqueue 
>>> / Transport Virtqueue Features}
>>> +
>>> +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if 
>>> VIRTIO_F_TRANSPT_VQ is not negotiated.
>>> +
>>>   \chapter{Device Types}\label{sec:Device Types}
>>>     On top of the queues, config space and feature negotiation 
>>> facilities
>>> -- 
>>> 2.35.3
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: 
>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: 
>> https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
>

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Michael S. Tsirkin]

On Tue, Aug 02, 2022 at 11:17:51AM +0800, Zhu, Lingshan wrote:
> 
> 
> On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
> > On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
> > > This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
> > > which depends on VIRTIO_F_TRANSPT_VQ.
> > > 
> > > Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
> > > devices are created, configured and destroyed through the
> > > transport virtqueue, the transport virtqueue is the
> > > transport layer for the managed devices.
> > I'd avoid the term MDEV if possible, it's already used in virt contexts.
> yes, let me work out a better name, it was vdev (virtual device) before, and
> maybe
> sdev(sub device) or a better one.

Besides mdev can thinkably stand for both managed and management.
Nvidia patches have the concept of a device group and
group member but do not have a name for the device that controls
the group. Admin device? Control device?

Also, creating/destroying is one thing that seems to only apply to
SIOV. configuring seems to apply to SRIOV as well?



> > 
> > 
> > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> > > ---
> > >   content.tex | 12 ++++++++++++
> > >   1 file changed, 12 insertions(+)
> > > 
> > > diff --git a/content.tex b/content.tex
> > > index 969ca46..143949e 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
> > >   Managed devices are created and discovered through a transport virtqueue.
> > > +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> > > +
> > > +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
> > > +and destroyed through the transport virtqueue.
> > Feature bit being present where? On the management device?
> Yes, the management device, I will add this.
> > 
> > > +This means the transport virtqueue is the transport layer for the managed devices.
> > confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
> VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
> VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the
> management device create / destroy and configure the managed device via the
> transport vq.
> 
> This reminds me that maybe I should use the term vdev(virtual device), so
> that other kinds of devices can add their own feature bits under
> VIRTIO_F_TRANSPT_VQ
> 
> Thanks,
> Zhu Lingshan
> > 
> > > +
> > > +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
> > > +
> > > +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> > > +
> > > +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
> > > +
> > >   \chapter{Device Types}\label{sec:Device Types}
> > >   On top of the queues, config space and feature negotiation facilities
> > > -- 
> > > 2.35.3
> > 
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> > 
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> > 
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> > 

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Zhu, Lingshan]

On 8/2/2022 2:52 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 02, 2022 at 11:17:51AM +0800, Zhu, Lingshan wrote:
>>
>> On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
>>> On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
>>>> This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
>>>> which depends on VIRTIO_F_TRANSPT_VQ.
>>>>
>>>> Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
>>>> devices are created, configured and destroyed through the
>>>> transport virtqueue, the transport virtqueue is the
>>>> transport layer for the managed devices.
>>> I'd avoid the term MDEV if possible, it's already used in virt contexts.
>> yes, let me work out a better name, it was vdev (virtual device) before, and
>> maybe
>> sdev(sub device) or a better one.
> Besides mdev can thinkably stand for both managed and management.
> Nvidia patches have the concept of a device group and
> group member but do not have a name for the device that controls
> the group. Admin device? Control device?
maybe management device?
>
> Also, creating/destroying is one thing that seems to only apply to
> SIOV. configuring seems to apply to SRIOV as well?
In this series, yes, because actually we can not provision a SRIOV VF 
with dynamic config, and VF provisioning is done through SRIOV cap

Thanks
Zhu Lingshan
>
>
>
>>>
>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>>> ---
>>>>    content.tex | 12 ++++++++++++
>>>>    1 file changed, 12 insertions(+)
>>>>
>>>> diff --git a/content.tex b/content.tex
>>>> index 969ca46..143949e 100644
>>>> --- a/content.tex
>>>> +++ b/content.tex
>>>> @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
>>>>    Managed devices are created and discovered through a transport virtqueue.
>>>> +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>> +
>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
>>>> +and destroyed through the transport virtqueue.
>>> Feature bit being present where? On the management device?
>> Yes, the management device, I will add this.
>>>> +This means the transport virtqueue is the transport layer for the managed devices.
>>> confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
>> VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
>> VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the
>> management device create / destroy and configure the managed device via the
>> transport vq.
>>
>> This reminds me that maybe I should use the term vdev(virtual device), so
>> that other kinds of devices can add their own feature bits under
>> VIRTIO_F_TRANSPT_VQ
>>
>> Thanks,
>> Zhu Lingshan
>>>> +
>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
>>>> +
>>>> +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>> +
>>>> +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
>>>> +
>>>>    \chapter{Device Types}\label{sec:Device Types}
>>>>    On top of the queues, config space and feature negotiation facilities
>>>> -- 
>>>> 2.35.3
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Michael S. Tsirkin]

On Tue, Aug 02, 2022 at 07:54:41PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 8/2/2022 2:52 PM, Michael S. Tsirkin wrote:
> > On Tue, Aug 02, 2022 at 11:17:51AM +0800, Zhu, Lingshan wrote:
> > > 
> > > On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
> > > > On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
> > > > > This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
> > > > > which depends on VIRTIO_F_TRANSPT_VQ.
> > > > > 
> > > > > Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
> > > > > devices are created, configured and destroyed through the
> > > > > transport virtqueue, the transport virtqueue is the
> > > > > transport layer for the managed devices.
> > > > I'd avoid the term MDEV if possible, it's already used in virt contexts.
> > > yes, let me work out a better name, it was vdev (virtual device) before, and
> > > maybe
> > > sdev(sub device) or a better one.
> > Besides mdev can thinkably stand for both managed and management.
> > Nvidia patches have the concept of a device group and
> > group member but do not have a name for the device that controls
> > the group. Admin device? Control device?
> maybe management device?

This term has been overused in this context with people saying things
like "the management interface is used by management tools to
manage managed devices" to the point where I'd
rather we avoided it altogether.

> > 
> > Also, creating/destroying is one thing that seems to only apply to
> > SIOV. configuring seems to apply to SRIOV as well?
> In this series, yes, because actually we can not provision a SRIOV VF with
> dynamic config, and VF provisioning is done through SRIOV cap
> 
> Thanks
> Zhu Lingshan

I am not sure what dynamic config means here exactly. Yes creating a VF
itself is not possible. But allocating VQs to it and
e.g. configuring the MAC address of a networking VF can be
done dynamically.

> > 
> > 
> > 
> > > > 
> > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> > > > > ---
> > > > >    content.tex | 12 ++++++++++++
> > > > >    1 file changed, 12 insertions(+)
> > > > > 
> > > > > diff --git a/content.tex b/content.tex
> > > > > index 969ca46..143949e 100644
> > > > > --- a/content.tex
> > > > > +++ b/content.tex
> > > > > @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
> > > > >    Managed devices are created and discovered through a transport virtqueue.
> > > > > +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> > > > > +
> > > > > +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
> > > > > +and destroyed through the transport virtqueue.
> > > > Feature bit being present where? On the management device?
> > > Yes, the management device, I will add this.
> > > > > +This means the transport virtqueue is the transport layer for the managed devices.
> > > > confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
> > > VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
> > > VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the
> > > management device create / destroy and configure the managed device via the
> > > transport vq.
> > > 
> > > This reminds me that maybe I should use the term vdev(virtual device), so
> > > that other kinds of devices can add their own feature bits under
> > > VIRTIO_F_TRANSPT_VQ
> > > 
> > > Thanks,
> > > Zhu Lingshan
> > > > > +
> > > > > +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
> > > > > +
> > > > > +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> > > > > +
> > > > > +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
> > > > > +
> > > > >    \chapter{Device Types}\label{sec:Device Types}
> > > > >    On top of the queues, config space and feature negotiation facilities
> > > > > -- 
> > > > > 2.35.3
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > 
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > > 
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > > 

Re: [virtio-comment] Re: [PATCH V2 3/5] Introduce the commands set of the transport vq

[Zhu, Lingshan]

On 8/2/2022 4:53 AM, Michael S. Tsirkin wrote:
> On Mon, Aug 01, 2022 at 05:32:14PM +0800, Zhu Lingshan wrote:
>> This commit introduces the commands set of the
>> transport virtqueue, including:
>>
>> The command to query available resources of the management device
>> The commands to create / destroy the managed devices.
>> The commands to config the managed devices.
>> The commands to config virtqueues of the managed devices.
>>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>> ---
>>   content.tex | 656 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 656 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index 143949e..e625faf 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -2980,6 +2980,662 @@ \subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Vi
>>   
>>   The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
>>   
>> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +statistical information of available resources of the management device could be fetched
>> +by the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
>> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
>> +\end{lstlisting}
>> +
>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
>> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
>> +command. The management device fills command-in-data in the following format:
>> +
>> +\begin{lstlisting}
>> +struct mgmt_dev_avail_res{
> space before {. And pls prefix things with virtio_ .
will fix
>
>> +        /* The number of total remaining virtqueues for the managed devices */
> remaining after what?
this tries to show the available free virtqueues that can be used to 
create managed devices, I will use a better term in the next version.
>
>> +        u16 num_avail_vqs;
>> +        /* The minimal number of virtqueues for a managed device */
>> +        u16 min_mdev_vqs;
>> +        /* The maximum number of virtqueues for a managed device */
>> +        u16 max_mdev_vqs;
>> +        /* The number of managed devices that can be created with min_mdev_vqs virtqueues.
> why with min_mdev_vqs specifically?
>
> can be created generally? or additionally to ones already created?
yes, additional ones already have been "pre-created", like 4 queues 
virtio-net, in this case,
both min_mdev_vqs and max_mdev_vqs are 4.
>
>> +         * This number may not equal to num_avail_vqs divided by min_mdev_vqs due to
>> +         * the limitations of other resources
>> +         */
>> +        u16 num_avail_mdev;
>> +};
>> +\end{lstlisting}
>> +
>> +If \field{min_mdev_vqs} equals to \field{min_mdev_vqs},
> I don't understand what this says
my typo, this should be min_mdev_vqs == max_mdev_vqs
>
>> that means the management device
>> +can only create managed devices with a fixed number of virtqueues, then the managed devices
>> +can be pre-allocated.
> what is this saying? what does pre-allocated mean? is it worth saying
> here?
if min_mdev_vqs == max_mdev_vqs, it implies the management device can 
only create fixed size / capabilities managed devices,
that's usually pre-created.

But I think I should remove this, it does not has to be pre-created, it 
is totally the hardware implementation thing.
>
>> +
>> +\devicenormative{\subsubsection}{Available resource of the management device}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Available resource of the management device}
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
>> +
>> +\drivernormative{\subsubsection}{Available resource of the management device}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Available resource of the management device}
>> +
>> +The management driver MUST use 0 as \field{device_id} for
>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
> Now I am tempted to ask why not put it all in config space.
> Why prefer it there you ask?
> Look how it is handy since it allows using config struct to create
> devices.
> Worth considering.
An implementation in the config space is truly attractive, my concern is 
that has to be PCI.
This seems can not be nested since a nested SIOV ADI can only have the 
transport vq as its
transport layer, no PCI for it.
>
>
>> +
>> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +managed devices must be created and destroyed through the transport virtqueue.
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPORTQ_CTRL_MDEV    2
>> + #define VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE        0
>> + #define VIRTIO_TRANSPORTQ_CTRL_MDEV_DESTROY       1
>> +
>> +struct transportq_ctrl_mdev_attribute {
>> +       u32 virtio_device_id;
>> +       u64 device_features;
>
> alignment issues.
will fix this
>
>
>> +       u8 dev_config[];
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE command is used to create a managed device.
>> +The command-out-data for VIRTIO_TRANSPTQ_CTRL_MDEV_CREATE is the virtio device id (\field{virtio_device_id}),
>
> yes but you do not actually say what this field is.
I will add a description here, define VRITO_ID_NET == 1, BLK == 2, etc
>
>> +feature bits and device type  specific configuration (\field{dev_config}) for creating the managed device.
>> +(as described in struct transportq_ctrl_mdev_attribute)
>> +
>> +The field \field{dev_config} is device type specific configurations. E.g., for a virtio-net device,
>> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
>>
>> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
>> +
>> +The field \field{device_features} is the device feature bits for the managed device.
>> +
>> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
>> +
>> +The VIRTIO_TRANSPORTQ_CTRL_MDEV_DESTROY command is used to destroy a
>> +managed device which is identified by its 64 bits UUID
>> +\field{device_id}. There's no command-in-data nor command-out-data for
>> +VIRTIO_TRANSPTQ_CTRL_MDEV_DESTROY command.
>> +
>> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>> +
>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_MDEV_CREATE command
>> +if \field{device_id} is not 0.
>> +
>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_MDEV CREATE command
>> +if \field{device_features} exceeds the feature bits of the management device.
> this is a bit vague.
this means, the management device should fail command 
VIRTIO_TRANSPOTRQ_CTRL_MDEV_CREATE if the driver asking for a feature 
bit the management device can not fulfill.

E.g., the management device does not have RSS, but the driver wants to 
create a managed device with RSS.

I will work on a better description.

By the way, it may be better remove all mdev/MDEV in the structure and 
the commands macros, only leave _dev_ and "managed device" in the 
descriptions.
>
>> +
>> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_MDEV_DESTROY command
>> +if \field{device_id} is 0.
>> +
>> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>> +
>> +The management driver MUST use 0 as \field{device_id} for
>> +TRANSPORTQ_CTRL_MDEV_CREATE command.
>> +
>> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Features Negotiation}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
> this is very confusing. I assume this refers to the management device?
> we need to somehow move discussion of management device out of this
> section, just focus on commands.
Yes, the management device.
This tries to explain if the feature bit VIRTIO_F_TRANSPT_VQ_MDEV is 
negotiated with the management device,
we can send these commands to control the manged device through the 
transport vq.

But I think it may be better remove the feature bit 
VIRTIO_F_TRANSPT_VQ_MDEV as you pointed out before.
So I would rework it like "When VIRTIO_F_TRANSPT_VQ is negotiated with 
the management device".
Do you think this can make it clear?
>
>
>> +the features negotiation of managed devices is done by the
>> +following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
>> +by a managed device.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
>> +bits offered by the managed device.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
>> +by both the virtual driver and the device.
> what is virtual driver?
my typo, should be the managed device driver
>
>> +
>> +The features is a 64 bits mask of the virtio features bit.
> let's find a way not to limit this to 64 bit please.
> why should we? why not let driver specify how many driver wants.
OK
>
>> +For VIRTIO_TRANSPTQ_CTRL_DRIVER_SET, the feature bits are passed to the device
>> +through command-out-data. For VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET and
>> +VIRTIO_TRANSPTQ_CTRL_DRIVER_GET the feature is returned for the device
>> +through command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Features Negotiation}
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
>> +\field{device_id} is 0.
>> +
>> +\drivernormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Features Negotiation}
>> +
>> +The management driver MAY mediate between the feature negotiation
>> +request of the managed devices and the transport virtqueue. E.g when
>> +offering features to the managed device, the management driver MAY
>> +exclude some features in order to limit the behaviors of the managed
>> +device.
> I don't understand what you are saying here. how is this different
> from any other transport?
Yes, this section is not necessary here.
>
>> +
>> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Device Status}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +the device status of a managed device can be accessed by the following
>> +commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        0
>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        1
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
>> +a managed device. The command-out-data is the one byte status
>> +to set to the device. There's no command-in-data for this command.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
>> +a managed device. The command-in-data is the one byte status
>> +returned from the device. There's no command-out-data for this
>> +command.
>> +
>> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Device Status}
>> +
>> +The management device MUST reset the managed device when 0
>> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
>> +command demonstrates the success of the reset.
>> +
>> +The management device MUST present 0 through
>> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
>> +
>> +The management device MUST fail the device status access if
>> +\field{device_id} is 0.
>> +
>> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Device Status}
>> +
>> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
>> +for the success of the command before re-initializing the device.
>> +
>> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +the device generation count can be read from the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
>> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
>> +of the managed device. The command-in-data is the one byte device
>> +generation returned from the device. There's no command-out-data for
>> +this command.
>> +
>> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>> +
>> +The management device MUST present a changed generation count after the driver
>> +has read a device-specific configuration value which has changed since
>> +any part of the device-specific configuration was last read.
> I know PCI says this too but I was never sure what this means ;)
yes, it is confusing, I will try to fix this both in this patch and in 
the PCI section.
>
>> +
>> +The management device MUST fail the device generation access if \field{device_id} is 0.
>> +
>> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Device Specific Configuration}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +the device config space contents of a managed device can be accessed through
>> +the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
>> +
>> +struct transportq_ctrl_mdev_config_get {
>> +       u32 offset;
>> +       u32 size;
>> +};
>> +
>> +struct transportq_ctrl_mdev_config_set {
>> +       u32 offset;
>> +       u32 size;
>> +       u8  data[];
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
>> +device configuration space.  The command-out-data is the \field{offset}
>> +from the start of the config space and the \field{size}
>> +of the data (as described in struct transportq_ctrl_mdev_config_get).
>> +The command-in-data is an array of the data that read from the config space.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
>> +configuration space. The command-out-data contains the
>> +\field{offset} from the start of the config space, the \field{size} of the data and
>> +the \field{data} to be written (as described in struct transportq_ctrl_mdev_config_set).
>> +There's no command-in-data for this command.
> alignment limitations? I think what pci does here is good ...
I didn't get it, do you mean the alignment in struct 
transportq_ctrl_mdev_config_set?
I think it may be hard to place padding there, because it's \field{size} 
is dynamic.
>
>> +
>> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
>> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
>> +
>> +The management device MUST fail the device configuration space access
>> +if the driver wants to access the range which is outside the config space.
> wants? pls be more specific.
I will remove this "wants to"
>
>> +
>> +The management device MUST fail the device configuration space access
>> +if \field{device_id} is 0.
>
> 0 seems to just be for the get command. Let's get rid of it
> and then we don't need it?
not only for the get command, there can be set commands, like
set some fields of the device config space, or set vq size.

The management device has it's own transport like PCI,
so here we don't allow get/set the device config space through the 
transport vq
>
>> +
>> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / MSI Configuration}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +the MSI entries of a managed device are set through the following command:
> wait what? there are msi entries? what are they? we need a chapter
> about signalling.
I will add a new section to explain how the managed devices do interrupts
>
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        0
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK       2
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET    3
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 4
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_MASK   5
>
> I think all these SET things really need GET commands too,
> so driver can query device for how it's been configured.
sure
>
>
>> +
>> +struct transportq_ctrl_mdev_msi_vq_config {
>> +       u16 queue_index;
>> +       u64 address;
>> +       u32 data;
>> +};
>> +
>> +struct transportq_ctrl_mdev_msi_vq_enable {
>> +       u16 queue_index;
>> +       u8 enable;
>> +};
>> +
>> +struct transportq_ctrl_mdev_msi_vq_mask {
>> +       u16 queue_index;
>> +       u8 mask;
>> +};
>> +
>> +struct transportq_ctrl_mdev_msi_config {
>> +       u64 address;
>> +       u32 data;
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
>> +specific virtqueue. The command-out-data is the \field{queue_index} and
>> +the MSI \field{address} and \field{data} (as described in struct transportq_ctrl_mdev_msi_vq_config).
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
>> +the MSI interrupt for a specific virtqueue. The command-out-data is the
>> +\field{queue_index} and an \field{enable} which representing ENABLE or DISABLE the MSI.
>> +(as described in struct transportq_ctrl_mdev_msi_vq_enable).
> and what happens if it's disabled? is the interrupt remembered until
> reenabled? or discarded?
once disable, it won't send any MSI interrupts anymore.
>
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK command is used to mask or unmask the
>> +MSI interrupt for a specific virtqueue. The command-out-data is the
>> +\field{queue_index} and the \field{mask} status
>> +(as described in struct transportq_ctrl_mdev_msi_vq_mask).
> how is mask/unmask different from enable/disable?
mask/unmask means the device should store/re-generate the pending MSI 
interrupts

Maybe I should remove this mask/unmask, only enable/disable is enough.
>
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_MASK       0
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_UNMASK     1
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
>> +for the config interrupt. The command-out-data is the MSI \field{address} and
>> +\field{data} (as described in struct transportq_ctrl_mdev_msi_config).
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable and disable
>> +MSI for config space. The command-out-data is an u8 byte which representing
>> +ENABLE or DISABLE the config MSI.
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_MASK command is used to mask and unmask the
>> +MSI interrupt for config space. The command-out-data is an u8 byte which representing
>> +the mask status.
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_MASK      0
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_UNMASK    1
>> +\end{lstlisting}
>> +
>> +There's no command-in-data for all the above MSI commands.
>> +
>> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>> +ver Transport Virtqueue / MSI Configuration}
>> +
>> +The managed device MUST record the pending MSI interrupts when masking, and
>> +re-generate the pending MSI interrupts when unmasking.
>> +
>> +The managed device MUST disable the MSI for both virtqueues and config space
>> +upon reset.
>> +
>> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / MSI Configuration}
>> +
>> +The driver MUST allocate transport or platform specific MSI entries
>> +for both virtqueues and config space if it wants to use interrupts.
>> +
>> +The driver MAY choose to disable the MSI if polling mode is used.
>> +
>> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Address}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +the addresses of a specific virtqueue are accessed through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       0
>> +
>> +struct transportq_ctrl_mdev_vq_addr {
>> +       u16 queue_index;
>> +       u64 descriptor_area;
>> +       u64 device_area;
>> +       u64 driver_area;
>> +};
>> +\end{lstlisting}
>> +
>> +The command-out-data is the \field{queue_index}, the addresses of \field{device_area},
>> +\field{descriptor_area} and \field{driver_area} (as described in struct
>> +transportq_ctrl_mdev_vq_addr. There's no command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueeu Address}
>> +
>> +The management device MUST fail the commands of class
>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
>> +
>> +The management device MUST fail the commands of class
>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is invalid.
>> +
>> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Status}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +virtqueue status is accessed through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       0
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       1
>> +
>> +struct transportq_ctrl_mdev_vq_status_set {
>> +  u16 queue_index;
>> +  u8 status;
>> +
>> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
>> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
>> +
>
> padding to align things to 64 bit pls.
sure
>
>> +};
>> +
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
>> +specific virtqueue. The command-out-data is the \field{queue_index} and the
>> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
>> +(as described in struct virtio_transportq_ctrl_vq_status_set).
>> +There's no command-in-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
>> +specific virtqueue. The command-out-data is the queue
>> +index. The command-in-data is the virtqueue status.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Status}
>> +
>> +When disabled, the managed device MUST stop processing requests from
>> +this virtqueue.
>> +
>> +The management device MUST present a 0 via
>> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
>> +
>> +The management device MUST fail the virtqueue status access if
>> +\field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue status access if
>> +the queue_index is invalid.
>> +
> invalid how?
Like queue_index == 5, but only 4 queues. I will rework on this to be 
"out-of-range invalid"
>
>> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Status}
>> +
>> +The driver MUST configure other virtqueue fields before enabling
>> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
>> +
>> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Size}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +virtqueue size is accessed through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       0
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       1
>> +
>> +struct transportq_ctrl_mdev_vq_size_set {
>> +       u16 queue_index;
>> +       u16 size;
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
>> +size. The command-out-data is the \field{queue_index} and the \field{size}
>> +of the virtqueue (as described in struct transportq_ctrl_mdev_vq_size_set).
>> +There's no command-in-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
>> +size. On reset, the maximum queue size supported by the device is
>> +returned. The command-out-data is an u16 of the queue index. The
>> +command-in-data is an u16 of the queue size of the virtqueue.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Size}
>> +
>> +The management device MUST fail the virtqueue size access if
>> +\field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue size access if
>> +the queue index is invalid.
>> +
>> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Notification}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +the virtqueue notification area information can be get through the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          1
>> +
>> +struct transportq_ctrl_mdev_vq_notification_area {
>> +       u64 address;
>> +       u64 size;
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
>> +specific address area that can be used to notify a virtqueue. The
>> +command-out-data is an u16 of the queue index. The command-in-data
>> +contains the \field{address} and the \field{size}
>> +of the notification area (as described in struct transportq_ctrl_mdev_vq_notification_area).
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>> +
>> +The management device MUST fail the virtqueue notification area information
>> +access if \field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue notification area information
>> +access if the queue index is invalid.
>> +
>> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>> +
>> +The driver MAY choose to notify the virtqueue by writing the queue
>> +index at address \field{address} which is fetched from the
>> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
>> +
>> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue State}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated,
>> +the virtqueue state is accessed through the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       0
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       1
>> +\end{lstlisting}
>> +
>> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
>> +
>> +\begin{lstlisting}
>> +struct transportq_ctrl_mdev_split_vq_state_set {
>> +    u16 queue_index;
>> +    u16 avail_index;
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
>> +split virtqueue. The command-out-data contains the \field{queue_index} and the
>> +available index for the virtqueue (as described in struct transportq_ctrl_mdev_split_vq_state_set).
>> +There is no command-in-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
>> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
>> +
>> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
>> +
>> +\begin{lstlisting}
>> +struct transportq_ctrl_mdev_packed_vq_state_set {
>> +    u16 queue_index;
>> +    u16 avail_counter;
>> +    u16 avail_index;
>> +    u16 used_counter;
>> +    u16 used_index;
>> +};
>> +
>> +struct transportq_ctrl_mdev_packed_vq_state_get {
>> +    u16 avail_counter;
>> +    u16 avail_index;
>> +    u16 used_counter;
>> +    u16 used_index;
>> +};
> reuse the structure here to avoid duplication?
There is a u16 queue_index in struct 
transportq_ctrl_mdev_packed_vq_state_set.
And I think it may be a little messy if we define a struct in another struct
to reuse struct transportq_ctrl_mdev_packed_vq_state_get
>
>
>> +\end{lstlisting}
>> +
>> +The state of a packed virtqueue includes :\\
>> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
>> +\field{avail_index}: virtqueue available index.\\
>> +\field{used_counter}: device ring wrap counter.\\
>> +\field{used_index}: virtqueue used index.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
>> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
>> +\field{used_counter} and \field{used_index} for the virtqueue
>> +(as described in transportq_ctrl_mdev_packed_vq_state_set).
>> +There is no command-in-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
>> +The command-out-data is the queue index, the command-in-data contains \field{avail_counter}, \field{avail_index},
>> +\field{used_counter} and \field{used_index} of the virtqueue
>> +(as described in transportq_ctrl_mdev_packed_vq_state_set).
>> +
>> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue State}
>> +
>> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue index access if \field{queue_index} is invalid.
>> +
>> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue ASID}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV is negotiated, the address space id of a virtqueue
>> +could be set through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
>> +
>> +struct transportq_ctrl_mdev_vq_asid_set {
>> +	u16 queue_index;
>> +	u32 asid;
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
>> +of a virtqueue.
>> +
>> +The address space id is an identifier of a memory space which is used to convey the address space targeted by the
>> +memory accesses, and to distinguish memory accesses performed by different virtqueues
>> +
>> +One example of the address space id is PASID (Process Address Space Identifier) which is
>> +defined in \hyperref[intro:PCIe]{[PCIe]} specification.
>> +
>> +The command-out-data of VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is the \field{queue_index}
>> +and the address space id \field{asid}
>> +(as described in struct transportq_ctrl_mdev_vq_asid_set).
>> +
>> +\devicenormative{\subsubsection}{Virtqueue ASID}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue ASID}
>> +
>> +Once a virtqueue has been set an asid, it MUST perform any memory accesses with the asid.
>> +
>> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue index access if \field{queue_index} is invalid.
>> +
>> +\subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Reset}
>> +
>> +When VIRTIO_F_TRANSPT_VQ_MDEV and VIRTIO_F_RING_RESET are negotiated,
>
> wait this is a mess. VIRTIO_F_TRANSPT_VQ_MDEV is on management device.
> VIRTIO_F_RING_RESET is I assume on the managed one?
Yes, there should only be VIRTIO_F_TRANSPT_VQ_MDEV negotiated with the 
management device,
I will remove this VIRTIO_F_RING_RESET requirement because it is the 
guest negotiated feature.
>
>
>> +a virtqueue can be reset through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_RESET        14
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET    1
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET command is used to reset a virtqueue.
>> +The command-out-data is the virtqueue index, there is no command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Reset}
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
>> +\field{device_id} is 0.
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
>> +the virtqueue index is invalid.
>> +
>> +The managed device MUST stop consuming the descriptors in the virtqueue once reset it.
>> +
>> +The managed device MUST present default states of a virtqueue after reset it.
>> +
>> +\drivernormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Reset}
>> +
>> +After reseting a virtqueue, the driver MUST wait until the reset is successfully
>> +done (by verifying the virtqueue state has been reset to default)
> verifying how? why isn't completion of the command sufficient?
Yes, I will use the completion of the command to indicate the reset is done.
>
>> +before re-enabling it.
>> +
>>   \chapter{Device Types}\label{sec:Device Types}
>>   
>>   On top of the queues, config space and feature negotiation facilities
>> -- 
>> 2.35.3
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Zhu, Lingshan]

On 8/2/2022 8:55 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 02, 2022 at 07:54:41PM +0800, Zhu, Lingshan wrote:
>>
>> On 8/2/2022 2:52 PM, Michael S. Tsirkin wrote:
>>> On Tue, Aug 02, 2022 at 11:17:51AM +0800, Zhu, Lingshan wrote:
>>>> On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
>>>>> On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
>>>>>> This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
>>>>>> which depends on VIRTIO_F_TRANSPT_VQ.
>>>>>>
>>>>>> Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
>>>>>> devices are created, configured and destroyed through the
>>>>>> transport virtqueue, the transport virtqueue is the
>>>>>> transport layer for the managed devices.
>>>>> I'd avoid the term MDEV if possible, it's already used in virt contexts.
>>>> yes, let me work out a better name, it was vdev (virtual device) before, and
>>>> maybe
>>>> sdev(sub device) or a better one.
>>> Besides mdev can thinkably stand for both managed and management.
>>> Nvidia patches have the concept of a device group and
>>> group member but do not have a name for the device that controls
>>> the group. Admin device? Control device?
>> maybe management device?
> This term has been overused in this context with people saying things
> like "the management interface is used by management tools to
> manage managed devices" to the point where I'd
> rather we avoided it altogether.
They are like the parent device and the sub-device, but this is not good 
enough.
It has ever been suggest to use "upstream device" and "downstream 
device" to replace "master device" and "slave device",
but looks not good enough either.

The concept should be the management device and the managed device, we 
never use these terms in virtio spec,
any better suggestions?

>
>>> Also, creating/destroying is one thing that seems to only apply to
>>> SIOV. configuring seems to apply to SRIOV as well?
>> In this series, yes, because actually we can not provision a SRIOV VF with
>> dynamic config, and VF provisioning is done through SRIOV cap
>>
>> Thanks
>> Zhu Lingshan
> I am not sure what dynamic config means here exactly. Yes creating a VF
> itself is not possible. But allocating VQs to it and
> e.g. configuring the MAC address of a networking VF can be
> done dynamically.
I think this config can only be done through the CVQ after provisioning.
>
>>>
>>>
>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>>>>> ---
>>>>>>     content.tex | 12 ++++++++++++
>>>>>>     1 file changed, 12 insertions(+)
>>>>>>
>>>>>> diff --git a/content.tex b/content.tex
>>>>>> index 969ca46..143949e 100644
>>>>>> --- a/content.tex
>>>>>> +++ b/content.tex
>>>>>> @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
>>>>>>     Managed devices are created and discovered through a transport virtqueue.
>>>>>> +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>>>> +
>>>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
>>>>>> +and destroyed through the transport virtqueue.
>>>>> Feature bit being present where? On the management device?
>>>> Yes, the management device, I will add this.
>>>>>> +This means the transport virtqueue is the transport layer for the managed devices.
>>>>> confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
>>>> VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
>>>> VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the
>>>> management device create / destroy and configure the managed device via the
>>>> transport vq.
>>>>
>>>> This reminds me that maybe I should use the term vdev(virtual device), so
>>>> that other kinds of devices can add their own feature bits under
>>>> VIRTIO_F_TRANSPT_VQ
>>>>
>>>> Thanks,
>>>> Zhu Lingshan
>>>>>> +
>>>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
>>>>>> +
>>>>>> +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>>>> +
>>>>>> +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
>>>>>> +
>>>>>>     \chapter{Device Types}\label{sec:Device Types}
>>>>>>     On top of the queues, config space and feature negotiation facilities
>>>>>> -- 
>>>>>> 2.35.3
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>

Re: [virtio-comment] Re: [PATCH V2 0/5] Introduce virtio transport virtqueue

[Zhu, Lingshan]

On 8/2/2022 4:56 AM, Michael S. Tsirkin wrote:
> On Mon, Aug 01, 2022 at 05:32:11PM +0800, Zhu Lingshan wrote:
>> This series introduces virtio transport virtqueue as a new
>> transport layer for virtio devices.
>>
>> This series defines the concepts of virtio management device
>> and virtio managed devices.
>
> Overall I'm happy with the progress made here. Pls keep iterating.
Sure, will do, thanks!
> I would like to see whether we can unify this with the admin
> queue work somehow. The technology is different (SIOV vs SRIOV)
> but there are also similarities.
>
> Do you care to review some patches from the admin queue proposals maybe?
IMHO, I think this transport vq series focus on introducing a new virtio 
transport layer for
hyper scaling, like SIOV. And the admin vq proposal looks focusing on 
administration, to control
the sub-devices. so maybe the transport vq and the admin vq are works in 
parallel.
>
>
>> This series also defines a commands set over the transport virtqueue,
>> including:
>> The command to query available resources of the management device
>> The commands to create / destroy the managed devices.
>> The commands to config the managed devices.
>> The commands to config virtqueues of the managed devices.
>>
>> Changes form V1:
>> 1)Add a command to set per-vq asid(Jason)
>> 2)Add a command to reset a virtqueue(Jason)
>> 3)Add support for packed virtqueue state(Jason)
>> 4)Allow nested managed devices(Jason)
>> 5)Provides device feature bits in the provisioning(Jason)
>> 6)Removed most of the error codes, only left OK and ERR(Jason)
>> 7)“virtual devices” is renamed to “managed devices”, to be more general(Jason)
>>
>> Zhu Lingshan (5):
>>    Introduce virito transport virtqueue
>>    Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV
>>    Introduce the commands set of the transport vq
>>    Describe the process to present a managed device
>>    Add transport vq number for virtio blk and net
>>
>>   content.tex      | 784 +++++++++++++++++++++++++++++++++++++++++++++++
>>   introduction.tex |   3 +
>>   2 files changed, 787 insertions(+)
>>
>> -- 
>> 2.35.3
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Michael S. Tsirkin]

On Tue, Aug 02, 2022 at 09:15:28PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 8/2/2022 8:55 PM, Michael S. Tsirkin wrote:
> > On Tue, Aug 02, 2022 at 07:54:41PM +0800, Zhu, Lingshan wrote:
> > > 
> > > On 8/2/2022 2:52 PM, Michael S. Tsirkin wrote:
> > > > On Tue, Aug 02, 2022 at 11:17:51AM +0800, Zhu, Lingshan wrote:
> > > > > On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
> > > > > > On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
> > > > > > > This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
> > > > > > > which depends on VIRTIO_F_TRANSPT_VQ.
> > > > > > > 
> > > > > > > Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
> > > > > > > devices are created, configured and destroyed through the
> > > > > > > transport virtqueue, the transport virtqueue is the
> > > > > > > transport layer for the managed devices.
> > > > > > I'd avoid the term MDEV if possible, it's already used in virt contexts.
> > > > > yes, let me work out a better name, it was vdev (virtual device) before, and
> > > > > maybe
> > > > > sdev(sub device) or a better one.
> > > > Besides mdev can thinkably stand for both managed and management.
> > > > Nvidia patches have the concept of a device group and
> > > > group member but do not have a name for the device that controls
> > > > the group. Admin device? Control device?
> > > maybe management device?
> > This term has been overused in this context with people saying things
> > like "the management interface is used by management tools to
> > manage managed devices" to the point where I'd
> > rather we avoided it altogether.
> They are like the parent device and the sub-device, but this is not good
> enough.
> It has ever been suggest to use "upstream device" and "downstream device" to
> replace "master device" and "slave device",
> but looks not good enough either.
> 
> The concept should be the management device and the managed device, we never
> use these terms in virtio spec,

It's not terrible but it's verbose and if you try to shorten
you can't tell managed from management apart.

Also the concept of a group of all devices controlled
through a single one is missing.

> any better suggestions?

I gave some above.

> > 
> > > > Also, creating/destroying is one thing that seems to only apply to
> > > > SIOV. configuring seems to apply to SRIOV as well?
> > > In this series, yes, because actually we can not provision a SRIOV VF with
> > > dynamic config, and VF provisioning is done through SRIOV cap
> > > 
> > > Thanks
> > > Zhu Lingshan
> > I am not sure what dynamic config means here exactly. Yes creating a VF
> > itself is not possible. But allocating VQs to it and
> > e.g. configuring the MAC address of a networking VF can be
> > done dynamically.
> I think this config can only be done through the CVQ after provisioning.
> > 
> > > > 
> > > > 
> > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> > > > > > > ---
> > > > > > >     content.tex | 12 ++++++++++++
> > > > > > >     1 file changed, 12 insertions(+)
> > > > > > > 
> > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > index 969ca46..143949e 100644
> > > > > > > --- a/content.tex
> > > > > > > +++ b/content.tex
> > > > > > > @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
> > > > > > >     Managed devices are created and discovered through a transport virtqueue.
> > > > > > > +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> > > > > > > +
> > > > > > > +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
> > > > > > > +and destroyed through the transport virtqueue.
> > > > > > Feature bit being present where? On the management device?
> > > > > Yes, the management device, I will add this.
> > > > > > > +This means the transport virtqueue is the transport layer for the managed devices.
> > > > > > confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
> > > > > VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
> > > > > VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the
> > > > > management device create / destroy and configure the managed device via the
> > > > > transport vq.
> > > > > 
> > > > > This reminds me that maybe I should use the term vdev(virtual device), so
> > > > > that other kinds of devices can add their own feature bits under
> > > > > VIRTIO_F_TRANSPT_VQ
> > > > > 
> > > > > Thanks,
> > > > > Zhu Lingshan
> > > > > > > +
> > > > > > > +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
> > > > > > > +
> > > > > > > +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
> > > > > > > +
> > > > > > > +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
> > > > > > > +
> > > > > > >     \chapter{Device Types}\label{sec:Device Types}
> > > > > > >     On top of the queues, config space and feature negotiation facilities
> > > > > > > -- 
> > > > > > > 2.35.3
> > > > > > This publicly archived list offers a means to provide input to the
> > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > 
> > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > to minimize spam in the list archive, subscription is required
> > > > > > before posting.
> > > > > > 
> > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > 

Re: [virtio-comment] Re: [PATCH V2 0/5] Introduce virtio transport virtqueue

[Zhu, Lingshan]

On 8/2/2022 4:56 AM, Michael S. Tsirkin wrote:
> On Mon, Aug 01, 2022 at 05:32:11PM +0800, Zhu Lingshan wrote:
>> This series introduces virtio transport virtqueue as a new
>> transport layer for virtio devices.
>>
>> This series defines the concepts of virtio management device
>> and virtio managed devices.
>
> Overall I'm happy with the progress made here. Pls keep iterating.
Sure, will do, thanks!
> I would like to see whether we can unify this with the admin
> queue work somehow. The technology is different (SIOV vs SRIOV)
> but there are also similarities.
>
> Do you care to review some patches from the admin queue proposals maybe?
IMHO, I think this transport vq series focus on introducing a new virtio 
transport layer for
hyper scaling, like SIOV. And the admin vq proposal looks focusing on 
administration, to control
the sub-devices. so maybe the transport vq and the admin vq are works in 
parallel.
>
>
>> This series also defines a commands set over the transport virtqueue,
>> including:
>> The command to query available resources of the management device
>> The commands to create / destroy the managed devices.
>> The commands to config the managed devices.
>> The commands to config virtqueues of the managed devices.
>>
>> Changes form V1:
>> 1)Add a command to set per-vq asid(Jason)
>> 2)Add a command to reset a virtqueue(Jason)
>> 3)Add support for packed virtqueue state(Jason)
>> 4)Allow nested managed devices(Jason)
>> 5)Provides device feature bits in the provisioning(Jason)
>> 6)Removed most of the error codes, only left OK and ERR(Jason)
>> 7)“virtual devices” is renamed to “managed devices”, to be more general(Jason)
>>
>> Zhu Lingshan (5):
>>    Introduce virito transport virtqueue
>>    Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV
>>    Introduce the commands set of the transport vq
>>    Describe the process to present a managed device
>>    Add transport vq number for virtio blk and net
>>
>>   content.tex      | 784 +++++++++++++++++++++++++++++++++++++++++++++++
>>   introduction.tex |   3 +
>>   2 files changed, 787 insertions(+)
>>
>> -- 
>> 2.35.3
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Zhu, Lingshan]

On 8/2/2022 9:20 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 02, 2022 at 09:15:28PM +0800, Zhu, Lingshan wrote:
>>
>> On 8/2/2022 8:55 PM, Michael S. Tsirkin wrote:
>>> On Tue, Aug 02, 2022 at 07:54:41PM +0800, Zhu, Lingshan wrote:
>>>> On 8/2/2022 2:52 PM, Michael S. Tsirkin wrote:
>>>>> On Tue, Aug 02, 2022 at 11:17:51AM +0800, Zhu, Lingshan wrote:
>>>>>> On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
>>>>>>> On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
>>>>>>>> This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
>>>>>>>> which depends on VIRTIO_F_TRANSPT_VQ.
>>>>>>>>
>>>>>>>> Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
>>>>>>>> devices are created, configured and destroyed through the
>>>>>>>> transport virtqueue, the transport virtqueue is the
>>>>>>>> transport layer for the managed devices.
>>>>>>> I'd avoid the term MDEV if possible, it's already used in virt contexts.
>>>>>> yes, let me work out a better name, it was vdev (virtual device) before, and
>>>>>> maybe
>>>>>> sdev(sub device) or a better one.
>>>>> Besides mdev can thinkably stand for both managed and management.
>>>>> Nvidia patches have the concept of a device group and
>>>>> group member but do not have a name for the device that controls
>>>>> the group. Admin device? Control device?
>>>> maybe management device?
>>> This term has been overused in this context with people saying things
>>> like "the management interface is used by management tools to
>>> manage managed devices" to the point where I'd
>>> rather we avoided it altogether.
>> They are like the parent device and the sub-device, but this is not good
>> enough.
>> It has ever been suggest to use "upstream device" and "downstream device" to
>> replace "master device" and "slave device",
>> but looks not good enough either.
>>
>> The concept should be the management device and the managed device, we never
>> use these terms in virtio spec,
> It's not terrible but it's verbose and if you try to shorten
> you can't tell managed from management apart.
there is only one struct using the shorten term of management device,
struct virtio_mgmt_dev_avail_res{}, and for the managed device,
I think maybe only _dev_ is fine, like
VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE and struct transportq_ctrl_dev_config_get
>
> Also the concept of a group of all devices controlled
> through a single one is missing.
I am not sure what kinds of commands against the device group? Destroy all?
>
>> any better suggestions?
> I gave some above.
>
>>>>> Also, creating/destroying is one thing that seems to only apply to
>>>>> SIOV. configuring seems to apply to SRIOV as well?
>>>> In this series, yes, because actually we can not provision a SRIOV VF with
>>>> dynamic config, and VF provisioning is done through SRIOV cap
>>>>
>>>> Thanks
>>>> Zhu Lingshan
>>> I am not sure what dynamic config means here exactly. Yes creating a VF
>>> itself is not possible. But allocating VQs to it and
>>> e.g. configuring the MAC address of a networking VF can be
>>> done dynamically.
>> I think this config can only be done through the CVQ after provisioning.
>>>>>
>>>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>>>>>>> ---
>>>>>>>>      content.tex | 12 ++++++++++++
>>>>>>>>      1 file changed, 12 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/content.tex b/content.tex
>>>>>>>> index 969ca46..143949e 100644
>>>>>>>> --- a/content.tex
>>>>>>>> +++ b/content.tex
>>>>>>>> @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
>>>>>>>>      Managed devices are created and discovered through a transport virtqueue.
>>>>>>>> +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>>>>>> +
>>>>>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
>>>>>>>> +and destroyed through the transport virtqueue.
>>>>>>> Feature bit being present where? On the management device?
>>>>>> Yes, the management device, I will add this.
>>>>>>>> +This means the transport virtqueue is the transport layer for the managed devices.
>>>>>>> confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
>>>>>> VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
>>>>>> VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the
>>>>>> management device create / destroy and configure the managed device via the
>>>>>> transport vq.
>>>>>>
>>>>>> This reminds me that maybe I should use the term vdev(virtual device), so
>>>>>> that other kinds of devices can add their own feature bits under
>>>>>> VIRTIO_F_TRANSPT_VQ
>>>>>>
>>>>>> Thanks,
>>>>>> Zhu Lingshan
>>>>>>>> +
>>>>>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
>>>>>>>> +
>>>>>>>> +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>>>>>> +
>>>>>>>> +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
>>>>>>>> +
>>>>>>>>      \chapter{Device Types}\label{sec:Device Types}
>>>>>>>>      On top of the queues, config space and feature negotiation facilities
>>>>>>>> -- 
>>>>>>>> 2.35.3
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH V2 2/5] Introduce feature bit VIRTIO_F_TRANSPT_VQ_MDEV

[Zhu, Lingshan]

On 8/2/2022 9:20 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 02, 2022 at 09:15:28PM +0800, Zhu, Lingshan wrote:
>>
>> On 8/2/2022 8:55 PM, Michael S. Tsirkin wrote:
>>> On Tue, Aug 02, 2022 at 07:54:41PM +0800, Zhu, Lingshan wrote:
>>>> On 8/2/2022 2:52 PM, Michael S. Tsirkin wrote:
>>>>> On Tue, Aug 02, 2022 at 11:17:51AM +0800, Zhu, Lingshan wrote:
>>>>>> On 8/2/2022 4:27 AM, Michael S. Tsirkin wrote:
>>>>>>> On Mon, Aug 01, 2022 at 05:32:13PM +0800, Zhu Lingshan wrote:
>>>>>>>> This commit introduces feature bit VIRTIO_F_TRANSPT_VQ_MDEV,
>>>>>>>> which depends on VIRTIO_F_TRANSPT_VQ.
>>>>>>>>
>>>>>>>> Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed
>>>>>>>> devices are created, configured and destroyed through the
>>>>>>>> transport virtqueue, the transport virtqueue is the
>>>>>>>> transport layer for the managed devices.
>>>>>>> I'd avoid the term MDEV if possible, it's already used in virt contexts.
>>>>>> yes, let me work out a better name, it was vdev (virtual device) before, and
>>>>>> maybe
>>>>>> sdev(sub device) or a better one.
>>>>> Besides mdev can thinkably stand for both managed and management.
>>>>> Nvidia patches have the concept of a device group and
>>>>> group member but do not have a name for the device that controls
>>>>> the group. Admin device? Control device?
>>>> maybe management device?
>>> This term has been overused in this context with people saying things
>>> like "the management interface is used by management tools to
>>> manage managed devices" to the point where I'd
>>> rather we avoided it altogether.
>> They are like the parent device and the sub-device, but this is not good
>> enough.
>> It has ever been suggest to use "upstream device" and "downstream device" to
>> replace "master device" and "slave device",
>> but looks not good enough either.
>>
>> The concept should be the management device and the managed device, we never
>> use these terms in virtio spec,
> It's not terrible but it's verbose and if you try to shorten
> you can't tell managed from management apart.
there is only one struct using the shorten term of management device,
struct virtio_mgmt_dev_avail_res{}, and for the managed device,
I think maybe only _dev_ is fine, like
VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE and struct transportq_ctrl_dev_config_get
>
> Also the concept of a group of all devices controlled
> through a single one is missing.
I am not sure what kinds of commands against the device group? Destroy all?
>
>> any better suggestions?
> I gave some above.
>
>>>>> Also, creating/destroying is one thing that seems to only apply to
>>>>> SIOV. configuring seems to apply to SRIOV as well?
>>>> In this series, yes, because actually we can not provision a SRIOV VF with
>>>> dynamic config, and VF provisioning is done through SRIOV cap
>>>>
>>>> Thanks
>>>> Zhu Lingshan
>>> I am not sure what dynamic config means here exactly. Yes creating a VF
>>> itself is not possible. But allocating VQs to it and
>>> e.g. configuring the MAC address of a networking VF can be
>>> done dynamically.
>> I think this config can only be done through the CVQ after provisioning.
>>>>>
>>>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>>>>>>> ---
>>>>>>>>      content.tex | 12 ++++++++++++
>>>>>>>>      1 file changed, 12 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/content.tex b/content.tex
>>>>>>>> index 969ca46..143949e 100644
>>>>>>>> --- a/content.tex
>>>>>>>> +++ b/content.tex
>>>>>>>> @@ -2968,6 +2968,18 @@ \subsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio
>>>>>>>>      Managed devices are created and discovered through a transport virtqueue.
>>>>>>>> +\subsection{Transport Virtqueue Features}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>>>>>> +
>>>>>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV indicates that the managed devices are created, configured
>>>>>>>> +and destroyed through the transport virtqueue.
>>>>>>> Feature bit being present where? On the management device?
>>>>>> Yes, the management device, I will add this.
>>>>>>>> +This means the transport virtqueue is the transport layer for the managed devices.
>>>>>>> confused. how does this differ from VIRTIO_F_TRANSPT_VQ?
>>>>>> VIRTIO_F_TRANSPT_VQ means the management device offers a transport vq.
>>>>>> VIRTIO_F_TRANSPT_VQ_MDEV is a feature of the transport vq, means the
>>>>>> management device create / destroy and configure the managed device via the
>>>>>> transport vq.
>>>>>>
>>>>>> This reminds me that maybe I should use the term vdev(virtual device), so
>>>>>> that other kinds of devices can add their own feature bits under
>>>>>> VIRTIO_F_TRANSPT_VQ
>>>>>>
>>>>>> Thanks,
>>>>>> Zhu Lingshan
>>>>>>>> +
>>>>>>>> +Feature bit VIRTIO_F_TRANSPT_VQ_MDEV depends on VIRTIO_F_TRANSPT_VQ
>>>>>>>> +
>>>>>>>> +\devicenormative{\subsubsection}{Transport Virtqueue Features}{Virtio Transport Options / Virtio Over Transport Virtqueue / Transport Virtqueue Features}
>>>>>>>> +
>>>>>>>> +The management device MUST not accept VIRTIO_F_TRANSPT_VQ_MDEV if VIRTIO_F_TRANSPT_VQ is not negotiated.
>>>>>>>> +
>>>>>>>>      \chapter{Device Types}\label{sec:Device Types}
>>>>>>>>      On top of the queues, config space and feature negotiation facilities
>>>>>>>> -- 
>>>>>>>> 2.35.3
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/